{"title": "Connection Topology and Dynamics in Lateral Inhibition Networks", "book": "Advances in Neural Information Processing Systems", "page_first": 98, "page_last": 104, "abstract": null, "full_text": "Connection Topology and Dynamics \n\nin Lateral Inhibition Networks \n\nC. M. Marcus, F. R. Waugh, and R. M. Westervelt \n\nDepartment of Physics and Division of Applied Sciences, Harvard University \n\nCambridge, MA 02138 \n\nABSTRACT \n\nWe show analytically how the stability of two-dimensional lateral \ninhibition neural networks depends on the local connection topology. \nFor various network topologies, we calculate the critical time delay for \nthe onset of oscillation in continuous-time networks and present \nanalytic phase diagrams characterizing the dynamics of discrete-time \nnetworks. \n\nINTRODUCTION \n\n1 \nMutual inhibition in an array of neurons is a common feature of sensory systems \nincluding vision, olfaction, and audition in organisms ranging from invertebrates to man. \nA well-studied instance of this configuration is lateral inhibition between neighboring \nphotosensitive neurons in the retina (Dowling, 1987). Inhibition serves in this case to \nenhance the perception of edges and to broaden the dynamic range by setting a local \nreference point for measuring intensity variations. Lateral inhibition thus constitutes the \nfirst stage of visual information processing. Many artificial vision systems also take \nadvantage of the computational power of lateral inhibition by directly wiring inhibition \ninto the photodetecting electronic hardware (Mead, 1989). \n\nLateral inhibition may create extensive feedback paths, leading to network-wide collective \noscillations. Sustained oscillations arising from lateral inhibition have been observed in \nbiological visual system~specifically. in the compound eye of the horseshoe crab \nLimulus (Barlow and Fraioli, 1978; Coleman and Renninger, 1978)-as well as in \nartificial vision systems, for instance plaguing an early version of the electronic retina \nchip built by Mead et al. (Wyatt and Standley, 1988; Mead, 1989). \n\nIn this paper we study the dynamics of simple neural network models of lateral inhibition \nin a variety of two-dimensional connection schemes. The lattice structures we study are \nshown in Fig. 1. Two-dimensional lattices are of particular importance to artificial \nvision systems because they allow an efficient mapping of an image onto a network and \nbecause they are well-suited for implementation in VLSI circuitry. We show that the \n\n98 \n\n\fConnection Topology and Dynamics in Lateral Inhibition Networks \n\n99 \n\nstability of these networks depends sensitively on such design considerations as local \nconnection topology, neuron self-coupling, the steepness or gain of the neuron transfer \nfunction, and details of the network dynamics such as connection delays for continuous(cid:173)\ntime dynamics or update rule for discrete-time dynamics. \n\n(a) \n\n0 \n\n0 \n\n0 \n\n0 \n\n(b) 0 000 \n\n(c) 0 \n\n0 \n\n0 \n\no \n\n:~ \no :To \n\no \n\n00* 00 \n\no \n\no \n\no \n\no \n\no \n\no \n\n(d) 0 \n\no \n\no \n\no \no \n\n0 \n\no \n\n0 \n\no \n\no \n\n0 \n\no \n\no \n0 000 00 000 o \n\n0 \n\no \n\no \n\n0 \n\n0 \n\n0 \n\n0 \n\nFigure 1: Connection schemes for two-dimensional lateral inhibition networks \nconsidered in this paper: (a) nearest-neighbor connections on a square lattice; (b) \nnearest-neighbor connections on a triangular lattice; (c) 8-neighbor connections \non a square lattice; and (d) 12-neighbor connections on a square lattice. \n\nThe paper is organized as follows. Section 2 introduces the dynamical equations \ndescribing continuous-time and discrete-time lateral inhibition networks . Section 3 \ndiscusses the relationship between lattice topology and critical time delay for the onset of \noscillation in the continuous-time case. Section 4 presents analytic phase diagrams \ncharacterizing the dynamics of discrete-time lateral inhibition networks as neuron gain, \nneuron self-coupling, and lattice structure are varied. Our conclusions are presented in \nSection 5. \n\n2 NETWORK DYNAMICS \nWe begin by considering a general neural network model defined by the set of electronic \ncircuit equations \n\nCi dUi(t')/dt'= - ui(t')/Ri + ~T;jfA uAt'-'l'i/)) + Ii \n\n,i=l .... ,N, \n\n(1) \n\nJ \n\nwhere u \u00b7 is the voltage. C. the capacitance. and Rj -1 = 1'. j l1ij I the total conductance at \nthe inpJt of neuron i. I~put to the network is througli the applied currents Ii. The \nnonlinear transfer function Ii is taken to be sigmoidal with odd symmetry and maximum \nslope at the origin. A time delay 'l'i/ in the communication from neuron i to neuron j \nhas been explicitly included. Such a delay could arise from the finite operating speed of \nthe elements-neurons or amplifiers-or from the finite propagation speed of the \ninterconnections. For the case of lateral inhibition networks with self-coupling. the \nconnection matrix is given by \n\nTij = -1 for i, j connected neighbors \n\nr for i = j \n\n{ \n\no otherwise, \n\n(2) \n\nwhich makes Ri- 1 = Irl+ z for all i, where z is the number of connected neighbors. \nFor simplicity, we take all neurons to have the same delay and characteristic relaxation \n\n\f100 Marcus, Waugh, and Westervelt \n\ntime (t.'=tdelgy ' R.C .=trelax for all i) and identical transfer functions. With these \nassumptions, hq. (1) ~an be rescaled and written in terms of the neuron outputs Xi(t) as \n\ndxi(t)/dt = - Xi(t) + F(Ij I;:;xj(t - t) + Ii)' i=l, ... , N, \n\n(3) \n\nwhere the odd, sigmoidal function F now appears outside the sum. The function F is \ncharacterized by a maximum slope f3 (> 0), and its saturation amplitude can be set to \u00b11 \nwithout loss of generality. The commonly used form F(h) = tanh(f3h) satisfies these \nrequirements; we will continue to use F to emphasize generality. As a result of \nrescaling, the delay time t is now measured in units of network relaxation time (i.e. \nt = tdelay/trelax )' and the connection matrix is normalized such that Ljl1ijl = 1 for \nall i. Stability of Eq. (3) against coherent oscillation will be discussed in Section 3. \n\nThe discrete-time iterated map, \n\n, \n\ni=l, ... , N, \n\n(4) \n\nwith parallel updating of neuron states Xi(t), corresponds to the long-delay limit of Eq. \n(3) (care must be taken in considering this limit; not all aspects of the delay system carry \nover to the map (Mallet-Paret and Nussbaum, 1986)). The iterated map network, Eq. (4), \nis particularly useful for implementing fast, parallel networks using conventional \ncomputer clocking techniques. The speed advantage of parallel dynamics, however, comes \nat a price: the parallel-update network may oscillate even when the corresponding \nsequential update network is stable. Section 4 gives phase diagrams based on global \nstability analysis which explicitly define the oscillation-free operating region of Eq. (4) \nand its generalization to a multistep updating rule. \n\n3 STABILITY OF LATTICES WITH DELAYED INHIBITION \nIn the absence of delay (t = 0) the continuous-time lateral inhibition network, Eq. (3), \nalways converges to a fixed point attractor. This follows from the famous stability \ncriterion based on a Liapunov (or \"energy\") function (Cohen and Grossberg, 1983; \nHopfield, 1984), and relies on the symmetry of the lateral inhibitory connections (Le. \n'Tjj = Tji for all connection schemes in Fig. I). This guarantee of convergence does not \nhold for nonzero delay, however, and it is known that adding delay can induce sustained, \ncoherent oscillation in a variety of symmetrically connected network configurations \n(Marcus and Westervelt, 1989a). Previously we have shown that certain delay networks \nof the form ofEq. (3)--including lateral inhibition network~will oscillate coherently, \nthat is with all neurons oscillating in phase, for sufficiently large delay. As the delay is \nreduced, however, the oscillatory mode becomes unstable, leaving only fixed point \nattractors. A critical value of delay tcrit below which sustained oscillation vanishes for \nany value of neuron gain f3 is given by \n\ntcrit=-ln(I+Amax/Amin) \n\n(5) \nwhere Amax and Amin are the extremal eigenvalues of the connection matrix Tij. The \nanalysis leading to (5) is based on a local stability analysis of the coherent oscillatory \nmode. Though this local analysis lacks the rigor of a global analysis (which can be done \nfor t = 0 and for the discrete-time case, Eq. (4)) the result agrees well with experiments \nand numerical simulations (Marcus and Westervelt, 1989a). \n\n(0< A max <-Amin) \n\n\fConnection Topology and Dynamics in Lateral Inhibition Networks \n\n101 \n\nIt is straightforward to find the spectrum of eigenvalues for the lattices in Fig. 1. \nAssuming periodic boundary conditions, one can expand the eigenvalue equation Tx = A x \nin terms of periodic functions x) = Xo exp(i q. Rj ) ,where Rj is the 2D vector position of \nneuron j and q is the reciprocal lattice vector characterizing a particular eigenmode. In \nthe large network limit, this expansion leads to the following results for the square and \ntriangular lattices with nearest neighbor connections and self-connection r [see next \nsection for a table of eigenvalues]: \n\n'r crit ~ In( 1/2 - 2/ r ) \n\n(-4 In 2 == 0.693 . \n\n[n.n. triangular lattice, Fig. 1(b)]. \n\n2 \n\n1 \n\nr 0 \n\n-1 \n\n-3 \n\n-4 \n\nr 2.5 ~ \n\n~* \n+ \n\nFigure 2: Critical delay 'rcrit as a function of self-connection r. from Eq. (6). \nNote that for r = 0 only triangular lattice oscillates at finite delay. The \nanalysis does not apply at exactly 'r = 0, where both networks are stable for all \nvalues of r. \n\nThe important difference between these two lattices--and the quality which accounts for \ntheir dissimilar stability properties-is not simply the number of neighbors, but is the \npresence of frustration in the triangular lattice but not in the square lattice. Lateral \ninhibition, like antiferromagnetism, forms closed loops in the triangular lattice which do \nnot allow all of the connections to be satisfied by any arrangement of neuron states. In \ncontrast, lateral inhibition on the square lattice is not frustrated, and is, in fact, exactly \nequivalent to lateral excitation via a gauge transformation. We note that a similar \nsituation exists in 2D magnetic models: while models of 2D ferromagnetism on square \nand triangular lattices behave nearly identically (both are nonfrustrated). the corresponding \n2D antiferromagnets are quite different, due to the presence of frustration in the triangular \nlattice, but not the square lattice (Wannier, 1950). \n\n\f102 Marcus, Waugh, and Westervelt \n\n4 LATTICES WITH ITERATED-MAP DYNAMICS \nNext we consider lateral inhibition networks with discrete-time dynamics where all neuron \nstates are updated in parallel. The standard parallel dynamics fonnulation was given above \nas Eq. (4), but here we will consider a generalized updating rule which offers some \nimportant practical advantages. The generalized system we consider updates the neuron \nstates based on an average over M previous time steps, rather than just using a single \nprevious state to generate the next This multistep rule is somewhat like including time \ndelay, but as we will see, increasing M actually makes the system more stable compared \nto standard parallel updating. This update rule also differs from the delay-differential \nsystem in pennitting a rigorous global stability analysis. The dynamical system we \nconsider is defined by the following set of coupled iterated maps: \n\nzlt)=M-1 :L Xj(t-'r) , \n\nM-l \n\n-r=O \n\n(7) \n\nwhere i,j = l, ... ,N and ME {1,2,3, ... }. The standard parallel updating rule, Eq.(4), is \nrecovered by setting M = 1. \n\nA global analysis of the dynamics of Eq. (7) for any symmetric Tij is given in (Marcus \nand Westervelt, 1990), and for M=1 in (Marcus and Westervelt, 1989b). It is found that \nfor any M, if all eigenvalues A satisfy 131,1,1 < 1 then there is a single attractor which \ndepends only on the inputs Ii. For Ii = 0, this attractor is the origin, Le. all neurons at \nzero output. Whenever 13IAI > 1 for one or more eigenvalues, multiple fixed points as \nwell as periodic attractors may exist. There is, in addition, a remarkably simple glopal \nstability criterion associated with Eq. (7): satisfying the condition 1/13 >-Amin(1i')/~ \ninsures that no periodic attractors exist, though there may be a multiplicity of fixed' point \nattractors. As in the previous section, Amin is the most negative eigenvalue of Tij. If \nTij has no negative eigenvalues, then Amin is the smallest positive eigenvalue, and the \nstability criterion is satisfied trivially since 13 is defined to be positive. \nThese stability results may be used to compute analytic phase diagrams for the various \nconnection schemes shown in Fig. 1 and defined in Eq. (3). The extremal eigenvalues of \nTij are calculated using the Fourier expansion described above. In the limit of large \nlattice size and assuming periodic boundary conditions, we find the following: \n\nsquare n.n. \n\ntriangle n. n. \n\nsquare 8-n. \n\nsquare 12-n. \n\nAmax: \n\nAmin: \n\nr+ 4 \nIrl+4 \nr- 4 \nIrl+4 \n\nr+3 \nIrl+6 \nr-6 \nIrl+6 \n\nr+4 \nIrl+8 \nr- 8 \nIrl+8 \n\nr+ 13/3 \nIrl+12 \nr- 12 \nIrl + 12 \n\nThe resulting phase diagrams characterizing regions with different dynamic properties are \nshown in Fig. 3. The four regions indicated in the diagrams are characterized as follows: \n(1) orig: low gain regime where a unique fixed point attractor exists (that attractor is the \norigin for Ii = 0); (2) fp: for some inputs Ii multiple fixed point attractors may exist, \neach with an attracting basin, but no oscillatory attractors exist in this region (i.e. no \nattractors with period >1); (3) osc: at most one fixed point attractor, but one or more \noscillatory modes also may exist; (4) fp + osc: multiple fixed points as well as \noscillatory attractors may exist. \n\n\fConnection Topology and Dynamics in Lateral Inhibition Networks \n\n103 \n\n(a) 3 \n\n2 \nr 1 \n\norig \n\no f----lt-----:'\"--=--~ \n3 f3 4 \n\n1 \n\nosc \n\n-1 \n\n-2 \n\n-3 \n\n(c) \n\nr \n\n4 \n\n3 \n2 \n\norig \n\n1 \no 1----4-~,..__--'---' \n-1 \n-2 \n\n1 \n\n(b) 6 \n\nr \n\n4 \n\n2 ong \n\nOf-----j'-----lo.~--'---' \n\n1 \n\n-2 \n\n-4 \n\n(d) 6 \n\n4 \nr 2 \n\n-2 \n\n-4 \n\n(f) 6 \n\nosc \n\nosc \n\nfp \n\n(e) 2 \n\n1 \n\nr \n\n-1 \n\n-2 \n\n-3 \n\nfp \n\nr \n\n4 \n\n2 \nfp+ \no 1------'--~::--1'-----'\"---'5 osc \n\nosc \n\n-2 \n\n-4 \n\norig \n\nosc \n\nFigure 3: Phase diagrams based on global analysis for lateral inhibition \nnetworks with discrete-time parallel dynamics [Eq.(7)] as a function of neuron \ngain f3 and self-connection r. Regions orig, jp, OSC, and jp+osc are \ndefmed in text. (a) Nearest-neighbor connections on a square lattice and single(cid:173)\nstep updating (M=l); (b) nearest-neighbor connections on a triangular lattice, \nM=l; (c) 8-neighbor connections on a square lattice, M=l; (d) 12-neighbor \nconnections on a square lattice, M=l; (e) nearest-neighbor connections on a \nsquare lattice, M=3; (0 nearest-neighbor connections on a triangular lattice, \nM=3. \n\n\f104 Marcus, Waugh, and Westervelt \n\n5 CONCLUSIONS \nWe have shown analytically how the dynamics of two-dimensional neural network models \nof lateral inhibition depends on both single-neuron properties-such as the slope of the \nsigmoidal transfer function, delayed response, and the strength of self-connection--and \nalso on the topological properties of the network. \n\nThe design rules implied by the analysis are in some instances what would be expected \nintuitively. For example, the phase diagrams in Fig. 4 show that in order to eliminate \noscillations one can either include a positive self-connection term or decrease the gain of \nthe neuron. It is also not surprising that reducing the time delay in a delay-differential \nsystem eliminates oscillation. Less intuitive is the observation that for discrete-time \ndynamics using a multistep update rule greatly expands the region of oscillation-free \noperation (compare, for example Figs. 4(a) and 4(e\u00bb. One result emerging in this paper \nthat seems quite counterintuitive is the dramatic effect of connection topology, which \npersists even in the limit of large lattice size. This point was illustrated in a comparison \nof networks with delayed inhibition on square and triangular lattices, where it was found \nthat in the absence of self-connection, only the triangular lattices will show sustained \noscillation. \n\nFinally, we note that it is not clear to us how to generalize our results to other network \nmodels, for example to models with asymmetric connections which allow for direction(cid:173)\nselective motion detection. Such questions remain interesting challenges for future work. \n\nAcknowledgments \n\nWe thank Bob Meade and Cornelia Kappler for informative discussions. One of us \n(C.M.M.) acknowledges support as an IBM Postdoctoral Fellow, and one (F.R.W.) from \nthe Army Research Office as a JSEP Graduate Fellow. This work was supported in part \nby ONR contract NOOOI4-89-J-1592, JSEP contract NOOOI4-89-J-1D23, and DARPA \ncontract AFOSR-89-0506. \n\nReferences \n\nBarlow, R. B. and A. J. Fraioli (1978), J. Gen. Physiol., 71, 699. \nCohen, M. A., and S. Grossberg (1983), IEEE Trans. SMC-13, 815. \nColeman, B. D. and G.H. Renninger (1978), Math. Biosc. 38, 123. \nDowling, J. E. (1987), The Retina: An Approachable Part of the Brain (Harvard \n\nUniversity Press, Cambridge, MA). \n\nHopfield, J. J. (1984), Proc. Nat. Acad. Sci. USA 81, 3008. \nMallet-Paret, 1. and R. D. Nussbaum (1986) in Chaotic Dynamics and Fractals, edited \n\nby M. F. Barnsley and S. G. Demko, (Academic Press, Orlando) p. 263. \n\nMarcus, C. M. and R. M. Westervelt (1989a), Phys. Rev. A 39, 347. \nMarcus, C. M. and R. M. Westervelt (1989b), Phys. Rev. A 40, 501. \nMarcus, C. M. and R. M. Westervelt (1990), Phys. Rev. A 42, 2410. \nMead, Carver A. (1989), Analog VLSI and Neural Systems (Addison-Wesley, Reading, \n\nMA). \n\nWyatt, Jr., J. L., and D. L. Standley (1988), in Neural Information Processing Systems, \n\nDenver CO, 1987, edited by D. Z. Anderson, (AlP, New York), p. 860. \n\nWannier, G. M. (1950), Phys. Rev. 79, 357. \n\n\f", "award": [], "sourceid": 346, "authors": [{"given_name": "C.M", "family_name": "Marcus", "institution": null}, {"given_name": "F.", "family_name": "Waugh", "institution": null}, {"given_name": "R.", "family_name": "Westervelt", "institution": null}]}