{"title": "Destabilization and Route to Chaos in Neural Networks with Random Connectivity", "book": "Advances in Neural Information Processing Systems", "page_first": 549, "page_last": 555, "abstract": null, "full_text": "Destabilization and Route to Chaos \n\nin Neural Networks \n\nwith Random Connectivity \n\nBernard Doyon \nUnite INSERM 230 \nService de Neurologie \n\nCHUPurpan \n\nF-31059 Toulouse Cedex, France \n\nBruno Cessac \n\nCentre d'Etudes et de Recherches \n\nde Toulouse \n\n2, avenue Edouard Belin, BP 4025 \nF-31055 Toulouse Cedex, France \n\nMathias Quoy \n\nCentre d'Etudes et de Recherches \n\nde Toulouse \n\n2, avenue Edouard Belin, BP 4025 \nF-31055 Toulouse Cedex, France \n\nManuel Samuelides \nEcole Nationale Superieure \n\nde I'Aeronautique et de l'Espace \n\n10, avenue Edouard Belin, BP 4032 \nF-31055 Toulouse Cedex, France \n\nAbstract \n\nThe occurence of chaos in recurrent neural networks is supposed to \ndepend on the architecture and on the synaptic coupling strength. It is \nstudied here for a randomly diluted architecture. By normalizing the \nvariance of synaptic weights, we produce a bifurcation parameter, \ndependent on this variance and on the slope of the transfer function but \nindependent of the connectivity, that allows a sustained activity and the \noccurence of chaos when reaching a critical value. Even for weak \nconnectivity and small size, we find numerical results in accordance \nwith the theoretical ones previously established for fully connected \ninfinite sized networks. Moreover the route towards chaos is \nnumerically checked to be a quasi-periodic one, whatever the type of the \nfirst bifurcation is (Hopf bifurcation, pitchfork or flip). \n\n549 \n\n\f550 \n\nDoyon, Cessac, Quoy, and Samuelides \n\n1 \n\nINTRODUCTION \n\nMost part of studies on recurrent neural networks assume sufficient conditions of \nconvergence. Models with symmetric synaptic connections have dynamical properties \nstrongly connected with those of spin-glasses. In particular, they have relaxationnal \ndynamics caracterised by the decreasing of a function which is analogous to the energy in \nspin-glasses (or free energy for models submitted to thermal noise). Networks with \nasymmetric synaptic connections lose this convergence property and can have more \ncomplex dynamics. but searchers try to obtain such a convergence because the relaxation \nto a stable network state is simply interpreted as a stored pattern. \n\nHowever, as pointed out by Hirsch (1989), it might be very interesting, from an \nengineering point of view. to investigate non convergent networks because their \ndynamical possibilities are much richer for a given number of units. Moreover, the real \nbrain is a highly dynamic system. Recent neurophysiological findings have focused \nattention on the rich temporal structures (oscillations) of neuronal processes (Gray et al., \n1989), which might play an important role in information processing. Chaotic behavior \nhas been found out in the nervous system (Gallez & Babloyantz, 1991) and might be \nimplicated in cognitive processes (Skarda & Freeman. 1987). \n\nWe have studied the emergent dynamics of a general class of non convergent networks. \nSome results are already available in this field. Sompolinsky et al. (1988) established \nstrong theoretical results concerning the occurrence of chaos forfully connected networks \n00) by using the Dynamic Mean Field Theory. Their \nin the thermodynamic limit (N -\nmodel is a continuous time. continuous state dynamical system with N fully connected \nneurons. Each connection Jij is a gaussian random variable with zero mean and a \nnormalized variance fllN. As the Jij'S are independent. the constant term fl can be seen \nas the variance of the sum of the weights connected to a given unit. Thus. the global \nstrength of coupling remains constant for each neuron as N increases. The output \nfunction of each neuron is sigmoidal with a slope g. Sompolinsky et al. established that, \nin the limit N -\n00, there is a sharp transition from a stationary state to a chaotic flow. \nThe onset of chaos is given by the critical value gJ=l. For gJ<1 the system admits the \nonly fixed point zero, while for gJ >1 it is chaotic. The same authors performed \nsimulations on finite and large values of N and showed the existence of an intermediate \nregime (nonzero stationary states or limit cycles) separating the stationary and the chaotic \nphase. but the routes to chaos were not systematically explored. The range of gJ where \nthis intermediate behavior is observed shrinks as N increases. \n\n2 THE MODEL \n\nThe hypothesis of a fully connected network being not biologically plausible. it could be \ninteresting to inspect how far these results could be extended as the dilution increases for \na general class of networks. The model we study is defined as follows: the number of \nunits is N. and K is the fixed number of connections received by one unit (K>I). There is \nno connection from one unit to itself. The K connections are randomly selected (with an \nuniform law) among the N-l' s. The state of each neuron i at time t is characterized by its \n\n\fDestabilization and Route to Chaos in Neural Networks with Random Connectivity \n\n551 \n\noutput xi (t) which is a real variable varying between -1 and 1. The discrete and parallel \ndynamics is given by: \n\nJij is the synaptic weight which couples the output of unit j to the input of unit i. \nThese weights are random independent variables chosen with a uniform law, with zero \nmenn and a normalized variance J21 K. Notice that, with such a normalization, the \nstandard deviation of the sum of the weights afferent to a given neuron is the constant J. \n\nOne has to distinguish two effects of coupling on the behavior of such a class of models. \nThe first effect is due to the strength of coupling, independent of the number of \nconnections. The second one is due to the architecture of coupling, which can be studied \nby keeping constant the global synaptic effect of coupling. The genericity of our model \ncancels the peculiar dynamic features which may occur due to geometrical effects. \nMoreover it allows to study a model at different scales of dilution. \n\n3 FIRST BIFURCATION \n\nFor such a system, zero is always a fixed point and for low bifurcation parameter value it \nis the only fixed point and it is stable. Let us call Amax the eigenvalue of the matrix of \nsynaptic weights with the greatest modulus and p = I Amaxl the spectral radius of this \nmatrix. The loss of stability arises when the product gp is larger than 1. Our numerical \nsimulations allow us to state that p is approximately equal to J for sufficiently large(cid:173)\nsized networks. This statement can be derived rigorously for an approximate regularized \nmodel in the thermodynamic limit (Doyon et aI., 1993). \n\nTable 1: Mean Value of the Bifurcation Parameter gJ over 30 Networks. \n\nDestabilization of the zero fixed point 1 Onset of Chaos \n\nConnectivity K \n\nNumber of neurons \n\n4 \n8 \n16 \n32 \n\n128 \n\n.954 / 1.337 \n.950 / 1.449 \n.951/1.434 \n.961 / 1.360 \n\n256 \n\n.9651 1.298 \n.966 1 1.301 \n.9651 1.315 \n.958 1 1.333 \n\n512 \n\n.9701 1.258 \n.9781 1.233 \n.969/ 1.239 \n.972 I 1.246 \n\nWe have studied by intensive simulations on a Cray I-XMP computer the statistical \nspectral distribution for N ranging from 4 to 512 and for K ranging from 2 to 32. Figure \n1 shows two examples of spectra (for convenience, J is set to 1). The apparent drawing of \na real axis is due to the real eigenvalue density but the distribution converges to a \nuniform one over the J radius disk, as N increases. A similar result has been theoretically \n\n\f552 \n\nDoyon, Cessac, Quay, and Samuelides \n\nachieved for full gaussian matrices (Girko. 1985; Sommers et al.. 1988). Thus pquicldy \ndecreases to J, so the loss of stability arises for a mean gJ value that increases to 1 for \nincreasing size (Tab. 1). For a given N value, p is nearly independent of K . \n\n..... :. .. \n\n. . '.-\n\n'. ~ \" , (cid:173)\n\nFigure 1: Plot of the Unit Disk and of the Eigenvalues in the Complex Plane. \n\nLeft: 100 Spectra for N=64. K=4. Right: 10 Spectra for N=512. K=4. \n\nThree types of first bifurcation can occur, depending on the eigenvalue Ama.'t : \n\na) Hopf Bifurcation: this corresponds to the appearance of oscillations. There are \n\ntwo complex conjugate eigenvalues with maximal modulus p. \n\nb) Pitchfork bifurcation: if Amax is real positive, the bifurcation arises when \n\ngAmax = 1. Zero loses its stability and two branches of stable equilibria emerge. \n\nc) Flip Bifurcation: for Amaxreal and negative a flip bifurcation occurs when \n\ng Amax = - 1. This corresponds to the appearance of a period two oscillation. \n\nAs the network size increases, the proportion of Hopf bifurcations increases because the \nproportion of real Amax decreases, nearly independent of K . \n\n4 ROUTE TO CHAOS \n\nTo study the following bifurcations, we chose the global observable: \n\nThe value m(t) correctly characterizes all types of first bifurcation that can occur. Indeed \nthe route to chaos is qualitatively well described by this observable, as we checked it by \n\n\fDestabilization and Route to Chaos in Neural Networks with Random Connectivity \n\n553 \n\nstudying simultaneously xi (I). The onset- of chaos was computed by testing the \nsensitivity on initial conditions for m(1) . We observed the onset of chaos occurs for quite \nlow parameter values. The transient zone from fixed point to chaos shrinks slowly to \nzero as the network size increases (fab. 1). \n\nThe qualitative study of the routes to chaos was made on a span of networks with various \nconnectivity and quite important size. The route towards chaos that was observed was a \nquasi-periodic one in all cases with some variations due to the particular symmetry x(cid:173)\n- x. '~\"he following figures are obtained by plotting m(l+l) versus m(1) after discarding the \ntransient (Fig. 2). They are not qualitatively different with a reconstruction in a higher \ndimensional space. The dominant features are the following ones. \n\na) \n\nb) \n\n0.0 \n\n~. II \n\n~,--------~--------~---\n-4.1 \ninti) \n\n0.1 \n\n0.0 \n\n! \n\"'+J~ \n0.1-\n\n0.0 \n\n-4.1, \n\n~-4~.I----------------~o:o----------------~o.i----~-')-\n\nr \n\n\",: ~_ .. \n\nd) \n\n0.0 \n\n-4.1 \n\n\"r''<~ \n.. -.~~,;b:t\" \n\n0.0 \n\n0.1 \n\n\"'(II \n\nFigure 2: Example of route to chaos when the fust bifurcation is a Hopf one. \n\n(N=128, K=16). \n\na) After the first bifurcation, the zero fixed point has lost its stability. \nThe series of points (m(I), m(t+l) densely covers a cycle (gJ=l.O). \n\nb) After the second Hopf bifurcation: projection of a T2 torus (gJ=l.23). \nc) Frequency locking on the T2 torus (gJ=1.247). \nd) Chaos (gJ=1.26). \n\n\f554 \n\nDoyon, Cessac, Quoy, and Samuelides \n\nWhen the first bifurcation is a Hopf one (Fig. 2a), it is followed by a second Hopf \nbifurcation (Fig. 2b). Then there is a frequency locking occuring on the T2 torus born \nfrom the second Hopf bifurcation (Fig. 2c), followed by chaos (Fig. 2d). This route is \nthen a quasi-periodic one (Ruelle & Takens, 1971 ; Newhouse et al., 1978). A slightly \ndifferent feature emerges when the first bifurcation is followed by a stable resonance due \nto discrete time occuring before the second Hopf bifurcation. Then the limit cycle reduces \nto periodic points. When the second bifurcation occurs, the resonance persists until chaos \nis reached. \n\nWhen the first bifurcation is a pitchfork, it is followed by a Hopf bifurcation for each \nstable point of the pitchfork (due to the symmetry x -\n-x). Then a second Hopf \nbifurcation occurs followed, via a frequency locking, by chaos. It follows then, despite \nthe pitchfork bifurcation, a quasi-periodicity route. Notice that in this case, we get two \nsymmetric strange attractors. When gJ increases, the two attractors fuse. \n\nFor a first bifurcation of flip type, the route followed is like the one described by Bauer \n& Martienssen (1989). The flip bifurcation leads to an oscillatory system with two \nstates. A first Hopf bifurcation arises followed by a second one leading to a quasi-periodic \nstate, followed by a frequency locking preceeding chaos. \n\n5 CONCLUSION \n\nWe have presented a type of neural network exhibiting a chaotic behavior when \nincreasing a bifurcation parameter. As in Sompolinsky's model, gJ is the control \nparameter of the network dynamics. The variance of the synaptic weights being \nnormalized, the bifurcation values are nearly independent of the connectivity K. The \nmagnitude of dilution is not important for the behavior. The route to chaos by quasi(cid:173)\nperiodicity seems to be generic. It suggests that such high-dimensional networks behave \nlike low-dimensional dynamical systems. It could be much simpler to control such \nnetworks than a priori expected. \n\nFrom a biological point of view, we built our model to provide a tool that could be used \nto investigate the influence of chaotic dynamics in the cognitive processes in the brain. \nWe clearly chose to simplify the biological complexity in order to understand a complex \ndynamic. We think that, if chaos plays a role in cognitive processes, it does neither \ndepend on a specific architecture, nor on the exact internal modelling of the biological \nneuron. However, it could be interesting to introduce some biological caracteristics in the \nmodel. The next step will be to study the influence of non-zero entries on the behavior of \nthe system, leading to the modelling of learning in a chaotic network. \n\nAcknowledgements \n\nThis research has been partly supported by the COGNISCIENCE research program of the \nC.N.R.S. through PRESCOT, the Toulouse network of searchers in Cognitive Sciences. \n\n\fDestabilization and Route to Chaos in Neural Networks with Random Connectivity \n\n555 \n\nReferences \n\nM. Bauer & W. Martienssen. (1989) Quasi-Periodicity Route to Chaos in Neural \nNetworks. Europhys. Lett. 10: 427-431. \n\nB. Doyon, B. Cessac, M. Quoy & M. Samuelides. (1993) Control of the Transition to \nChaos in Neural Networks with Random Connectivity. Int. 1. Bifurcation and Chaos (in \npress). \n\nD. Gallez & A. Babloyantz. (1991) Predictability of human EEG: a dynamical approach. \nBiGI. Cybern. 64: 381-392. \n\nV.l.. Girko. (1985) Circular Law. Theory Prob. Its Appl. (USSR) 29: 694-706. \n\nC.M. Gray, P. Koenig, A.K. Engel & W. Singer. (1989) Oscillatory responses in cat \nvisual cortex exhibit intercolumnar synchronisation which reflects global stimulus \nproperties. Nature 338: 334-337. \n\nM. W. Hirsch. (1989) Convergent Activation Dynamics in Continuous Time Networks. \nNeural Networks 2: 331-349. \n\nS. Newhouse, D. Ruelle & F. Takens. (1978) Occurrence of Strange Axiom A Attractors \nNear Quasi Periodic Flows on rm, m ~ 3. Commun. math. Phys. 64: 35-40. \n\nD. Ruelle & F. Takens. (1971) On the nature of turbulence. Comm. math. Phys. 20: \n167-192. \n\nC.A. Skarda & W.J. Freeman. (1987) How brains makes chaos in order to make sense of \nthe world. Behav. Brain Sci. 10: 161-195. \n\nH.J. Sommers, A. Crisanti, H. Sompolinsky & Y. Stein. (1988) Spectrum of large \nrandom asymmetric matrices. Phys. Rev. Lett. 60: 1895-1898. \n\nH. Sompolinsky, A. Crisanti & H.J. Sommers. (1988) Chaos in random neural \nnetworks. Phys. Rev. Lett. 61: 259-262. \n\n\f", "award": [], "sourceid": 676, "authors": [{"given_name": "Bernard", "family_name": "Doyon", "institution": null}, {"given_name": "Bruno", "family_name": "Cessac", "institution": null}, {"given_name": "Mathias", "family_name": "Quoy", "institution": null}, {"given_name": "Manuel", "family_name": "Samuelides", "institution": null}]}