{"title": "Predicting Complex Behavior in Sparse Asymmetric Networks", "book": "Advances in Neural Information Processing Systems", "page_first": 556, "page_last": 563, "abstract": null, "full_text": "Predicting Complex Behavior \nin Sparse Asymmetric Networks \n\nAn A. Minai and William B. Levy \n\nDepartment of Neurosurgery \n\nBox 420. Health Sciences Center \n\nUniversity of Virginia \n\nCharlottesville. V A 22908 \n\nAbstract \n\nRecurrent networks of threshold elements have been studied inten(cid:173)\nsively as associative memories and pattern-recognition devices. While \nmost research has concentrated on fully-connected symmetric net(cid:173)\nworks. which relax to stable fixed points. asymmetric networks show \nricher dynamical behavior. and can be used as sequence generators or \nflexible pattern-recognition devices. In this paper. we approach the \nproblem of predicting the complex global behavior of a class of ran(cid:173)\ndom asymmetric networks in terms of network parameters. These net(cid:173)\nworks can show fixed-point. cyclical or effectively aperiodic behavior. \ndepending on parameter values. and our approach can be used to set \nparameters. as necessary. to obtain a desired complexity of dynamics. \nThe approach also provides qualitative insight into why the system \nbehaves as it does and suggests possible applications. \n\n1 INTRODUCTION \nRecurrent neural networks of threshold elements have been intensively investigated in \nrecent years. in part because of their interesting dynamics. Most of the interest has \nfocused on networks with symmetric connections. which always relax to stable fixed \npoints (Hopfield. 1982) and can be used as associative memories or pattern-recognition \ndevices. Networks with asymmetric connections. however. have the potential for much \n\n556 \n\n\fPredicting Complex Behavior in Sparse Asymmetric Networks \n\n557 \n\nricher dynamic behavior and may be used for learning sequences (see, e.g., Amari, 1972; \nSompolinsky and Kanter, 1986). \nIn this paper, we introduce an approach for predicting the complex global behavior of an \ninteresting class of random sparse asymmetric networks in terms of network parameters. \nThis approach can be used to set parameter values, as necessary, to obtain a desired \nactivity level and qualitatively different varieties of dynamic behavior. \n\n2 NETWORK PARAMETERS AND EQUATIONS \nA network consists of n identical 011 neurons with threshold O. The fixed pattern of \nexcitatory connectivity between neurons is generated prior to simulation by a Bernoulli \nprocess with a probability p of connection from neuron j to neuron i. All excitatory \nconnections have the fixed value w, and there is a global inhibition that is linear in the \nnumber of active neurons. If m (t ) is the number of active neurons at time t, K the inhi(cid:173)\nbitory weight, y; (t) the net excitation and Z; (t) the firing status of neuron i at t, and C;j a \n011 variable indicating the presence or absence of a connection from j to i, then the \nequations for i are: \n\ny;(t) = \n\nw ~ C;jZj(t-1) \n\nj -\n\nw ~ C;jzj(t-I) +Km(t-I) \n\nl:t \n\n,I~m(t-I)~n \n\nZ;(t)={1 \n\nify;(t)2!O \no otherwise ' \n\n0<0<1 \n\n(1) \n\n(2) \n\n(3) \n\nIf m (t-I) = 0, y; (t) = 0 'Vi. Equation (1) is a simple variant of the shunting inhibition \nneuron model studied by several researchers, and the network is similar to the one pro(cid:173)\nposed by Marr (Marr, 1971). Note that (1) and (2) can be combined to write the neuron \nequations in a more familiar subtractive inhibition format Defining a == OK / (I-9)w , \n\nz; (t ) = 1 \n\nif Jt C;j Zj (t -1) - a ~ Zj (t -I) 2! 0 \no otherwise \n\n3 NETWORK BEHAVIOR \nIn this paper, we study the evolution of total activity, m(t), as the system relaxes. From \nEquation (3), the firing condition for neuron i at time t, given the activity m (t-l)=M at \ntime t-I, is: e;(t).= ~C;jZj(t-l) 2! aM. Thus, in order to fire at time t, neuron i must \nhave at least r aMl active inputs. This allows us to calculate the average firing probability \nof a neuron given the prior activity M as: \n\nl:t \n\nP{lo! active inputs 2!raMll= ~ (AfJpk (l-p)M.....t =p(M;n,p,a) \n\nk,{aM! \n\n(4) \n\nIf M is large enough, we can use a Gaussian approximation to the 'binomial distribution \n\n\f558 \n\nMinai and Levy \n\nand a hyperbolic tangent approximation to the error function to get \n\np(M; n~.a) = ~ [t-eif[ ~]] = ~ [l-mn+#x]] \n\nwhere \n\nFinally, when M is large enough to assume [aMl = aM, we get an even simpler form: \n\nx == [aM] -Mp \n\"Mp(l-p) \n\np(M; n~.a)= t[l-tanh ~] \n\n(5) \n\n(6) \n\nwhere \n\nT = _1_ ~ 1tp(l-p) \n\n- a-p 2 ' \n\na.~p \n\nAssuming that neurons fire independently, as they will tend to do in such large, sparse \nnetworks (Minai and Levy, I992a,b), the network's activity at time t is distributed as \n\nP {m (1 )=N I m (t-I)=M} = (Z) p(M)H (1- p(M\u00bb\"-N \n\n(7) \n\nwhich leads to a stochastic return map for the activity: \n\nm (1) = n p(m (t-I\u00bb + O(..Jii) \n\n(8) \nIn Figure 1, we plot m (t) against m (1 -1) for a 120 neuron network and two different \nvalues of a.. The vertical bars show two standard deviations on either side of \nn p(m (t-I\u00bb. It is clear that the network's activity falls within the range predicted by (8). \nAfter an initial transient period, the system either switches off permanently (correspond(cid:173)\ning to the zero activity fixed point) or gets trapped in an 0 ( ..Jii) region around the point \nm defined by m (t) = m (t -1). We call this the attracting region of the map. The size and \nlocation of the attracting region are determined by a and largely dictate the qualitative \ndynamic behavior of the network. \nAs a. ranges from 0 to 1, networks show three kinds of behavior: fixed points, short \ncycles, and effectively aperiodic dynamics. Before describing these behaviors, however, \nwe introduce the notion of available neurons. Let k; be the number of input connections \nto neuron i (the fan-in of n. Given m (t -1) = M, if k; < [aMl, neuron; cannot possibly \nmeet the firing criterion at time t. Such a neuron is said to be disabled by activity M. The \ngroup of neurons not disabled are considered available neurons. At any specific activity \nM, there is a unique set, Na (M), of available neurons in a given network, and only neu(cid:173)\nrons from this set can be active at the next time step. Clearly, Na (M 1) !:: Na (M 2) if \nM 1 ~ M 2. The average size of the available set at a given activity M is \n\nna (M; n,p ,a) == n [1 - P {k; < [aMl} ] = n t (~)pA: (l-p )\"-A: \n\nA:=faMl \n\n(9) \n\n\fPredicting Complex Behavior in Sparse Asymmetric Networks \n\n559 \n\n(a) Eft'edlytJ, Aperiodic B.Jllwior \n8 - 0.85, K - 0.016, w - 0.4 =- a - 0.227 \n\no-~da&a(2000\") \n\n. . \n\n. . \n\n.' \n\n(11) IB&h AcdYit, C,cIe \n8-0.85, K -0.012, w -0.4 =-a-o.l1 \n\n.' \n\n8-+~--------------------------~ .' \n\n8 \n\n8I(t) \n\n118 8 \n\na.(t) \n\nUO \n\nFigure 1: Predicted Distribution of m (t+l) given met), and Empirical Data (0) for Two \nNetworks A and B. The vertical bars represent 4 standard deviations of the predicted dis(cid:173)\ntribution for each m (t). Note that the empirical values fall in the predicted range. \n\n~~--------------------------------------------------------------~ \n\n8I(t) \n\n(a) Eft'edlyitly Aperiodic BebaYIor \n8-0.85,K -0.016, w -0.4 =-a-0.221 \n\nU8 1000 \n\n8-+--------------------------------------________________________ ~ \nr-----------------------------, \n\n10000 \n\n(e) Low AcdYit, C,cIe \n8-0.85, K -0.0241, w -0.4 ... a-0.35 \n\na.(t) \n\n(11) Hlet- At:thit, C,cIe \n8-0.85, K -0.012, w -0.4 ... 11-0.11 \n\n--+-----------------------------4 \n\n400 :zoo \n\nFigure 2: Activity time-series for three kinds of behavior shown by a 120 neuron net(cid:173)\nwork. Graphs (a) and (b) correspond to the data shown in Figure 1. \n\n\f560 \n\nMinai and Levy \n\nIt can be shown that na (M) ~ n p(M), so there are usually enough neurons available to \nachieve the average activity as per (8). \n\nWe now describe the three kinds of dynamic behavior exhibited by our networks. \n(1) \n\nFixed Point Behavior: If a is very small, m is close to n, inhibition is not strong \nenough to control activity and almost all neurons switch on permanently. If a is too \nlarge, iff is close to 0 and the stochastic dynamics eventually finds, and remains at, \nthe zero activity fixed point. \n\n(2) Effectively Aperiodic Behavior: While deterministic, finite state systems such as \n\nour networks cannot show truly aperiodic or chaotic behavior, the time to repeti(cid:173)\ntion can be so long as to make the dynamics effectively aperiodic. This occurs \nwhen the attracting region is at a moderate activity level, well below the ceiling \ndefined by the number of available neurons. In such a situation, the network, start(cid:173)\ning from an initial condition, successively visits a very large number of different \nstates, and the activity, m (t), yields an effvectively aperiodic time-series of ampli(cid:173)\ntude 0 (-{,1), as shown in Figure 2(a). \n\n(3) Cyclical Behavior: If the attracting region is at a high activity level, most of the \navailable neurons must fire at every time step in order to maintain the activity \npredicted by (8). This forces network states to be very similar to each other, which, \nin turn, leads to even more similar successor states and the network settles into a \nrelatively short limit cycle of high activity (Figure 2(b\u00bb. When the attracting \nregion is at an activity level just above switch-off, the network can get into a low(cid:173)\nactivity limit cycle mediated by a very small group of high fan-in neurons (Figure \n2(c\u00bb. This effect, however, is unstable with regard to initial conditions and the \nvalue of a; it is expected to become less significant with increasing network size. \n\nu-.-------------------------------------------------. \n\n(a) \n\nMean-0.2B7 \nV 8riaDce - 0.0689 \n\n(II) \n\nMean-0.310 \nV8riance - 0.0003 \n\n\u2022 \n\n\u2022 \n\nus \n\nFlrin& Probabilit, \n\n0.75 \n\n1 \n\n\u2022 \n\nG.25 \n\nFIrinc Probabilit, \n\n8.75 \n\n1 \n\nFigure 3: Neuron firing probability histograms for two 120-neuron networks in the effec(cid:173)\ntively aperiodic phase (a ~ 0.227). Graph (a) is for a network with random connectivity \ngenerated through a Bernoulli process with p = 0.2, while Graph (b) is for a network \nwith a fixed fan-in of exactly 24, which corresponds to the mean fan-in for p = 0.2. \n\n\fPredicting Complex Behavior in Sparse Asymmetric Networks \n\n561 \n\nOne interesting issue that arises in the context of effectively aperiodic behavior is that of \nstate-space sampling within the O(f,i) constraint on activity. We assess this by looking \nat the histogram of individual neuron firing rates. Figure 3(a) shows the histogram for a \n120 neuron network in the effectively aperiodic phase. Clearly, some subspaces are being \nsampled much more than others and the histogram is very broad. This is mainly due to \ndifferences in the fan-in of individual neurons, and will diminish in larger networks. Fig(cid:173)\nure 3(b) shows the neuron firing histogram for a 120 neuron network where each neuron \nhas a fan-in of 24. The sampling is clearly much more \"ergodic\" and the dynamics less \nbiased towards certain subspaces. \n\n0.75 \n\n0.2.5 \n\no-+----------------------~--------------------------------------------------~------------------------~ \n\no \n\n0.2.5 \n\n0.75 \n\nFigure 4: The complete set of non-zero activation values available to two identical neu(cid:173)\nrons i and j with fan-in 24 in a 12O-neuron network. \n\n\f562 \n\nMinai and Levy \n\n4 ACTIVATIONDYNAMICS \nWhile our modeling so far has focused on neural firing, it is instructive to look at the \nunderlying neuron activation values, Yi. If m (t -1) = M , the possible Yi (t) values for a \nneuron i with fan-in ki are given by the set \n\ny(M, ti) ={ wq :qKM I MAX(O,ki-n+M)SqSMIN(M, til} M > 0 \n\n(10) \n\nn \n\nwith Y (0, ki ) == {O}. Here q represents the number of active inputs to i, and the set \nYi == ~ Y(M, ki) represents the set of all possible activation values for the neuron. The \nnetwork's n -dimensional activation state, y(t) == [y It Y2, ... , Yn ], evolves upon the activa(cid:173)\ntion space Y 1 X Y 2 X \u2022\u2022\u2022 x Y n' which is an extremely complex but regular object. In \ncalled a Y -Y plot - of the \nFigure 4, we plot a 2-dimensional subspace projection -\nactivation space for a 120-neuron network excluding the zero states. Both neurons shown \nhave a fan-in of 24. In actuality, only a small subset of the activation space is sampled \ndue to the constraining effects of the dynamics and the improbability of most q values. \n\n5 RELATING THE ACTMTY LEVEL TO ex \nFrom a practical standpoint, it would be useful to know how the average activity in a net(cid:173)\nwork is related to its a. parameter. This can be done using the hyperbolic tangent approxi(cid:173)\nmation of Equation (6). First, we define the activity level at time t as r (t) == n-1m (t), i.e., \nthe proportion of active neurons. This is a macrostate variable in the sense of (Amari, \n1974). In the long term, the activity level becomes confined to a o (1/...Jn) region around \nthe value corresponding to the activity fixed point Thus, it is reasonable to use r as an \nestimate for the time-averaged activity level (r). To relate m (and thus r) to a, we must \nsolve the fixed point equation m = n p(m ). Substituting this and the definition of r into \n\nl-r--------~~--------------------~ \n\n0.75 \n\n., \n\n.-r----~~----~----~------.-----~ \n\n0.04 \n\nII ~ \n\n0.1 \n\n\u2022 \n\nFigure 5: Predicted and empirical activities for 1000 neuron networks with p = 0.05. \nEach data point is averaged over 7 networks. \n\n\fPredicting Complex Behavior in Sparse Asymmetric Networks \n\n563 \n\n(6) gives: \n\na(r) = p + ~ 1tp (I-p) tanh-I (1 - 2r ) \n\n2nr \n\n(11) \n\nWhile a can range from 0 to 1, the approximation of (11) breaks down at very high or \nvery small values of r. However, the range of its applicability gets wider as n increases. \nFigure 5 shows the perfonnance of (11) in predicting the average activity level in a \n1000-neuron network. Note that a = p always leads to r = 0.5 by Equation (11). \n6 CONCLUSION \nWe have studied a general class of asymmetric networks and have developed a statistical \nmodel to relate its dynamical behavior to its parameters. This behavior, which is largely \ncharacterized by a composite parameter a, is richly varied. Understanding such behavior \nprovides insight into the complex possibilities offered by sparse asymmetric networks, \nespecially with regard to modeling such brain regions as the hippocampal CA3 area in \nmammals. The complex behavior of random asymmetric networks has been discussed \nbefore by Parisi (Parisi, 1986), Niitzel (Niitzel, 1991), and others. We show how to con(cid:173)\ntrol this complexity in our networks by setting parameters appropriately. \n\nAcknowledgements: This research was supported by NIMH MH00622 and NIMH \nMH48161 to WBL, and by the Department of Neurosurgery, University of Virginia, Dr. \nJohn A. Jane, Chairman. \n\nReferences \n\nS. Amari (1972). Learning Patterns and Pattern Sequences by Self-Organizing Nets of \nThreshold Elements. IEEE Trans. on Computers C-21, 1197-1206 \nS. Amari (1974). A Method of Statistical Neurodynamics. Kybemetik 14,201-215 \nJ.1. Hopfield (1982). Neural Networks and Physical Systems with Emergent Collective \nComputational Abilities. Proc. Nat. Acad. Sci. USA 79, 2554-2558. \nD. Marr (1971). Simple Memory: A Theory for Archicortex. Phil. Trans. R. Soc. Lond. \nB 262, 23-81. \nA.A. Minai and W.B. Levy (1992a). The Dynamics of Sparse Random Networks. In \nReview. \nA.A. Minai and W.B. Levy (1992b). Setting the Activity Level in Sparse Random Net(cid:173)\nworks. In Review. \n\nK. Niitzel (1991). The Length of Attractors in Asymmetric Random Neural Networks \nwith Deterministic Dynamics. J. Phys. A: Math. Gen 24, LI51-157. \nG. Parisi (1982). Asymmetric Neural Networks and the Process of Learning. J. Phys. A: \nMath. Gen. 19, L675-L680. \nH. Sompolinsky and I. Kanter (1986), Temporal Association in Asymmetric Neural Net(cid:173)\nworks. Phys. Rev. Lett. 57,2861-2864. \n\n\f", "award": [], "sourceid": 592, "authors": [{"given_name": "Ali", "family_name": "Minai", "institution": null}, {"given_name": "William", "family_name": "Levy", "institution": null}]}