{"title": "Phase Transitions in Neural Networks", "book": "Neural Information Processing Systems", "page_first": 192, "page_last": 200, "abstract": null, "full_text": "192 \n\nPHASE TRANSITIONS IN NEURAL NETWORKS \n\nUniversity of Wisconsin, Madison, WI 53706 \n\nJoshua Chover \n\nABSTRACT \n\nVarious simulat.ions of cort.ical subnetworks have evidenced \nsomething like phase transitions with respect to key parameters. \nWe demonstrate that. such transi t.ions must. indeed exist. in analogous \ninfinite array models. For related finite array models classical \nphase transi t.ions (which describe steady-state behavior) may not. \nexist., but. there can be distinct. quali tative changes in \n(\"metastable\") transient behavior as key system parameters pass \nthrough crit.ical values . \n\nINTRODUCTION \n\nc \n\nc \n\nthe act.ivity tends to be much higher than \n\nSuppose that one st.imulates a neural network - actual or \nsimulated - and in some manner records the subsequent firing \nactivity of cells. Suppose further that. one repeats the experiment. \nfor different. values of some parameter (p) of the system: and that \none finds a \"cri t.ical value\" (p) of the parameter, such that. \n(say) for values p > p \nc \nit. is for values p < p. Then, by analogy with statist.ical \nmechanics (where, e.g., p may be temperature, with criUcal \nvalues for boiling and freezing) one can say that. the neural \nnetwork undergoes a \"phase transition\" at. p. Intracellular phase \ntransi t.ions, parametrized by membrane potential, are well mown. \nHere we consider intercellular phase transi t.ions. These have been \nevidenced in several detailed cort.ical simulations: e.g., of the \n2 \n1 \npiriform cortex and of the hippocampus \nthe parameter p \nspontaneous EPSPs \nhippocampal case, the parameter was the ratio of inhibitory to \nexcitatory cells in the system. \n\nreceived by a typical pyramidal cell; in the \n\nrepresented the frequency of high amplitude \n\nc \n\nIn the piriform case, \n\nBy what. mechanisms could approach to, and retreat. from, a \n\ncri t.ical value of some parameter be brought about.? An intriguing \nconjecture is that. neuromodulators can play such a role in certain \n3 \nnetworks; temporarily raising or depressing synaptic efficacies \nWhat. possible interesting consequences could approach to \ncriticality have for system performance. Good effects could be \nthese: \nto a stimulus can mean faster changes in synaptic efficacies, which \nwould bring about. faster memory storage. More and longer activi ty \ncould also mean faster access to memory. A bad effect. of \n\nfor a network with plasticity, heightened firing response \n\n\u00a9 American Institute of Physics 1988 \n\n\fnear-criticality - depending on other parameters - can be wild, \nepileptiform activity. \n\nPhase transitions as they might. relate to neural networks have \n\n193 \n\n4 \nbeen studied by many authors \nparticular category of network models - abstracted from the \npiriform cortex set.ting referred to above - and show the following: \n\nHere, for clarity, we look at. a \n\na) For \"elementary\" reasons, phase transition would have to \n\nexist if there were infinitely many cells; and the near-subcrit.ical \nstate involves prolonged cellular firing activity in response to an \nini t.ial stimulation. \n\nb) Such prolonged firing activity takes place for analogous \n\nlarge finite cellular arrays - as evidenced also by computer \nsimulat.ions. \n\nWhat. we shall be examining is space-time patterns which \n\ndescribe the mid-term transient. activity of (Markovian) systems \nthat. tend to silence (with high probability) in the long run. \n(There is no reference to energy functions, nor to long-run stable \nfiring rates - as such rates would be zero in most. of our cases.) \nIn the following models time will proceed in discrete steps. \n\n(In the more complicated set.tings these will be short. in comparison \nto other time constants, so that. the effect of quant.ization becomes \nsmaller.) The parameter p will be the probability that at. any \ngiven t.ime a given cell will experience a certain amount. of \nexci tatory \"spontaneous firing\" input.: by itself this amount. will \nbe insufficient. to cause the cell to fire, but. in conjunction wi th \nsufficiently many exci tatory inputs from other cells it. can assist. \nin reaching firing threshold. \naverage firing threshold value and average efficacy value give \nsimilar results.) \nIn all the models there is a refractory period \nafter a cell fires, during which it cannot fire again; and there \nmay be local (shunt. type) inhibition by a firing cell on near \nneighbors as well as on itself - but. there is no long-distance \ninhibi tion. We look first. at. limi ting cases where there are \ninfini tely many cells and - classically - phase transi tion appears \nin a sharp form. \n\n(Other related parameters such as \n\nA \"SIMPLE\" MODEL \n\nWe consider an infinite linear array of similar cells which \n\nobey the following rules, pictured in Fig. lA: \n\n(i) If cell k \n\nfires at. time n, \n\nthen it. must. be silent. \n\nat. t.ime n+l; \n\n(11) \n\nif cell k \n\nis silent. at. time n but. both of its \n\nneighbors k-l and k+l do fire at. time n, \nat. t .ime n+l; \n\nthen cell k \n\nfires \n\n(iii) \n\nif cell k \n(k-l or k+ I) \n\nneighbors \nfire at t .ime n+l with probability p and not. fire with \nprobability \ncells and at. other times. \n\nindependently of similar decisions at. other \n\nfires at. time n, \n\nthen ce 11 k wi 11 \n\nl-p, \n\nis silent at time n and just one of its \n\n\f194 \n\nA TIM~ \n\n~ CELLS\"\"\"'> \n\n~ \u00b7\u00b7\u00b70. 0 \n\n~ \n\ni \n\n1'\\ \n\n000 \n~ \noollio \n\ni \n\nDOD \n\nFig. 1. \"Simple model\". A: \n\nfiring rules; cells are represented \n\nhorizontally, time proceeds downwards: filled squares \ndenote firing. B: \n\nsample development. \n\nThus, effecUvely, signal propagat ion speed here is one cell \n\nIf we sUmulate ~ cell to fire at time n::O, will its \n\nper uni t. time, and a cell's firing threshold value is 2 (EPSP \nunits). \ninfluence necessarily die out or can it. go on forever? \n(See \nFig. lB.) For an answer we note that. in this simple case the \nfiring paUern (if any) at. Ume n must. be an alternat.ing stretch \nof firing/silent. cells of some length, call it. L. Moreover, \n\nn \n\nL \n\n2 \nI = L +2 with probability p \n\nn+ \nfiring assists on both ends of the stretch), or \n\nn \n\n(when there are sponteneous \n\nLn+l = Ln-2 with \n(when there is no assist at. either. end of the \n\nLn+l = Ln with probability 2p(l-p) \n\n(when there is \n\nprobability \nstretch), or \n\n2 \n(l-p) \n\nan assist. at. just. one end of the stretch). \n\nStart.ing wi th any fini te al ternating stretch La, \n\nthe \n\nsuccessive values L \nn \n\nconsUtute a \"random walk\" among the \n\nnonnegat.ive integers. \n\nIntui t.ion and simple analysis5 lead to the \n\nsame conclusion: \n\nto decrease \u00ab1_p)2) \n\nif the probability for L \nn \n2 \n\nis greater than that. for it. to increase (p) -\nstep taken by the random walk is negative -\n\ni.e. if the average \n\nthen ul t .imately L \nn \n\nwill reach a \n\nand the firing response dies out. COntrariwise, if \n\n\f195 \n\n2 \n) (l-p) \n\nthen the L \ncan drift. to even higher values wi th \nn \nIn Fig. 2A we sketch the probability for \n\n2 \nP \npositive probability. \nultimate die-out as a function of p: and in Fig. 2B, the average \ntime until die out. Figs. 2A and B show a classic example of phase \ntransition \n\nfor this infinite array. \n\n(p = 1/2) \n\nc \n\nA \n\n8 \n\n, \n\n\\ \n\n\\ \n\n-\n\nI \n\u00b71\u00b7-- - - I'~ l<-\n'I) \n\nf. \n\nFig. 2. Critical behavior. A: probability of ultimate die out. (or \n\naverage time until die-out (or for reaching other \n\nof reaching other traps. in finite array case). \nB: \ntraps). Solid curves refer to an infinite array; dashed, \nto finite arrays. \n\nMORE mMPLEX MODELS \n\nFor an infinite linear array of cells, as sketched in Fig. 3 \n\n. \nwe describe now a much more general (and hopefully more realistic) \nset of rules: \n\n(i') A cell cannot fire, nor receive excitatory inputs. at. \nif it has fired at any time during the preceding ~ Hme \n\ntime n \nunits (refraction and feedback inhibition). \n\n(11 .) Each cell x has a local \"inhibitory neighborhood\" \n\nif any other cell y \n\nconsisting of a number (j) of cells to its immediate right. and \nleft.. The given cell x cannot. fire or receive excitatory inputs \nat Hme n \nhas fired at. any t .ime between \nwhere \nto x at. a speed of VI cells per unit time. \nrepresents local shunt~type inhibition.) \n\nin its inhibi tory neighborhood \nt+mI uni ts preceding n, \n\nt . is the t .ime it. would take for a message to travel from y \n\n(This rule \n\nt. and \n\n(iii') Each cell x has an \"excitatory neighborhood\" \n\nconsisting of a number (e) of cells to the immediate right. and left \nof its inhibitory neighborhood. \nin that. neighborhood \nfires at a certain time. that firing causes a unit impulse to \ntravel to cell x at a speed of vE cells per uni t. time. The \nimpulse is received at. x subject to rules (i') and (11'). \n\nIf a cell y \n\n\f196 \n\n(s < 9). \n\n(iv') All cells share a \"firing threshold\" value 9 and an \n\n\"integraUon Ume constant.\" \nIn addition each cell. at. \neach t.ime n and independent ly of other times and other cells. can \nreceive a random amount. X \nn \n\nof \"spontaneous excitatory input.\". \n\ns \n\nThe variable Xn can have a general distribution: however. for \nsimplicity we suppose here that. it. assumes only one of two values: \nb or O. with probabilities p and 1-p respecUvely. \nsuppose that. b <. e. \ninsufficient. for firing.) The above quant.i ties enter into the \nfollowing firing rule: \nprevented by rules (i') and (ii') and if the total number of inputs \nfrom other cells. received during the integration \"window\" last.ing \nbetween t.imes n-s+1 and n \n\nso that. the spontaneous \"assist.\" itself is \n\na cell will fire at. time n \n\ninclusive. plus the assist. X , \nn \n\nif it. is not. \n\n(We \n\nequals or exceeds the threshold 9. \n\n(The propagat.ion speeds vI and VE and the neighborhoods \n\nare here given left.-right. syrrmetry merely for ease in exposi tion.) \n\no 0 0 0 tl 0 \u2022 \n\n[J \u2022 Jl tl tl [J U ' 0 0 0 D \u2022 11 0 \n\n0 Jl 0 '\" \n\n~Iit' \n\n'\"\" \n\nI~h I \nI \nt \n\nt ~ I \u2022 )l,. \n\nFig. 3. Message travel in complex model: \n\nsee text. rules \n\n(i')-(iv'). \n\nWi 11 such a mode 1 d i sp lay phase trans i t i on a t. some cr i t .i cal \n\nvalue of the spontaneous firing frequency p? The dependence of \nresponses upon the ini t.ial condi tions and upon the various \nparameters is intricate and wi 11 affect. the answer. We briefly \ndiscuss here conditions under which the answer is again yes. \n\n(1) For a given configuration of parameters and a given \n\nini Ual stimulation (of a stretch of cont.iguous cells) we compare \nthe development. of the model's firing response first. to that. of an \nauxil iary \"more act.ive\" system: Suppose that. L \nnow denotes the \nn \ndistance at. t.ime n between the left:- and right.-most cells which \nare either firing or in refractory mode. Because no cell can fire \nwi thout. influence from others and because such influence travels at. \na given speed, there is a maximal amount. \nexceed L. n \n\nThere is also a maximum probability Q(p) - which \n\n(D) whereby L 1 can \n\nn+ \n\n\f197 \n\na \n\nn \n\nn \n\nthat. L 1 ~ L \nn \n\nn+ \n\n(viz., DQ(p)+(-I)(I-Q(b\u00bb) \n\nl-Q(p). At each transition, An \n\n'We can compare L with a random walk \"A\" \nn \n\nis \nis more likely to die \ntends to zero as p \n\ndepends on the spontaneous firing parameter p -\n(whatever n). \ndefined so that. An+l = An+D with probability Q(p) and \nAn+l = An-1 with probability \nmore likely to increase than L. Hence L \nn \nou t . than A \nIn the many cases where Q(p) \nn \ndoes, the average step size of A \nn \nwi 11 become negat.ive for p below a \"cri tical\" value p. Thus, \nas in the \"simple\" model above, the probability of ultimate die-out \nfor the A, hence also for the L \nof the complex model, will be \nn \n1 when 0 ~ p < p . a \n(2) There will be a phase transition for the complex model if \nits probability of die out. - given the same parameters and initial \nstimulation is in (1) - becomes less than 1 for some p values \nwith p < p < 1. Comparison of the complex process with a simpler \n\"less act.ive\" process is difficul t. in general. However, there are \nparameter configurat.ions which ul timately can channel all or part. \nof the firing activity into a (space-t.ime) sublat.t .ice analgous to \nthat. in Fig. 1. Fig. 4 illustrates such a case. For p \nsufficiently large there is posi tive probabili ty that. the act.ivity \nwill not. die out, just as in the \"simple\" model. \n\nn \n\na \n\nFig. 4. Activity on a sublattice. \n\nMR=2, M1=I, VR=V1=I, 9=3, s=2, \nareas indicate refract.ionlinhibi tion: diagonal lines, \nexcitatory influence. \n\n(Parameter values: \n\nj=2, e=6, \nand b=I.) Rectangular \n\n\f198 \n\nLARGE FINITE ARRAYS \n\nConsider now a large finite array of N cells, again as \n\nsketched in Fig. 3 ; and operating according to rules similar to \n(i')-(iv') above, with suitable modifications near the edges. \nAppropriately encoded, its activity can be described by a (huge) \nMarkov transit.ion matrix, and - depending on the initial \nst.imulation - must. tend5 to one of a set. of steady-state \ndistribut.ions over firing patterns. For example, \nif N \nodd and the rules are those for Fig. I, then extinct.ion is the \nunique steady state, for any p (1 (since the L \nn \nwalk with \"reflecUng\" upper barrier). But, \nand the cells are arranged in a ring, then, for any P with \no < p < 1. both ext.inction and an alternate flip-flop firing \npat.tern of period 2 are \"traps\" for the system - wi th relative long \nrun probabilities determined by the initial state. See the dashed \n\u00ab(3) case, \nline in Fig. 2A for the extinction probability in the \nand in Fig. 2B for the expected time until hitting a trap in the \n(a) case \n\nform a random \nis even \n\n(P(2) and the {(3) case. \n\nif N \n\n(a) \n\n\u00ab(3) \n\nis \n\n1 \n\nWhat quali tat.ive properties related to phase transi tion and \n\n(a) \n\nexample above shows that long term activity \n\ncritical p values carryover from the infinite to the finite \narray case? The \nmay now be the same for all 0 ( p (1 but. that parameter \nintervals can exist. whose key feature is a particularly large \nexpected t.ime before the system hi ts a trap. \nregion can depend upon the ini tial st.imulation.) Prior to being \ntrapped the system spends its time among many states in a kind of \n\"metastable\" equilibrium. \n(We have some preliminary theoretical \nresults on this conditional equilibrium and on its relation to the \ninfinite array case. See also Ref. 6 concerning time scales for \nwhich certain corresponding infinite and finite stochastic automata \nsystems display similar behavior . ) \n\n(Again. the cri tical \n\nSimulat.ion of models satisfying rules (i' )-( iv') does indeed \n\ndisplay large changes in length of firing activity corresponding to \nparameter changes near a critical value. See Fig. 5 for a typical \nexample: As a function of p, \nthe expected time until the system \nis trapped (for the given parameters) rises approximately linearly \nin the interval \n- as is the case in Fig. 5A at. time n=115 \n(for p=.10). But. for \np).15 a relatively rigid patterning sets in which leads with high \nprobability to very long runs or to traps other than extinction -\nas is the case in Fig. 5B \n(p=.20) where the run is arbitrarity \ntruncated at. n=525. \n(The patterning is highly influenced by the \nlarge size of the excitatory neighborhoods.) \n\n.05