{"title": "Discontinuous Recall Transitions Induced by Competition Between Short- and Long-Range Interactions in Recurrent Networks", "book": "Advances in Neural Information Processing Systems", "page_first": 337, "page_last": 345, "abstract": null, "full_text": "Discontinuous Recall Transitions Induced By \nCompetition Between Short- and Long-Range \n\nInteractions in Recurrent Networks \n\nN.S. Skantzos, C.F. Beckmann and A.C.C. Coolen \n\nDept of Mathematics, King's College London, Strand, London WC2R 2LS, UK \n\nE-mail: skantzos@mth.kcl.ac.uktcoolen@mth.kcl.ac.uk \n\nAbstract \n\nWe present exact analytical equilibrium solutions for a class of recur(cid:173)\nrent neural network models, with both sequential and parallel neuronal \ndynamics, in which there is a tunable competition between nearest(cid:173)\nneighbour and long-range synaptic interactions. This competition is \nfound to induce novel coexistence phenomena as well as discontinuous \ntransitions between pattern recall states, 2-cycles and non-recall states. \n\n1 \n\nINTRODUCTION \n\nAnalytically solvable models of large recurrent neural networks are bound to be simplified \nrepresentations of biological reality. In early analytical studies such as [1,2] neurons were, \nfor instance, only allowed to interact with a strength which was independent of their spatial \ndistance (these are the so-called mean field models). At present both the statics of infinitely \nlarge mean-field models of recurrent networks, as well as their dynamics away from satura(cid:173)\ntion are well understood, and have obtained the status of textbook or review paper material \n[3,4]. The focus in theoretical research of recurrent networks has consequently turned to \nnew areas such as solving the dynamics of large networks close to saturation [5], the anal(cid:173)\nysis of finite size phenomenology [6], solving biologically more realistic (e.g. spike-based) \nmodels [7] or analysing systems with spatial structure. In this paper we analyse mod(cid:173)\nels of recurrent networks with spatial structure, in which there are two types of synapses: \nlong-range ones (operating between any pair of neurons), and short-range ones (operating \nbetween nearest neighbours only). In contrast to early papers on spatially structured net(cid:173)\nworks [8], one here finds that, due to the nearest neighbour interactions, exact solutions \nbased on simple mean-field approaches are ruled out. Instead, the present models can be \nsolved exactly by a combination of mean-field techniques and the so-called transfer ma(cid:173)\ntrix method (see [9]). In parameter regimes where the two synapse types compete (where \none has long-range excitation with short-range inhibition, or long-range Hebbian synapses \nwith short-range anti-Hebbian synapses) we find interesting and potentially useful novel \nphenomena, such as coexistence of states and discontinuous transitions between them. \n\n\f338 \n\nN. S. Skantzos, C. F Beckmann and A. C. C. Coo/en \n\n2 MODEL DEFINITIONS \n\nWe study models with N binary neuron variables Ui = \u00b11. which evolve in time stochas(cid:173)\ntically on the basis of post-synapi.ic potentials hi (8). following \n\nProb[ui(t + 1) = \u00b11] = ~ [1 \u00b1 tanh[,Bhi(8(t))Jl \n\nhi(8) = L JijUj + ()i \n\n(1) \n\n#i \n\nThe variables Jij and ()i represent synaptic interactions and firing thresholds. respectively. \nThe (non-negative) parameter ,B controls the amount of noise. with,B = 0 and,B = 00 corre(cid:173)\nsponding to purely random and purely deterministic response. respectively. If the synaptic \nmatrix is symmetric. both a random sequential execution and a fully parallel execution of \nthe stochastic dynamics (1) will evolve to a unique equilibrium state. The corresponding \nmicroscopic state probabilities can then both formally be written in the Boltzmann form \nPoo(a) ,...., exp[-,BH(a)], with [10] \n\ni..f- + >..I'!, in which \n>..\u00b1 are the eigenvalues of the 2 x 2 matrix T, enables us to take the limit N -+ 00 in our \nequations. The integral over (m, m) is for N -+ 00 evaluated by gradient descent, and is \ndominated by the saddle points of the exponent \u00a2 . We thus arrive at the transparent result \n\nf = extr \u00a2(m, m) \n\n{\n\nA \n\n) \n\n( \n\nA. \n'l'seq m,m = -~mm - m \nA. \n'l'par m, m = -~mm - m \n\n( \n\nA \n\nA \n\n\u2022 \n\nA \n\n\u2022 \n\n) \n\n0 \n\n-\n0 \n\n-\n\n1 I \n\\ seq \n73 og,,+ \n\n1 J \n2 \n2\" tm -\n\\ par \n1 I \n73 og \"+ \n\n(9) \n\nwhere >..~q and >..~ar are the largest eigenvalues of Tseq and T par. For simplicity, we will \nrestrict ourselves to the case where 0 = 0; generalisation of what follows to the case of \narbitrary 0, by using the full form of (9), is not significantly more difficult. The expressions \ndefining the value(s) of the order parameter m can now be obtained from the saddle point \nequations om\u00a2(m, m) = om\u00a2(m, m) = O. Straightforward differentiation shows that \n\nsequential: m = imJt , \nm =imJt , \nparallel: \nm = -imJt, \n\nm = G(m; Jt, Ja ) \n\nm = G(m; Jt, Ja ) \nm = G(m;-Jt,-Ja ) \n\nfor Jt 2: 0 \nfor Jt < 0 \n\nwith \n\nG( \n\n. J J) _ \n\nm, t, a \n\nsinh[{3Jtm ] \n\n- -r========== \nJsinh2[{3Jtm] + e-4{3J. \n\n(10) \n\n(11) \n\nNote that equations (to, 11) allow us to derive the physical properties of the parallel dy(cid:173)\nnamics model from those of the sequential dynamics model via simple transformations. \n\n\f340 \n\nN. S. Skantzos, C. F Beckmann and A. C. C. Coo/en \n\n4 PHASE TRANSITIONS \n\nOur main order parameter m is to be determined by solving an equation of the form m = \nG(m), in which G(m) = G(m; Jl. Js ) for both sequential and parallel dynamics with \nJl ~ 0, whereas G(m) = G(m;-Jl.-Js ) for parallel dynamics with Jl < O. Note that, \ndue to G(O; h, J s ) = 0, the trivial solution m = 0 always exists. In order to obtain a phase \ndiagram we have to perform a bifurcation analysis of the equations (10,11), and determine \nthe combinations of parameter values for which specific non-zero solutions are created or \nannihilated (the transition lines). Bifurcations of non-zero solutions occur when \n\nm = G(m) \n\nand \n\n1 = G'(m) \n\n(12) \n\nThe first equation in (12) states that m must be a solution of the saddle-point problem, the \nsecond one states that this solution is in the process of being created/annihilated. Nonzero \nsolutions of m = G (m) can come into existence in two qualitatively different ways: as con(cid:173)\ntinuous bifurcations away from the trivial solution m = 0, and as discontinuous bifurcations \naway from the trivial solution. These two types will have to be treated differently. \n\n4.1 Continuous Transitions \n\nAn analytical expression for the lines in the ({3Js \u2022 {3Jl) plane where continuous transitions \noccur between recall states (where m =f. 0) and non-recall states (where m = 0) is obtained \nby solving the coupled equations (12) for m = O. This gives: \n\ncont. trans . : \n\nsequential: \nparallel: \n\n{3Jl = e- 2/3J. \n{3Jl = e- 2/3J. \n\nand \n\n{3Jl = _e 2/3J. \n\n(13) \n\nIf along the transition lines (13) we inspect the behaviour of G(m) close to m = 0 we \ncan anticipate the possible existence of discontinuous ones, using the properties of G(m) \nfor m -+ \u00b1oo, in combination with G(-m) = -O(m). Precisely at the lines (13) we \nhave G(m) = m + i-G'''(O).m3 + O(m5 ) . Since liIllm-+oo G(m) = 1 one knows that \nfor G11I(0) > 0 the function G(m) will have to cross the diagonal G(m) = m again at \nsome value m > 0 in order to reach the limit G (00) = 1. This implies, in combination \nwith G (-m) = -0 (m), that a discontinous transition must have already taken place earlier, \nand that away from the lines (13) there will consequently be regions where one finds five \nsolutions of m = G(m) (two positive ones, two negative ones). Along the lines (13) the \ncondition G11I (0) > 0, pointing at discontinuous transitions elsewhere, translates into \n\nsequential : \nparallel: \n\n{3Jl > J3 and \nI{3Jll > J3 and \n\n{3Js < - i log 3 \nI{3Js l < - i log 3 \n\n(14) \n\n4.2 Discontinuous Transitions \n\nIn the present models it turns out that one can also find an analytical expression for the \ndiscontinuous transition lines in the ({3Js , {3h) plane, in the form of a parametrisation. For \nsequential dynamics one finds a single line, parametrised by x = {3Jlm E [0,00) : \n\ndiscont. trans. : \n\n{3Jl(X) = \n\nx3 \n\n( ). \n\nx-tanh x \n\n{3Js (x) = _~ 10 \ng \n\n4 \n\n[tanh(X) Sinh2 (X)] \n\n( ) \nx-tanh x \n\n(15) \nSince this parametrisation (15) obeys {3Js (O) = -i log3 and {3Jl(O) = J3, the discontin(cid:173)\nuous transition indeed starts precisely at the point predicted by the convexity of G (m) at \nm = 0, see (14). For sequential dynamics the line (15) gives all non-zero solutions of the \ncoupled equations (12). For parallel dynamics one finds, in addition to (15), a second 'mir(cid:173)\nror image' transition line, generated by the transformation {{3Jl, {3Js } H {-{3h, -{3Js } . \n\n\fCompetition between Short- and Long-Range Interactions \n\n341 \n\n5 PHASE DIAGRAMS \n\n4 r-\n\n6 .. ........ ---\n2 r \n\ncoex \n\n--\n---\n---\n-\n--\n-, \n\nm=O, aO, a>O \n\nm..(),a>O \n\n-6 \n\n-2 \n\n-I \n\n0 \nf3Js \n\n~ \n1 \n\n~ \n\n1 \nj \n, \n\n~ \n\n2 \n\n2 ~ \n\n0 \n\nm=O, aO, a>O \nfixed point \n\n1 \n1 \n\nf3lt \n\n> \n-2 ~ \nt \n-4 -r \n-6 \n\n-2 \n\n1m1>O, a 0), and (iii) a region where the m = 0 state and the two m #- 0 \nstates coexist. The (i) -+ (ii) and (ii) -+ (iii) transitions are continuous (solid lines), whereas the \n(i) -+ (iii) transition is discontinuous (dashed line). Right: phase diagram for parallel dynamics, \ninvolving the above regions and transitions, as well as a second set of transition lines (in the region \nIt < 0) which are exact reflections in the origin of the first set. Here. however, the m = 0 region has \na = tanh[2f3Jsl, the two m #- 0 physical solutions describe 2-cycles rather than fixed-points, and \nthe Jl < 0 coexistence region describes the coexistence of an m = 0 fixed-point and 2-cycles. \n\nHaving detennined the transition lines in parameter space. we can turn to the phase dia(cid:173)\ngrams. A detailed expose of the various procedures followed to detennine the nature of the \nvarious phases, which are also dependent on the type of dynamics used, goes beyond the \nscope of this presentation; here we can only present the resulting picture. 1 Figure 1 shows \nthe phase diagram for the two types of dynamics, in the (j3Js , j3Jl ) plane (note: of the \nthree parameters {j3, Js , Jd one is redundant). In contrast to models with nearest neigh(cid:173)\nbour interactions only (Jl = 0, where no pattern recall ever will occur), and to models with \nmean-field interactions only (Js = 0, where pattern recall can occur), the combination of \nthe two interaction types leads to qualitatively new modes of operation. This especially in \nthe competition region, where Jl > 0 and J s < 0 (Hebbian long-range synapses, com(cid:173)\nbined with anti-Hebbian short range ones). The novel features of the diagram can playa \nuseful role: phase coexistence ensures that only sufficiently strong recall cues will evoke \npattern recognition; the discontinuity of the transition subsequently ensures that in the lat(cid:173)\nter case the recall will be of a substantial quality. In the case of parallel dynamics, similar \nstatements can be made in the opposite region of synaptic competition, but now involving \n2-cycles. Since figure 1 cannot show the zero noise region (13 = T- 1 = 00), we have also \ndrawn the interesting competition region of the sequential dynamics phase diagram in the \n(Jl, T) plane, for Js = -1 (see figure 3, left picture). At T = 0 one finds coexistence of \nrecall states (m i:- 0) and non-recall states (m = 0) for any Jl > 0, as soon as Js < O. \nIn the same figure (right picture) we show the magnitude of the discontinuity in the order \nparameter m at the discontinuous transition, as a function of j3Jl. \n\nIDue to the occurrence of imaginary saddle-points in (10) and our strategy to eliminate the vari(cid:173)\nable m by using the equation om(m, m) = 0, it need not be true that the saddle-point with the low(cid:173)\nest value of <1>( m, m) is the minimum of * (complex conjugation can induce curvature sign changes. \nand in addition the minimum could occur at boundaries or as special limits). Inspection of the sta(cid:173)\ntus of saddle-points and identification of the physical ones in those cases where there are multiple \nsolutions is thus a somewhat technical issue, details of which will be published elsewhere [11]. \n\n\f342 \n\n6 \n\n5 \n\n4 \n\nT \n\n3 \n\nt \n\nr \n2 t \n1 t \n\nr \n\n0 \n\n0 \n\nN. S. Skantzos. C. F Beckmann and A. C. C. Coolen \n\nj \n\n1.0 \n\n0.8 \n\nm 06 f \n0.4 t \n0.2 r \n\nf \n\n0.0 \n\n0 \n\n~ \n\nj \n\n10 \n\n2 \n\n4 \n\n6 \n{31t \n\n8 \n\n, \n, , \n, \n, , \n, , \n, , \n, , \n, , \n, , \n\" \n\ncMSisttnct \n\nm=IJ, a*