{"title": "A Winner-Take-All Circuit with Controllable Soft Max Property", "book": "Advances in Neural Information Processing Systems", "page_first": 717, "page_last": 723, "abstract": null, "full_text": "A Winner-Take-All Circuit with \nControllable Soft Max Property \n\nShih-Chii Lin \n\nInstitute for Neuroinformatics, ETHjUNIZ \n\nWinterthurstrasse 190, CH-8057 Zurich \n\nSwitzerland \n\nshih@ini.phys.ethz.ch \n\nAbstract \n\nI describe a silicon network consisting of a group of excitatory neu(cid:173)\nrons and a global inhibitory neuron. The output of the inhibitory \nneuron is normalized with respect to the input strengths. This out(cid:173)\nput models the normalization property of the wide-field direction(cid:173)\nselective cells in the fly visual system. This normalizing property is \nalso useful in any system where we wish the output signal to code \nonly the strength of the inputs, and not be dependent on the num(cid:173)\nber of inputs. The circuitry in each neuron is equivalent to that in \nLazzaro's winner-take-all (WTA) circuit with one additional tran(cid:173)\nsistor and a voltage reference. Just as in Lazzaro's circuit, the \noutputs of the excitatory neurons code the neuron with the largest \ninput. The difference here is that multiple winners can be chosen. \nBy varying the voltage reference of the neuron, the network can \ntransition between a soft-max behavior and a hard WTA behav(cid:173)\nior. I show results from a fabricated chip of 20 neurons in a 1.2J.Lm \nCMOS technology. \n\n1 \n\nIntroduction \n\nLazzaro and colleagues (Lazzaro, 1988) were the first to implement a hardware \nmodel of a winner-take-all (WTA) network. This network consists of N excitatory \ncells that are inhibited by a global signal. Improvements of this network with ad(cid:173)\ndition of positive feedback and lateral connections have been described (Morris, \n1998; Indiveri, 1998). The dynamics and stability properties of networks of cou(cid:173)\npled excitatory and inhibitory neurons have been analyzed by many (Amari, 1982; \nGrossberg, 1988). Grossberg described conditions under which these networks will \nexhibit WTA behavior. Lazzaro's network computes a single winner as reflected by \nthe outputs of the excitatory cells. Several winners can be chosen by using more \nlocalized inhibition. \nIn this work, I describe two variants of a similar architecture where the outputs of \nthe excitatory neurons code the relative input strengths as in a soft-max compu(cid:173)\ntation. The relative values of the outputs depend on the number of inputs, their \nrelative strengths and two parameter settings in the network. The global inhibitory \n\n\f718 \n\n8.-c. Liu \n\nFigure 1: Network model of recurrent inhibitory network. \n\nsignal can also be used as an output. This output saturates with increasing num(cid:173)\nber of active inputs, and the saturation level depends on the input strengths and \nparameter settings. This normalization property is similar to the normalization be(cid:173)\nhavior of the wide-field direction-selective cells in the fly visual system. These cells \ncode the temporal frequency of the visual inputs and are largely independent of the \nstimulation size. The circuitry in each neuron in the silicon network is equivalent \nto that in Lazzaro et. al.'s hard WTA network with an additional transistor and \na voltage reference. By varying the voltage reference, the network can transition \nbetween a soft-max computation and a hard WTA computation. In the two vari(cid:173)\nants, the outputs of the excitatory neurons either code the strength of the inputs \nor are normalized with respect to a constant bias current. Results from a fabri(cid:173)\ncated network of 20 neurons in a 1.2J.Lm AMI CMOS show the different regimes of \noperation. \n\n2 Network with Global Inhibition \n\nThe generic architecture of a recurrent network with excitatory neurons and a single \ninhibitory neuron is shown in Figure 1. The excitatory neurons receive an external \ninput, and they synapse onto a global inhibitory neuron. The inhibitory neuron, in \nturn, inhibits the excitatory neurons. The dynamics of the network is described as \nfollows: \n\ndYi \ndt = -Yi + ei - g(~ WjYj) \n\nN \n~ \n\n(1) \n\nwhere Wj is the weight of the synapse between the jth excitatory neuron and the \ninhibitory neuron, and Yj is the state of the jth neuron. Under steady-state condi-\ntions, Yi = ei - YT, where YT = g(L:~l WjYj)\u00b7 \nAssume a linear relationship between YT and Yj, and letting Wj = W, \n\nj=l \n\n\"N \n_ ~ _ W L..Jj=l ej \n1 + wN \n\nYT - W ~ Yj -\n\nN \n\nj=l \n\nAs N increases, YT = L:;l ej \u2022 If all inputs have the same level, e, then YT = e. \n\n\fA Winner-Take-All Circuit with Controllable Soft Max Property \n\n719 \n\nFigure 2: First variant of the architecture. Here we show the circuit for two excita(cid:173)\ntory neurons and the global inhibition neuron, M 4 \u2022 The circuit in each excitatory \nneuron consists of an input current source, h, and transistors, M1 to M 3 . The \ninhibitory transistor is a fixed current source, lb . The inputs to the inhibitory \ntransistor, 101 and I~2 are normalized with respect to lb. \n\n3 First Variant of Network with Fixed Current Source \n\nIn Sections 3 and 4, I describe two variants of the architecture shown in Figure \n1. The two variants differ in the way that the inhibition signal is generated. The \nfirst network in Figure 2 shows the circuitry for two excitatory neurons and the \ninhibition neuron. Each excitatory neuron is a linear threshold unit and consists of \nan input current, h, and transistors, Ml, M 2 , and M 3 . The state of the neuron is \nrepresented by the current, Ir1 . The diode-connected transistor, M 2 , introduces a \nrectifying nonlinearity into the system since Ir1 cannot be negative. The inhibition \ncurrent, Ir, is sunk by M 1 , and is determined by the gate voltage, VT. The inhibition \nneuron consists of a current source, Ib, and VT is determined by the corresponding \ncurrent, Ir1 and the corresponding transistor, M3 in each neuron. Notice that IT \ncannot be greater than the largest input to the network and the inputs to this \nnetwork can only be excitatory. The input currents into the transistor, M 4 , are \ndefined as 101 and 102 and are normalized with respect to the current source, h. In \nthe hard WTA condition, the output current of the winning neuron is equal to the \nbias current, h. \nThis network exhibits either a soft-maximum behavior or a hard WTA behavior \ndepending on the value of an external bias, Va. The inhibition current, IT, is \nderived as: \n\n(2) \n\nwhere N is the number of \"active\" excitatory neurons (that is, neurons whose \nIi > IT), Ii is the same input current to each neuron, and Ia = Ioe\",vQ/uT. In \nderiving the above equation, we assumed that K, = 1. The inhibition current, IT, is \na linear combination of the states of the neurons because Ir = 2:f Iri x Ial h\u00b7 \nFigure 3(a) shows the response of the common-node voltage, VT, as a function of \nthe number of inputs for different input values measured from a fabricated silicon \nnetwork of 20 neurons. The input current to each neuron is provided by a pFET \ntransistor that is driven by the gate voltage, Yin. All input currents are equal in \nthis figure. The saturation behavior of the network as a function of the number \n\n\f720 \n\ns.-c. Liu \n\n0.8,---~-~-~-~-~----, \n\nVin=3.9V \n\n-' . \n\n0.7 \n\n0.6 \n\n.II'~''''--.'' ......... \u2022\u2022 ' .. - - - ... ... ..... .. \n\nVin=4.3V \n\n5 \n\n10 \n\n20 \nNumber of inputs \n\n15 \n\n25 \n\n30 \n\n5 \n\n10 \n\n20 \nNumber of inputs \n\n15 \n\n25 \n\n30 \n\n(a) \n\n(b) \n\nFigure 3: (a) Common-node voltage, VT, as a function of the number of input \nstimuli. Va = O.8V. (b) Common-node voltage, VT, as a function of the number \nof inputs with an input voltage of 4.3V and Vb = O.7V. The curves correspond to \ndifferent values of Va . \n\nof inputs can be seen in the different traces and the saturation level increases as \nVin decreases. As seen in Equation 2, the point at which the response saturates is \ndependent on the ratio, h / I a. In Figure 3(b), I show how the curve saturates at \ndifferent points for different values of Va and a fixed hand Vin. \nIn Figure 4, I set all inputs to zero except for two inputs, Vin1 and Vin2 that are set \nto the same value. I measured 101 and 101 as a function of Va as shown in Figure \n4(a). The four curves correspond to four values of Vin. Initially both currents 101 \nand 102 are equal as is expected in the soft-max condition. As Va increases, the \nnetwork starts exhibiting a WTA behavior. One of the output currents finally goes \nto zero above a critical value of Va. This critical value increases for higher input \ncurrents because of transistor backgate effects. In Figure 4(b), I show how the \noutput currents respond as a function of the differential voltage between the two \ninputs as shown in Figure 4. Here, I fixed one input at 4.3V and swept the second \ninput differentially around it. The different curves correspond to different values of \nVa. For a low value of Va, the linear differential input range is about lOOmV. This \nlinear range decreases as Va is increased (corresponding to the WTA condition). \n\n4 Second Variant with Diode-Connected Inhibition \n\nTransistor \n\nIn the second variant shown in Figure 5, the current source, M4 is replaced by a \ndiode-connected transistor and the output currents, 10i' follow the magnitude of \nthe input currents. The inhibition current, Ir, can be expressed as follows: \n\nwhere la is defined in Section 3. We sum Equation 3 over all neurons and assuming \nequal inputs, we get Ir = J'LJri x la. This equation shows that the feedback \nsignal has a square root dependence on the neuron states. As we will see, this \ncauses the feedback signal to saturate quickly with the number of inputs. \n\n(3) \n\n\fA Winner-Take-All Circuit with Controllable Soft Max Property \n\n721 \n\n6 \n\n5 \n~4 \n~ , \n\n2 \n\nr-... \n\n\".\\'0\" \n\n2.5 Va=O 5~'~\\ I /~. \" \nVa=O 6V Al\\ ',/' Va=O.4V \n~ 2 \n-: 1.5 \n..s \n\nVa=O.7V \n\n,i \nl~ \n) ' ~ ,. \n, <\\ ,,~ \n.. \\ \\> \n/ \n\n0.5 \n\n< .,.~J \\ \\ '\" \n\n0 \n\n0.1 \n\n0.2 \n\n0.3 \n\n-0.2 \n\n-0.1 \n\nVio2-Viol (V) \n\n(a) \n\n(b) \n\nFigure 4: (a) Output currents, 101 and 102 , as a function of Va: for a subthreshold \nbias current and Yin = 4.0V to 4.3V. (b) Outputs, 101 and 102 , as a function of the \ndifferential input voltage, ~ Vin, with Yinl = 4.3V. \n\nFigure 5: Second variant of network. The schematic shows two excitatory neurons \nwith diode-connected inhibition transistor. \n\nSubstituting lri = Ii - IT in Equation 3, we solve for Jr, \n\nIT = -1a:N + \n\n(Ia: N )2 + 41a: L Ii \n\nN \n\n(4) \n\nFrom measurements from a fabricated circuit with 20 neurons, I show the depen(cid:173)\ndence of VT (the natural logarithm of Jr) on the number of inputs in Figure 6(a). \nThe output saturates quickly with the number of inputs and the level of saturation \nincreases with increased input strengths. All the inputs have the same value. \nThe network can also act as a WTA by changing Va:. Again, all inputs are set to \nzero except for two inputs whose gate voltages are both set at 4.2V. As shown in \nFigure 6(b), the output currents, 101 and 102 , are initially equal, and as Va: increases \nabove 0.6V, the output currents split apart and eventually, 102 = OA. The final value \nof 101 depends on the maximum input current. This data shows that the network \nacts as a WTA circuit when Va: > 0.73V. If I set Vin2 = 4.25V instead, the output \ncurrents split at a lower value of Va:. \n\n\f722 \n\ns.-c. Liu \n\n0.45.--~-~-~-~-~------, \n\nVinl=4.2V. Vin2=4.25V \n\n5 \n\n4 \n$ \n~3 \n\n2 \n\n0.20'---:-2 -~4-~6:--8=---1-:':0:----:'12 \n\nNumber of inputs \n\n(a) \n\n0.6 \n\n0.7 \n\nVa (V) \n(b) \n\n0.8 \n\n0.9 \n\nFigure 6: (a) Common-node voltage, VT, as a function of the number of inputs for \ninput voltages, 3.9V, 4.06V, and 4.3V for Va = O.4V. (b) Outputs, 101 and 1 02 , as \na function of Va for Vinl = 4.2V, Vin2 = 4.25V for the 2 curves with asterisks and \nfor Vinl = Vin2 = 4.2V for the 2 curves with circles. \n\n5 \n\nInhibition \n\nThe WTA property arises in both variants of this network if the gain parameter, \nVa, is increased so that the diode-connected transistor, M 2 , can be ignored. Both \nvariants then reduce to Lazzaro's network. In the first variant, the feedback current \n(Ir) is a linear combination ofthe neuron states. However, when the gain parameter \nis increased so that M2 can be ignored, the feedback current is now a nonlinear \ncombination of the input states so the WTA behavior is exhibited by these reduced \nnetworks. \nUnder hard WTA conditions, if Ir is initially smaller than all the input currents, the \ncapacitances C at the nodes Vr1 and Vr2 are charged up by the difference between \nthe individual input current and IT, i.e., d~t = liCIT. Since the inhibition current \nis a linear combination of Iri and Iri is exponential in Vri , we can see that IT is \na sum of the exponentials of the input currents, h Hence the feedback current is \nnonlinear in the input currents. Another way of viewing this condition in electronic \nterms is that in the soft WTA condition, the output node of each neuron is a soft(cid:173)\nimpedance node, or a low-gain node. In the hard WTA case, the output node is now \na high-impedance node or a high-gain node. Any input differences are immediately \namplified in the circuit. \n\n6 Discussion \n\nHahnloser (Hahnloser, 1998) recently implemented a silicon network of linear thresh(cid:173)\nold excitatory neurons that are coupled to a global inhibitory neuron. The inhibitory \nsignal is a linear combination of the output states of the excitatory neurons. This \nnetwork does not exhibit WTA behavior unless the excitatory neurons include a \nself-excitatory term. The inhibition current in his network is also generated via a \ndiode-connected transistor. The circuitry in two variants described here is more \ncompact than the circuitry in his network. \n\nRecurrent networks with the architecture described in this paper have been proposed \nby Reichardt and colleagues (Reichardt, 1983) in modelling the aggregation property \n\n\fA Winner-Take-All Circuit with Controllable Soft Max Property \n\n723 \n\nof the wide-field direction-selective cells in flies. The synaptic inputs are inhibited \nby a wide-field cell that pools all the synaptic inputs. Similar networks have also \nbeen used to model cortical processing, for example, orientation selectivity (Douglas, \n1995). \n\nThe network implemented here can model the aggregation property of the direction(cid:173)\nselective cells in the fly. By varying a voltage reference, the network implements \neither a soft-max computation or a hard WTA computation. This circuitry will be \nuseful in hardware models of cortical processing or motion processing in inverte(cid:173)\nbrates. \n\nAcknowledgments \n\nI thank Rodney Douglas for supporting this work, and the MOSIS foundation for \nfabricating this circuit. I also thank Tobias Delbriick for proofreading this docu(cid:173)\nment. This work was supported in part by the Swiss National Foundation Research \nSPP grant and the U.S. Office of Naval Research. \n\nReferences \n\nAmari, S., and Arbib, M. A., \"Competition and cooperation in neural networks,\" \nNew York, Springer-Verlag, 1982. \n\nGrossberg, W., \"Nonlinear neural networks: Principles, mechanisms, and architec(cid:173)\ntures,\" Neural Networks, 1, 17-61, 1988. \nHanhloser, R., \"About the piecewise analysis of networks of linear threshold neu(cid:173)\nrons,\" Neural Networks, 11,691- 697, 1988. \nHahnloser, R., \"Computation in recurrent networks of linear threshold neurons: \nTheory, simulation and hardware implementation,\" Ph.D. Thesis, Swiss Federal \nInstitute of Technology, 1998. \n\nLazzaro, J., Ryckebusch, S. Mahowald, M.A., and Mead. C., \"Winner-take-all net(cid:173)\nworks of O(n) complexity,\" In Tourestzky, D. (ed), Advances in Neural Information \nProcessing Systems 1, San Mateo, CA: Morgan Kaufman Publishers, pp. 703-711, \n1988. \n\nMorris, T .G., Horiuchi, T. and Deweerth, S.P., \"Object-based selection within an \nanalog VLSI visual attention system,\" IEEE Trans. on Circuits and Systems II, \n45:12, 1564-1572, 1998. \n\nIndiveri, G., \"Winner-take-all networks with lateral excitation,\" Neuromorphic Sys(cid:173)\ntems Engineering, Editor, Lande, TS., 367-380, Kluwer Academic, Norwell, MA, \n1998. \nReichardt, W., Poggio, T., and Hausen, K., \"Figure-ground discrimination by rel(cid:173)\native movement in the visual system of the fly,\" BioI. Cybern., 46, 1-30, 1983. \nDouglas, RJ., Koch, C., Mahowald, M., Martin, KAC., and Suarez, HH., \"Recurrent \nexcitation in neocortical circuits,\" Science, 269:5226,981-985, 1995. \n\n\f", "award": [], "sourceid": 1725, "authors": [{"given_name": "Shih-Chii", "family_name": "Liu", "institution": null}]}