{"title": "An Analog Self-Organizing Neural Network Chip", "book": "Advances in Neural Information Processing Systems", "page_first": 739, "page_last": 747, "abstract": null, "full_text": "739 \n\nAN ANALOG SELF-ORGANIZING \n\nNEURAL NElWORK CHIP \n\nJames R. Mann \n\nSheldon Gilbert \n\nMIT Lincoln Laboratory \n\n244 Wood Street \n\nLexington, MA 02173\"()()73 \n\n4421 West Estes \n\nLincolnwood, IL 60646 \n\nABSTRACT \n\nA design for a fully analog version of a self-organizing feature map neural \nnetwork has been completed. Several parts of this design are in fabrication. \nThe feature map algorithm was modified to accommodate circuit solutions \nto the various computations required. Performance effects were measured \nby simulating the design as part of a frontend for a speech recognition \nsystem. Circuits are included to implement both activation computations and \nweight adaption 'or learning. External access to the analog weight values is \nprovided to facilitate weight initialization, testing and static storage. This \nfully analog implementation requires an order of magnitude less area than \na comparable digital/analog hybrid version developed earlier. \n\nINTRODUCTION \n\nThis paper describes an analog version of a self-organizing feature map circuit. The design \nimplements Kohonen's self-organizing feature map algorithm [Kohonen, 1988] with some \nmodifications imposed by practical circuit limitations. The feature map algorithm automatically \nadapts connection weights to nodes in the network such that each node comes to represent a \ndistinct class of features in the input space. The system also self-organizes such that neighboring \nnodes become responsive to similar input classes. The prototype circuit was fabricated in two \nparts (for testability); a 4 node, 4 input synaptic array, and a weight adaptation and refresh \ncircuit. A functional simulator was used to measure the effects of design constraints. This \nsimulator evolved with the design to the point that actual device characteristics and process \nstatistics were incorporated. The feature map simulator was used as a front-end processor to \na speech recognition system whose error rates were used to monitor the effects of parameter \nchanges on performance. \n\nThis design has evolved over the past two years from earlier experiments with a perceptron \nclassifier [Raffel, 1987] and an earlier version of a self-organizing feature map circuit [Mann, \n1988]. The perceptron classifier used a connection matrix built with multiplying D / A converters \nto perform the product operation for the sum-of-products computation common to all neural \nnetwork algorithms. The feature map circuit also used MDAC's to perform a more complicated \ncalculation to realize a squared Euclidean distance measure. The weights were also stored \ndigitally, but in a unary encoded format to simplify the weight adjustment operation. This circuit \ncontained all of the control necessary to perform weight adaptation, except for selecting a \nmaximum responder. \nThe new feature map circuit described in this paper replaces the digital weight storage with \ndynamic analog charge storage on a capacitor. This paper will describe the circuitry and discuss \nproblems associated with this approach to neural network implementations. \n\nReprinted with pennission of Lincoln Laboratory, Massachusetts Institute of Tedmology, Lexington, \nMassachusetts \n\n\f740 \n\nMann and Gilbert \n\nALGORITHM DESCRIPTION \n\nThe original Kohonen algorithm is based on a network topology such as shown in Figure 1. This \nillustrates a linear array of nodes, consistent with the hardware implementation being descnbed. \nEach node in the circuit computes a level of activity [Dj(t)] which indicates the similarity \nbetween the current input vector [Xi(t)] and its respective weight vector [Wij(t)]. Traditionally \nthis would be the squared Euclidean distance given by the activation equation in the figure. If \nthe inputs are normalized, a dot product operation can be substituted. The node most \nrepresentative of the current input will be the one with the minimum or maximum output \nactivity (classification), depending on which distance measure is used. The node number of the \nmin.fmax. responder U\u00b7] then comes to represent that class of which the input is a member. \nIf the network is still in its learning phase, an adaptation process is invoked. This process \nupdates the weights of all the nodes lying within a prescribed neighborhood [NEjj\u00b7(t)] of the \nselected node. The weights are adjusted such that the distance between the input and weight \nvector is diminished. This is accomplished by decreasing the individual differences between each \ncomponent pair of the two vectors. The rate of learningis controlled by the gain term [aCt)]. \nBoth the neighborhood and gain terms decrease during the learning process, stopping when the \ngain term reaches O. \nThe following strategy was selected for the circuit implementation. First, it was assumed that \ninputs are normalized, thereby permitting the simpler dot product operation to be adopted. \nSecond, weight adjustments were reducedto a simple increment / decrement operation determined \nby the sign of the difference between the components of the input and weight vector. Both of \nthese Simplifications were tested in the simulations described earlier and had negligible effects \non overall performance as a speech vector quantizer. In addition, the prototype circuits of the \nanalog weight version of the feature map vector quantizer do not include either the max. picker \nor the neighborhood operator. To date, a version of a max. picker has not yet been chosen, \nthough many forms exist. The neighborhood operator was included in the previous version of \nthis design, but was not repeated on this ftrst pass. \n\nHARDWARE DESCRIPTION \n\nSYNAPTIC ARRAY \n\nA transistor constitutes the basic synaptic connection used in this design. An analog input is \nrepresented by a voltage v(Xi) on the drain of the transistor. The weight is stored as charge \nq(Wij) on the gate of the transistor. If the gate voltage exceeds the maximum input voltage by \nan amount greater than the transistor threshold voltage, the device will be operating in the \nohmic region. In this region the current [i(Dj)] through the transistor is proportional to the \nproduct of the input and weight voltages. This effectively computes one contribution to the dot \nproduct. By connecting many synapses to a single wire, current summing is performed, in \naccordance with Kirchofrs current law, producing the desired sum of products activity. \n\nFigure 2 shows the transistor current as a function of the input and weight voltages. These \ncurves merely serve to demonstrate how a transistor operating in the ohmic region will \napproximate a product operation. \nAs the input voltage begins to approach the saturation region of the transistor, the curves begin \nto bend over. For use in competitive learning networks, like the feature map algorithm, it is \nonly important that the computation be monotonically increasing. These curves were the \ncharacteristics of the computation used in the simulations. The absolute values given for output \ncurrent do not reflect those produced in the actual circuit. \n\n\fAn Analog Self-Organizing Neural Network Chip \n\n741 \n\n\u2022 ACTIVATION : \n\nm \n\nDj(l) = 2: (x,(I) - W'j(I))2 \n\ni=1 \n\n\u2022 CLASSIFICATION : \n\nj' = M!N(D,(I)) \n\n, \n\n\u2022 ADAPTATION : \n\nFigure 1. Description of Kohonen's original feature map algorithm using a \n\nlinear array of nodes. \n\n/l. A \n\n200 ,-------,-------...---~---~ WEIGHT (Vgs) \n\n150 \n\nlao \n\n50 \n\n~ \nI-\n::;) \nc.. \nI-\n::;) \n0 \n\nS.OV \n4.BV \n\n4.6V \n\n4.4V \n4.2V \n\n4.0V \n\nJ.BV \nJ.6V \n\n~.4V \n3.2V \n\n3.0V \n\nC L-__ ~ ___ ~ ___ _ _L ___ __ _ \n\no \n\n0.5 \n\n1 \n\n1.5 \n\nINPUT (Vds) \n\n2 \nV \n\nFigure 2. Typical T -V curves for a transistor operating in the ohmic region. \n\n\f742 \n\nMann and Gilbert \n\nIt should also be noted that there is no true zero weight; even the zero weight voltage \ncontnbutes to the output current. But again, in a competitive network, it is only important that \nit contribute less than a higher weight value at that same input voltage. \nIn short, neither small non-linearities nor offsets interfere with circuit operation if the synapse \ncharacteristic is monotonic with weight value and input. \n\nSYSTEM \n\nFigure 3 is a block diagram of the small four-node hardware prototype. The nodes are oriented \nhorizontally, their outputs identified as 10 through 13 along the right-hand edge, representing the \naccumulated currents. The analog inputs [X3-XO] come in from the bottom and, traveling \nvertically, make connections with each node at the boxes identified as synapses. Each synapse \nperforms its product operation between the analog weight stored at that node and the input \npotential. \nAlong the top and left sides are the control circuits for accessing weight information. The two \nstorage registers associated with each synapse are the control signals used to select the reading \nand writing of weights. Weights are accessed serially by connecting to a global read and write \nwire, W- and W + respectively. Besides the need for modification, the weights also drift with \ntime, much like DRAM storage, and therefore must be refreshed periodically. This is also \nperformed by the adaptation circuit that will be presented separately. \nControl is provided by having a single \"1\" bit circulating through the DRAM storage bits \nassociated with each synapse. This process goes on continuously in the background after being \ninitialized, in parallel with the activity calculations. If the circuit is not being trained, the \nadaptation circuit continues to refresh the existing weights. \n\nWEIGHT MODIFICATION & REFRESH \nA complete synapse, along with the current to voltage conversion circuit used to read the weight \ncontents, is shown in Figure 4. The current synapse is approximately the size of two 6 tr~sistor \nstatic RAM bits. This approximation will be used to make synaptic population estimates from \ncurrent SRAM design experience. The six transistors along the top of the synapse circuit are \ntwo, three-transistor dynamic RAM cells used to control access to weight contents. These are \nrepresented in Figure 3 as the two storage elements associated with each synapse and are used \nas descnbed earlier. \n\nREADING THE WEIGHT \nThe two serial, vertically oriented transistors in the synapse circuit are used to sense the stored \nweight value. The bottom (sensing) transistor's channel is modulated by the charge stored on \nthe weight capacitor. The sensing transistor is selected through the binary state of the 3T \nDRAM bit immediately above it. These two transistors used for reading the weight are \nduplicated in the outpu~ circuit shown to the right of the synapse. The current produced in the \nglobal read wire through the sensing transistor, is set up in the cascode current mirror \narrangement in the output circuit. A mirrored version of the current, leaving the right hand side \nof the cascode mirror, is established in the duplicate transistor pair. The gate of this transistor \nis controlled by the operational amplifier as shown, and must be equivalent to the weight valueat \nthe connection being read, if the drains are both at the same potential. This is guaranteed by \nthe cascode mirror arrangement selected, and is set by the minus input to the amplifier. \n\nWRITING THE WEIGHT \nThe lone horizontal transistor at the bottom right comer of the synapse circuit is the weight \naccess transistor. This connects the global write wire[W +] to the weight capacitor [Wij]. This \n\n\fAn Analog Self-Organizing Neural Network Chip \n\n743 \n\n(581 x 320 microns) \n\nROW\u00b7CTRl\u00b7IN w+ \n\nW-X3 \n\nX2 \n\nXl \n\nXO \n\nFigure 3. A block diagram of the 4 x 4 synaptic array integrated circuit. \n\nSYNAPSE \n\n(82 x 32 MICRONS) \n\n,dl \n\nwrl \n\n,d2 \n\nwr2 \n\nRO \n\n\u2022 \nI \n\nOUTPUT CIRCUIT \n\nI'IR--;---~--+.--+--=--t--+-+-\n\nI \n, \n\ni \n\ni \n\n~~I' \nT \n: :fiT1 \nI I \n-: ~rJ I \nj ?' \n\nI r \n\n\\'IIi \n\n1'1. -r----r-+--;----+-_f_ \n\nX, \n\nW\u00b7 \n\nFigure 4. Full synapse circuit. Activation transistor is at bottom central \n\nposition in the synapse circuit. \n\n\f744 \n\nMann and Gilbert \n\noccurs whenever the DRAM element directly above it is holding a \"1\". When the access \ntransistor is off, current leakage takes place, causing the voltage on the capacitor to drift with \ntime. \n\nThere are two requirements on the weight drift for our application; That drift rates be as slow \nas possible, and that they drift in a known direction, in our case, toward ground. This is true \nbecause the refresh mechanism always raises the voltage to the top of a quantized voltage bin. \n\nA cross-section of the access transistor in Figure 5 identifies the two major leakage components; \nreverse diode leakage to the grounded substrate (or p-well) [10], and subthreshold channel \nconduction to the global write wire[Id]. The reverse diode leakage current is proportional to \nthe area of the diffusion while the channel conduction leakage is proportional to the channel \nW /L ratio. Maintaining a negative voltage drift can be accomplished by sizing the devices such \nthat reverse diode leakage dominates the channel conduction. This however would degrade the \noverall storage performance, and hence the minimum refresh cycle time. This can be relaxed by \nthe technique of holding the global write line at some low voltage during everything but write \ncycles. This then makes the average voltage seen across the channel less than the minimum \nweight voltage, always resulting in a net voltage drop. \nAlso, these leakage currents are exponentially dependent on temperature and can be decreased \nby an order of magnitude with just 10's of degrees of cooling [SChwartz, 1988]. \n\nWEIGHT REPRESENTATION \nWeights, while analog, are restricted to discrete voltages. This permits the stored voltage to drift \nby a restricted amount (a bin), and still be refreshed to its original value. The drift rate just \ndiscussed, combined with the bin size (determined by the levels of quantization (i.e. 'of bins) and \nweight range (i.e. column height\u00bb, determines the refresh cycle time. The refresh cycle time, \nin tum, determines how many synapses (or weights) can be served by a single adaptation circuit. \nThis means that doubling the range of the weight voltage would permit either doubling the \nnumber of quantization levels or doubling the number of synapses served by one adaptation \ncircuit. \nWeight adjustments during learning involve raising or lowering the current weight voltage to the \nbins immediately above or below the current bin. This constitutes a digital increment or \ndecrement operation. \n\nADAPTATION CIRCUITRY \nWeight adjustments are made based upon a comparison between the current weight value and \nthe input voltage connected to that weight. But, as these two ranges are not coincident, the \ncomparison is made between two binary values produced by parallel flash AID converters \n[Brown, 1987]. The two opposing AID converters in Figure 6, produce a 1-of-N code, used in \nthe comparison. The converters are composed of two stages to conserve area. The fIrst stage \nperforms a coarse conversion which in tum drives the upper and lower rails of the second stage \nconverter. The selection logic decides which of the voltages among those in the second stage \nweight conversion circuit to route back on the global write wire [W +]. \nThis conflguration provides an easy mechanism for setting the ranges on both the inputs and \nweights. This is accomplished merely by setting the desired maximum and minimum voltages \ndesired on the respective conversion circuits ([Xmin,Xmax] [Wmin,Wmax]). \n\nTEST RESULTS \n\nBoth circuits were fabricated in MOSIS. The synaptic array was fabricated in a 3 micron 2 metal \nCMOS process while the adaptation circuitry was fabricated in a similar 2 micron process. To \ndate, only the synaptic array has been tested. In these tests, the input was restricted to a 0 to1 \n\n\fAn Analog Self-Organizing Neural Network Chip \n\n745 \n\nI \n\nFigure 5. Cross-sectional view of a weight access transistor with leakage \n\ncurrents. \n\n~~~m \n~-~T! \n\\ I \n- '.---, \nI \n<;:, \n:; ~,compH \n- ... ~ !: \n_.... \n\nI \n\"\"'--\n\n~ \n\n: \n\nI \nI \n.... \n\na.1 \n\nSELECT \nLOGIC \n\nI---.-----J \n\nFigure 6. Block diagram of the weight adaptation and refresh circuit. \n\nComparison of digital A \\D outputs and new weight selection takes \nplace in the box marked SELECT LOGIC. \n\nW-mln \n\n\f746 \n\nMann and Gilbert \n\nV range while the weight range was 2 to 3 V. Most of these early tests were done with binary \nweights, either 2 V or 3 V, corresponding to a \"O\"and a \"1\". \nThe synapses and associated control circuitry all work as expected. The circuit can be clocked \nup to 7 MHz. The curves shown in Figure 7 display a typical neuron output during two modes \nof operation; a set of four binary weights with all of the inputs swept together over their \noperating range, and a single, constant input with its weight being swept through its operating \nrange. \nThe graphs in Figure 8 show the temporal behavior of the weight voltage stored at a single \nsynapse. On the left is plotted the output current to weight VOltage, for converting between the \ntwo quantities. The right hand plot is the output current of the synapse plotted against time. \nIf the weight VOltage bin size is set to 15 m V (2V range, 128 bins), a 3 to 4 second refresh cycle \ntime limit would be required. This is a very lenient constraint and may permit a much finer \nquantization than expected. \nThe circuitry for reading the weights was tested and appears to be inoperative. The casco de \nmirror requires a very high potential at the p-channel sources which causes the circuit to latch \nup when the clocks are turned on. This circuit will be isolated and tested under static \nconditions. \n\nCONCLUSIONS \n\nIn summary, a design for an analog version of a self- organizing feature map has been completed \nand prototype versions of the synaptic array and the adaptation circuitry have been fabricated. \nThe devices are still undergoing testing and characterization, but the basic DRAM control and \nsynaptic operation have been demonstrated. Simulations have provided the guidance on design \nchoices. These have been instrumental in providing information on effects due to quantization, \ncomputational non-linearities, and process variations. The new design offers a significant \nincrease in density over a digital/analog hybrid approach. The 84 pin standard frame package \nfrom MOSIS will accommodate more than 8000 synapses of from 6 to 8 bits accuracy. \nIt \nappears that control modifications may offer even greater densities in future versions. \n\nThis work was sponsored by the Department of the Air Force, and the Defense Advanced \nResearch Projects Agency, the views expressed are those of the author and do not reflect the \nofficial policy or pOSition of the U.S. Government. \n\nREFERENCES \n\nP. Brown, R Millecchia M. Stinely. Analog Memory for Continuous-Voltage, Discrete-Time \nImplementation of Neural Networks. Proc. IEEE IntI. Conf. on Neural Networks. 1987. \n\nT. Kohonen. Self-Organization and Associative Memory. Springer-Verlag. 1988. \n\nJ. Mann, R Lippmann, R Berger J. Raffel. A Self-Organizing Neural Net Chip. IEEE 1988 \nCustom Integrated Circuits Conference. pp. 103.1-103.5. 1988. \n\nJ. Raffel, J. Mann, R Berger, A. Soares S. Gilbert. A Generic Architecture for Wafer-Scale \nNeuromorphic Systems. Proc. IEEE IntI. Conf. on Neural Networks.1987. \n\nD.B. Schwartz RE. Howard. A Programmable Analog Neural Network Chip. IEEE 1988 \nCustom Integrated Circuits Conference. pp. 10.2.1-10.2.4. 1988. \n\n\fAn Analog Self-Organizing Neural Network Chip \n\n747 \n\nI . \n\n:... \n\nI \nI \nI \nI \n\n'J l \n\nI , \n\nI \n\nI \n\n~ \ni \nI \nIl \n\nX1 (V) \n\nX1 (V) \n\na) \n\nb) \n\nFigure 7. a) plot of output current (lj) as a function of imput voltage (Xi) \nbetween 0 and I volt for 0 (top curve) to 4 (bottom curve) weights \n\"DN\". b) plot of output current (Ij) vs. input voltage (Xi) from \no to IV for a weight voltage between 2V (top) and 3V (bottom) \nin O.IV steps. \n\n0.212 \n\nY I \"\". y(lon) \n\nI \nO.261------~------~---\n\nI \n\nI \n\nI \n\nI \n\nI \n\nI \nI \nV(lou,) (V) I \nI \nI \n\nI \n\n0 . 2S ' r - - - - - -+ - -\n\n--1'------\n\n0 . 2Ut--- ---+-------+..------\n\nI \n\nI \n\nI \nI \n\nI \n\nI \nI \nI \nI \n\nI \nI \nI \n\nI \nI \n\nI \n\nI \nI \nI \nI \n\n0.25C=\",,; =-----:-h----..\"..+,,--___=' \n:s \n\n2.t7 \n\nYel (V) \n\nO .\u2022 II.....-_~--.:.;~~:....:,:::=---~-___. \n\n, \n\nI \n\nI \n\nI \nI \nI \nI \n\nV(lo.,) (V) I \n\nI \nI \n\n\u2022 \n\n0 .\n\nI \nI \n\n261----1 --:---r---l---\no.m ,---1----:-- -r---t---\nI \no.258~---l---~---~. _ L __ _ \n\nI \n\nI \n\nI \n\nI \n\n. \n\nI \n\nI \n\nI \n\n0 . 257 \n\n\" \n\no \n\nI \n\n: \n\nI \n\nI \n\n: \n\nI \n\nI \n\nI \nI \n2 :S \nn \u2022\u2022 (I'c) \n\nI \nI \nI \n4 \n\n6 \n\na) \n\nb) \n\nFigure 8. a) plot of output current V (lout) vs. weight voltage. b) plot of \noutput current as a function of time with W + held at OVX and the \nlocal weight initially set to 3V. \n\n\f", "award": [], "sourceid": 155, "authors": [{"given_name": "James", "family_name": "Mann", "institution": null}, {"given_name": "Sheldon", "family_name": "Gilbert", "institution": null}]}