{"title": "Pulse-Firing Neural Chips for Hundreds of Neurons", "book": "Advances in Neural Information Processing Systems", "page_first": 785, "page_last": 792, "abstract": null, "full_text": "Pulse-Firing Neural Chips for Hundreds of Neurons \n\n785 \n\nPULSE-FIRING NEURAL CIDPS \nFOR HUNDREDS OF NEURONS \n\nMichael Brownlow \nLionel Tarassenko \nDept.  Eng.  Science \nUniv.  of Oxford \nOxford  OX1  3PJ \n\nAlan F.  Murray \n\nDept.  Electrical Eng. \nUniv.  of Edinburgh \n\nMayfield  Road \n\nEdinburgh EH9 3JL \n\nAlister Hamilton \nII  Song Han(l) \nH. Martin Reekie \nDept.  Electrical Eng. \nU niv.  of Edinburgh \n\nABSTRACT \n\nWe  announce  new  CMOS  synapse  circuits  using  only  three \nand four  MOSFETsisynapse.  Neural states are asynchronous \npulse  streams,  upon  which  arithmetic  is  performed  directly. \nChips  implementing  over  100  fully  programmable  synapses \nare  described  and  projections  to  networks  of  hundreds  of \nneurons are made. \n\n1 OVERVIEW OF PULSE FIRING NEURAL  VLSI \nThe  inspiration  for  the  use  of  pulse  firing  in  silicon  neural  networks  is \nclearly  the  electrical/chemical  pulse  mechanism  in  \"real\"  biological  neurons. \nAsynchronous,  digital  voltage  pulses  are  used  to  signal  states  t Si  )  through \nsynapse  weights  {  Tij  }  to  emulate  neural  dynamics.  Neurons  fire  voltage \npulses  of a frequency  determined  by  their  level  of activity  but of a  constant \nmagnitude  (usually  5  Volts)  [Murray,1989a].  As  indicated  in  Fig.  1, \nto \nsynapses  perform  arithmetic  directly  on  these  asynchronous  pulses, \nincrement  or  decrement  the  receiving  neuron's  activity.  The  activity  of  a \nreceiving  neuron  i,  Xi  is  altered  at  a  frequency  controlled  by  the  sending \nneuron  j,  with  state  Sj  by  an  amount  determined  by  the  synapse  weight \n(here,  T ij ). \n\n1 On secondment from  the Korean Telecommunications Authority \n\n\f786 \n\nBrownlow, Tarassenko, Murray, Hamilton, Han and Reekie \n\nSj> 0 \nlij > 0 \n\nSj> 0 \nlij < 0 \n\nSj  =  0 \nlij > 0 \n\nSj  > 0 \nTij = 0 \n\nSj > 0 \nlij < 0 \n\nI \n\nveo \n\nx\u00b7 I \nt=:= \n\nS. I \n\n.fL.fL.n-IL \n\nFigure 1  : Pulse stream synapse functionality \n\nthis \n\nis \n\ntechnique \n\ntherefore  an \nA  silicon  neural  network  based  on \nasynchronous,  analog  computational  structure. \nIt  is  a  hybrid  between \nanalog  and  digital  techniques in  that  the individual  neural  pulses  are  digital \nvoltage  spikes,  with  all  the  robustness to noise  and  ease  of regeneration that \nthis  implies.  These  and  other  characteristics  of  pulse  stream  networks  will \nbe  discussed  in  detail  later  in  this  paper.  Pulse  stream  methods,  developed \nin  Edinburgh,  have  since  been  investigated  by  other  groups  - see  for \ninstance [EI-Leithy,1988, Daniell, 1989]. \n1.1. WHY PULSE STREAMS? \nThere  are  some  advantages  in  the  use  of  pulse  streams,  and  pulse  rate \nencoding,  in implementing neural networks.  It should  be admitted here that \nthe  initial  move  towards  pulse  streams  was  motivated  by  the  desire  to \nimplement pseudo-analog circuits on an essentially digital  CMOS process.  It \nwas  a  decision  based  at  the  time  on  expediency rather  than  on great  vision \non  our  part,  as  we  did  not  initially  appreciate  the full  benefits  of this form \nof pulse stream arithmetic [Murray,1987]. \n\n\fPulse-Firing Neural Chips for Hundreds of Neurons \n\n787 \n\nFor  example,  the  voltages  on  the  terminals  of  a  MOSFET,  V GS  and  V DS \ncould  clearly  be  used  to code a  neural synapse weight  and state respectively, \ndoing  away  with  the  need  for  pulses.  In  the  pulse  stream  form,  however, \nwe  can  arrange  that  only  VGS  is  an  \"unknown\".  The  device  equations  are \ntherefore  easily  simplified,  and  furthermore \nthe  body  effect  is  more \npredictable.  In  an  equivalent  continuous  - time  circuit,  VDS  will  also  be  a \nvariable,  which  codes  information.  Predicting  the  transistor's  operating \nregime  becomes  more  difficult,  and  the  equation  cannot  be  simplified. \nAside  of the  transistor  - level  advantages,  giving  rise  to  extremely  compact \nsynapse  circuits,  there  may be architectural  advantages.  There are  certainly \narchitectural consequences.  Digital  pulses  are easier to regenerate,  easier to \npass  between  chips,  and  generally  far  more  noise  - insensitive  than  analog \nvoltages,  all  of  which  are  significant  advantages  in  the  VLSI  context. \nFurthermore,  the  relationship  to  the  biological  exemplar  should  not  be \nignored.  It is  at  least  interesting  - whether  it  is  significant  remains  to  be \nseen. \n\n2 FULLY ANALOG PULSE STREAM SYNAPSES \nOur  early  pulse  stream  chips  proved  the  viability  of  the  pulse  stream \ntechnique  [Murray,1988a].  However,  the  area  occupied  by  the  digital \nweight  storage  memory  was  unacceptably  large.  Furthermore,  the  use  of \npseudo-clocks  in  an  analog  circuit  was  both  aesthetically  unsatisfactory  and \ndetrimental  to smooth  dynamical  behaviour,  and  using  separate signal  paths \nfor  excitation  and  inhibition  was  both clumsy  and  inefficient.  Accordingly, \nwe  have  developed  a  family  of  fully  programmable,  fully  analog  synapses \nusing  dynamic weight storage,  and operating on individual pulses to perform \narithmetic.  We  have  already  reported  time-modulation  synapses  based  on \nthis  technique,  and  a  later  section  of this  paper  will  present  the  associated \nchips [Murray,1988b, Murray,1989b]. \n\n2.1. TRANSCONDUCTANCE MULTIPLIER SYNAPSES \nThe  equation  of  interest  is  that  for  the  drain-source  current,  IDs,  for  a \nMOSFET in  the linear or triode  region:-\n\nIDS  =  -1:-- (VGS  - VT \n\nj.l.Cox W  [ \n\nVDs2] \n)  VDS  - 2--\n\n(1) \n\nHere,  Cox \ntransistor  gate  width,  L  the  transistor  gate  length,  and  V GS,  V T,  V DS \ntransistor gate-source,  threshold  and drain-source voltages respectively. \n\nis  the  oxide  capacitance/area,  j.l.  the  carrier  mobility,  W  the \nthe \n\n. \n\nf \n\nod \n\nj.l.C ox W \n\nL \n\n. \n\n. \n\nf \n\nThIs expressIon  or IDS  contams a  use  ul  pr  uct  term:-\nHowever,  it also contains two other terms in VDS  x VT  and VDs2. \nOne  approach  might  be  to  ignore  this  imperfection  in  the  multiplication,  in \nthe  hope  that  the  neural  parallelism  renders it  irrelevant.  We have  chosen, \nrather,  to  remove  the  unwanted  terms via  a  second  M OSFET,  as  shown  in \nFig.  2. \n\nx VGS  X  VDS . \n\n\f788 \n\nBrownlow, Tarassenko, Murray, Hamilton, Han and Reekie \n\n13  =  11-12 \n\nFigure 2  : Use of a second MOSFET to remove nonlinearities \n\n(a transconductance multiplier). \n\nThe output current 13  is now given  by:-\n\n13  =  JJ.Cox  L1  (VGSl  - VT  )  VDS1  - L1 \n\n2 \n\nW1 \n\n[ \n\nW 1  vDsl \n\n(2) \n\nW 2 \nL 2  (V GS2  - V T  )  V DS2  + \n\nW 2  VDsL] \nL2 \n\n2 \n\nThe  secret  now  is  to  select  W 1, L 1,  W 2, L 2,  VGSb  VGS2 ,  VDS1  and VDS2  to \ncancel  all terms except \n\nW1 \n\nJJ.Cox  L1  V GSl X VDS1 \n\n(3) \n\nThis  is  a  fairly  well-known  circuit,  and  constitutes  a  Transconductance \nMultiplier.  It was  reported  initially  for  use  in  signal  processing  chips  such \nas filters  [Denyer,1981 , Han,1984].  It would  be feasible  to  use  it directly in \na  continuous time network,  with  analog voltages representing the {Sj}.  We \nchoose  to  use  it  within  a  pulse-stream  environment,  to  minimise  the \nuncertainty  in  determining  the  operating  regime,  and  terminal  voltages,  of \nthe MOSFETs, as described above. \nFig.  3  shows two related  pulse stream synapse  based  on this technique.  The \npresynaptic  neural  state  Sj  is  represented  by  a  stream  of  0-5V  digital, \nasynchronous voltage  pulses Vj \u2022  These are used  to switch a  current sink and \nsource  in  and  out  of the  synapse,  either  pouring  current  to  a  fixed  voltage \nnode  (excitation  of  the  postsynaptic  neuron),  or  removing  it  (inhibition). \nThe magnitude and  direction  of the  resultant  current  pulses  are determined \nby the synapse weight,  currently stored as a  dynamic,  analog voltage Tij. \n\n\fPulse-Firing Neural Chips for Hundreds of Neurons \n\n789 \n\n(b) \n\nReference 1 \n\nr1 \n:r: \n\nReferen~ \n\nReference 3 \n\nVfixed \n\nVfixed \n\n(a) \n\nState VJ \n\nReference V r \n\nTij \n\nI \n\nFigure 3  : Use of a  transconductance multiplier to \nform fully  programmable pulse-stream synapses. \n\nThe  fixed  voltage  V Jixed  and  the  summation  of the  current  pulses  to  give  an \nactivity  Xj  = 'LTjjSj  are  both  provided  by  an  Operational  Amplifier \nintegrator  circuit,  whose  saturation  characteristics  incidentally  apply  a \nsigmoid  nonlinearity.  The  transistors  Tl  and  T4  act  as  power  supply \n\"on/off\"  switches  in  Fig.  3a,  and  in  Fig  3b  are  replaced  by  a  single \ntransistor,  in the output \"leg\"  of the synapse,  Transistors T2 and  T3 form  the \ntransconductance  multiplier.  One  of the  transistors  has  the  synapse  voltage \nTij  on  its  gate,  the  other  a  reference  voltage,  whose  value  determines  the \ncrossover  point  between  excitation  and  inhibition.  The gate-source voltages \non  T2  and  T3  need  to  be  substantially  greater  than  the  drain-source \nvoltages,  to  maintain  linear  operation.  This  is  not  a  difficult  constraint  to \nsatisfy. \nThe attractions of these cells  are that all  the transistors are n-type,  removing \nthe  need  for  area-hungry  isolation  well  structures,  and  In  Fig.  3a,  the \nvertical \ntopologically  attractive, \nproducing  very  compact  layout,  while  Fig.  3b  has  fewer  devices.  It is  not \nyet clear which will  prove optimal. \n\nline  of  drain-source  connections \n\nis \n\n2.2.  ASYNCHRONOUS  \"SWITCHED  CAPACITOR\" SYNAPSE \nFig.  4  shows  a  further  variant,  in  the  form  of  a  \"switched  capacitor\"  pulse \nstream  synapse.  Here  the  synapse  voltage  Tij  is  electrically  buffered  to \nswitched  capacitor  structure,  clocked  by \nthe  presynaptic  neural  pulse \nwaveforms.  Packets  of  charge  are  therefore  \"metered  out\"  to  the  current \nintegrator  whose  magnitude  is  controlled  by  Tij  (positive  or  negative),  and \n\n\f790 \n\nBrownlow, Tarassenko, 1\\1urray, Hamilton, Han and I{eekie \n\nwhose  frequency  by  the  presynaptic  pulse  rate.  The  overall  principle  is \ntherefore  the  same  as  that  described  for  the  transconductance  multiplier \nsynapses,  although  the circuit level  details are different. \n\nBuffer \n\nVj \n\nX \n\n/ \n\n1 \nI \n\nTij \n\n~ \n\nT \nI \n\n-\nVj \n\nIntegrator \n\n-\n\nFigure 4  : Asynchronous,  \"switched capacitor\"  pulse stream synapse. \n\nConventional  synchronous  switched  capacitor  techniques  have  been  used  in \nneural  integration  [Tsividis,1987],  but  nowhere  as  directly  as  in \nthis \nexample. \n\n2.3.  CHIP DETAILS AND  RESULTS \nBoth  the  time-modulation  and  switched  capacitor  synapses  have  been  tested \nfully  in  silicon,  and  Fig.  5  shows  a  section  of the  time-modulation  test  chip. \nThis synapse currently occupies 174x73jl.m. \n\nFigure 5  : Section,  and single synapse,  from  time-modulation chip. \n\n\fPulse-Firing Neural Chips for Hundreds of Neurons \n\n791 \n\nThree  distinct  pulse-stream  synapse \ntypes  have  been  presented,  with \ndifferent  operating  schemes  and  characteristics.  None  has  yet  been  used  to \nconfigure  a  large  network,  but  this  is  now  being  done.  Current  estimates \nfor \nthe  number  of  synapses  implementable  using  the  two  techniques \ndescribed  above  are  as  shown  in  Table  1,  using  an  8mmx8mm  die  as  an \nexample. \nThe  lack  of direct  scaling  between  transistor  count  and  synapse  count  (e.g. \nwhy  does  the  factor  4111  not  manifest  itself  as  a  much  larger  increase  in \nsynapse  count)  can  be  explained.  The raw  number  of transistors  is  not  the \nonly  factor  in  determining circuit area.  Routing of power supplies,  synapse \nweight  address  lines,  as  well  as  storage capacitor  size  all  take  their  toll,  and \nare  common  to  both  of the  above  synapse  circuits.  Furthermore,  in  analog \ncircuitry,  transistors  are  almost  certainly  larger  than  minimum  geometry, \nand generally significantly larger,  to minimise noise problems.  This all gives \nrise  to  a  larger  area  than  might  be  expected  from  simple  arguments. \nClearly,  however,  we  are  in  position  to  implement  serious  sized  networks, \nfirstly  with  the time-modulation synapse,  which  is fully  tested  in silicon,  and \nlater with  the  transconductance type,  which is still  under detailed  design  and \nlayout. \n\nTable 1  : Estimated  synapse count on 8mm die \n\nSYNAPSE \n\nNO. OF \n\nTRANSISTORS \n\nESTIMATED \n\nNETWORK SIZE \n\nTime modulation \nTransconductance \nSwitched  Capacitor \n\n11 \n4 \n4 \n\n= 6400  synapses \n= 15000 synapses \n= 14000 synapses \n\ntechniques \n\nIn  addition,  we  are  developing  new  oscillator  forms, \nto \ncounteract leakage from  dynamic nodes,  novel inter-chip signalling strategies \nspecifically  for  pulse-stream  systems,  and  non-volatile  (a-Si)  pulse  stream \nsynapses.  These  are  to  be  used  for  applications  in  text-speech  synthesis, \npattern  analysis  and  robotics.  Details  will  be  published  as  the  work \nprogresses. \nAcknowledgements \nThe  authors  are  grateful  to  the  UK  Science  and  Engineering  Research \nCouncil,  and  the  European  Community  (ESPRIT  BRA)  for  its  support  of \nthe  Korean  Telecommunications \nthis  work.  Dr.  Han  is  grateful \nAuthority, \nin  Edinburgh,  and \nKOSEF(Korea) for  partial financial  support. \n\nis  on  secondment \n\nfrom  whence  he \n\nto \n\n\f792 \n\nBrownlow, Tarassenko, Murray, Hamilton, Han and Reekie \n\nReferences \nDaniell, 1989. \n\nP.  M.  Daniell,  W.  A.  J.  Waller,  and  D.  A.  Bisset,  \"An \nImplementation  of  Fully  Analogue  Sum-of-Product  Neural  Models,\" \nProc.  lEE Conf.  on Artificial Neural Networks,  pp.  52-56,  ,1989. \n\nDenyer ,1981. \n\nP.  B.  Denyer and J.  Mavor,  \"MOST Transconductance Multipliers for \nArray Applications,\" lEE Proc.  Pt.  1, vol.  128,  no.  3,  pp.  81-86,  June \n,1981. \n\nEI-Leithy,1988. \n\nN.  EI-Leithy,  M.  Zaghloul,  and  R.  W.  Newcomb,  \"Implementation of \nPulse-Coded  Neural  Networks,\"  Proc.  27th  Conj.  on  Decision  and \nControl,  pp.  334-336,  ,1988. \n\nHan,1984. n S.  Han  and  Song  B.  Park,  \"Voltage-Controlled  Linear  Resistors  by \nMaS  Transistors  and  their  Application  to  Active  RC  Filter  MaS \nIntegration,\" Proc.  IEEE,  pp.  1655-1657,  Nov.,  ,1984. \n\nMurray,1987. \n\nA.  F.  Murray  and  A.  V.  W.  Smith,  \"Asynchronous  Arithmetic  for \nVLSI  Neural  Systems,\"  Electronics  Letters,  vol.  23,  no.  12,  pp.  642-3, \nJune,  ,1987. \n\nMurray,1988a. \n\nA.  F.  Murray  and  A.  V.  W.  Smith,  \"Asynchronous  VLSI  Neural \nNetworks  using  Pulse  Stream  Arithmetic,\"  IEEE  Journal  of Solid-State \nCircuits and Systems,  vol.  23,  no.  3, pp.  688-697,  June,  ,1988. \n\nMurray,1988b. \n\nA.  F.  Murray,  L.  Tarassenko,  and  A.  Hamilton,  \"Programmable \nAnalogue  Pulse-Firing  Neural  Networks,\"  Neural \nInformation \nProcessing  Systems  Conference,  pp.  671-677,  Morgan  Kaufmann, \n,1988. \n\nMurray,1989a. \n\nA.  F.  Murray,  \"Pulse  Arithmetic  in  VLSI  Neural  Networks,\"  IEEE \nMICRO,  vol.  9, no.  6,  pp.  64-74,  ,1989. \n\nMurray,1989b. \n\nA.  F.  Murray,  A.  Hamilton,  H.  M.  Reekie,  and  L.  Tarassenko, \n\"Pulse  - Stream  Arithmetic  in  Programmable  Neural  Networks,\"  Int. \nSymposium  on  Circuits  and  Systems,  Portland,  Oregon,  pp.  1210-1212, \nIEEE,  ,1989. \n\nTsividis,1987. \n\nY.  P.  Tsividis  and  D.  Anastassiou,  \"Switched  - Capacitor  Neural \nNetworks,\" Electronics Letters,  vol.  23,  no.  18,  pp.  958  - 959,  August, \n,1987. \n\n\f", "award": [], "sourceid": 215, "authors": [{"given_name": "Michael", "family_name": "Brownlow", "institution": null}, {"given_name": "Lionel", "family_name": "Tarassenko", "institution": null}, {"given_name": "Alan", "family_name": "Murray", "institution": null}, {"given_name": "Alister", "family_name": "Hamilton", "institution": null}, {"given_name": "Il", "family_name": "Han", "institution": null}, {"given_name": "H.", "family_name": "Reekie", "institution": null}]}