{"title": "Analog Neural Networks of Limited Precision I: Computing with Multilinear Threshold Functions", "book": "Advances in Neural Information Processing Systems", "page_first": 702, "page_last": 709, "abstract": null, "full_text": "702 \n\nObradovic and Pclrberry \n\nAnalog  Neural  Networks  of Limited Precision I: \nComputing with  Multilinear Threshold Functions \n\n(Preliminary Version) \n\nZoran Obradovic and Ian Parberry \n\nDepartment of Computer Science. \n\nPenn State University. \n\nUniversity Park. Pa.  16802. \n\nABSTRACT \n\nExperimental  evidence  has  shown  analog  neural  networks  to  be  ex(cid:173)\n~mely fault-tolerant;  in  particular.  their  performance  does  not  ap(cid:173)\npear  to  be  significantly  impaired  when  precision  is  limited.  Analog \nneurons  with  limited  precision  essentially  compute  k-ary  weighted \nmultilinear threshold  functions.  which  divide  R\"  into k  regions  with \nk-l hyperplanes.  The behaviour of k-ary  neural networks  is  investi(cid:173)\ngated.  There  is  no  canonical  set  of  threshold  values  for  k>3. \nalthough  they  exist  for  binary  and  ternary  neural  networks.  The \nweights  can be  made  integers of only  0 \u00abz +k ) log  (z +k \u00bb bits. where \nz  is  the  number  of processors.  without  increasing  hardware  or  run(cid:173)\nning  time.  The  weights  can  be  made  \u00b11  while  increasing  running \ntime  by a constant multiple and hardware by  a small polynomial  in  z \nand  k.  Binary  neurons  can  be  used  if the  running  time  is allowed  to \nincrease  by  a larger constant  multiple  and  the  hardware  is  allowed  to \nincrease  by  a  slightly  larger polynomial  in  z  and k.  Any  symmetric \nk-ary  function  can  be  computed \nin  constant  depth  and  size \no (n k- 1/(k-2)!).  and  any  k-ary  function  can  be  computed  in  constant \ndepth and  size  0  (nk\").  The alternating neural  networks of Olafsson \nand  Abu-Mostafa.  and  the  quantized  neural  networks  of Fleisher  are \nclosely related  to this model. \n\n\fAnalog Neural Networks of Limited Precision I \n\n703 \n\n1  INTRODUCTION \nNeural  networks  are  typically  circuits  constructed  from  processing  units  which  com(cid:173)\npute  simple  functions  of the  form  f(Wl> ... ,wlI):RII-+S  where SeR, wieR for  1~~, \nand \n\nf  (Wl> ... ,WII)(Xl,  .\u2022. ,xlI)=g (LWi X;) \n\nII \n\ni=1 \n\nfor  some  output  function  g :R-+S.  There  are  two  choices  for  the  set  S  which  are \ncurrently  popular  in  the  literature.  The  first  is  the  discrete  model,  with  S=B  (where B \ndenotes  the Boolean  set  (0,1)).  In  this  case,  g  is  typically  a  linear  threshold function \ng (x)= 1  iff x~. and  f \nis  called  a  weighted  linear  threshold function.  The  second  is \nthe  analog  model,  with  S=[O,I]  (where  [0,1]  denotes  (re RI~~I}).  In  this  case.  g \nis \nthe  sigmoid  function \ng (x)=(1 +c -% r 1  for  some constant c e R.  The analog  neural  network model  is popular \nbecause  it is  easy  to  construct processors  with  the  required characteristics  using  a  few \ntransistors.  The digital  model  is popular because its behaviour is easy to  analyze. \n\ntypically  a  monotone \n\nfunction,  such  as \n\nincreasing \n\nExperimental  evidence  indicates  that  analog  neural  networks  can  produce  accurate \ncomputations  when  the  precision  of their  components  is  limited.  Consider  what actu(cid:173)\nally  happens  to  the  analog  model  when  the  precision  is  limited.  Suppose  the  neurons \ncan  take  on  k  distinct  excitation  values  (for example,  by  restricting  the  number  of di(cid:173)\ngits  in  their  binary  or decimal  expansions).  Then  S  is  isomorphic  to  Zk={O, ... ,k-l}. \nWe  will \nfunction \ng (hloh2 .... ,hk-l):R-+Zk  defined by \n\nthe  multilinear \n\nis  essentially \n\nthreshold \n\nshow \n\nthat  g \n\nHere  and  throughout  this  paper,  we  will  assume  that  hl~h2~ ... ~hk-1> and  for  conveni(cid:173)\nence  define  ho=-oo  and  h/c=oo.  We  will  call  f  a  k-ary  weighted multilinear  threshold \nfunction  when  g  is a  multilinear threshold  function. \n\nWe will  study  neural  networks  constructed  from  k-ary  multilinear  threshold  functions. \nWe  will  call  these  k-ary  neural  networks,  in  order to  distinguish  them  from  the  stan(cid:173)\ndard 2-ary or binary neural network.  We  are  particularly concerned with  the resources \nof  time,  size  (number  of processors),  and  weight  (sum  of  all  the  weights)  of  k-ary \nneural  networks  when  used  in  accordance  with  the  classical  computational  paradigm. \nThe reader is referred to (parberry,  1990) for similar results  on binary neural networks. \nA  companion  paper (Obradovic  & Parberry,  1989b) deals  with  learning  on  k-ary  neur(cid:173)\nal  networks.  A  more  detailed version  of this  paper appears  in  (Obradovic  & Parberry, \n1989a). \n\n2  A K-ARY  NEURAL  NETWORK MODEL \nA  k-ary  neural  network  is  a  weighted  graph  M =(V ,E ,W ,h),  where  V  is  a  set of pro(cid:173)\ncessors  and  E cVxV  is  a  set  of  connections  between  processors.  Function \nw:VxV -+R  assign  weights  to  interconnections  and  h:V -+Rk-\nassign  a  set  of  k-l \nthresholds  to  each  of the  processors.  We  assume  that  if  (u ,v) eE,  W (u ,v )=0.  The \nsize of M  is  defined to be the number of processors, and  the weight  of M is \n\n\f704 \n\nObradovic and Parberry \n\nThe  processors  of a  k-ary  neural  network  are  relatively  limited  in  computing  power. \nA  k-ary function  is  a  function  f  :Z:~Z\".  Let F; denote  the  set of all  n-input k-ary \nfunctions.  Define e::R,,+Ir;-l~F; by e:(w l ..... w\".h It .\u2022\u2022\u2022 h''_l):R;~Z,,. where \n\ne;(w It \u2022\u2022\u2022\u2022 w\" .h h\u00b7\u00b7\u00b7.h,,-l)(X 1o ... ,% .. )=i  iff hi ~~Wi xi <h; +1\u00b7 \n\n.. \n\ni=1 \n\nThe  set  of k-ary  weighted  multilinear  threshold functions  is  the  union.  over  all  n e N. \nof the  range  of e;.  Each  processor  of a  k-ary  neural  network  can  compute  a  k-ary \nweighted multilinear threshold  function  of its  inputs. \nEach  processor  can  be  in  one  of k  states,  0  through  k-l.  Initially.  the  input proces(cid:173)\nsors  of M  are  placed  into  states  which  encode  the  input  If processor  v  was  updated \nduring  interval  t,  its  state  at time  t -1 was  i  and output  was  j. then  at time  t  its  state \nwill  be  j.  A k-ary  neural  network  computes by having the processors change  state  un(cid:173)\ntil  a  stable  configuration  is  reached.  The output of M  are  the states of the output pro(cid:173)\ncessors  after a  stable state  has been  reached.  A neural  network M 2 is  said  to be f  (t )(cid:173)\nequivalent  to  M 1  iff for  all  inputs  x. for  every  computation  of M 1  on  input x  which \nterminates  in  time  t  there is a computation of M 2 on  input x  which terminates  in  time \nf  (t)  with  the  same output.  A  neural  network M 2  is  said  to be  equivalent  to M 1  iff it \nis t -equivalent to it. \n\n3  ANALOG  NEURAL NETWORKS \nLet f  be  a  function  with  range  [0.1].  Any  limited-precision device  which  purports  to \ncompute  f  must  actually  compute  some  function  with  range  the  k  rational  values \nR\"={ilk-llieZ,,,~<k} (for some keN).  This is sufficient for all practical purposes \nprovided  k  is  large  enough.  Since  R\"  is  isomorphic  to  Z\".  we  will  formally  define \nthe  function  f\" :X ~Z\"  defined  by \nthe \nf,,(x)=round(j (x).(k-l\u00bb, where  round:R~N is  the natural rounding  function  defined \nby  round(x)=n  iff n-o.5~<n-tO.5. \nTheorem  3.1  : Letf(Wlo ... ,w .. ):R\"~[O,I] where WieR for  1~~. be defined by \n\nlimited  precision  variant  of  f \n\nto  be \n\nf  (w1O.\u00b7.,W,,)(X 10  .\u2022\u2022 ,x .. )=g (LWiXi) \n\n.. \ni=l \n\nwhere  g:R~[O,I] is  monotone  increasing  and  invertible.  Then f(Wlo ... ,W .. )\":R\"~Z,, \nis a k-ary  weighted  multilinear threshold function. \n\nIt \n\nis  easy \n\nto  verify \n\nProof: \nhi=g-1\u00ab2i-l)/2(k-l\u00bb.  0 \nThus  we  see  that  analog  neural  networks  with  limited  precision  are  essentially  k-ary \nneural networks. \n\nf(Wlo ...\u2022 W\")\"=S;(Wl' ... ,w\",hl, ...\u2022 h,,_l)'  where \n\nthat \n\n\fAnalog Neural Networks of Limited Precision I \n\n70S \n\n4  CANONICAL THRESHOLDS \nBinary  neural  networks  have  the  advantage  that  all  thresholds  can  be  taken  equal  to \nzero  (see.  for  example.  Theorem  4.3.1  of Parberry,  1990).  A  similar  result  holds  for \nternary  neural  networks. \n\nTheorem  4.1  :  For every  n-input ternary  weighted  multilinear  threshold  function  there \nis  an  equivalent  (n + I)-input  ternary  weighted  multilinear  threshold  function  with \nthreshold values equal to zero and one. \nProof:  Suppose  W=(W1o \u2022\u2022\u2022 ,WII )E R\",  hloh2E R.  Without  loss  of  generality  assume \nI~!0t,  and \nh l<h 2.  Define  W=(Wl \u2022...\u2022 wlI+l)e RII+I \nIt  can  be  demonstrated  by  a  simple  case  analysis  that  for  all \nwlI+I=-h I/(h2-h 1). \nx =(x 1 , \u2022\u2022\u2022 ,xll)e Z;. \n\nby  wj=wjl(hrh 1) \n\nfor \n\n8;(w,h l,hz)(x )=8;+I(W ,0,I)(x l, ... ,xll ,1). \n\no \nThe  choice  of threshold  values  in  Theorem  4.1  was  arbitrary.  Unfortunately  there  is \nno canonical  set of thresholds for k >3. \nTheorem  4.2  :  For every  k>3,  n~2, m~. h1o \u2022\u2022\u2022 ,hk - 1E R.  there  exists  an  n-input k-ary \nweighted multilinear threshold  function \n\nsuch that for all  (n +m )-input k-ary  weighted multilinear threshold functions \n\n8 \"+m(\" \n\n)\u00b7zm+1I  Z \nk  WI.\u00b7\u00b7\u00b7 .WII+m.  10\u00b7\u00b7\u00b7.  k-l'  k  ~ k \n\nA \n\nh \n\nh \n\nProof (Sketch):  Suppose that t I \u2022.. . .tk-l e R  is  a canonical set of thresholds.  and w.t.o.g. \nassume  n =2.  Let  h =(h 1o \u2022\u2022\u2022 ,hk - 1),  where  h l=h z=2.  h j=4,  hi =5  for  4Si <k.  and \nf=8i(1,I.h). \nBy hypothesis there exist wlo \u2022\u2022\u2022\u2022 wm+2  and y=(ylo ...\u2022 ym)eRm such that for all xeZi, \n\nf  (x )=8r+2(w 1.\u00b7 .. ,Wm+2,t 1 , \u2022\u2022\u2022 ,tk-l)(X ,y). \n\nm \n\nLet S= I:Wi+2Yi.  Since f  (1.0)=0. f  (0.1)=0, f  (2,1)=2, f  (1,2)=2.  it follows  that \n\n;=1 \n\n2(Wl+Wz+S )<tl+t 3. \n\n(1) \n\nSince f  (2,0)=2, f  (1.1 )=2. and f  (0.2)=2,  it follows  that \n\n\f706 \n\nObradovic and Pdrberry \n\nInequalities (1) and  (2)  imply  that \n\nWl+W2+S~2\u00b7 \n\n2t2<ll+13. \n\nBy  similar arguments  from  g=S;(1,l,l.3.3.4 \u2022...\u2022 4) we can conclude that \n\n(2) \n\n(3) \n\n(4) \n\nBut (4) contradicts (3).  0 \n\nS  NETWORKS  OF BOUNDED WEIGHT \nAlthough  our  model  allows  each  weight  to  take  on  an  infinite  number  of  possible \nvalues.  there  are  only  a  finite  number  of threshold  functions  (since  there  are  only  a \nfinite  number  of k-ary  functions)  with  a  fixed  number  of inputs.  Thus  the  number  of \nn -input  threshold  functions  is  bounded  above  by  some  function  in  n  and  k.  In  fact. \nsomething  stronger  can  be  shown.  All  weights  can  be  made \nintegral.  and \no ((n +k) log  (n +k\u00bb  bits are sufficient to describe each one. \nTheorem  5.1  :  For every  k-ary  neural  network M 1 of size z there  exists an  equivalent \nk-ary  neural  network  M2  of size  z  and  weight  ((k_l)/2)Z(z+I)(z+k)'2+0(1)  with  integer \nweights. \nProof (Sketch):  It  is  sufficient  to  prove  that  for  every  weighted  threshold  function \nf:(Wlt ...\u2022 wll.hh ...\u2022 h\"-I):Z:~Z,, for  some  neN. there  is  an  equivalent  we1f.hted  thres(cid:173)\nhold  function  g:(w~ \u2022...\u2022 w:.hi \u2022...\u2022 h;-d  such  that  Iwtl~((k-l)/2)I(n+l)'\"  )12+0(1)  for \nl~i~.  By  extending  the  techniques  used  by  Muroga.  Toda and  Takasu  (1961)  in  the \nbinary  case.  we  see  that  the  weights  are  bounded  above  by  the  maximum  determinant \nof a matrix  of dimension  n +k -lover Z\".  0 \nThus if k  is  bounded  above  by  a polynomial  in  n.  we  are  guaranteed of being  able  to \ndescribe  the  weights  using  a polynomial  number of bits. \n\n6  THRESHOLD CIRCUITS \nA k-ary  neural  network  with  weights  drawn  from  {\u00b11}  is said to  have  unit  weights.  A \nunit-weight  directed  acyclic  k-ary  neural  network  is  called  a  k-ary  threshold  circuit. \nA  k-ary  threshold  circuit  can  be  divided  into  layers.  with  each  layer  receiving  inputs \nonly  from  the  layers  above  it.  The  depth  of a  k-ary  threshold  circuit  is  defined  to  be \nthe  number  of layers.  The  weight  is  equal  to  the  number  of edges.  which  is bounded \nabove  by  the  square  of the  size.  Despite  the  apparent handicap  of limited weights.  k(cid:173)\nary threshold circuits are surprisingly powerful. \nMuch  interest  has  focussed  on  the  computation  of symmetric  functions  by  neural  net(cid:173)\nworks.  motivated by  the  fact  that the  visual system appears  to  be able  to recognize  ob(cid:173)\njects  regardless  of their  position  on  the  retina  A  function  f :Z: ~Z\" is  called  sym(cid:173)\nmetric if its output remains the  same  no  matter how  the input is permuted. \n\n\fAnalog Neural Networks of Limited Precision I \n\n707 \n\nTheorem  6.1  : Any  symmetric  k-ary  function  on  n inputs can be computed  by  a k-ary \nthreshold circuit of depth  6 and size (n+1)k-l/(k-2)!+ o (kn). \nProof: Omitted.  0 \nIt has  been  noted  many  times  that neural  networks  can compute any  Boolean  function \nin  constant  depth.  The  same  is  true  of k-ary  neural  networks,  although  both  results \nappear  to  require exponential  size for many  interesting functions. \nTheorem  6.2  :  Any  k-ary  function  of n  inputs  can  be  computed  by  a k-ary  threshold \ncircuit with  size (2n+1)k\"+k+1  and  depth 4. \nProof:  Similar to that for k=2  (see Chandra et.  al.,  1984;  Parberry,  1990). 0 \nThe  interesting  problem  remaining  is  to determine  which  functions  require  exponential \nsize  to  achieve  constant  depth,  and  which  can  be  computed  in  polynomial  size  and \nconstant  depth.  We  will  now  consider  the  problem  of adding  integers  represented  in \nk-ary  notation. \nTheorem  6.3  :  The  sum  of two k-ary  integers  of size  n  can  be computed  by  a k-ary \nthreshold circuit with  size 0 (n 2)  and depth  5. \nProof:  First compute  the  carry  of x and y in  'luadratic size and depth  3 using  the  stan(cid:173)\ndard  elementary  school  algorithm.  Then  the it  position  of the  result can  be computed \nfrom  the  i tit  position  of the  operands  and  a  carry  propagated  in  that  position  in  con(cid:173)\nstant size and depth  2.  0 \nTheorem  6.4  :  The  sum  of  n  k-~ integers  of size  n  can  be  computed  by  a  k-ary \nthreshold circuit with  size 0 (n 3+kn  ) and  constant depth. \nProof:  Similar  to  the  proof for k=2 using Theorem  6.3  (see  Chandra et.  al.,  1984; Par(cid:173)\nberry,  1990).  0 \nTheorem  6.S  :  For  every  k-ary  neural  network  M 1  of  size  z  there  exists  an  0 (t)(cid:173)\nequivalent unit-weight k-ary  neural  network M2  of size o ((z+k)410g3(z+k\u00bb. \nProof:  By  Theorem  5.1  we  can  bound  all  weights  to  have  size  0 ((z+k)log(z+k\u00bb  in \nbinary  notation.  By  Theorem  6.4  we  can  replace  every  processor  with  non-unit \nweights  by a threshold circuit of size o ((z+k)310g3(z+k\u00bb  and constant depth.  0 \nTheorem  6.5  implies  that  we  can  assume  unit weights  by  increasing  the  size  by  a po(cid:173)\nlynomial  and  the  running  time  by  only  a  constant  multiple  provided  the  number  of \nlogic  levels  is  bounded  above  by  a  polynomial  in  the  size  of  the  network.  The \nnumber  of thresholds  can  also  be  reduced  to  one  if the  size  is  increased  by  a  larger \npolynomial: \nTheorem  6.6  :  For  every  k-ary  neural  network  M 1  of size  z  there  exists  an  0 (t )(cid:173)\nequivalent  unit-weight  binary  neural  network  M 2  of  size  0 (z 4k4)(log  z  + log k)3 \nwhich outputs  the  binary encoding of the required  result \nProof:  Similar to the proof of Theorem  6.5.  0 \nThis  result  is  primarily  of theoretical  interest.  Binary  neural  networks  appear  simpler, \nand  hence  more  desirable  than  analog  neural  networks.  However,  analog  neural  net(cid:173)\nworks  are  actually  more  desirable  since  they  are  easier  to  build.  With  this  in  mind, \nTheorem  6.6  simply  serves  as  a  limit  to  the  functions  that  an  analog  neural  network \n\n\f708 \n\nObradovic and Parberry \n\ncan  be  expected  to  compute  efficiently.  We  are  more  concerned  with  constructing  a \nmodel  of the  computational  abilities  of neural  networks,  rather  than  a  model  of their \nimplementation details. \n\n7  NONMONOTONE MULTILINEAR  NEURAL  NETWORKS \nOlafsson  and  Abu-Mostafa \n(1988)  study \nf(Wlt ... ,wl):R\"-+B for w;ER,  1~~, where \n\ninformation  capacity  of \n\nfunctions \n\nf  (Wlt .. \u00b7\u2022WII)(X1 \u2022... , xlI)=g (~W;X;) \n\nII \n\n;=1 \n\nand  g  is  the  alternating  threshold function  g (h loh2 ..... hk-1):R-+B  for  some  monotone \nincreasing  h;ER,  1~<k, defined  by  g(x)=O  if  h2i~<h2i+1 for  some  ~5:nI2.  We \nwill  call  f  an  alternating  weighted  multilinear  threshold function,  and  a  neural  net(cid:173)\nwork  constructed  from  functions  of this  form  alternating  multilinear  neural  networks. \nAlternating  multilinear neural  networks are closely  related  to k-ary  neural networks: \nTheorem  7.1  :  For  every  k-ary  neural  network  of size  z  and  weight  w  there  is  an \nequivalent  alternating  multilinear  neural  network  of  size  z log  k  and  weight \n(k -l)w log  (k -1) which produces the output of the  former in  binary  notation. \nProof (Sketch):  Each  k-ary  gate  is replaced  by  log k  gates  which  together essentially \nperform  a  \"binary search\"  to determine each bit of the k-ary  gate.  Weights  which  in(cid:173)\ncrease exponentially are used  to provide the correct output value.  0 \nTheorem  7.2  :  For every  alternating  multilinear  neural  network  of size  z  and  weight \nw there  is a  3t-equivalent k-ary neural  network  of size 4z  and  weight w+4z. \nProof (Sketch):  Without  loss of generality.  assume  k  is odd.  Each  alternating  gate is \nreplaced by a k-ary  gate with  identical weights and thresholds.  The  output of this gate \ngoes  with  weight  one  to  a  k-ary  gate  with  thresholds  1,3,S \u2022... ,k-1  and  with  weight \nminus  one  to  a  k-ary  gate  with  thresholds  -(k-1), ... ,-3,-1.  The  output of these  gates \ngoes to a binary  gate with  threshold k.  0 \nBoth  k-ary  and  alternating  multilinear neural  networks  are  a special  case  of nonmono(cid:173)\ntone  multilinear  neural  networks,  where  g :R-+R  is  the  defined  by  g (x )=Ci \nhi~<h;+lt for  some  monotone  increasing  h;ER,  1~<k, and  co, ... ,Ck-1EZk.  Non(cid:173)\nmonotone  neural  networks  correspond  to  analog  neural  networks  whose  output  func(cid:173)\ntion  is  not  necessarily  monotone  nondecreasing.  Many  of the  result  of this  paper,  in(cid:173)\ncluding  Theorems  5.1,  6.5,  and  6.6, also  apply  to  nonmonotone  neural  networks.  The \nsize,  weight and  running  time  of many  of the upper-bounds  can  also be improved by  a \nsmall  amount  by  using  nonmonotone  neural  networks  instead  of k-ary  ones.  The  de(cid:173)\ntails are  left  to  the  interested reader. \n\niff \n\n8  MUL TILINEAR HOPFIELD  NETWORKS \nA  multilinear version of the  Hopfield  network called  the  quantized neural network  has \nbeen  studied  by  Fleisher  (1987).  Using  the  terminology  of (parberry,  1990),  a  quan(cid:173)\ntized  neural  network  is  a  simple  symmetric  k-ary  neural  network  (that is,  its  intercon(cid:173)\nnection  pattern  is  an  undirected  graph  without self-loops)  with  the  additional  property \nthat  all  processors  have  an  identical  set of thresholds.  Although  the  latter  assumption \n\n\fAnalog Neural Networks of Limited Precision I \n\n709 \n\nis reasonable  for binary  neural  networks (see,  for example,  Theorem 4.3.1  of Parberry, \n1990),  and  ternary  neural  networks  (Theorem  4.1),  it  is  not  necessarily  so  for  k-ary \nneural  networks  with  k>3  (Theorem  4.2).  However,  it  is  easy  to  extend  Fleisher's \nmain  result to give the following: \nTheorem  8.1  :  Any  productive  sequential  computation  of a  simple  symmetric  k-ary \nneural network  will  converge. \n\n9  CONCLUSION \nIt  has  been  shown  that  analog  neural  networks  with  limited  precision  are  essentially \nk-ary  neural  networks.  If k  is  limited to a polynomial,  then  polynomial  size,  constant \ndepth  k-ary  neural  networks  are  equivalent  to  polynomial  size,  constant  depth  binary \nneural  networks.  Nonetheless,  the  savings  in  time  (at  most  a  constant  multiple)  and \nhardware  (at  most  a  polynomial)  arising  from  using  k-ary  neural  networks  rather  than \nbinary ones can  be quite  significant.  We do  not suggest that one should actually con(cid:173)\nstruct binary or k-ary  neural  networks.  Analog  neural  networks can  be constructed by \nexploiting  the analog behaviour of transistors,  rather  than  using extra hardware  to inhi(cid:173)\nbit it  Rather,  we suggest that k-ary  neural  networks are  a  tool  for reasoning about the \nbehaviour of analog neural  networks. \n\nAcknowledgements \nThe  financial  support  of the  Air  Force  Office  of Scientific  Research,  Air Force  S ys(cid:173)\nterns  Command,  DSAF,  under  grant  numbers  AFOSR  87-0400  and  AFOSR  89-0168 \nand NSF grant CCR-8801659  to Ian Parberry is gratefully  acknowledged. \n\nReferences \nChandra A.  K.,  Stockmeyer L. J.  and Vishkin D., (1984)  \"Constant depth  reducibility,\" \nSIAM 1.  Comput.,  vol.  13,  no.  2, pp.  423-439. \nFleisher  M.,  (1987)  \"The  Hopfield  model  with  multi-level  neurons,\"  Proc.  IEEE \nConference  on Neural Information Processing Systems, pp. 278-289,  Denver,  CO. \nMuroga  S.,  Toda  1.  and  Takasu  S.,  (1961)  \"Theory  of majority  decision  elements,\"  1. \nFranklin Inst.,  vol.  271.,  pp.  376-418. \nObradovic  Z.  and  Parberry  1.,  (1989a)  \"Analog  neural  networks of limited precision  I: \nComputing  with  multilinear  threshold  functions  (preliminary  version),\"  Technical  Re(cid:173)\nport CS-89-14, Dept of Computer Science,  Penn. State Dniv. \nObradovic  Z.  and  Parberry I., (1989b)  \"Analog  neural  networks of limited precision  II: \nLearning  with  multilinear  threshold  functions  (preliminary  version),\"  Technical Report \nCS-89-15, Dept.  of Computer Science, Penn.  State Dniv. \nOlafsson  S.  and  Abu-Mostafa Y.  S.,  (1988) \"The capacity of multilevel threshold func(cid:173)\ntions,\"  IEEE  Trans.  Pattern  Analysis  and  Machine  Intelligence,  vol.  10,  no.  2,  pp. \n277-281. \nParberry I., (To  Appear in  1990)  \"A  Primer  on  the  Complexity  Theory  of Neural  Net(cid:173)\nworks,\"  in  A  Sourcebook  of Formal  Methods  in  Artificial  Intelligence,  ed.  R.  Banerji, \nNorth-Holland. \n\n\f", "award": [], "sourceid": 232, "authors": [{"given_name": "Zoran", "family_name": "Obradovic", "institution": null}, {"given_name": "Ian", "family_name": "Parberry", "institution": null}]}