{"title": "Are Hopfield Networks Faster than Conventional Computers?", "book": "Advances in Neural Information Processing Systems", "page_first": 239, "page_last": 245, "abstract": null, "full_text": "Are  Hopfield Networks Faster Than \n\nConventional Computers? \n\nIan Parberry* and  Hung-Li Tsengt \n\nDepartment of Computer Sciences \n\nUniversity of North Texas \n\nP.O.  Box  13886 \n\nDenton,  TX  76203-6886 \n\nAbstract \n\nIt is shown that conventional computers can be exponentiallx faster \nthan planar Hopfield  networks:  although there  are planar Hopfield \nnetworks that take exponential time to converge,  a stable state of an \narbitrary  planar Hopfield  network  can be found  by  a  conventional \ncomputer  in  polynomial  time.  The  theory  of 'P.cS-completeness \ngives  strong evidence  that such  a separation is  unlikely for  nonpla(cid:173)\nnar  Hopfield  networks,  and it is  demonstrated  that  this is  also  the \ncase  for  several  restricted  classes  of nonplanar  Hopfield  networks, \nincluding  those  who  interconnection  graphs are  the class  of bipar(cid:173)\ntite graphs,  graphs of degree  3, the dual of the knight's graph,  the \n8-neighbor mesh,  the hypercube ,  the butterfly,  the cube-connected \ncycles,  and the shuffle-exchange  graph. \n\n1 \n\nIntroduction \n\nAre  Hopfield  networks  faster  than  conventional  computers?  This  apparently \nstraightforward  question  is  complicated  by  the  fact  that  conventional  computers \nare  universal  computational  devices,  that  is,  they  are  capable  of simulating  any \ndiscrete  computational  device  including  Hopfield  networks.  Thus,  a  conventional \ncomputer could in  a  sense  cheat by imitating the fastest  Hopfield  network  possible. \n\n* Email:  ianGcs. unt .edu.  URL:  http://hercule .csci. unt. edu/ian. \nt Email:  ht sengGponder. csci. unt . edu. \n\n\f240 \n\nI.  Parberry and H.  Tseng \n\nBut the question remains, is  it faster for  a  computer to imitate a Hopfield  network , \nor to use other computational methods?  Although the  answer  is  likely to be  differ(cid:173)\nent for different benchmark problems, and even for different  computer architectures , \nwe  can  make our results  meaningful in  the long term by  measuring scalability, that \nis, how the running time of Hopfield networks and conventional computers increases \nwith the size  of any  benchmark problem to be  solved. \n\nStated  more technically, we  are  interested  in  the  computational complexity of the \nstable  state  problem  for  Hopfield  networks,  which  is  defined  succinctly  as  follows : \ngiven  a  Hopfield  network,  determine  a  stable  configuration.  As  previously  stated, \nthis stable  configuration  can  be  determined  by  imitation, or by  other  means.  The \nfollowing results are known about the scalability of Hopfield network imitation.  Any \nimitative algorithm for the stable state problem must take exponential time on som e \nHopfield networks, since there exist Hopfield networks that require exponential time \nto converge  (Haken  and  Luby  [4] , Goles  and Martinez  [2]) .  It is  unlikely that even \nnon-imitative  algorithms  can  solve  the  stable  state  problem  in  polynomial  time , \nsince  the  latter  is  PeS-complete  (Papadimitriou ,  Schaffer,  and  Yannakakis  [9]). \nHowever ,  the  stable  state  problem  is  more  difficult  for  some  classes  of  Hopfield \nnetworks  than others.  Hopfield  networks  will  converge  in  polynomial time if their \nweights  are  bounded  in  magnitude  by  a  polynomial of the  number  of nodes  (for \nan  expository  proof  see  Parberry  [11 ,  Corollary  8.3.4]) .  In  contrast ,  the  stable \nstate  problem  for  Hopfield  networks  whose  interconnection  graph  is  bipartite  is \npeS-complete (this  can  be  proved  easily  by  adapting techniques  from  Bruck  and \nGoodman  [1])  which  is  strong  evidence  that  it  too  requires  superpolynomial time \nto  solve even  with  a nonimitative algorithm. \n\nWe show in this paper that although there exist planar Hopfield networks that t ake \nexponential time to converge  in the worst  case, the  stable state problem for  planar \nHopfield  networks  can  be  solved  in polynomial time by  a  non-imitative algorithm. \nThis demonstrates  that imitating planar Hopfield  networks is  exponentially slower \nthan  using  non-imitative algorithmic techniques.  In  contrast , we  discover  that the \nstable  state  problem  remains peS-complete for  many simple classes  of nonplanar \nHopfield  network,  including  bipartite  networks,  networks  of  degree  3,  and  some \nnetworks that are  popular in neurocomputing and parallel computing. \n\nThe  main part of this  manuscript  is  divided  into four  sections.  Section  2  contains \nsome  background  definitions  and  references.  Section  3  contains  our  results  about \nplanar Hopfield networks.  Section 4 describes  our peS-completeness results , based \non a pivotal lemma about  a  nonstandard type of graph embedding. \n\n2  Background \n\nThis  section  contains  some  background  which  are  included  for  completeness  but \nmay  be  skipped  on  a  first  reading.  It is  divided  into two  subsections,  the  first  on \nHopfield  networks, and the second  on PeS-completeness. \n\n2.1  Hopfield Networks \n4- Hopfield  network  [6]  is  a  discrete  neural  network  model with symmetric connec(cid:173)\ntions.  Each  processor  in  the  network  computes  a  hard  binary  weighted  threshold \n\n\fAre Hopfield Networks Faster than Conventional Computers? \n\n241 \n\nfunction.  Only one  processor  is  permitted to change state at any given  time.  That \nprocessor  becomes  active  if its excitation  level  exceeds  its  threshold,  and  inactive \notherwise.  A  Hopfield  network  is  said  to  be  in  a  stable  state  if the states  of all  of \nits processors  are consistent  with their respective excitation levels.  It is  well-known \nthat all  Hopfield  networks  converge  to a  stable state.  The proof defines  a  measure \ncalled  energy,  and  demonstrates  that  energy  is  positive  but  decreases  with  every \ncomputation step.  Essentially  then,  a  Hopfield  network  finds  a  local  minimum in \nsome energy  landscape. \n\n2.2  P .cS-completeness \n\nWhile the theory of NP-completeness measures the complexity of global optimiza(cid:173)\ntion,  the  theory  of p.cS-completeness  developed  by  Johnson,  Papadimitriou,  and \nYannakakis  [7]  measures  the  complexity of local optimization.  It is  similar to the \ntheory  of NP-completeness  in  that  it  identifies  a  set  of difficult  problems  known \ncollectively  as  p.cS-complete  problems.  These  are  difficult  in  the  sense  that  if  a \nfast  algorithm  can  be  developed  for  any  P .cS-complete  problem,  then  it  can  be \nused  to  give  fast  algorithms for  a  substantial  number  of other  local  optimization \nproblems including many important problems for  which  no fast  algorithms are cur(cid:173)\nrently  known.  Recently,  Papadimitriou,  Schaffer,  and  Yannakakis  [9]  proved  that \nthe problem of finding  stable states  in  Hopfield  networks  is  P .cS-complete. \n\n3  Planar Hopfield  Networks \n\nA planar Hopfield network is one whose interconnection graph is planar, that is,  can \nbe  drawn  on  the  Euclidean  plane  without  crossing  edges.  Haken  and  Luby  [4]  de(cid:173)\nscribe  a  planar Hopfield network that provably takes exponential time to converge, \nand hence  any imitative algorithm for the stable state problem must take exponen(cid:173)\ntial  time on  some  Hopfield  network.  Yet  there  exists  a  nonimitative algorithm for \nthe stable state problem that runs  in  polynomial time on  all  Hopfield  networks: \n\nTheorem 3.1  The  stable state problem for Hopfield networks with planar intercon(cid:173)\nnection  pattern  can  be  solved  in  polynomial time. \n\nPROOF:  (Sketch.)  The prooffollows from the fact that the maximal cut in a  planar \ngraph  can  be  found  in  polynomial time  (see,  for  example,  Hadlock  [3]),  combined \nwith  results of Papadimitriou, Schaffer,  and Yannakakis [9].  0 \n\n4  P .cS-completeness Results \n\nOur  P .cS-completeness  results  are  a  straightforward  consequence  of a  new  result \nthat  characterizes  the  difficulty  of the  stable  state  problem  of an  arbitrary  class \nof Hopfield  networks  based  on  a  graph-theoretic  property  of their  interconnection \npatterns.  Let  G =  (V, E)  and  H  =  (V', E')  be graphs.  An  embedding  of G  into  H \nis  a  function  f: V -+ 2 Vi  such  that the following properties  hold.  (1)  For  all  v  E V, \nthe subgraph of H  induced by  f(v)  is  connected.  (2)  For all  (u, v)  E  E, there exists \na  path  (which  we  will  denote  f(u , v))  in  H  from  a  member of  f(u)  to  a  member \nof f(v).  (3)  Each  vertex  w  E  H  is  used  at  most once,  either  as  a  member of f(v) \n\n\f242 \n\nI.  Parberry and H.  Tseng \n\nfor  some v  E  V,  or as  an  internal vertex  in  a  path feu, v)  for  some u, v  E V.  The \ngraph  G  is  called  the  guest  graph,  and  H  is  called  the  host  graph.  Our  definition \nof embedding is  different  from the standard notion of embedding (see,  for  example, \nHong,  Mehlhorn,  and  Rosenberg  [5])  in  that we  allow  the  image of a  single  guest \nvertex to be  a  set  of host  vertices,  and we  insist  in  properties  (2)  and  (3)  that the \nimages of guest edges be distinct paths.  The latter property is crucial to our results, \nand forms the major difficulty in  the  proofs. \nLet  5, T  be  sets  of graphs.  5  is  said  to  be  polynomial-time  embeddable  into  T, \nwritten  5  ::;e  T,  if there  exists  polynomials Pl(n),P2(n)  and a  function  f  with  the \nfollowing properties:  (1)  f  can be computed in time PI(n),  and (2)  for every  G  E 5 \nwith  n  vertices,  there  exists  H  E  T  with  at  most  p2(n)  vertices  such  that  G  can \nbe  embedded  into  H  by  f.  A set  5  of graphs  is  said  to  be  pliable  if the set  of all \ngraphs is  polynomial-time embeddable into 5. \n\nLemma 4.1  If 5  is  pliable,  then  the  problem  of finding  a stable  state  in  Hopfield \nnetworks  with  interconnection  graphs  in  5  is 'P \u00a3S-complete. \n\n(Sketch.)  Let  5  be  a  set  of graphs  with  the  property  that the  set  of all \nPROOF: \ngraphs  is  polynomial-time  embeddable  into  5 .  By  the  results  of Papadimitriou, \nSchaffer,  and  Yannakakis  [9],  it  is  enough  to show  that  the  max-cut  problem  for \ngraphs in  5  is  'P \u00a3S-complete. \nLet G be  an arbitrary labeled graph.  Suppose G is embedded into H  E 5  under the \npolynomial-time embedding.  For  each  edge  e  in  G  of cost  c,  select  one  edge  from \nthe  path  connecting  the  vertices  in  f( e)  and  assign  it  cost  c.  We  call  this  special \nedge  f' ( e ).  Assign  all  other edges  in  the  path cost  -00.  For  all  v  E  V,  assign  the \nedges  linking the  vertices  in  f(v)  a  cost  of -00.  Assign  all other edges  of H  a cost \nof zero. \nIt can be shown that every cut in  G induces a cut of the same cost  in H,  as follows. \nSuppose  G  ~ E  is  a  cut  in  G,  that  is,  a  set  of edges  that  if  removed  from  G, \ndisconnects it into two components containing vertices  VI  and V2 respectively.  Then, \nremoving vertices  f'(G)  and all  zero-cost  edges  from  H  will  disconnect  it into two \ncomponents  containing  vertices  f(VI )  and  f(V2 )  respectively.  Furthermore,  each \ncut of positive cost  in  H  induces  a  cut  of the same cost  in  G,  since  a  positive  cost \ncut in H  cannot contain any edges of cost  -00, and hence must consist only of f'(e) \nfor  some  edges  e  E  E.  Therefore,  every  max-cost  cut  in  H  induces  in  polynomial \ntime a  max-cost  cut in  G.  0 \n\nWe  can  now  present  our  'P \u00a3S-completeness  results.  A  graph  has  degree  3  if all \nvertices  are  connected  to  at most 3 other  vertices  each. \n\nTheorem 4.2  The  problem  of finding  stable  states  in  Hopfield  networks  of degree \n3  is 'P \u00a3S-complete. \n\nPROOF: \n\n(Sketch.)  By  Lemma 4.1,  it  suffices  to  prove  that  the  set  of degree-3 \ngraphs is pliable.  Suppose  G = (V, E) is  an arbitrary graph.  Replace each  degree-k \nvertex  x  E V  by  a path consisting of k  vertices,  and attach each edge  incident  with \nv  by  a  new  edge  incident  with  one of the  vertices  in  the  path.  Figure  1 shows  an \nexample of this embedding.  0 \n\n\fAre Hopfield Networks Faster than Conventional Computers? \n\n243 \n\nFigure  1:  A guest  graph of degree  5  (left),  and the  corresponding  host  of degree  3 \n(right).  Shading  indicates  the  high-degree  nodes  that  were  embedded  into  paths. \nAll other  nodes were  embedded into single nodes. \n\nFigure 2:  An 8-neighbor  mesh with  25  vertices  (left),  and the  8 X  8 knight's graph \nsuperimposed on an 8 x  8 board (right). \n\nThe  8-neighbor  mesh  is  the  degree-8  graph  G  =  (V, E)  defined  as  follows:  V  = \n{1,2, ... ,m}  x  {1,2, ... ,n},  and  vertex  (u,v)  is  connected  to  vertices  (u,v\u00b1 1), \n(u \u00b1 1, v),  (u \u00b1 1, v \u00b1 1).  Figure 2 shows  an 8-neighbor mesh  with  25  vertices. \n\nTheorem 4.3  The  problem  of finding  stable  states  in  H opfield  networks  on  the \n8-neighbor mesh  is  peS-complete. \n\n(Sketch.)  By  Lemma 4.1,  it suffices  to prove  that the 8-neighbor  mesh is \nPROOF: \npliable.  An  arbitrary graph  can be embedded  on  an  8-neighbor  mesh  by  mapping \neach  node to a set of consecutive nodes in the bottom row of the grid, and mapping \nedges  to  disjoint  rectilinear  paths  which  use  the  diagonal  edges  of  the  grid  for \ncrossovers.  0 \nThe  knight's  graph  for  an  n  X  n  chessboard  is  the  graph  G  = (V, E)  where  V  = \n{(i, j) 11  ~ i, j  ~ n},  and  E  = {((i, j), (k, i\u00bb I {Ii - kl, Ij - il} = {I, 2}}.  That  is, \n\nthere  is  a  vertex  for  every  square  of the  board  and  an  edge  between  two  vertices \nexactly when there is  a knight's move from one to the other.  For example, Figure 2 \nshows  the  knight's graph for  the 8 x  8 chessboard.  Takefuji  and  Lee  [15]  (see  also \nParberry  [12])  use  the  dual  of the  knight's  graph  for  a  Hopfield-style  network  to \nsolve  the  knight's tour  problem.  That is,  they  have  a  vertex  Ve  for  each  edge  e of \nthe knight's graph,  and an edge between  two vertices  Vd  and  Ve  when  d and e share \na  common vertex in the knight's graph. \n\n\f244 \n\nI.  Parberry and H.  Tseng \n\nTheorem 4.4  The problem of finding  stable states in H opfield networks on  the  dual \nof the  knight's  graph  is  pes -complete. \n\nPROOF:  (Sketch.)  By  Lemma4.1,  it suffices  to prove that the dual of the knight 's \ngraph  is  pliable. \nIt  can  be  shown  that  the  knight 's  graph  is  pliable  using  the \ntechnique of Theorem 4.3.  It can  also he proved that if a set  S  of graphs is pliable, \nthen the set  consisting of the duals of graphs in  S  is  also pliable.  0 \n\nThe  hypercube  is  the  graph  with  2d  nodes  for  some  d,  labelled  with  the  binary \nrepresentations  of the  d-bit natural numbers, in which  two  nodes  are connected  by \nan  edge  iff  their  labels  differ  in  exactly  one  bit.  The  hypercube  is  an  important \ngraph for  parallel computation (see, for  example,  Leighton  [8],  and Parberry  [lOD . \n\nTheorem 4.5  The  problem  of finding  stable  states  in  Hopfield  networks  on  the \nhypercube  is  peS-complete. \n\nPROOF :  (Sketch.)  By  Lemma 4.1, it suffices to prove that the hypercube is pliable. \nSince the  \"~e\"  relation is transitive, it further suffices  by Theorem 4.2 to show that \nthe  set  of degree-3  graphs  is  polynomial-time embeddable into the  hypercube.  To \nembed  a  degree-3  graph G  into the  hypercube, first  break  it into a  degree-1  graph \nG 1  and  a  degree-2  graph  G2 .  Since  G2  consists  of cycles,  paths,  and  disconnected \nvertices,  it  can  easily  be  embedded  into  a  hypercube  (since  a  hypercube  is  rich \nin  cycles).  G 1  can  be  viewed  as  a  permutation of vertices  in  G  and  can  hence  be \nrealized using a hypercube implementation of Waksman 's permutation network [16] . \no \nWe  conclude  by  stating  PeS-completeness  results  for  three  more graphs  that  are \nimportant  in  the  parallel  computing  literature  the  butterfly  (see,  for  example, \nLeighton  [8]) ,  the  cube-connected  cycles  (Preparata  and  Vuillemin  [13D ,  and  the \nshuffle-exchange  (Stone [14]).  The proofs  use  Lemma 4.1  and Theorem 4.5 , and are \nomitted for  conciseness. \n\nTheorem 4.6  The  problem  of finding  stable  states  in  Hopfield  networks  on  the \nbutterfly,  the  cube-connected  cycles,  and  the  shuffle-exchange  is peS-complete. \n\nConclusion \n\nAre Hopfield networks faster than conventional computers?  The answer seems to be \nthat it depends on the interconnection graph of the Hopfield network.  Conventional \nnonimitative algorithms can be exponentially faster  than planar Hopfield networks. \nThe  theory  of peS-completeness  shows  us  that  such  an  exponential  separation \nresult  is  unlikely  not  only  for  nonplanar  graphs,  but  even  for  simple  nonplanar \ngraphs such  as bipartite graphs, graphs of degree  3, the dual of the knight's graph, \nthe 8-neighbor  mesh, the  hypercube,  the  butterfly,  the  cube-connected  cycles,  and \nthe shuffle-exchange  graph. \n\nAcknowledgements \n\nThe research  described  in this paper was  supported  by the  National Science  Foun(cid:173)\ndation under grant number CCR- 9302917, and by the Air  Force Office  of Scientific \n\n\fAre Hopfield Networks Faster than Conventional Computers? \n\n245 \n\nResearch,  Air Force  Systems Command, USAF,  under grant number F49620-93-1-\n0100. \n\nReferences \n\n[1]  J.  Bruck  and J.  W.  Goodman.  A  generalized  convergence  theorem for  neural \nnetworks.  IEEE  Transactions  on  Information  Theory,  34(5):1089-1092, 1988. \n[2]  E.  Goles  and  S.  Martinez.  Exponential  transient  classes  of symmetric neural \nnetworks  for  synchronous  and sequential  updating.  Complex  Systems,  3:589-\n597,  1989. \n\n[3]  F.  Hadlock.  Finding  a  maximum cut  of a  planar  graph  in  polynomial time. \n\nSIAM  Journal  on  Computing,  4(3) :221-225,  1975. \n\n[4]  A.  Haken  and  M.  Luby.  Steepest  descent  can  take  exponential  time for  sym(cid:173)\n\nmetric conenction  networks.  Complex  Systems,  2:191-196,1988. \n\n[5]  J.-W.  Hong,  K.  Mehlhorn,  and  A.L.  Rosenberg.  Cost  tradeoffs  in  graph em(cid:173)\n\nbeddings.  Journal  of the  ACM, 30:709-728,1983. \n\n[6]  J.  J . Hopfield.  Neural  networks  and physical systems  with emergent  collective \ncomputational  abilities.  Proc.  National  Academy  of Sciences,  79:2554-2558 , \nApril  1982. \n\n[7]  D.  S.  Johnson ,  C.  H.  Papadimitriou,  and  M.  Yannakakis.  How  easy  is  local \nIn  26th  Annual  Symposium  on  Foundations  of  Computer  Science, \n\nsearch? \npages  39-42. IEEE Computer Society  Press,  1985. \n\n[8]  F.  T.  Leighton.  Introduction  to  Parallel Algorithms and  Architectures:  Arrays \n\n.  Trees\u00b7  Hypercubes.  Morgan Kaufmann,  1992. \n\n[9]  C.  H.  Papadimitrioll, A.  A.  Schaffer,  and  M.  Yannakakis.  On  the  complexity \nof local search.  In  Proceedings  of the  Twenty Second  Annual ACM Symposium \non  Theory  of Computing,  pages 439-445. ACM  Press,  1990. \n\n[10]  I.  Parberry.  Parallel Complexity  Theory.  Research  Notes  in Theoretical Com(cid:173)\n\nputer Science.  Pitman Publishing,  London , 1987. \n\n[11]  I.  Parberry.  Circuit  Complexity  and  Neural Networks.  MIT Press,  1994. \n[12]  I.  Patberry.  Scalability  of  a  neural  network  for  the  knight's  tour  problem. \n\nNeurocomputing,  12 :19-34,  1996. \n\n[13]  F.  P.  Preparata  and  J.  Vuillemin.  The  cube-connected  cycles:  A  versatile \nnetwork  for  parallel  computation.  Communications  of the  ACM,  24(5):300-\n309 , 1981. \n\n[14]  H.  S.  Stone.  Parallel  processing  with  the  perfect  shuffle.  IEEE  Transactions \n\non  Computers,  C-20(2):153-161, 1971. \n\n[15]  Y.  Takefuji  and  K.  C.  Lee.  Neural  network  computing for  knight's tour prob(cid:173)\n\nlems.  Neurocomputing,  4(5):249-254 , 1992. \n\n[16]  A.  Waksman.  A  permutation network.  Journal  of the  ACM,  15(1):159-163, \n\nJanuary  1968. \n\n\f", "award": [], "sourceid": 1259, "authors": [{"given_name": "Ian", "family_name": "Parberry", "institution": null}, {"given_name": "Hung-Li", "family_name": "Tseng", "institution": null}]}