{"title": "A Connectionist Expert System that Actually Works", "book": "Advances in Neural Information Processing Systems", "page_first": 248, "page_last": 255, "abstract": null, "full_text": "248 \n\nA CONNECTIONIST EXPERT SYSTEM \n\nTHAT ACTUALLY WORKS \n\nGary  Bradshaw \nPsychology \n\nRichard  Fozzard \nComputer Science \n\nLouisCeci \nComputer Science \n\nUniversity of Colorado \n\nBoulder, CO 80302 \n\nfozzard@boulder.colorado.edu \n\nABSTRACf \n\nThe Space Environment Laboratory in Boulder has collaborated \nwith  the  University  of  Colorado  to  construct  a  small  expert \nsystem for solar flare forecasting, called THEa. It performed as \nwell  as  a  skilled  human  forecaster.  We  have  constructed \nTheoNet,  a  three-layer  back-propagation  connectionist  net(cid:173)\nwork  that  learns  to  forecast  flares  as  well  as  THEa  does. \nTheoNet's  success  suggests  that  a  connectionist  network  can \nperform the task of knowledge engineering automatically.  A \nstudy of the internal representations constructed by the network \nmay give insights to the \"microstructure\" of reasoning processes \nin the human brain. \n\nINTRODUCTION \n\nCan  neural  network  learning  algorithms  let  us  build  \"expert  systems\" \nautomatically, merely by presenting the network with data from  the  problem \ndomain?  We  tested  this  possibility  in  a  domain  where a  traditional  expert \nsystem has been developed  that is at least as good as the expert, to  see if the \nconnectionist approach could stand up to tough competition. \nKnowledge-based expert systems attempt to capture in a computer program the \nknowledge of a  human expert in a  limited doma!n and make  this knowledge \navailable to a  user with less experience.  Such systeins could be valuable as an \nassistant to a forecaster or as a training tool.  In the past three years, the Space \nEnvironment Laboratory (SEL)  in Boulder has collaborated with the Computer \nScience and Psychology Departments at the University of Colorado to construct \na  small  expett  system  emulating  a  methodology  for  solar  flare  forecasting \ndeveloped  by  Pat  McIntosh,  senior  solar  physicist  at  SEL.  The  project \nconvincingly demonstrated the possibilities of this type of computer assistance, \nwhich also proved to be a  useful  tool  for formally expressing a  methodology, \nverifying  its  performance,  and  instructing  novice  forecasters.  The  system, \n\n\fA Connectionist Expert System that Actually Warks \n\n249 \n\nnamed THEO (an OPS-83 production system with about 700 rules), performed as \nwell  as a  skilled  human forecaster  using the same methods, and scored  well \ncompared with actual forecasts  in the period covered  by the test data [Lewis \nand Dennett 1986]. \nIn  recent  years  connectionist  (sometimes  called  \"non-symbolic\"  or  \"neural\") \nnetwork approaches have been used with varying degrees of success to simulate \nhuman behavior in such areas as vision and speech learning and  recognition \n[Hinton 1987, Lehky and Sejnowski 1988, Sejnowski and Rosenberg 1986, Elman \nand Zipser 1987].  Logic (or \"symbolic\") approaches have been used to simulate \nhuman (especially expert)  reasoning [see Newell 1980 and Davis 1982].  There \nhas  developed  in  the  artificial  intelligence  and  cognitive  psychology \ncommunities quite a  schism between the two areas of research and  the same \nproblem has rarely been attacked by both approaches.  It is hardly our intent to \ndebate the relative merits of the two paradigms.  The intent of this project is to \ndirectly  apply  a  connectionist  learning  technique  (multi-layer  back(cid:173)\npropagation)  to  the  same  problem,  even  the  very same database  used  in an \nexisting successful rule-based expert system.  At this time we know of no current \nwork attempting to do this. \nForecasting, as described by those who practice it, is a unique combination of \ninformal  reasoning within very soft constraints supplied by often incomplete \nand  inaccurate  data.  The  type  of  reasoning  involved  makes  it  a  natural \napplication for  traditional  rule-based approaches.  Solar and flare  occurrence \ndata are often inconsistent and noisy.  The nature of the data, therefore, calls \nfor  careful handling of rule  strengths and certainty factors.  Yet  dealing with \nthis  sort  of  data  is  exactly  one  of  the  strengths  claimed  for  connectionist \nnetworks.  It may also be that some of the reasoning involves pattern matching \nof  the  different  categories  of  data.  This  is  what  led  us  to  hope  that  a \nconnectionist  network  might  be  able  to  learn  the  necessary  internal \nrepresentations to cope with this task. \n\nTECHNICAL APPROACH \n\nThe TheoNet  network model has three layers of simple, neuron-like processing \nelements called \"units\".  The lowest layer is the input layer and is clamped to  a \npattern that is a  distributed  representation  of the solar data for  a  given day. \nFor  the  middle  (\"hidden\")  and  upper  (\"output\")  layers,  each  unit's  output \n(called  \"activation\")  is  the  weighted  sum of all  inputs from  the  units  in  the \nlayer below: \n\nYj  = ~1 __ \n\n1 + e-Xj \n\nwhere:  Xj  = ~Y'Wji - 8j \n\n, \n\n(1) \n\nwhere Yi  is the activation of the ith  unit in  the layer below, Wji  is  the weight \non the connection from  the ith to the jth  unit, and 9j is the threshold of the jth \n\n\f250 \n\nFozzard, Bradshaw and Ceci \n\nunit.  The weights are initially set to  random values between -1.0 and +1.0, but \nare allowed  to  vary beyond  that  range.  A least mean  square error learning \nprocedure called back-propagation  is  used  to  modify the weights incrementally \nfor each input data pattern presented to the network.  This compares the output \nunit  activations  with  the  \"correct\"  (what  actually  happened)  solar  flare \nactivity for  that day.  This gives the weight update rule: \n\n(2) \n\nwhere  V E(t)  is  the  partial  derivative  of  least  mean  square  error,  \u00a3  is  a \nparameter  called  the  learning  rate  that  affects  how  quickly  the  network \nattempts to  converge on the appropriate weights (if possible), and a  is called \nthe momentum  which affects the amount of damping in the procedure.  This is \nas in  [Hinton  1987],  except  that  no  weight  decay  was  used.  Weights  were \nupdated after each presentation of an input/output pattern. \nThe network was constructed as shown in Figure 1.  The top three output units \nare intended to code for each of the three classes of solar flares to be forecasted. \nThe individual activations are currently intended to correspond  to  the relative \nlikelihood of a  flare  of that class within the next 24  hours (see  the analysis of \nthe results below).  The 17 input units provide a  distributed coding of the ten \ncategories of input data that are currently fed  into  the \"default\" mode of the \nexpert system THEO.  That is,  three binary  (on/off) units code for  the  seven \nclasses of  sunspots,  two  for  spot distribution,  and  so  on.  The  hidden  units \nmediate  the  transfer  of  activation  from  the  input  to  the  output  units  and \nprovide  the  network  with  the  potential  of forming  internal  representations. \nEach layer is fully  interconnected  with  the layer above and/or below it, but \nthere are no connections within layers. \n\nRESULTS \n\nThe  P3  connectionist  network  simulator  from  David  Zipser of University of \nCalifornia at San Diego's parallel distributed processing (PDP) group was used \nto  implement  and  test  TheoNet  on  a  Symbolics  3653  workstation.  This \nsimulator allowed  the  use of Lisp code  to  compile  statistics and provided an \ninteractive environment for working with the network simulation. \n\nThe  network  was  trained  and  tested  using  two  sets  of  data  of  about  500 \ninput/ output pairs (solar data/flare occurrence) each from the THEO database. \nMany of these resulted in the same input pattern (there  were only about 250 \ndifferent input patterns total), and in many cases the same input would result in \ndifferent flare results in the following 24  hours.  The data was from a low flare \nfrequency period (about 70-80 flares total).  These sorts of inconsistencies in the \ndata make the job of prediction difficult to systematize.  The network would be \n\n\fA Connectionist Expert System that Actually Works \n\n251 \n\nOUTPUT: \n\nFlare probability by class \nx \n\nc \n\nM \n\n\"D\"=011 \n\n\"A\"=011 \n\n\"0\"=01 \n\ngrowth \n\nyes  no \n\nabove 5 \n\n\u2022 \n\n'W' \nen \n~ \n() \n.r::. \n.52 \n.... \n:::l \nN \n\n,. \n\n'  , \n\nreduced \n\n''--'' \n\n.~ \n.~ \n~ \n\niJ \nc \n.2 \nS \ng \nw \n\niJ \nc \n.2 \n:; \n..c \n\u00b7c \n.~ \nCI \n\n-\n-\n\n'W' \n8. \nen \nen \nCD \n~ \n~ \n\nINPUT SOLAR DATA \n\nless than M1 \n\nsmall \n\n, . \n\n(G \n\niJ \n! \nu:: \nen \n:::l \n.2 \n> \n! \nQ. \n\n, '--' '--' '--' '--' \n\n)( \nCD \n0-\n\n(G \n.... \nCD \n\n(G \n)( \n.... \noS! \nCD \nE  \u00ab  \u00ab \n0-\n8. \nen \niii \nCD \n~ \nas \n...J \n\nE  8 \n8 \n~ \n~ \n\u00b7c \n.e \n.!a \n:I: \n\n>. \nC \nCD \n~ \na::: \n\n1.  Modified Zurich class (7 possible values: A/B/C/D/E/F/H) \n2.  Largest spot size (6 values: X/R/S/ A/H/K) \n3.  Spot distribution (4 values: X/O/I/C) \n4.  Activity (reduced /  unchanged) \n5.  Evolution (decay /  no growth /  or growth) \n6.  Previous flare activity (less than Ml  /  Ml /  more than Ml) \n7.  Historically complex (yes/no) \n8.  Recently became complex on this pass (yes/no) \n9.  Area (small/large) \n10. Area of the largest spot (up to 5/ above 5) \n\nFigure 1.  Architecture of TheoNet \n\n\f252 \n\nFozzard, Bradshaw and Ceci \n\ntrained on one data set and  then tested on the other (it did not matter which \none was used for which). \nTwo ways of measuring performance were used.  An earlier simulation tracked a \nsimple  measure  called  overall-prediction-error.  This  was  the  average \ndifference over one complete epoch of input patterns between the activation of \nan output unit and the \"correct\" value it was supposed to  have.  This is directly \nrelated to the sum-squared error used by the back-propagation method. \nWhile  the  overall-prediction-error  would  drop  quickly  for  all  flare  classes \nafter a  dozen  epoches  or so  (about  5 minutes on  the  Symbolics),  individual \nweights would take much longer to stabilize.  Oscillations were seen in weight \nvalues if a large learning rate was used.  When this was reduced to 0.2 or lower \n(with a momentum of 0.9), the weights would converge more smoothly to  their \nfinal  values. \nOverall-prediction-error however, is not a good measure of performance since \nthis could be reduced  simply by reducing average activation (a  \"Just-Say-No\" \nnetwork).  Analyzing  performance  of  an  expert  system  is  best  done  using \nmeasures  from  the  problem  domain.  Forecasting  problems  are  essentially \nprobabilistic,  requiring  the  detection  of  signal  from  noisy  data.  Thus \nforecasting  techniques and systems are often analyzed  using signal detection \ntheory [Spoehr and Lehmkuhle 1982]. \nThe  system  was  modified  to  calculate  P(H),  the  probability  of a  hit,  and \nP(FA),  the  probability of a  false  alarm,  over each  epoch.  These  parameters \ndepend on the response  bias,  which determines the  activation level  used  as a \nthreshold  for  a  yes/no  response\u00b7  .  A graph of P(H)  versus P(FA)  gives  the \nreceiver  operating  characteristic  or ROC  curve.  The amount  that  this curve is \nbowed away from a  1:1  slope is  the degree to which a  signal is being detected \nagainst background.  This was the method used for measuring the performance \nof THEO [Lewis and Dennett 1986]. \nAs in  the earlier simulation, the network was exposed  to  the  test data before \nand  after  training.  After  training,  the  probability  of  hits  was  consistently \nhigher  than  that  of  false  alarms  in  all  flare  classes  (Figure  2).  Given  the \nlimited  data and very low activations for X-class flares,  it  mayor may not be \nreasonable to draw conclusions about the network's ability to detect these - in \nthe  test  data  set  there  were  only  four  X-flares  in  the  entire  data  set.  The \ndegree to which the hits exceed false alarms is given by  a', the area under the \ncurve.  The performance of TheoNet  was at least as good as the THEO expert \nsystem . \n\n..  Even though both THEO and TheoNet  have a continuous output (probability \nof  flare  and  activation),  varying  the  response  bias  gives  a  continuous \nevaluation of performance at any output level. \n\n\fA Connectionist Expert System that Actually Works \n\n253 \n\n1.0  -\n\nQ::' \n\n/ \n\n1.0 \n\na/ \n\n1.0 \n\n\u2022 \n\u2022 \n\n/ \n\n~5 \n\nQ::' \n\na' \u2022. 71 \n\na \n\n/ \n\n\u2022 \n\nC-class \n\n0 \n/ \n\n0 \n\n/ \n\n, \n1/ \n\n/'~/ \n\n~ \ni  ~.5  a \n~ \n\n\u2022 \\/  ~5 \n. / \n\n/  \"// \n@ ~5  . 1' /  ~5  .1, /  \u00a7:5 .,,-// \n.  / \n\n~  a' \u2022. 78 \n\n.5 \nP(FA) \n\n.5 \nP(FA) \n\n.5 \nP(FA) \n\nM-class \n\n1/ \n\na' \u2022. 90 \n\n\u2022 \n\n/ \n\n0.:' \n\nI \nl-\n\n/ \n\n/ \n\n.a \n\n\u2022 \n\n-/ \n\n1.0 \n\n1.0 \n\n/ \n\n\u2022 \n\nX-class \n\n1.0 \n\n1.0 \n\nQ::. \n\n/ \n\n/ \n\n1.0 \n\n/ \n\n/ \n\n1.0 \n\n/ \n\n/ \n\n\u2022 \n/ \n\nC-c1ass \n\n/ \n\na' \u2022. 68 \n\nQ::. \n\n/ \n\na \n\ni /  M-class \n\na' \u2022. 70 \n\n/ \n\n/ \n\n/ \n\nX-class \n\n/ \n\na' \u2022. 78 \n\n.5 \nP(FA) \n\n1.0 \n\n.5 \nP(FA) \n\n1.0 \n\n.5 \nP(FA) \n\n1.0 \n\nFigure 2.  ROC perfonnance measures of TheoNet and THEO \n\nCONCLUSIONS \n\nTwo  particularly intriguing prospects are raised by these  results.  The  first  is \nthat  if  a  connectionist  network  can  perform  the  same  task  as  a  rule-based \nsystem, then a study of the internal representations constructed by the network \nmay give insights to the \"microstructure\" of how reasoning processes occur in the \nhuman brain.  These are the  same  reasoning  processes delineated  at a  higher \nlevel  of description by the rules in an expert system.  How this  sort of work \nmight  affect  the  schism  between  the  symbolic  and  non-symbolic  camps \n(mentioned  in  the  introduction) is  anyone's guess.  Our hope is  that  the  two \nparadigms may  eventually come  to  complement and  support  each  other  in \ncognitive science research. \nThe  second  prospect is more of an engineering nature.  Though cortt1ectionist \nnetworks  do  offer  some  amount  of  biological  plausibility  (and  hence  their \ntrendy status right now), it is difficult to imagine a  neural mechanism for  the \nback-propagation algorithm.  However, what do engineers care?  As a lot, they \nare more interested  in implementing a  solution than explaining the  nature of \nhuman thought.  Witness the current explosion of expert system technology in \nthe  marketplace  today.  Yet  for  all  its  glamor,  expert  systems  have  usually \nproved  time  consuming  and  expensive  to  implement.  The  \"knowledge(cid:173)\nengineering\" step of interviewing experts and transferring their knowledge to \n\n\f254 \n\nFozzard, Bradshaw and Ceci \n\nrules that work successfully together has been the most difficult and expensive \npart,  even  with  advanced  knowledge  representation  languages  and  expert \nsystem  shells.  TheoNet  has  shown  that at  least  in  this  instance,  a  standard \nback-propagation  network can quickly  learn  those  necessary  representations \nand interactions (rules?) needed to do the same sort of reasoning.  Development \nof THEO  (originally presented as one of the quickest developments of a usable \nexpert system)  required  more  than a  man-year of  work and  700  rules,  while \nTheoNet  was  developed  in  less  than  a  week  using  a  simple  simulator.  In \naddition, THEO requires about five minutes to process a single prediction while \nthe network requires only a few milliseconds, thus promising better perfonnance \nunder real-time conditions. \nMany questions remain to be answered.  TheoNet has only been tested on a small \nsegment of  the  ll-year solar cycle.  It has  yet  to  be  determined  how many \nhidden units are needed for generalization of performance (is a simple pattern \nassociator sufficient?).  We  would like to examine the internal representations \nformed  and  see  if there  is  any  relationship  to  the  rules  in THEO.  Without \nthose  interpretations, connectionist networks cannot easily offer  the help and \nexplanation  facilities  of  traditional  expert  systems  that  are  a  fallout  of  the \nrule-writing process. \nSince the categories of data used  were  what is input to  THEO,  and therefore \nknown to be significant, we need to ask if the network can eliminate redundant \nor unnecessary categories.  We also would like to attempt to  implement other \nwell-known expert systems to determine the generality of this approach. \n\nAcknowledgements \nThe authors would like to acknowledge the encouragement and advice of Paul \nSmolensky (University of Colorado) on this project and the desktop publishing \neqUipment of Fischer Imaging Corporation in Denver. \n\nREFERENCES \n\nRandall  Davis \"Expert Systems: Where Are We?  And Where Do We Go  From \n\nHere?\",  The  AI Magazine,  Spring 1982 \n\nJ.L.  Elman  and  David  Zipser  Discovering  the  Hidden  Structure  of  Speech  ICS \n\nTechnical Report 8701,  University of California, San Diego \n\nGeoffrey  Hinton \"Learning Translation  Invariant  Recognition  in a  Massively \nParallel  Network\",  in  Proc.  Conf.  Parallel  Architectures  and  Languages \nEurope, Eindhoven, The Netherlands, June 1987 \n\n\fA Connectionist Expert System that Actually Works \n\n255 \n\nSidney Lehky and Terrence Sejnowski, \"Neural Network Model for the Cortical \nRepresentation of Surface Curvature from  Images of Shaded Surfaces\", \nin Sensory Processing, J.S. Lund, ed., Oxford 1988 \n\nClayton Lewis and Joann Dennett \"Joint CU/NOAA Study Predicts Events on \nthe Sun with Artificial  Intelligence Technology\", CUEngineering,  1986 \n\nAllen  Newell  ''Physical  Symbol Systems\",  Cognitive  Science  4:135-183 \nDavid  Rumelhart,  Jay  McClelland,  and  the  PDP  research  group,  Parallel \nDistributed  Processing.  Volume  1.  Cambridge,  MA,  Bradford  books, \n1986 \n\nTerrence  Sejnowski  and  C.R.  Rosenberg,  NETtalk:  A  parallel  network  that \nlearns  to  read  aloud  Technical  Report  86-01  Dept.  of  Electrical \nEngineering  and  Computer Science,  Johns  Hopkins  University,  Bal(cid:173)\ntimoreMD \n\nK.T.  Spoehr  and  S.W.  Lehmkuhle  \"Signal  Detection  Theory\"  in  Visual \n\nInformation  Processing,  Freeman  1982 \n\n\f", "award": [], "sourceid": 135, "authors": [{"given_name": "Richard", "family_name": "Fozzard", "institution": null}, {"given_name": "Gary", "family_name": "Bradshaw", "institution": null}, {"given_name": "Louis", "family_name": "Ceci", "institution": null}]}