{"title": "Heterogeneous Neural Networks for Adaptive Behavior in Dynamic Environments", "book": "Advances in Neural Information Processing Systems", "page_first": 577, "page_last": 585, "abstract": null, "full_text": "577 \n\nHETEROGENEOUS NEURAL NETWORKS FOR \n\nADAPTIVE BEHAVIOR IN DYNAMIC ENVIRONMENTS \n\nHillel J. Chiel \nBiology Dept. \n& CAISR \nCWRU \n\nRandall D. Beer \n\nDept. of Computer Engineering and Science and \n\nCenter for Automation and Intelligent Systems Research \n\nCase Western Reserve University \n\nCleveland, OH 44106 \n\nLeon S. Sterling \nCS Dept. \n& CAISR \nCWRU \n\nABSTRACT \n\nResearch \nin artificial neural networks has genera1ly emphasized \nhomogeneous architectures. In contrast, the nervous systems of natural \nanimals exhibit great heterogeneity in both their elements and patterns \nof interconnection. This heterogeneity is crucial to the flexible \ngeneration of behavior which is essential for survival in a complex, \ndynamic environment. It may also provide powerful insights into the \ndesign of artificial neural networks. \nIn this paper, we describe a \nheterogeneous neural network for controlling \nthe wa1king of a \nsimulated insect. This controller is inspired by the neuroethological \nIt exhibits a \nand neurobiological literature on insect locomotion. \nvariety of statically stable gaits at different speeds simply by varying \nthe tonic activity of a single cell. It can also adapt to perturbations as a \nnatural consequence of its design. \n\nINTRODUCTION \n\nEven very simple animals exhibit a dazzling variety of complex behaviors which they \ncontinuously adapt to the changing circumstances of their environment. Nervous systems \nevolved in order to generate appropriate behavior in dynamic, uncertain situations and \nthus insure the survival of the organisms containing them. The function of a nervous \nsystem is closely tied to its structure. Indeed, the heterogeneity of nervous systems has \nbeen found to be crucial to those few behaviors for which the underlying neura1 mecha(cid:173)\nnisms have been worked out in any detail [Selverston, 1988]. There is every reason to \nbelieve that this conclusion will remain valid as more complex nervous systems are stud(cid:173)\nied: \n\nThe brain as an \"organ\" is much more diversified than, for example, the \nkidney or the liver. If the performance of relatively few liver cells is \nknown in detail, there is a good chance of defining the role of the whole \norgan. In the brain, different ce))s perform different, specific tasks ... \nOnly rarely can aggregates of neurons be treated as though they were \nhomogeneous. Above all, the cells in the brain are connected with one \nanother according to a complicated but specific design that is of far \ngreater complexity than the connections between cells in other organs. \n([Kuffler, Nicholls, & Martin, 1984], p. 4) \n\n\f578 \n\nBeer, Chiel and Sterling \n\nIn contrast to research on biological nervous systems, work in artificial neural networks \nhas primarily emphasized uniform networks of simple processing units with a regular in(cid:173)\nterconnection scheme. These homogeneous networks typically depend upon some gener(cid:173)\nal learning procedure to train them to perform specific tasks. This approach has certain \nadvantages. Such networks are analytically tractable and one can often prove theorems \nabout their behavior. Furthermore, such networks have interesting computational proper(cid:173)\nties with immediate practical applications. In addition, the necessity of training these net(cid:173)\nworks has resulted in a resurgence of interest in learning, and new training procedures are \nbeing developed. When these procedures succeed, they allow the rapid construction of \nnetworks which perform difficult tasks. \n\nHowever, we believe that the role of learning may have been overemphasized in artificial \nneural networks, and that the architectures and heterogeneity of biological nervous sys(cid:173)\ntems have been unduly neglected. We may learn a great deal from more careful study of \nthe design of biological nervous systems and the relationship of this design to behavior. \nToward this end, we are exploring the ways in which the architecture of the nervous \nsystems of simpler organisms can be utilized in the design of artificial neural networks. \nWe are particularly interested in developing neural networks capable of continuously \nsynthesizing appropriate behavior \nin dynamic, underspecified, and uncertain \nenvironments of the sort encountered by natural animals. \n\nTHE ARTIFICIAL INSECT PROJECT \n\nIn order to address these issues, we have begun to construct a simulated insect which we \ncall Periplaneta compUlatrix. Our ultimate goal is to design a nervous system capable of \nendowing this insect with a1l of the behaviors required for long-term survival in a com(cid:173)\nplex and dynamic simulated environment similar to that of natural insects. The skills re(cid:173)\nquired to survive in this environment include the basic abilities to move around, to find \nand consume food when necessary, and to escape from predators. In this paper, we focus \non the design of that portion of the insect's nervous system which controls its locomo(cid:173)\ntion. \n\nIn designing this insect and the nervous system which controls it, we are inspired by the \nbiological literature. It is important to emphasize, however, that this is not a modeling \nproject. We are not altempting to reproduce the experimental data on a particular animal; \nrather, we are using insights gleaned from Biology to design neural networks capable of \ngenerating similar behaviors. In this manner, we hope to gain a better understanding of \nthe role heterogeneity plays in the generation of behavior by nervous systems, and to ab(cid:173)\nstract design principles for use in artificial neural networks. \n\nFigure 1. Periplaneta computatrix \n\n\fHeterogeneous Neural Networks for Adaptive Behavior \n\n579 \n\nBODY \n\nThe body of our artificial insect is shown in Figure 1. It is loosely based on the American \nCockroach. Periplaneta americana [Bell & Adiyodi. 1981]. However. it is a reasonable \nabstraction of the bodies of most insects. It consists of an abdomen. head. six legs with \nfeet. two antennae. and two cerci in the rear. The mouth can open and close and contains \ntactile and chemical sensors. The antennae also contain tactile and chemical sensors. \nThe cerci contain tactile and wind sensors. The feet may be 'either up or down. When a \nfoot is down. it appears as a black square. Finally. a leg can apply forces which translate \nand rotate the body whenever its foot is down. \n\nIn addition. though the insect is only two-dimensional, it is capable of \"falling down.\" \nWhenever its center of mass falls outside of the polygon formed by its supporting feet, \nthe insect becomes statically unstable. If this condition persists for any length of time. \nthen we say that the insect has \"fallen down\" and the legs are no longer able to move the \nbody. \n\nNEURAL MODEL \n\nThe essential challenge of the Artificial Insect Project is to design neural controllers ca(cid:173)\npable of generating the behaviors necessary to the insect's survival. The neural model \nthat we are currently using to construct our controllers is shown in Figure 2. It represents \nthe firing frequency of a cell as a function of its input potential. We have used saturating \nlinear threshold functions for this relationship (see inset). The RC characteristics of the \ncell membrane are also represented. These cells are interconnected by weighted synapses \nwhich can cause currents to flow through this membrane. Finally, our model includes the \npossibility of additional intrinsic currents which may be time and voltage dependent. \nThese currents aJlow us to capture some of the intrinsic propenies which make real neu(cid:173)\nrons unique and have proven to be important components of the neural mechanisms un(cid:173)\nderlying many behaviors. \n\nSynaptic \nCurrents \n\nIntrinsic \nCurrents \n\nI(V)le \n\nv \n\nv \n\nC \n\nCell \nMembrane \n\nFiring \nProperties \n\nFigure 2. Neural Model \n\nFiring \nFrequency \n\n\f580 \n\nBeer, Chiel and Sterling \n\nFor example, a pacemaker cell is a neuron which is capable of endogenously producing \nrhythmic bursting. Pacemakers have been implicated in a number of temporally pat(cid:173)\nterned behaviors and playa crucial role in our locomotion controller. As described by \nKandel (1976, pp. 260-268), a pacemaker cell exhibits the following characteristics: (1) \nwhen it is sufficiently inhibited, it is silent, (2) when it is sufficiently excited, it bursts \ncontinuously, (3) between these extremes, the interburst interval is a continuous function \nof the membrane potential, (4) a transient excitation which causes the cell to fire between \nbursts can reset the bursting rhythm, and (5) a transient inhibition which prematurely ter(cid:173)\nminates a burst can also reset the bursting rhythm. \n\nThese characteristics can be reproduced with our neural model through the addition of \ntwo intrinsic currents. IH is a depolarizing current which tends to pull the membrane po(cid:173)\ntential above threshold. IL is a hyperpolarizing current which tends to pull the membrane \npotential below threshold. These currents change according to the following rules: (1) \nIH is triggered whenever the cell goes above threshold or IL terminates, and it then re(cid:173)\nmains active for a fixed period of time, and (2) IL is triggered whenever IH terminates, \nand it then remains acti ve for a variable period of time whose duration is a function of the \nmembrane potential. In our work to date, the voltage dependence of IL has been linear. \n\nLOCOMOTION \n\nAn animal's ability to move around its environment is fundamental to many of its other \nbehaviors. In most insects, this requirement is fulfilled by six-legged walking. Thus, this \nwas the first capability we sought to provide to P. computatrix. Walking involves the \ngeneration of temporally patterned forces and stepping movements such that the insect \nmaintains a steady forward motion at a variety of speeds without falling down. Though \nwe do not address all of these issues here, it is worth pointing out that locomotion is an \ninteresting adaptive behavior in its own right. An insect robustly solves this complex co(cid:173)\nordination problem in real Lime in the presence of variations in load and terrain, develop(cid:173)\nmental changes, and damage to the walking apparatus itself [Graham, 1985]. \n\nLEG CONTROLLER \n\nThe most basic components of walking are the rhythmic movements of each individual \nleg. These consist of a swing phase, in which the foot is up and the leg is swinging for(cid:173)\nward, and a stance phase, in which the foot is down and the leg is swinging back, propel(cid:173)\nling the body forward. In our controller, these rhythmic movements are produced by the \nleg controller circuit shown in Figure 3. There is one command neuron, C, for the entire \ncontroller and six copies of the remainder of this circuit, one for each leg. \n\nThe rhythmic leg movements are primarily generated centrally by the portion of the leg \ncontroller shown in solid lines in Figure 3. Each leg is controlled by three motor neurons. \nThe stance and swing motor neurons determine the force with which the leg is swung \nbackward or forward, respectively, and the foot motor neuron controls whether the foot is \nup or down. Normally, the foot is down and the stance motor neuron is active, pushing \n\n\fHeterogeneous Neural Networks for Adaptive Behavior \n\n581 \n\nthe leg back and producing a stance phase. Periodically, however, this state is interrupted \nby a burst from the pacemaker neuron P. This burst inhibits the foot and stance motor \nneurons and excites the swing motor neuron, lifting the foot and swinging the leg for(cid:173)\nward. When this burst terminates, another stance phase begins. Rhythmic bursting in P \nthus produces the basic swing/stance cycle required for walking. The force applied dur(cid:173)\ning each stance phase as well as the time between bursts in P depend upOn the level of ex(cid:173)\ncitation supplied by the command neuron C. This basic design is based on the flexor \nburst-generator model of cockroach walking [pearson, 1976]. \n\nIn order to properly time the transitions between the swing and stance phases, the control(cid:173)\nler must have some information about where the legs actually are. The simplest way to \nprovide this information is to add sensors which signal when a leg has reached an ex(cid:173)\ntreme forward or backward angle, as shown with dashed lines in Figure 3. When the leg \nis all the way back, the backward angle sensor encourages P to initiate a swing by excit(cid:173)\ning it. When the leg is all the way forward, the forward angle sensor encourages P to ter(cid:173)\nminate the swing by inhibiting it. These sensors serve to reinforce and fine-tune the cen\u00b7 \ntrally generated stepping rhythm. They were inspired by the hair plate receptors in P. \namericana, which seem LO playa similar role in its locomotion [Pearson, 1976]. \n\nThe RC characteristics of our neural model cause delays at the end of each swing before \nthe next stance phase begins. This pause produces a \"jerky\" walk which we sought to \navoid. In order to smooth out this effect, we added a stance reflex comprised of the dot(cid:173)\nted connections shown in Figure 3. This reflex gives the motor neurons a slight \"kick\" in \nthe right direction to begin a stance whenever the leg is swung all the way forward and is \nalso inspired by the cockroach [Pearson, 1976]. \n\nStance \n\nFoot \n\nSwing \n\nBackward Angle \n\nSensor \n\nO~\u00b7\n\n\"-\n\n\u00b7 ... : ........ '. '. ' ... ' .. ~:~ :D \n\n\u2022...................... \n\nForward Angle \n\nSensor \n\nE>- Excitatory Connection \n\n_ \n\nInhibitory Connection \n\nFigure 3. Leg Controller Circuit \n\n\f582 \n\nBeer, Chiel and Sterling \n\nFigure 4. Central Coupling between Pacemakers \n\nLOCOMOTION CONTROLLER \n\nIn order for these six individual leg controllers to serve as the basis for a locomotion con(cid:173)\ntroller, we must address the issue of stability. Arbitrary patterns of leg movements will \nnot, in general, lead to successful locomotion. Rather, the movements of each leg must \nbe synchronized in such a way as to continuously maintain stability. \n\nA good rule of thumb is that adjacent legs should be discouraged from swinging at the \nsame time. As shown in Figure 4, this constraint was implemented by mutual inhibition \nbetween the pacemakers of adjacent legs. So, for example, when leg L2 is swinging, legs \nLI, L3 and R2 are discouraged from also swinging, but legs RI and R3 are unaffected (see \nFigure Sa for leg labelings). This coupling scheme is also derived from Pearson's (1976) \nwork. \n\nThe gaits adopted by the controller described above depend in general upon the initial an(cid:173)\ngles of the legs. To further enhance stability, it is desirable to impose some reliable order \nto the stepping sequence. Many animals exhibit a stepping sequence known as a metach(cid:173)\nronal wave, in which a wave of stepping progresses from back to front. In insects, for ex(cid:173)\nample, the back leg swings, then the middle one, then the front one on each side of the \nbody. This sequence is achieved in our controller by slightly increasing the leg angle \nranges of the rear legs, lowering their stepping frequency. Under these conditions, the \nrear leg oscillators entrain the middle and front ones, and produce metachronal waves \n[Graham, 1977]. \n\nRESULTS \n\nWhen this controller is embedded in the body of our simulated insect, it reliably produces \nsuccessful walking. We have found that the insect can be made to walk at different \nspeeds with a variety of gaits simply by varying the flring frequency of the command \nneuron C. Observed gaits range from the wave gait, in which the metachronal waves on \neach side of the body are very nearly separated, to the tripod gait, in which the front and \nback legs on each side of the body step with the middle leg on the opposite side. These \ngaits fall out of the interaction between the dynamics of the neural controller and the \nbody in which it is embedded. \n\n\fHeterogeneous Neural Networks for Adaptive Behavior \n\n583 \n\nL~R \n\nI \nZ \n\nI \nt \n\nS \n\n3 \n\n-\n\n_ \n\n_ \n\n_ \n\n.$1q'plq ,dIrII \n_ \n_ \n-\n_ \n\nIt3 \na, \nJI2 \nL3 \nLl ii i ,nT, iii ii i i., 1m. iii' iii i 1m: i Ii iii, , \nL2_ \n\n-\n25~dlv \n\n-\n-\n\n\u2022 \n\n-\n\n-\n\n-\n\n_ \n\nIt3 \n112. ___ -\nal ___ _ \n\nstq'pfq 'dIrII \n\n_ \n\n_ \n\n-\n\nL3. \nU \n_ \nLl \n\n_ \n\n_ \n\n-\n\n_ \n\n-\n\n-\n\n\u2022 \n\n-\n\n_ \n\n.\"iI .. hi 'liihi \"\"\"\" Ii i hi iii Ii liiii i i \n\n-\n25 --=vdlv \n\n_ \n\nIt3 R2 _ \n\n_IIIJ11l'aMm \n\n- - -- - --\u2022 \n.1 _ - - -\nL3 __ - - \u2022 \nL2 __ - -\nLl ~'iii Ii ,.\"'ii.i 'irm \u2022\u2022 ; \u2022\u2022 .wriii'iilWt \n- - - - -\n\n&ea'p.ptq I'dIrII \n1t3 __ -\n-\nal ___ -\n1.2 ___ _ \nu ___ -\nL 1 \"Ii mT II Ii 1i'l'lT! \" Ii i \"\"Ii \" i ~'mT i Ii ill \n\n25 -'Ct,v \n\n-\n-\n-\n\n25~d,v \n\n.. -.... \n\n.. \n\nAS \"...,., ,,--'.. \\ . - ''''_ \n\n~ -\nI :~I . p . - - . \n\ntI,: \n\n1 t ,1_11 , _ _ \n\n_ \n\nI \nLS! \n\n;: \n,1 .. _ : , \u2022 . \u2022 \u2022 \u2022 _ \n\n: \n\n'...-,', . \u2022 -\n\nL, \nll \u00b7\u00b7 .. ~ - .... ' ~ -\n\n\u2022.\u2022\u2022 _ \n\n\" \n\n-\n\n-\n\nA. \n\nB. \n\nFigure S. \n1966]). (B) Selected Gaits Observed in P. computatrix. \n\n(A) Description of Some Gaits Observed in Natural Insects (from [Wilson, \n\nIf the legs .are labeled as shown at the top of Figure Sa, then gaits may be conveniently \ndescribed by their stepping patterns. In this representation, a black bar is displayed dur(cid:173)\ning the swing phase of each leg. The space between bars represents the stance phase. \nSelected gaits observed in P. computatrix at different speeds are shown in Figure 5b as \nthe command neuron firing frequency is varied from lowest (top) to highest (bottom). At \nthe lower speeds, the metachronal waves on each ~ide of the body are very apparent. The \nmetachronal waves can still be discerned in fas~r walks. However, they increasingly \noverlap as the stance phases shorten, until the tripod gait appears at the highest speeds. \nThis sequence of gaits bears a strong resemblance to some of those that have beep de(cid:173)\nscribed for natural insects, as shown in Figure Sa [Wilson, 1966]. \n\nIn order to study the robustness of this controller and to gain insight into the detailed \nmechanisms of its operation, we have begun a series of lesion studies. Such studies ex-\n\n\f584 \n\nBeer, Chiel and Sterling \n\namine the behavioral effects of selective damage to a neural controller. This study is still \nin progress and we only report a few preliminary results here. In general, we have been \nrepeatedly surprised by the intricacy of the dynamics of this controller. For example, re(cid:173)\nmoval of all of the forward angle sensors resulted in a complete breakdown of the \nmetachronal wave at low speeds. However, at higher speeds, the gait was virtually unaf(cid:173)\nfected. Only brief periods of instability caused by the occasional overlap of the slightly \nlonger than normal swing phases were observed in the tripod gait, but the insect did not \nfall down. Lesioning single forward angle sensors often dynamically produced compen(cid:173)\nsatory phase shifts in the other legs. Lesions of selected central connections produced \nsimilarly interesting effects. In general, our studies seem to suggest subtle interactions \nbetween the central and peripheral components of the controller which deserve much \nmore exploration. \n\nFinally, we have observed the phenomena of reflex stepping in P. computalrix. When the \ncentral locomotion system is completely shut down by strongly inhibiting the command \nneuron and the insect is continuously pushed from behind, it is still capable of producing \nan uncoordinated kind of walking. As the insect is pushed forward, a leg whose foot is \ndown bends back until the backward angle sensor initiates a swing by exciting the pace(cid:173)\nmaker neuron P. When the leg has swung all the way forward, the stance reflex triggered \nby the forward angle sensor puts the foot down and the cycle repeats. \n\nBrooks (1989) has described a semi-distributed locomotion controller for an insect-like \nautonomous robot. We are very much in agreement with his general approach. \nHowever, his controller is not as fully distributed as the one described above. It relies on \na central leg lift sequencer which must be modified to produce different gaits. Donner \n(1985) has also implemented a distributed hexapod locomotion controller inspired by an \nearly model of Wilson's (1966). His design used individual leg controllers driven by leg \nload and position information. These leg controllers were coupled by forward excitation \nfrom posterior legs. Thus, his stepping movements were produced by reflex-driven pe(cid:173)\nripheral oscillators rather than the central oscillators used in our model. He did not report \nthe generation of the series of gaits shown in Figure Sa. Donner also demonstrated the \nability of his controller to adapt to a missing leg. We have experimented with leg ampu(cid:173)\ntations as well, but with mixed success. We feel that more accurate three-dimensional \nload information than we currently model is necessary for the proper handling of amputa(cid:173)\ntions. Neither of these other locomotion controllers utilize neural networks. \n\nCONCLUSIONS AND FUTURE WORK \n\nWe have described a heterogeneous neural network for controlling the walking of a simu(cid:173)\nlated insect. This controller is completely distributed yet capable of reliably producing a \nrange of statically stable gaits at different walking speeds simply by varying the tonic ac(cid:173)\ntivity of a single command neuron. Lesion studies have demonstrated that the controller \nis robust, and suggested that subtle interactions and dynamic compensatory mechanisms \nare responsible for this robustness. \n\nThis controller is serving as the basis for a number of other behaviors. We have already \nimplemented wandering, and are currently experimenting with controllers for recoil re-\n\n\fHeterogeneous Neural Networks for Adaptive Behavior \n\n585 \n\nsponses and edge following. In the near future, we plan to implement feeding behavior \nand an escape response, resulting in what we feel is the minimum complement of behav(cid:173)\niors necessary for survival in an insect-like environment Finally, we wish to introduce \nplasticity into these controllers so that they may better adapt to the exigencies of particu(cid:173)\nlar environments. We believe that learning is best viewed as a means by which additional \nflexibility can be added to an existing controller. \n\nThe locomotion controller described in this paper was inspired by the literature on insect \nlocomotion. The further development of P. compUlalrix will continue to draw inspiration \nfrom the neuroethology and neurobiology of simpler natural organisms. In trying to de(cid:173)\nsign autonomous organisms using principles gleaned from Biology, we may both im(cid:173)\nprove our understanding of natural nervous systems and discover design principles of use \nto the construction of artificial ones. A robot with \"only\" the behavioral repertoire and \nadaptability of an insect would be an impressive achievement indeed. In particular, we \nhave argued in this paper for a more careful consideration of the intrinsic architecture and \nheterogeneity of biological nervous systems in the design of artificial neural networks. \nThe locomotion controller we have described above only hints at how productive such an \napproach can be. \n\nReferences \n\nBell, W.J. and K.G. Adiyodi eds (1981). The American Cockroach. New York: Chapman \nand Hall. \n\nBrooks, R.A. (1989). A robot that walks: emergent behaviors from a carefully evolved \nnetwork. Neural Computation 1(1). \n\nDonner, M. (1987). Real-time control of walking \nVolume 7). Cambridge, MA: Birkhauser Boston, Inc. \n\n(Progress in Computer Science, \n\nGraham, D. (1977). Simulation of a model for the coordination of leg movements in free \nwalking insects. Biological Cybernetics 26:187-198. \n\nGraham, D. (1985). Pattern and control of walking in insects: Advances in Insect \nPhYSiology 18:31-140. \n\nKandel, E.R. (1976). Cellular Basis of Behavior: An Introduction to Behavioral \nNeurobiology. W.H. Freeman. \n\nKuffler, S.W., Nicholls, J.G., and Martin, A. R. (1984). From Neuron to Brain: A \nCellular Approach to the Function of the Nervous System. Sunderland, MA: Sinauer \nAssociates Inc. \n\nPearson, K. (1976). The control of walking. Scientific American 235:72-86. \n\nSelverston, A.I. (1988). A consideration of invertebrate central pattern generators as com(cid:173)\nputational data bases. Neural Networks 1:109-117. \n\nWilson, D.M. (1966). Insect walking. Annual Review of Entomology 11:103-122. \n\n\f", "award": [], "sourceid": 115, "authors": [{"given_name": "Randall", "family_name": "Beer", "institution": null}, {"given_name": "Hillel", "family_name": "Chiel", "institution": null}, {"given_name": "Leon", "family_name": "Sterling", "institution": null}]}