{"title": "Optical Implementation of a Self-Organizing Feature Extractor", "book": "Advances in Neural Information Processing Systems", "page_first": 821, "page_last": 828, "abstract": null, "full_text": "Optical Implementation of a Self\u00b7Organizing \n\nFeature Extractor \n\nDana Z. Anderson*, Claus Benkert, Verena Hebler, Ju-Seog Jang, \n\nDon Montgomery, and Mark Saffinan. \n\nJoint Institute for Laboratory Astrophysics, University of Colorado and the \nDepartment of Physics, University of Colorado, Boulder Colorado 80309-0440 \n\nAbstract \n\nWe demonstrate a self-organizing system based on photorefrac(cid:173)\ntive ring oscillators.  We  employ the system in two ways that \ncan  both be thought of as feature extractors; one acts on  a  set \nof images exposed repeatedly to the system strictly as a linear \nfeature extractor, and the other serves as a signal demultiplex(cid:173)\ner  for  fiber  optic  communications.  Both  systems  implement \nunsupervised competitive learning embedded within the mode \ninteraction dynamics between the modes of a set of ring oscilla(cid:173)\ntors.  After a training period, the modes of the rings become as(cid:173)\nsociated  with \nfrequencies within the incoming data stream. \n\nimage  features  or  carrier \n\nthe  different \n\n1 Introduction \nSelf-organizing networks (Kohonen, Hertz, Domany) discover features or qual(cid:173)\nities about their input environment on their own; they learn without a teacher \nmaking explicit  what is  to  be  learned.  This  property  reminds  us  of severa] \nubiquitous behaviors we  see in the physical and natural sciences such  as pat(cid:173)\ntern formation, morphogenesis and phase transitions (Domany).  While in the \nnatural case one is usually satisfied simply to analyze and understand the be(cid:173)\nhavior of a self-organizing system, we  usually have a specific function in mind \nthat we wish a neural network to perform.  That is, in the network case we wish \nto synthesize a  system that will  perform the desired function.  Self-organizing \nprinciples are particularly valuable when one does not know ahead of time ex(cid:173)\nactly what to  expect from  the input to be processed and when it is some prop(cid:173)\nerty  of the  input  itself that is  of interest.  For  example,  one  may  wish  to \ndetermine some quality about the input statistics - this one can often do by ap(cid:173)\nplying  self-organization  principles.  However,  when  one  wishes  to  attribute \nsome meaning to  the data, self-organization principles are probably poor can(cid:173)\ndidates for this task. \n\n821 \n\n\f822 \n\nAnderson,  Benkert,  Hebler. lang. Montgomery. and Saffman \n\nIt is  the  behavioral  similarity  between  self-organizing network  models  and \nphysical systems that has lead us to investigate the possibility of implementing \na self-organizing network function by designing the dynamics for a set of opti(cid:173)\ncal  oscillators.  Modes  of sets  of oscillators  undergo  competition  (Anderson, \nBenkert)  much  like  that employed  in  competitive  learning network  models. \nUsing photorefractive elements, we have tailored the dynamics of the mode in(cid:173)\nteraction to perfonn a learning task.  A physical optical implementation of self(cid:173)\norganized  learning serves two  functions.  Unlike a  computer simulation,  the \nphysical  system must obey certain physical laws just like a biological  system \ndoes.  We  have in mind  the consequences  of energy conservation, finite  gain \nand the effects of noise.  Therefore, we  might expect to learn something about \ngeneral principles applicable to biological systems from  our physical versions. \nSecond, there are some applications where an optical system serves as an ideal \n\"front end\" to  signal processing. \n\nHere  we  take a  commonly  used  supervised  approach  for  extracting features \nfrom  a  stream of images and demonstrate how this task can be done in a self(cid:173)\norganizing manner.  The conventional approach employs a holographic corre(cid:173)\nlator (Vander Lugt).  In this technique, various patterns are chosen for recog(cid:173)\nnition  by  the  optical  system  and  then recorded  in  holographic  media  using \nangle-encoded reference beams.  When a specific pattern is presented to the ho(cid:173)\nlographic correlator, the output is detennined by the correlation between the \npresented pattern and the patterns that have been recorded as holograms dur(cid:173)\ning the 'learning phase'.  The angles and intensities of the reconstructed refer(cid:173)\nence  beams  identify  the  features  present  in  the  pattern.  Because  the \nprocessing time-scale in holographic systems is detennined by the time neces(cid:173)\nsary for  light to  scatter off of the holographic  grating, the optical  correlation \ntakes place virtually instantaneously.  It is  the  speed of this correlation that \nmakes the holographic approach so interesting. \n\nWhile its speed is an asset, the holographic correlator approach to feature ex(cid:173)\ntraction from images is a supervised approach to the problem:  an external su(cid:173)\npervisor  must choose  the  relevant  image  features  to  store  in  the  correlator \nholograms.  Moreover the supervisor must provide an angle-encoded reference \nbeam for each stored feature.  For many applications, it is desirable to have an \nadaptive system that has the innate capacity to discover,  in  an unsupervised \nfashion, the underlying structure within the input data. \n\nA photorefractive ring resonator circuit that learns to extract spatially orthog(cid:173)\nonal features from images is illustrated schematically in figure  1.  The resona(cid:173)\ntor rings in figure  1 are constructed physically from  optical fibers cables.  The \nresonator is self-starting and is pumped by images containing the input data \n(White).  The resonator learns to associate each feature in the input data set \nwith one and only one of the available resonator rings.  In other words,  when \nthe proper feature is present in the input data, the resonator ring with which \nit has become associated will  light up.  When  this feature is absent from  the \ninput data, the corresponding resonator ring will be dark. \n\nThe self-organizing capabilities of this system arise from the nonlinear dynam-\n\n\fOptical Implementation of a Self-Organizing Feature Extractor \n\n823 \n\nt 1111\"\" \n\nFigure1: Schematic diagram of the self-organizing photorefractive ring \nresonator. The two signal frequencies,  (01  amd (02,  are identical when \nthe circuit is used as a feature extractor and are separated by 280 MHz \nwhen the system is used as a frequency demultiplexer. \n\nics of competition between resonator modes for optical energy within the com(cid:173)\nmon photorefractive pump crystal (Benkert).  We have used this system to ac(cid:173)\ncomplish two optical signal processing tasks.  In the first case,  the resonator \ncan learn to distinguish between  two  spatially orthogonal input images that \nare impressed on the common pump beam in a piece-wise constant fashion.  In \nthe  second  case,  frequency  demultiplexing of a  composite  input image  con(cid:173)\nstructed from  two  spatially orthogonal image components of different optical \nfrequencies can be accomplished (Saffman,  1991b).  In both cases,  the optical \nsystem has no a priori knowledge of the input data and self-discovers the im(cid:173)\nportant structural elements. \n\n2 A Self\u00b7Organizing Photorefractive Ring Resonator \nThe experimental design that realizes an optical self-organizing feature extrac(cid:173)\ntor is shown in figure  l.  The optical system consists of a two ring, multimode, \nunidirectional photorefractive ring resonator in which  the rings are spatially \ndistinct.  The resonator rings are defined by loops of 100 Jl  core multimode op(cid:173)\ntical fiber.  The gain for both modes is provided by a  common  BaTi03  crystal \nthat is pumped by optical images presented as speckle patterns from  a  single \n100 Jl multimode optical fiber.  The light source is a single frequency argon-ion \nlaser operating at 514.5 nm.  The second BaTi03 crystal provides reflexive cou-\npling within the resonator, which ensures that each resonator ring becomes as(cid:173)\nsociated with only one input feature. \n\nThe  input images are generated by splitting the source beam  and passing it \nthrough  two  acousto-optic  modulator cells.  The optical  signals generated by \nthe acousto-optic modulators are then focused into a single 1.5 meter long step(cid:173)\nindex, 100 Jl core, multimode optical fiber.  The difference in the angle of inci(cid:173)\ndence for the two signal beams at the fiber end face is sufficient to ensure that \nthe corresponding speckle  pattern  images are  spatially  orthogonal (Safi'man, \n\n\f824 \n\nAnderson,  Benkert, Hebler, Jang,  Montgomery, and Saffman \n\n1991a).  The acousto-optic cells are used in a conventional fashion  to  shift the \noptical frequency of the carrier signal, and are also used as shutters to impress \ntime modulated information on  the input signals.  When the resonator is oper(cid:173)\nating as a feature extractor, both input signals are carried on the same optical \nfrequency, but are presented to  the resonator sequentially.  The presentation \ncycle  time  of 500  Hz  was  chosen  to  be  much  smaller than the characteristic \ntime constan t of the BaTi03 pump crystal.  When operating as a frequency de-\nmultiplexer, the acousto-optic modulators shift the optical carrier frequencies \nof the input signals such that they are separated by 280 MHz.  The two  input \ncarrier signals are time modulated and mixed into the optical fiber to  form  a \ncomposite image composed of two spatially orthogonal speckle patterns having \ndifferent optical frequencies.  This composite image is used as the pump beam \nfor  the resonator. \n\n3 Unsupervised Competitive Learning \nCorrelations between the optical electric fields in images establish the criterion \nfor  a  measure of similarity between different image features.  The best mea(cid:173)\nsure of these correlations is the inner product between the complex-valued spa(cid:173)\ntial electric field distribution across the input images, \n\nWhen S 12 = 0 the images are uncorrelated and we  define such images as spa(cid:173)\ntially  orthogonal.  When  the resonator begins to  oscillate,  neither resonator \nring has any preference for a particular input feature or frequency.  The system \nmodes have no internal bias (i.e.,  no a priori knowledge) for the input data.  As \nthe gain for photorefractive two-beam  coupling in  the common  BaTi03 pump \ncrystal saturates, the two resonator rings begin to compete with each other for \nthe available pump energy.  This competitive coupling leads to 'winner-takes(cid:173)\nall' dynamics in the resonator in which each resonator ring becomes associated \nwith one  or the other spatially orthogonal input images.  In other words,  the \nrings become labels for each spatially orthogonal feature present in the input \nimage set. \n\nPhenomenologically,  the dynamics of this mode  competition  can be  described \nby Lotka-Volterra equations (Benkert, Lotka, Volterra), \n\ndli,p  -\n-\ndt \n\nI.p ( \n\n- /.  a \u00b7  -\n\nI,p  P\"p  I,p \n\n.  I \n\nL \n9\u00b7\nl,p;,,1  J,J) \n\n\u00b7 I \n\n. \nJ,J \n\n-\n\nWhere  Ii,p  is  the intensity of the  oscillating energy  in  ring  i  due  to  energy \ntransferred from the input feature p, ai,p is the gain for two-beam coupling be(cid:173)\ntween ring i and feature p, ~i,p is the self-saturation coefficient, and 9i,pj,l are \nthe cross-saturation coefficients.  The self-organizing dynamics are determined \nby the values of the cross coupling coefficients.  Thus the competitive learning \nalgorithm that drives the self-organization in this optical system is embedded \n\n\fOptical Implementation of a Self-Organizing Feature Extractor \n\n825 \n\nresonalor \n\nbeam \n\nFigure 2: Reflexive gain interaction. A fraction, 0, of the incident inten(cid:173)\nsity is removed from  the resonator beam,  and then coupled back into \nitself b~ photorefractive  two  beam  coupling.  This  ensures  'Winner(cid:173)\ntakes-all' competitive dynamics between the resonator rings. \n\nwithin the nonlinear dynamics of mode competition in the pump crystal. \n\nOnce the system has learned, the spatially orthogonal features in the training \nset are represented  as holograms  in  the  BaTi03  pump crystal.  These holo(cid:173)\ngrams act as linear projection operators, and any new image constructed from \nfeatures in the training set will be projected in a linear fashion onto the learned \nfeature basis set.  The relative intensity of light oscillating in each ring corre(cid:173)\nsponds to the fraction of each learned feature in the new image.  Thus, the res(cid:173)\nonator functions as a feature extractor (Kohonen). \n\n4 Reflexive Gain \nIf each resonator ring was single mode, then competitive dynamics in the com(cid:173)\nmon pump crystal would be sufficient for feature extraction.  However, a mul(cid:173)\ntimode ring system allows stability for certain pathological feature extracting \nstates.  The multimode  character of each resonator ring can  permit simulta(cid:173)\nneous oscillation of two spatially orthogonal modes within a single ring.  Osten(cid:173)\nsibly, the system is performing feature extraction, but this form of output is not \nuseful for further processing.  These pathological states are excluded by intro(cid:173)\nducing reflexive gain into the cavity. \n\nAny system that twists back upon itself and closes a loop  is referred to  as re(cid:173)\nflexive (Hofstadter, pg. 3).  A reflexive gain interaction is achieved by removing \na portion of the oscillating energy from each ring and then coupling it back into \nthe same ring by photorefractive two-beam coupling, as illustrated in figure 2. \nThe  standard  equations  for  photorefractive  two-beam  coupling  (Kukhtarev, \nHall) can be used to derive an expression for the steady-state transmission, T, \nthrough the reflexive gain element in terms of the number of spatially orthog(cid:173)\nonal modes, N,  that are oscillating simultaneously within a single ring, \n\nHere,  exp(Go) is  the  small  signal  gain and 0 is  the fraction  of light removed \n\n\f826 \n\nAnderson,  Benkert, Hebler, lang, Montgomery, and Saffman \n\nRing \n\n1 \n\n.... \n. !  I \n~ ... _ill \n\nRing \n\n2 \n\nI, \n\n/ \n\n.L \nI \nv \nlr \n\n/ \n\nI, \nr \n! \n1/ \n\n- --.. ..,.,... \n\nI, \n\n-~/ \n\n./ \n\nI, \n\nFigure 3: Time evolution of the in(cid:173)\ntensities within each resonator ring \ndue to 0>1  (11) and 0>2(12).  After \nabout 30 seconds, tne system has \nlearned to  demultiplex the two input \nfrequencies. Ring 1 has become as(cid:173)\nsociated with 0>1  and Ring 2 has \nbecome associated with 0l2.  The con(cid:173)\ntrast ratio between 11 anal2 in each \nring is about 40: 1. \n\nfrom the resonator.  The transmission decreases for N > 1 causing larger cavity \nlosses for  the  case  of simultaneous  oscillation  of spatially orthogonal  modes \nwithin a single ring.  Therefore, the system favors 'winner-takes-all' dynamics \nover other pathological feature extracting states. \n\n5 Experimental Results \nThe self-organizing dynamics within the optical circuit require several seconds \nto reach steady state.  In the case of frequency de multiplexing, the dynamical \nevolution of the system was observed by detecting the envelopes of the carrier \nmodulation, as shown in figure 3.  In the case of the feature extractor, transient \nsystem  dynamics  were  observed  by  synchronizing  the  observation  with  the \nmodulation of one feature or the other, as shown in figure 4.  The frequency de(cid:173)\nmultiplexing (figure 3)  and feature extracting (figure 4) states develop a high \ncontrast ratio and are stable for as long as the pump beam is present.  Measure(cid:173)\nments with a  spectrum analyzer show an output contrast ratio of better than \n40:1 in the frequency demultiplexing case. \nThe circuit described here extracts spatially orthogonal features while contin-\n~ I  +----+-~:___-+--_+_-_+ \n(n ..... ) \n\nRing 1  1 \n\nFigure 4: Time evolution of the \nintensities in each resonator ring \ndue to the two input pictures. \nThe system requires about 30 sec(cid:173)\nonds to learn to extract features \nfrom the input images. Picture 1 \nis associated with Ring 1 and pic(cid:173)\nture 2 is associated with Ring2. \n\n:  \u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7l\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7l\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7r\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7t\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7 \n\u2022\u2022\u00b7\u2022\u2022\u2022\u2022\u2022\u00b7\u00b7 .. r\u00b7\u00b7\u00b7\u00b7 \u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7;\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7 .. \u00b7\u00b7\u00b7r\u00b7\u00b7\u00b7\u00b7\u00b7 .. \u00b7 .. \u00b7r .... \u00b7 .. \u00b7\u00b7 .. \n\n. \n\n0.. \n\n: \n\n: \n\n: \n\n02 \n\n...........  r ...... \u00b7\u00b7\u00b7 .... ;\u00b7 .......... \u00b7.;-............ .:. .. \u00b7 \n\n0+-\"\"\"\"'--;----;'\"--;...---;--_+ \n50 \n\no \n\n10 \n\n:P .... 2 \n\n. \nRmg2  ~ \n\n~~ \n\n(n ..... ) \n\n0.8 \n\n10 \n\n20 \n\n10 \n\n'0 \n\n50 \n\n\fOptical Implementation of a Self-Organizing Feature Extractor \n\n827 \n\nuously  adapting to  slow  variation s in  the  spatial  mode  superposition  due  to \ndrifts in the carrier frequency or perturbations to the fibers.  Thus, the system \nis adaptive as well as unsupervised. \n\n6 Summary \nAn  optical implementation of a  self-organizing feature extractor that is adap(cid:173)\ntive has been demonstrated.  The circuit exhibits the desirable dynamical prop(cid:173)\nerty  that  is  often  referred  to  in  the  parlance  of  the  neural  networks  as \n'unsupervised learning'.  The essential properties of this system arise from the \nnonlinear dynamics of mode competition within the optical ring resonator.  The \nlearning algorithm is embedded in these dynamics and they contribute to its \ncapacity to adapt to slow changes in the input signal.  The circuit learns to as(cid:173)\nsociate different spatially orthogonal images with different rings in an optical \nresonator.  The learned feature set can represent orthogonal basis vectors in an \nimage or different frequencies in a multiplexed optical signal.  Because a wide \nvariety of information can be encoded onto the input images presented to the \nfeature extractor described here, it has the potential to find general application \nfor  tasks where  the  speed and adaptability of self-organizing and  all-optical \nprocessing is desirable. \n\nAcknowledgements \nWe  are grateful for the support of both the Office of Naval Research, contract \n#N00014-91.J-1212  and  the Air  Force  Office  of Scientific  Research,  contract \n#AFOSR-90-0198.  Mark Saffman would like to acknowledge support provided \nby a U.S. Air Force Office of Scientific Research laboratory graduate fellowship. \n\nReferences \nD.Z. Anderson and R.  Saxena, Theory of Multimode Operation of a  Unidirec(cid:173)\ntional Ring Oscillator having Photorefractive Gain: Weak Field Limit, J. Opt. \nSoc. Am.  B, 4, 164 (1987). \n\nC.  Benkert and D.Z.  Anderson, Controlled competitive dynamics in a photore(cid:173)\nfractive ring oscillator: 'Winner-takes-all\" and the \"voting-paradox\" dynamics, \nPhys. Rev. A,  44,4633 (1991). \n\nE.  Domany, J.L. van Hemmen and K  Schulten, eds., Models of Neural Net(cid:173)\nworks; Springer-Verlag (1991). \n\nT.J. Hall, R. Jaura, L.M.  Connors and P.D. Foote, The Photorefractive Effect(cid:173)\nA Review;  Prog. Quant. Electr., 10, 77 (1985). \n\nJ. Hertz, A.  Krogh and R.G. Palmer, Introduction to the Theory of Neural Com(cid:173)\nputation;  Addison-Wesley (1991). \n\nD.  R.  Hofstadter, Metamagical Themas: Questing for the Essence of Mind and \nPattern; Bantam Books (1985). \n\n\f828 \n\nAnderson, Benkert, Hebler, lang, Montgomery, and Saffman \n\nT.  Kohonen, Self-Organization and Associative Memory, 3rd Edition; Springer(cid:173)\nVerlag (1989). \n\nN.  V.  Kukhtarev, V.B.  Markov, S.G. Odulov, M.S. Soskin and V.L. Vinetskii, \nHolographic Storage in Electrooptic Crystals. 1.  Steady State; Ferroelectrics, \n22, 949 (1979). \n\nN.  V.  Kukhtarev, V.B.  Markov, S.G. Odulov, M.S. Soskin and V.L. Vinetskii, \nHolographic Storage in Electrooptic Crystals. II. Beam Coupling - Light Ampli(cid:173)\nfication;  Ferroelectrics, 22, 961 (1979). \n\nA.J.  Lotka, Elements of Physical Biology; Baltimore (1925). \n\nM.  Saffman and D.Z. Anderson, Mode multiplexing and holographic demulti(cid:173)\nplexing communications channels on a multimode fiber,  Opt. Lett., 16,300 \n(1991a). \n\nM. Safi'man, C. Benkert and D.Z. Anderson, Self-organizing photorefractive fre(cid:173)\nquency demultiplexer, Opt. Lett., 16,  1993 (1991b). \n\nA  Vander Lugt, Signal Detection by Complex Spatial Filtering;  IEEE Trans. \nInfonn. Theor., IT-10,  139 (1964). \n\nV.  Volterra, Lecons sur la Theorie Mathematiques de la Lutte pour la Vie; \nGauthier-Villars (1931). \n\nJ.O. White, M.  Cronin-Golomb, B.  Fischer, and A  Yariv, Coherent Oscillation \nby Self-Induced Gratings in the Photorefractive Crystal BaTiO.j; Appl. Phys. \nLett., 40, 450 (1982). \n\n\fPART XII \n\nLEARNING AND \nGENERALIZATION \n\n\f\f", "award": [], "sourceid": 501, "authors": [{"given_name": "Dana", "family_name": "Anderson", "institution": null}, {"given_name": "Claus", "family_name": "Benkert", "institution": null}, {"given_name": "Verena", "family_name": "Hebler", "institution": null}, {"given_name": "Ju-Seog", "family_name": "Jang", "institution": null}, {"given_name": "Don", "family_name": "Montgomery", "institution": null}, {"given_name": "Mark", "family_name": "Saffman", "institution": null}]}