{"title": "A Hierarchical Model of Visual Rivalry", "book": "Advances in Neural Information Processing Systems", "page_first": 48, "page_last": 54, "abstract": null, "full_text": "An Hierarchical Model of Visual Rivalry \n\nPeter Dayan \n\nDepartment of Brain and Cognitive Sciences \nE25-21O Massachusetts Institute of Technology \n\nCambridge, MA 02139 \ndayan@psyche.mit.edu1 \n\nAbstract \n\nBinocular rivalry is  the alternating percept that can result when \nthe two eyes see different scenes.  Recent psychophysical evidence \nsupports an account for one component of binocular rivalry similar \nto that for other bistable percepts.  We  test the hypothesisl9, 16, 18 \nthat  alternation  can be  generated by competition  between  top(cid:173)\ndown cortical explanations for  the inputs,  rather than by direct \ncompetition between the  inputs.  Recent neurophysiological ev(cid:173)\nidence  shows  that some binocular neurons  are  modulated  with \nthe changing percept; others are not, even if they are selective be(cid:173)\ntween the stimuli presented to the eyes.  We  extend our model to \na hierarchy to address these effects. \n\n1  Introduction \n\nAlthough binocular  rivalry  leads  to  distinct  perceptual  distress,  it  is  revealing \nabout the  mechanisms  of visual information  processing.  The  first  accounts  for \nrivalry argued on the basis of phenomena such as increases in thresholds for  test \nstimuli presented in  the  suppressed eye24 , 8, 3  that there was a  early competitive \nprocess,  the  outcome  of  which  meant that  the  system  would just ignore  input \nfrom  one  eye  in favour  of  the  other.  Various  experiments  have  suggested  that \nsimple  input competition cannot be the  whole  story.  For instance,  in  a  case  in \nwhich  rivalry  is  between a vertical  grating in the  left  eye  and  a  horizontal one \nin the right, and in which a vertical grating is presented prior to rivalry to cause \nadaptation,  the  relative suppression of vertical during rivalry is  independent of \n\n1 I am very grateful to Bart Anderson, Adam Elga, Geoff Goodhill, Geoff Hinton, David \nLeopold,  Earl  Miller,  Read  Montague,  Bruno  Olshausen,  Pawan Sinha,  Rich  Zemel,  and \nparticularly  Zhaoping Li and  Tommi  Jaakkola  for  their  comments  on  earlier  drafts  and \ndiscussions. This work was supported by the NIH. \n\n\fA Hierarchical Model of Visual Rivalry \n\n49 \n\nthe eye of origin of the adapting grating.4  Even more compelling, if the rivalrous \nstimuli in the two eyes are switched rapidly,  the percept switches only slowly -\ncompetition is more between coherent percepts than merely inputs.  Rivalry is an \nattractive paradigm for studying models of cortex like the Helmholtz machine12, 7 \nthat construct coherent percepts, and in particular for studying hierarchical models, \nbecause  of electrophysiological  data on the behaviour during  rivalry  of cells  at \ndifferent levels of the visual processing hierarchy.16 \n\nLeopold &  Logothetis16 trained monkeys to report their percepts during rivalrous \nand non-rivalrous stimuli whilst recording from neurons VI/2 and V4.  Important \nfindings are that striate monocular neurons are unaffected by rivalry; some striate \nbinocular neurons that are selective between the stimuli modulate their activities \nduring rivalry;  others  do not;  some  fire  more  when their  preferred  stimuli  are \nsuppressed; others still are only selective during rivalry.  In this paper we consider \none form of analysis-by-synthesis model of cortical processing7  and show how it \ncan exhibit rivalry between explanations in the case that the eyes receive different \ninput.  This model can provide an account for many of the behaviours described \nabove. \n\n2  The Model \n\nFigure  Ia shows the full generative model.  Units in layers y  (modeling VI) and \nx  and w  (modeling early and late extra-striate areas) are all binocular and jointly \nexplain successively more complex features in the input z according to a top-down \ngenerative  model.  Apart from  the  half bars  in  y,  the  model  is  similar  to  that \nlearned by the Helmholtz machine12, 7 for  which increasing complexity in higher \nlayers rather than the increasing input scale is key.  In this case, for instance,  w2 \nspecifies  the  occurrence  of  vertical bars  anywhere in  the 8  x  8  input grids;  X16 \nspecifies the rightmost vertical bar; and Y31  and Y32  the top and bottom half of this \nvertical bar.  These specifications are provided by a top-down generative model in \nwhich the activations of units are specified by probabilities such as P[Yi  = Ilx]  = \na (by  + Lk xkJ!~) where the sum  k  is  over all the units  in the  x  layer,  and 0'0 \nis  a  robust normal distribution function.  We  model the  percept in  terms  of the \nactivation in the w layer. \n\nWe  model differing input contrasts by representing the input to  Zi  by di ,  where \nP[Zi =  1]  =  O'(di )  and all the Zi  are independent.  Recognition is formally the sta(cid:173)\ntistical inverse to generation, and should produce distribution P[w, x, yld] over all \nthe choices of the hidden activations.  We  use  a mean field  inversion method,13 \nusing a factorised  approximation Q[w,x, y; Il,~,~] =  Q[w; Il]Q][x; ~]Q[y; ~], with \nQ[w; Il]  =  TIi O'(lli)Wi  (1  - O'(lli))l-Wi, etc, and fitting the parameters Il,~, ~ to min(cid:173)\nimise the approximation cost: \n\n[ \n\n:F Il,~, ~ = L.J \n\n]  \"'P[  d]  '\" Q[ \n\nz; \n\nz \n\nL.J  w, x, y; Il, ,-, 'f/  og  P[w  x \nwxy \n,  , \n\n, \n\n,y \n\nC  .1']1 \n\nQ[w,x,y;Il,~,~] \n. \n\nIz] \n\nWe report the mean activities of the units in the graphs and use a modified gradient \ndescent method  to  find  appropriate parameters.  Figure  Ib shows  the  resulting \nactivities of units in response to binocular horizontal (i) and vertical (ii) bars, and \nalso the two equally likely explanations for rivalrous input (iii and iv).  For rivalry, \n\n\f50 \n\nLAYER \nwi \n\n(a) \n\nP.  Dayan \n\n(b) \n\nlib \n\nb \u2022 \u2022  _  \u2022 \u2022 \u2022 \u2022  \n\n\u2022\u2022\u2022\u2022\u2022\u2022\u2022 \nYII \u2022\u2022\u2022\u2022\u2022\u2022 \n\u2022\u2022\u2022\u2022\u2022\u2022 \n\n........ \n\n~ \n\ny \n\n% \n\nIa:::::QJ \n\nwi \n\"II~I \n\nyl-\n\u00b71_ \n\nL  R \n\nwl~ \n\n,,11l1lil1li1111 II \nylllllill \n\u00b7111111111 \n\nL \n\nR \n\n(i) \n\n(ii) \n\n!I:::::iJI \n\nwi \n\"II~I \n\nyllil \n\u00b71_ \n\nL \n\nR \n\n(iii) \n\nwl~ \n\n\"llmBEl \nyl-\n\u00b71_ \n\nL \n\nR \n(iv) \n\nL \n\nR \n\nFigure  1:  a)  Hierarchical generative model for  8  x 8 bar patterns across  the two eyes.  Units  are \ndepicted by their net projective (generative) fields, and characteristic weights are shown.  Even though \nthe net projective field of Xl is the top horizontal bar in both eyes, note that it generates this by increasing \nthe probability that units YI  and Y9  in the y  layer will be active,  not by having direct connections to \nthe input z.  Unit  WI  connects to  Xl, X2, .\u2022\u2022 Xs  through Jwx = 0.8;  XI6  connects to  Y31 , Y32  through \nJ XY  = 1.0  and  Y32  connects  to  the bottom  right  half  vertical  bar  through  J yz  = 5.8.  Biases  are \nbw  =  -0.75,bx  = -1.5, by  = -2.7 and  bz  = -3.3.  b)  Recognition activity in  the network for  four \ndifferent input patterns.  The units are arranged in the same order as (a), and white and black squares \nimply activities for the units whose means are less than and greater than 0.5.  (i) and (ii) represent normal \nbinocular stimulation; (iii)  and (iv) show the two alternative stable states during rivalrous stimulation, \nwithout the fatigue process. \n\nthere is direct competition in the top left hand quadrant of z, which is reflected in the \ncompetition between YI, Y3  and Y17, Y21.  However, the input regions (top right of L \nand bottom left of R) for which there is no competition, require the constant activity \nof explanations Y9, Yu ,Y18  and Y22.  Under the generative model, the coactivation \nof YI  and Y9  without Xl  is quite unlikely  (P[XI  = OIYI  = 1, Y3  = 1]  =  0.1), which is \nwhy XI, X3  and also WI  become active with YI  and Y3. \n\nGiven just gradient descent for the rivalrous stimulus, the network would just find \none of the two equally good (or rather bad) solutions in figure 1b(iii,iv). Alternation \nensues when descent is augmented by a fatigue process: \n\n= \n\n'l/JI(t)  +<5(-\\7I/JIF[1L,~,'l/J] + a ({3'l/JI (t))  -'l/J~(t)) \n'l/J~ (t) + <5('l/JI (t)  - {3'l/J~ (t)), \n\nwhere {3  is a decay term.  In all the simulations, a = 0.5, {3 = 0.1 and <5 = 0.01. \n\nWe adopted various heuristics to simplify the process of using this rather cumber(cid:173)\nsome mean field model.  First, fatigue is only implemented for the units in the y \n\n\fA Hierarchical Model of Visual Rivalry \n\n51 \n\nlayer, and the 'I.jJ  follow the equivalent of the dynamical equations above. Although \nadaptation processes can clearly occur at many levels in the system, and indeed \nhave been used to try to diagnose the mechanisms of rivalry,15  their exact form is \nnot clear.  Bialek &  DeWeesel  argue that the rate of a switching process should be \nadaptive to the expected rate of change of the associated signal on the basis of prior \nobservations. This is clearly faster nearer to the input. \n\nThe  second heuristic  is  that rather  than perform gradient  descent  for  the  non(cid:173)\nfatiguing units, the optimal values of f.1.  and ~ are calculated on each iteration by \nsolving numerically equations such as \n\nThe dearth of connections in the network of figure la allows f.1.  and ~ to be calculated \nlocally at each unit in an efficient manner.  Whether this is reasonable depends on \nthe time constants of settling in the mean field model with respect to the dynamics \nof switching, and, more particularly on the way that this deterministic model is \nmade appropriately stochastic. \n\nFigure 2a shows the resulting activities during rivalry of units at various levels of \nthe hierarchy including the fatigue process.  Broadly, the competing explanations \nin figure Ib(iii;iv), ie horizontal and vertical percepts, alternate, and units without \ncompeting inputs, such as Y9, are much less modulated than the others, such as Yl. \nThe activity of Y9  is slightly elevated when horizontal bars are dominant, based on \ntop-down connections. The activities of the units higher up, such as Xl  and WI, do \nnot decrease to 0 during the suppression period for horizontal bars, leaving weak \nactivity during suppression.  Many of the modulating cells in monkeys were not \ncompletely silent during their periods of less activity.16  Figure 2b shows that the \nhierarchical version of the model also behaves in accordance with experimental \nresults on the effects  of varying the input contrast,17, 10, 22, 16  which suggest that \nincreasing the contrast in both eyes decreases the period of the oscillation (ie  in(cid:173)\ncreases the frequency),  and increasing the contrast in just one eye decreases the \nsuppression period for that eye much more than it increases its dominance period. \n\n3  Discussion \n\nFollowing  Logothetis  and his  colleaguesl9 , 16, 18  (see  also  Grossbergll )  we  have \nsuggested  an  account  of  rivalry based  on competing top-down  hierarchical  ex(cid:173)\nplanations, and have shown how it models various experimental observations on \nrivalry.  Neurons explain inputs in virtue of being capable of generating their activ(cid:173)\nities through a top-down statistical generative model.  Competition arises between \nhigher-level explanations of overlapping active regions (ie those involving contrast \nchanges) of the input rather than between inputs themselves.  Note that alternating \nthe  input between the  two eyes  would  have  no effect  on this behaviour  of the \nmodel, since explanations  are competing rather than inputs.  Of course, the model \nis greatly simplified - for instance, it only has units that are not modulating with \nthe percept in the earliest binocular layer (layer y), whereas in the monkeys, more \nthan half the cells in V4 were unmodulated during rivalry.I6 \n\nThe model's accounts of the neurophysiological findings described in the introduc(cid:173)\ntion are: i) monocular cells will generally not be modulated if they are involved in \n\n\f52 \n\nP.  Dayan \n\nIterations  - - - - - -+  \n\na) \n\no ::r \n\n0.0 \n\nb) \n\nContrast Dependence \n\n600  r - - - - - - - - - - - - - - - - - - - - - ,  \n\n:: \n\n00 \n\nx,  ~ \n~::n, 1 I 11.1  11-\" I \n\n~ \n3 \n\n~ \nQ \n\nQ) \n\n-\n\n- -\n\nequal contrast \nhorizontal dominance (1=1 .25) \n\n-\n-\n---- vertical dominance (r) \n\n------------\n\n'.0~ \n\n0.5~-Y. \n\n500 \n\n0,0 O'-----,:-:':OOO:::-----::2000:::-:-----:::3000~---::.OOO \n\n1.0 \n\n12 \nTest vertical 'contrast' (r) \n\nlA \n\n1~ \n\nFigure 2:  a)  Mean activities of units at three levels of the hierarchy in response to rivalrous stimuli \nwith input strengths I = r = 1.75.  b)  Contrast dependence of the oscillation periods for equal input \nstrengths, and when I = 1.25 and r  is varied. \n\nexplaining local correlations in the input from a single eye.  This model does not \ndemonstrate this explicitly, but would if, for instance, each of the inputs Zi actually \nconsisted of two units, which are always on or off together.  In this case one could \nget a compact explanation of the joint activities with a set of monocular units which \nwould then not be modulated.  ii) Units such as Y9  in the hierarchical model are \nbinocular, are selective between the binocular version of the stimuli, and are barely \nmodulated  with  the  percept.  iii)  Units  such  as  YI, Xl  and  WI  are binocular,  are \nselective between the stimuli, and are significantly modulated with the percept. \n\nThe final neurophysiological finding is to do with cells that fire when their preferred \nstimuli are suppressed, or fire selectively between the stimuli only during rivalry. \nThere  are  no  units  in  this  model that are  selective between the  stimuli  and  are \npreferentially activated during suppression of their preferred stimuli.  However, in \na  model  with  more  complicated  stimulus  contingencies,  they  would  emerge  to \naccount for the parts of the stimulus in the suppressed eye that are not accounted \nfor by the explanation of the overlying parts of the dominant explanation, at least \nprovided that this residual between the true monocular stimulus and the current \nexplanation is sufficiently complex as to require explaining itself. \n\nWe would expect to find two sets of cells that are activated during the suppressed \nperiod by this residual, some of which will form part of the representation of the \nstimulus when presented binocularly and some of which will not.  Those that do \nnot  (class  A)  will  only  even  appear to  be  selective between the  stimuli  during \nrivalry, and will represent parts of the residual that are themselves explained by \nmore overarching explanations for parts of the complete (binocularly presented) \nstimulus.  This suggests the experimental test of presenting binocularly a putative \nform of the residual (eg dotted lines for competing horizontal and vertical gratings). \nWe predict that these cells should be activated. \nIf there are cells that do participate in the binocular representation, then they will \nbe selective, but will preferentially fire during suppression (class B). Certainly, the \n\n\fA Hierarchical Model of Visual Rivalry \n\n53 \n\nresidual will have a  high  correlation with the  full  suppressed pattern,  and so a \ncell  that is  selective  for  part of  the  residual could  have  appropriate  properties. \nHowever, why should such a cell not fire when the full, but currently suppressed, \npattern is dominant?  In monkeys,16  there are fewer class B than class A  cells  (0 \nversus 3 of 33  cells in Vl/2; 6 versus 8 of 68  cells in V4).  Under the model, we \naccount for  these cells based on a  competition between units that represent the \nresidual and those  that represent overlapping parts of the  complete pattern.  In \nbinocular viewing,  explanations  are  generally  stronger  than during  rivalry.  So \neven if both such units participate in representing a binocular stimulus, the cells \nrepresenting the residual might not reach threshold during the dominance period. \nHowever, during suppression, they no longer suffer from competition, and so will \nbe activated.  The model's explanation for class B cells seems far less natural than \nthat for  class  A  cells.  One  experimental test  would be  to present the  preferred \npattern binocularly, reduce the contrast, and see if these cells are suppressed more \nstrongly. \n\nThe overall model mechanistically has much in common with models which place \nthe competition in rivalry at the level of binocular oriented cells rather than between \nmonocular cells.11,2  Indeed, the model is based on an explanation-driven account \nfor  normal binocular  processing,  so  this  is  to  be  expected.  The  advantage  of \ncouching rivalry in terms  of explanations is  that this  provides a  natural way of \naccounting  for  top-down  influences.  In  fact,  one  can hope  to  study  top-down \ncontrol through studying its effects on the behaviour of cells during rivalry. \n\nThe  model suffers from various lacunce.  Foremost,  it is  necessary to model  the \nstochasticity  of  switching between  explanations.9,17  The  distributions  of  domi(cid:173)\nnance times for both humans and monkeys is well characterised by a r distribution \n(Lehky14  argues  that this  is  descriptive  rather than normative),  with  strong in(cid:173)\ndependence between successive dominance periods.  Our mean field recognition \nprocess is deterministic.  The stochastic analogue would be some form of Markov \nchain Monte-Carlo method such as Gibbs sampling.  However,  it is not obvious \nhow to incorporate the equivalent of fatigue in a computationally reasonable way. \nIn any case, the nature of neuronal randomness is subject to significant debate at \npresent.  Note  that the  recognition  model of a  stochastic  Helmholtz machine7,6 \nwould be unsuitable, since it is purely feedforward and does not integrate bottom(cid:173)\nup and top-down information. \n\nWe  have adopted a  very simple  mean field  approach  to  recognition,  giving  up \nneurobiological plausibility for convenience.  The determinism of the mean field \nmodel  in  any  case  rules  it  out  as  a  complete  explanation,  but  it  does  at  least \nshow clearly the nature of competition between explanations.  The architecture of \nthe model is  also incomplete.  The cortex is replete  with what we  would model \nas lateral connections between units within a  single layer.  We  have constructed \ngenerative  models  in which  there  are no  such direct connections,  because  they \nsignificantly complicate the mean field recognition method.  It could be that these \nconnections are important for  the recognition process,6  but modeling their effect \nwould require representing them explicitly. This would also allow modeling of the \napparent diffusive process by which patches of dominance spread and alter.  In a \ncomplete model, it would also be necessary to account for  competition between \neyes in addition to competition between explanations. 24,8,3 \n\n\f54 \n\nP.  Dayan \n\nAnother gap is some form of contrast gain control.s The model is  quite sensitive \nto input contrast.  This is  obviously important for  the effects shown in figures 2, \nhowever the range of contrasts over which it works should be larger.  It would be \nparticularly revealing to explore the effects of changing the contrast in some parts \nof images and examine the consequent effects on the spreading of dominance. \n\nReferences \n\n[1]  Bialek, W & DeWeese, M (1995).  Random switching and optimal processing in the perception of \n\nambiguous Signals. Physical Review Letters, 74, 3077-3080. \n\n[2]  Blake, R (1989).  A neural theory of binocular rivalry. Psyclwlogical Review, 96,  145-167. \n[3]  Blake,  R  &  Fox,  R  (1974).  Binocular rivalry  suppression:  Insensitive to spatial frequency  and \n\norientation change. Vision  Research, 14, 687-692. \n\n[4]  Blake,  R,  Westendorf,  DH &  Overton,  R (1980).  What is  suppressed during binocular  rivalry? \n\nPerception, 9, 223-231. \n\n[5]  Carandini, M & Heeger, DJ (1994). Summation and division by neurons in primate visual cortex. \n\nScience, 264, 1333-1336. \n\n[6]  Dayan, P & Hinton, GE (1996).  Varieties of Helmholtz machine. Neural Networks, 9, 1385-1403. \n[7]  Dayan, P, Hinton, GE, Neal, RM & Zemel, RS (1995). The Helmholtz machine. Neural Computation, \n\n7,889-904. \n\n[8]  Fox,  R  &  Check,  R  (1972).  Independence between binocular rivalry suppression duration and \n\nmagnitude of suppression. Journal of Experimental Psyclwlogy, 93, 283-289. \n\n[9]  Fox, R & Herrmann, J (1%7). Stochastic properties of binocular rivalry alternations. Perception and \n\nPsychophysics, 2, 432-436. \n\n[10]  Fox, R & Rasche, F (1969). Binocular rivalry and reciprocal inhibition. Perception and Psychophyics, \n\n5,215-217. \n\n[11]  Grossberg, S (1987).  Cortical dynamiCS of three-dimensional form, color and brightness percep(cid:173)\n\ntion:  2.  Binocular theory. Perception & Psychphysics, 41, 117-158. \n\n[12]  Hinton, GE,  Dayan, P,  Frey, BJ  &  Neal,  RM (1995).  The wake-sleep algorithm for  unsupervised \n\nneural networks. Science, 268,1158-1160. \n\n[13]  Jaakkola, T, Saul, LK & Jordan, MI (1996). Fast learning by bounding likelihoods in sigmoid type \n\nbelief networks. Advances in Neural Information  Processing Systems, 8, forthcoming. \n\n[14]  Lehky, SR (1988). An astable multivibrator model of binocular rivalry. Perception, 17, 215-228. \n[15]  Lehky, SR &  Blake, R (1991).  Organization of binocular pathways:  Modeling and data related to \n\nrivalry. Neural Computation, 3,44-53. \n\n[16]  Leopold, DA &  Logothetis, NK (1996).  Activity changes in early visual cortex reflect monkeys' \n\npercepts during binocular rivalry. Nature, 379, 549-554. \n\n[17]  Levelt, WJM (1968). On Binocular Rivalry.  The Hague, Paris:  Mouton. \n[18]  Logothetis, NK, Leopold, DA & Sheinberg, DL (1996).  What is rivalling during binocular rivalry. \n\nNature, 380, 621-624. \n\n[19]  Logothetis, NK &  Schall, JD (1989).  Neuronal correlates of subjective visual perception. Science\" \n\n245,761-763. \n\n[20]  Matsuoka, K (1984). The dynamic model of binocular rivalry. Biological Cybernetics, 49, 201-208. \n[21]  Mueller, 11 (1990).  A physiological model of binocular rivalry. Visual Neuroscience, 4, 63-73. \n[22]  Mueller, 11 & Blake, R (1989). A fresh look at the temporal dynamiCS of binocular rivalry. Biological \n\nCybernetics, 61, 223-232. \n\n[23]  Pearl,  J  (1988).  Probabilistic  Reasoning  in  Intelligent  Systems:  Networks  of Plausible  Inference.  San \n\nMateo, CA: Morgan Kaufmann. \n\n[24]  Wales, R &  Fox,  R (1970). Increment detection thresholds during binocular rivalry suppression. \n\nPerception and PsychophysiCS, 8, 90-94. \n\n[25]  Wheatstone, C (1838).  Contributions to the theory of vision. I:  On some remarkable and hitherto \nunobserved phenomena of binocular vision. Philosophical Transactions of the Royal Society of London, \n128,371-394. \n\n[26]  Wolfe, JM (1986). Stereopsis and binocular rivalry. PsycholOgical Review, 93,269-282. \n\n\f", "award": [], "sourceid": 1277, "authors": [{"given_name": "Peter", "family_name": "Dayan", "institution": null}]}