{"title": "\u201cCongruent\u201d and \u201cOpposite\u201d Neurons: Sisters for Multisensory Integration and Segregation", "book": "Advances in Neural Information Processing Systems", "page_first": 3180, "page_last": 3188, "abstract": "Experiments reveal that in the dorsal medial superior temporal (MSTd) and the ventral intraparietal (VIP) areas, where visual and vestibular cues are integrated to infer heading direction, there are two types of neurons with roughly the same number. One is \u201ccongruent\u201d cells, whose preferred heading directions are similar in response to visual and vestibular cues; and the other is \u201copposite\u201d cells, whose preferred heading directions are nearly \u201copposite\u201d (with an offset of 180 degree) in response to visual vs. vestibular cues. Congruent neurons are known to be responsible for cue integration, but the computational role of opposite neurons remains largely unknown. Here, we propose that opposite neurons may serve to encode the disparity information between cues necessary for multisensory segregation. We build a computational model composed of two reciprocally coupled modules, MSTd and VIP, and each module consists of groups of congruent and opposite neurons. In the model, congruent neurons in two modules are reciprocally connected with each other in the congruent manner, whereas opposite neurons are reciprocally connected in the opposite manner. Mimicking the experimental protocol, our model reproduces the characteristics of congruent and opposite neurons, and demonstrates that in each module, the sisters of congruent and opposite neurons can jointly achieve optimal multisensory information integration and segregation. This study sheds light on our understanding of how the brain implements optimal multisensory integration and segregation concurrently in a distributed manner.", "full_text": "\u201cCongruent\u201d and \u201cOpposite\u201d Neurons: Sisters for\n\nMultisensory Integration and Segregation\n\nWen-Hao Zhang1,2 \u2217 , He Wang1, K. Y. Michael Wong1, Si Wu2\n\nwenhaoz@ust.hk, hwangaa@connect.ust.hk, phkywong@ust.hk, wusi@bnu.edu.cn\n\n1Department of Physics, Hong Kong University of Science and Technology, Hong Kong.\n\n2State Key Lab of Cognitive Neuroscience and Learning, and\n\nIDG/McGovern Institute for Brain Research, Beijing Normal University, China.\n\nAbstract\n\nExperiments reveal that in the dorsal medial superior temporal (MSTd) and the\nventral intraparietal (VIP) areas, where visual and vestibular cues are integrated\nto infer heading direction, there are two types of neurons with roughly the same\nnumber. One is \u201ccongruent\" cells, whose preferred heading directions are similar in\nresponse to visual and vestibular cues; and the other is \u201copposite\" cells, whose pre-\nferred heading directions are nearly \u201copposite\" (with an offset of 180\u25e6) in response\nto visual vs. vestibular cues. Congruent neurons are known to be responsible for\ncue integration, but the computational role of opposite neurons remains largely\nunknown. Here, we propose that opposite neurons may serve to encode the dispar-\nity information between cues necessary for multisensory segregation. We build\na computational model composed of two reciprocally coupled modules, MSTd\nand VIP, and each module consists of groups of congruent and opposite neurons.\nIn the model, congruent neurons in two modules are reciprocally connected with\neach other in the congruent manner, whereas opposite neurons are reciprocally\nconnected in the opposite manner. Mimicking the experimental protocol, our model\nreproduces the characteristics of congruent and opposite neurons, and demonstrates\nthat in each module, the sisters of congruent and opposite neurons can jointly\nachieve optimal multisensory information integration and segregation. This study\nsheds light on our understanding of how the brain implements optimal multisensory\nintegration and segregation concurrently in a distributed manner.\n\n1\n\nIntroduction\n\nOur brain perceives the external world with multiple sensory modalities, including vision, audition,\nolfaction, tactile, vestibular perception and so on. These sensory systems extract information\nabout the environment via different physical means, and they generate complementary cues (neural\nrepresentations) about external objects to the multisensory areas. Over the past years, a large volume\nof experimental and theoretical studies have focused on investigating how the brain integrates multiple\nsensory cues originated from the same object in order to perceive the object reliably in an ambiguous\nenvironment, the so-called multisensory integration. They found that the brain can integrate multiple\ncues optimally in a manner close to Bayesian inference, e.g., integrating visual and vestibular cues to\ninfer heading direction [1] and so on [2\u20134]. Neural circuit models underlying optimal multisensory\nintegration have been proposed, including a centralized model in which a dedicated processor receives\nand integrates all sensory cues [5, 6], and a decentralized model in which multiple local processors\nexchange cue information via reciprocal connections, so that optimal cue integration is achieved at\neach local processor [7].\n\n\u2217Current address: Center for the Neural Basis of Cognition, Carnegie Mellon University.\n\n30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain.\n\n\fFigure 1: Congruent and opposite neurons in MSTd. Similar results were found in VIP [12]. (A-B)\nTuning curves of a congruent neuron (A) and an opposite neuron (B). The preferred visual and\nvestibular directions are similar in (A) but are nearly opposite by 180\u25e6 in (B). (C) The histogram of\nneurons according to their difference between preferred visual and vestibular directions. Congruent\nand opposite neurons are comparable in numbers. (A-B) adapted from [1], (C) from [13].\n\nHowever, multisensory integration is only half of the story of multisensory information processing,\nwhich works well when the sensory cues are originated from the same object. In cases where the\nsensory cues originate from different objects, the brain should segregate, rather than integrate, the\ncues. In a noisy environment, however, the brain is unable to differentiate the two situations at\n\ufb01rst sight. The brain faces a \u201cchicken vs. egg\" dilemma in multisensory integration: without \ufb01rst\nintegrating multiple cues to eliminate uncertainty, the brain is unable to estimate the objects reliably to\ndifferentiate whether the cues are from the same or different objects; but once the cues are integrated,\nthe disparity information between the cues is lost, and the brain can no longer discriminate objects\nclearly when the cues actually come from different objects. To solve this dilemma, here we argue that\nthe brain needs to carry out multisensory integration and segregation concurrently in the early stage\nof information processing, that is, a group of neurons integrates sensory cues while another group of\nneurons extracts the cue disparity information, and the interplay between two networks determines\nthe \ufb01nal action: integration vs. segregation. Concurrent processing has the advantage of achieving\nrapid object perception if the cues are indeed from the same object, and avoiding information loss if\nthe cues are from different objects. Psychophysical data tends to support this idea, which shows that\nthe brain can still sense the difference between cues in multisensory integration [8, 9].\nWhat are the neural substrates of the brain to implement concurrent multisensory integration and\nsegregation? In the experiments of integrating visual and vestibular cues to infer heading direction, it\nwas found that in the dorsal medial superior temporal area (MSTd) and the ventral intraparietal area\n(VIP) which primarily receive visual and vestibular cues respectively, there exist two types of neurons\ndisplaying different cue integrative behaviors [1, 10]. One of them is called \u201ccongruent\" cells, since\ntheir preferred heading directions are similar in response to either a visual or a vestibular cue (Fig. 1A);\nand the other type is called \u201copposite\" cells, since their preferred visual and vestibular directions are\nnearly \u201copposite\" (with an offset of 180\u25e6, half of the period of direction, Fig. 1B). Data analyses and\nmodelling studies revealed that congruent neurons are responsible for cue integration [1, 10, 6, 7].\nHowever, the computational role of opposite neurons remains largely unknown, despite the fact that\ncongruent and opposite neurons are comparably numerous in MSTd and VIP (Fig. 1C). Notably,\nthe responses of opposite neurons hardly vary when a single cue is replaced by two congruent cues\n(i.e., no cue integration behavior), whereas their responses increase signi\ufb01cantly when the disparity\nbetween visual and vestibular cues increases [11], indicating that opposite neurons may serve to\nextract the cue disparity information necessary for multisensory segregation. Motivated by the above\nexperimental \ufb01ndings, we explore how multisensory integration and segregation are concurrently\nimplemented in a neural system via sisters of congruent and opposite cells.\n\n2 Probabilistic Model of Multisensory Information Processing\n\nIn reality, because of noise, the brain estimates stimulus information relying on ambiguous cues in a\nprobabilistic manner. Thus, we formulate multisensory information processing in the framework of\nprobabilistic inference. The present study mainly focuses on information processing at MSTd and\nVIP, where visual and vestibular cues are integrated/segregated to infer heading direction. However,\nthe main results of this work are applicable to the processing of cues of other modalities.\n\n2\n\n\u00b040302010Firing rate (spikes s\u20131)081081\u2013\u00b1180\u00b0\u201390\u00b0900\u00b0\u2013909000VestibularVisualHeading direction (\u00b0)6020081081\u2013\u20139090080400Heading direction (\u00b0)BCANumber of Neurons0609012018001020304050| \u2206 Preferred direction (\u00b0)|Visual vs. vestibularCongruent neuronOpposite neuronCongruentOpposite\f2.1 The von Mises distribution for circular variables\nBecause heading direction is a circular variable whose values are in range (\u2212\u03c0, \u03c0], we adopt the\nvon Mises distribution [14] (Supplementary Information Sec. 1). Compared with the Gaussian\ndistribution, the von Mises distribution is more suitable and also more accurate to describe the\nprobabilistic inference of circular variables, and furthermore, it gives a clear geometrical interpretation\nof multisensory information processing (see below).\nSuppose there are two stimuli s1 and s2, each of which generates a sensory cue xm, for m = 1, 2\n(visual or vestibular), independently. We call xm the direct cue of sm, and xl (l (cid:54)= m) the indirect\ncue to sm. Denote as p(xm|sm) the likelihood function, whose form in von Mises distribution is\n\n1\n\n2\u03c0I0(\u03bam)\n\nexp [\u03bam cos(xm \u2212 sm)] \u2261 M(xm \u2212 sm, \u03bam),\n\n(1)\n\np(xm|sm) =\n\nwhere I0(\u03ba) = (2\u03c0)\u22121(cid:82) 2\u03c0\n\n0 e\u03ba cos(\u03b8)d\u03b8 is the modi\ufb01ed Bessel function of the \ufb01rst kind and order zero.\nsm is the mean of the von Mises distribution, i.e., the mean value of xm. \u03bam is a positive number\ncharacterizing the concentration of the distribution, which is analogous to the inverse of the variance\n(\u03c3\u22122) of Gaussian distribution. In the limit of large \u03bam, a von Mises distribution M[xm \u2212 sm, \u03bam]\napproaches to a Gaussian distribution N [xm \u2212 sm, \u03ba\u22121\nm ] (SI Sec. 1.2). For small \u03bam, the von Mises\ndistribution deviates from the Gaussian one (Fig.2A).\n\n2.2 Multisensory integration\n\nWe introduce \ufb01rst a probabilistic model of Bayes-optimal multisensory integration. Experimental data\nrevealed that our brain integrates sensory cues optimally in a manner close to Bayesian inference [2].\nAssuming that noises in different channels are independent, the posterior distribution of two stimuli\ncan be written according to Bayes\u2019 theorem as\n\n(2)\nwhere p(s1, s2) is the prior of the stimuli, which speci\ufb01es the concurrence probability of a stimulus\npair. As an example in the present study, we choose the prior to be\n\np(s1, s2|x1, x2) \u221d p(x1|s1)p(x2|s2)p(s1, s2),\n\np(s1, s2) =\n\nM(s1 \u2212 s2, \u03bas) =\n\n1\n2\u03c0\n\n1\n\n(2\u03c0)2I0(\u03bas)\n\nexp [\u03bas cos(s1 \u2212 s2)] .\n\n(3)\n\nThis form of prior favors the tendency for two stimuli to have similar values. Such a tendency has\nbeen modeled in multisensory integration [7, 15\u201317]. \u03bas determines the correlation between two\nstimuli, i.e., how informative one cue is about the other, and it regulates the extent to which two cues\nshould be integrated. The fully integrated case, in which the prior becomes a delta function in the\nlimit \u03bas \u2192 \u221e, has been modeled in e.g., [4, 5].\nSince the results for two stimuli are exchangeable, hereafter, we will only present the result for\ns1, unless stated speci\ufb01cally. Noting that p(sm) = p(xm) = 1/2\u03c0 are uniform distributions, the\nposterior distribution of s1 given two cues becomes\n\np(s1|x1, x2) \u221d p(x1|s1)\n\np(x2|s2)p(s2|s1)ds2 \u221d p(s1|x1)p(s1|x2).\n\n(4)\n\nThe indirect cue x2 is informative to s1 via the prior p(s1, s2). By using Eqs. (1,3) and under\nreasonable approximations (SI Sec. 1.4), we obtain\n\n(cid:90)\n\n(cid:90)\n\nwhere A(\u03ba12) = A(\u03ba2)A(\u03bas) with A(\u03ba) \u2261(cid:82) \u03c0\n\np(s1|x2) \u221d\n\n\u2212\u03c0 cos(\u03b8)e\u03ba cos \u03b8d\u03b8/(cid:82) \u03c0\n\np(x2|s2)p(s2|s1)ds2 (cid:39) M (s1 \u2212 x2, \u03ba12) ,\n\n\u2212\u03c0 e\u03ba cos \u03b8d\u03b8.\n\nFinally, utilizing Eqs. (1,5), Eq. (4) is written as\n\np(s1|x1, x2) \u221d M(s1 \u2212 x1, \u03ba1)M(s1 \u2212 x2, \u03ba12) = M(s1 \u2212 \u02c6s1, \u02c6\u03ba1),\n\nwhere the mean and concentration of the posterior given two cues are (SI Sec. 1.3)\n\n\u02c6s1 = atan2(\u03ba1 sin x1 + \u03ba12 sin x2, \u03ba1 cos x1 + \u03ba12 cos x2),\n\n\u02c6\u03ba1 = (cid:2)\u03ba2\n\n12 + 2\u03ba1\u03ba12 cos(x1 \u2212 x2)(cid:3)1/2\n\n,\n\n1 + \u03ba2\n\n3\n\n(5)\n\n(6)\n\n(7)\n(8)\n\n\fwhere atan2 is the arctangent function of two arguments (SI Eq. S17).\nEqs. (7,8) are the results of Bayesian integration in the form of von Mises distribution, and they are\nthe criteria for us to judge whether optimal cue integration is achieved in a neural system.\nTo understand these optimality criteria intuitively, it is helpful to see their equivalence of the Gaussian\ndistribution in the limit of large \u03ba1, \u03ba2 and \u03bas. Under the condition x1 \u2248 x2, Eq. (8) is approximated\nto be \u02c6\u03ba1 \u2248 \u03ba1 + \u03ba12 (SI Sec. 2). Since \u03ba \u2248 1/\u03c32 when von Mises distribution is approximated as\nGaussian one, Eq. (8) becomes 1/\u02c6\u03c32\n12, which is the Bayesian prediction on Gaussian\nvariance conventionally used in the literature [4]. Similarly, Eq. (7) is associated with the Bayesian\nprediction on the Gaussian mean [4].\n\n1 \u2248 1/\u03c32\n\n1 + 1/\u03c32\n\n2.3 Multisensory segregation\n\nWe introduce next the probabilistic model of multisensory segregation. Inspired by the observation\nin multisensory integration that the posterior of a stimulus given combined cues is the product of\nthe posteriors given each cue (Eq.4), we propose that in multisensory segregation, the disparity\nD(s1|x1; s1|x2) between two cues is measured by the ratio of the posteriors given each cue, that is,\n(9)\nBy taking the expectation of log D over the distribution p(s1|x1), we get the Kullback-Leibler\ndivergence between the two posteriors given each cue. This disparity measure was also used to\ndiscriminate alternative moving directions in [18].\nInterestingly, by utilizing the property of von Mises distributions and the condition cos(s1 +\u03c0\u2212x2) =\n\u2212 cos(s1 \u2212 x2), Eq. (9) can be rewritten as\n\nD(s1|x1; s1|x2) \u2261 p(s1|x1)/p(s1|x2),\n\nD(s1|x1; s1|x2) \u221d p(s1|x1)p(s1 + \u03c0|x2),\n\n(10)\nthat is, the disparity information between two cues is proportional to the product of the posterior\ngiven the direct cue and the posterior given the indirect cue but with the stimulus value shifted by \u03c0.\nBy utilizing Eqs. (1,5), we obtain\n\nD(s1|x1; s1|x2) \u221d M(s1 \u2212 x1, \u03ba1)M(s1 + \u03c0 \u2212 x2, \u03ba12) = M (s1 \u2212 \u2206\u02c6s1, \u2206\u02c6\u03ba1) ,\n\n(11)\n\nwhere the mean and concentration of the von Mises distribution are\n\n\u2206\u02c6\u03ba1 = (cid:2)\u03ba2\n\n\u2206\u02c6s1 = atan2(\u03ba1 sin x1 \u2212 \u03ba12 sin x2, \u03ba1 cos x1 \u2212 \u03ba12 cos x2),\n\n(12)\n(13)\nThe above equations are the criteria for us to judge whether the disparity information between two\ncues is optimally encoded in a neural system.\n\n12 \u2212 2\u03ba1\u03ba12 cos(x1 \u2212 x2)(cid:3)1/2\n\n1 + \u03ba2\n\n.\n\n3 Geometrical Interpretation of Multisensory Information Processing\n\nA bene\ufb01t of using the von Mises distribution is that it gives us a clear geometrical interpretation of\nmultisensory information processing. A von Mises distribution M (s \u2212 x, \u03ba) can be interpreted as\na vector in a two-dimensional space with its mean x and concentration \u03ba representing respectively\nthe angle and length of the vector (Fig. 2B-C). This \ufb01ts well with the circular property of heading\ndirection. When the posterior of a stimulus is interpreted as a vector, the vector length represents the\ncon\ufb01dence of inference. Interestingly, under such a geometrical interpretation, the product of two von\nMises distributions equals summation of their corresponding vectors, and the ratio of two von Mises\ndistributions equals subtraction of the two vectors. Thus, from Eq. (4), we see that multisensory\nintegration is equivalent to vector summation, with each vector representing the posterior of the\nstimulus given a single cue, and from Eq. (9), multisensory segregation is equivalent to vector\nsubtraction (see Fig. 2D).\nOverall, multisensory integration and segregation transform the original two vectors, the posteriors\ngiven each cue, into two new vectors, the posterior given combined cues and the disparity between\nthe two cues. The original two vectors can be recovered from their linear combinations. Hence,\nthere is no information loss. The geometrical interpretation also helps us to understand multisensory\ninformation processing intuitively. For instance, if two vectors have a small intersection angle, i.e., the\n\n4\n\n\fFigure 2: Geometrical interpretation of multisensory information processing in von Mises distribution.\n(A) The difference between von Mises and Gaussian distributions. For large concentration \u03ba, their\ndifference becomes small. (B) A von Mises distribution in the polar coordinate. (C) A von Mises\ndistribution M (s \u2212 x, \u03ba) can be represented as a vector in a 2D space with its angle given by\nx and length by \u03ba. (D) Geometrical interpretations of multisensory integration and segregation.\nThe posteriors of s1 given each cue are represented by two vectors (blue). Inverse of a posterior\ncorresponds to rotating it by 180\u25e6. Multisensory integration corresponds to the summation of two\nvectors (green), and multisensory segregation the subtraction of two vectors (red).\n\nposteriors given each cue tend to support each other, the length of summed vector is long, implying\nthat the posterior of cue integration has strong con\ufb01dence; and the length of subtracting vector is short,\nimplying that the disparity between two cues is small. If the two vectors have a large intersection\nangle, the interpretation becomes the opposite.\n\n4 Neural Implementation of Multisensory Information Processing\n\n4.1 The model Structure\n\nWe adopt a decentralized architecture to model multisensory information processing in the brain [7,\n19]. Compared with the centralized architecture in which a dedicated processor carries out all\ncomputations, the decentralized architecture considers a number of local processors communicating\nwith each other via reciprocal connections, so that optimal information processing is achieved at\neach local processor distributively [7]. This architecture was supported by a number of experimental\n\ufb01ndings, including the involvement of multiple, rather than a single, brain areas in visual-vestibular\nintegration [1, 10], the existence of intensive reciprocal connections between MTSd and VIP [20, 21],\nand the robustness of multisensory integration against the inactivation of a single module [22]. In a\nprevious work [7], Zhang et al. studied a decentralized model for multisensory integration at MSTd\nand VIP, and demonstrated that optimal integration can be achieved at both areas simultaneously,\nagreeing with the experimental data. In their model, MSTd and VIP are congruently connected, i.e.,\nneurons in one module are strongly connected to those having the similar preferred heading directions\nin the other module. This congruent connection pattern naturally gives rise to congruent neurons.\nSince the number of opposite neurons is comparable with that of congruent neurons in MSTd and VIP,\nit is plausible that they also have a computational role. It is instructive to compare the probabilistic\nmodels of multisensory integration and segregation, i.e., Eqs. (4) and (10). They have the same form,\nexcept that in segregation the stimulus value in the posterior given the indirect cue is shifted by \u03c0.\nFurthermore, since congruent reciprocal connections lead to congruent neurons, we hypothesize that\nopposite neurons are due to opposite reciprocal connections, and their computational role is to encode\nthe disparity information between two cues. The decentralized model for concurrent multisensory\nintegration and segregation in MSTd and VIP is shown in Fig.3.\n\n5\n\n0Lp(s1|x2)p(s1|x2)-1p(s1|x1,x2)p(s1|x1)D(s1|x1;s1|x2)D(s1|x1;s1|x2)Geometric representation of integration and segregation0xkLGeometric representation of a von Mises distribution0.0050.010.01590\u00b0 270\u00b0180\u00b00\u00b0von Mises distribution(polar coordinate)\u2212180\u22129009018000.0040.0080.012xProbability densityvon Mises M(x,k)Gaussian N(x,k-1)k=1k=3ABCD\fFigure 3: The model structure. (A) The model is composed of two modules, representing MSTd and\nVIP respectively. Each module receives the direct cue via feedforward input. In each module, there\nare two nets of excitatory neurons, each connected recurrently. Net c (blue) consists of congruent\nneurons. Congruent neurons between modules are connected reciprocally in the congruent manner\n(blue lines). On the other hand, net o (red) consists of opposite neurons, and opposite neurons between\nmodules are connected in the opposite manner (brown lines). Moreover, to implement competition\nbetween information integration and segregation, all neurons in a module are connected to a common\ninhibitory neuron pool (purple, only shown in module 1). (B) The recurrent, congruent, and opposite\nconnection patterns between neurons. (C) Network\u2019s peak \ufb01ring rate re\ufb02ects its estimation reliability.\n\n4.2 The model dynamics\n\nDenote as um,n(\u03b8) and rm,n(\u03b8) respectively the synaptic input and \ufb01ring rate of a n-type neuron in\nmodule m whose preferred heading direction with respect to the direct cue m is \u03b8. n = c, o represents\nthe congruent and opposite cells respectively, and m = 1, 2 represents respectively MSTd and VIP.\nFor simplicity, we assume that the two modules are symmetric, and only present the dynamics of\nmodule 1.\nThe dynamics of a congruent neuron in module 1 is given by\n\n\u03c4\n\n\u2202u1,c(\u03b8, t)\n\n\u2202t\n\n= \u2212u1,c(\u03b8, t) +\n\n\u03c0(cid:88)\n\n\u03c0(cid:88)\n\n(cid:48)\n\n(cid:48)\n\n(cid:48)\n\n(cid:48)\n\nWr(\u03b8, \u03b8\n\n)r1,c(\u03b8\n\n, t) +\n\nWc(\u03b8, \u03b8\n\n)r2,c(\u03b8\n\n, t) + I1,c(\u03b8, t), (14)\n\n(cid:48)\n\n(cid:48)\n\n\u03b8(cid:48)=\u2212\u03c0\n\n\u03b8(cid:48)=\u2212\u03c0\n\n2\u03c0a)\u22121 exp(cid:2)\u2212(\u03b8 \u2212 \u03b8(cid:48))2/(2a2)(cid:3)\n2\u03c0a)\u22121 exp(cid:2)\u2212(\u03b8 \u2212 \u03b8(cid:48))2/(2a2)(cid:3). The reciprocal connection strength Jc controls\n\nwhere I1,c(\u03b8, t) is the feedforward input to the neuron. Wr(\u03b8, \u03b8(cid:48)) is the recurrent connection between\n\u221a\nneurons in the same module, which is set to be Wr(\u03b8, \u03b8(cid:48)) = Jr(\nwith periodic condition imposed, where a controls the tuning width of the congruent neurons.\nWc(\u03b8, \u03b8(cid:48)) is the reciprocal connection between congruent cells in two modules, which is set to be\n\u221a\nWc(\u03b8, \u03b8(cid:48)) = Jc(\nthe extent to which cues are integrated between modules and is associated with the correlation\nparameter \u03bas in the stimulus prior (see SI Sec. 3.3).\nThe dynamics of an opposite neuron in module 1 is given by\n\n\u03c4\n\n\u2202u1,o(\u03b8, t)\n\n\u2202t\n\n= \u2212u1,o(\u03b8, t) +\n\n\u03c0(cid:88)\n2\u03c0a)\u22121 exp(cid:2)\u2212(\u03b8 + \u03c0 \u2212 \u03b8(cid:48))2/(2a2)(cid:3) = Wc(\u03b8 + \u03c0, \u03b8(cid:48)), that is, opposite\n\n\u221a\nIt has the same form as that of a congruent neuron except that the pattern of reciprocal connections are\ngiven by Wo(\u03b8, \u03b8(cid:48)) = Jc(\nneurons between modules are oppositely connected by an offset of \u03c0. We choose the strength and\nwidth of the connection pattern Wo to be the same as that of Wc. This is based on the \ufb01nding that\nthe tuning functions of congruent and opposite neurons have similar tuning width and strength [12].\nNote that all connections are imposed with periodic conditions.\nIn the model, we include the effect of inhibitory neurons through a divisive normalization to the\nresponses of excitatory neurons [23], given by\n\n, t) + I1,o(\u03b8, t). (15)\n\nWo(\u03b8, \u03b8\n\n)r2,o(\u03b8\n\nWr(\u03b8, \u03b8\n\n)r1,o(\u03b8\n\n\u03c0(cid:88)\n\n\u03b8(cid:48)=\u2212\u03c0\n\n(cid:48)\n\n, t) +\n\n\u03b8(cid:48)=\u2212\u03c0\n\n(cid:48)\n\n[u1,n(\u03b8, t)]2\n\n+ ,\n\n(16)\n\nr1,n(\u03b8, t) =\n\n1\nDu\n\n6\n\n\u03b8\u2212\u03b8'Connection strengthW(\u03b8,\u03b8')\u221218000180Wr(\u03b8,\u03b8'), Wc(\u03b8,\u03b8'), Wo(\u03b8,\u03b8')ABC\u03b8Cue 1Cue 2Module 1(MSTd)Module 2(VIP)90\u00b0 180\u00b00\u00b0\u03b8CongruentOppositeInhibitory poolExcitatory connectionInhibitory connection270\u00b0Reliability (concentration of net\u2019s estimate)0102030036Peak firing rate (Hz)x103\fFigure 4: Bayes-optimal multisensory integration and segregation with congruent and opposite\nneurons. (A-B) Tuning curves of an example congruent neuron and an example opposite neuron in\nmodule 1. The preferred direction of the congruent neuron in response to two single cues are the same\nat \u221290\u25e6, but the preferred direction of the opposite neuron under two single cues are opposite by 180\u25e6.\n(C-E) The neuronal population activities at module 1 under three cuing conditions: only the direct\ncue 1 (C), only the indirect cue 2 (D), and combination of the two cues (E). (F) The activity levels\nof the congruent and opposite neuronal networks (measured by the corresponding bump heights)\nvs. the cue disparity. (G-H). Comparing the mean and concentration of the stimulus posterior given\ntwo cues estimated by the congruent neuronal network with that predicted by Bayesian inference,\nEqs. (7,8). Each dot is a result obtained under a parameter set. (I-J). Comparing the mean and\nconcentration of the cue disparity information estimated by the opposite neuronal network with that\npredicted by probabilistic inference, Eqs. (12,13). Parameters: Jr = 0.4 \u00afJ, Jc = Jo \u2208 [0.1, 0.5]Jr,\n\u03b11 = \u03b12 \u2208 [0.8, 1.6]U 0\n\nm, Ib = 1, F = 0.5. (G-J) x1 = 0\u25e6, x2 \u2208 [0\u25e6, 180\u25e6].\n\nwhere Du \u2261 1+\u03c9(cid:80)\n(cid:20)\n\u2212 (\u03b8 \u2212 x1)2\n\nthe magnitude of divisive normalization.\nThe feedforward input conveys the direct cue information to a module (e.g., the feedforward input to\nMSTd is from area MT which extracts the heading direction from optical \ufb02ow), which is set to be\n\n(cid:80)\u03c0\n+. [x]+ \u2261 max(x, 0), and the parameter \u03c9 controls\n\u03b8(cid:48)=\u2212\u03c0 [u1,n(cid:48)(\u03b8(cid:48), t)]2\n(cid:20)\n(cid:21)\n\u2212 (\u03b8 \u2212 x1)2\n\nI1,n(\u03b8, t) = \u03b11 exp\n\n\u03be1(\u03b8, t)+Ib +\n\nF Ib\u00011,n(\u03b8, t), (17)\n\n(cid:112)\n\nn(cid:48)=c,o\n\n(cid:112)\n\n+\n\nF \u03b11 exp\n\n4a2\n\n8a2\n\n(cid:21)\n\nwhere \u03b11 is the signal strength, Ib the mean of background input, and F the Fano factor. \u03be1(\u03b8, t) and\n\u00011,n(\u03b8, t) are Gaussian white noises of zero mean with variance satisfying (cid:104)\u03bem(\u03b8, t)\u03bem(cid:48)(\u03b8(cid:48), t(cid:48))(cid:105) =\n\u03b4mm(cid:48)\u03b4(\u03b8\u2212\u03b8(cid:48))\u03b4(t\u2212t(cid:48)), (cid:104)\u0001m,n(\u03b8, t)\u0001m(cid:48),n(cid:48)(\u03b8(cid:48), t(cid:48))(cid:105) = \u03b4mm(cid:48)\u03b4nn(cid:48)\u03b4(\u03b8\u2212\u03b8(cid:48))\u03b4(t\u2212t(cid:48)). The signal-associated\nnoises \u03be1(\u03b8, t) to congruent and opposite neurons are exactly the same, while the background noises\n\u00011,n(\u03b8, t) to congruent and opposite neurons are independent of each other. At the steady state, the\nsignal drives the network state to center at the cue value x1, whereas noises induce \ufb02uctuations of the\nnetwork state. Since we consider multiplicative noise with a constant Fano factor, the signal strength\n\u03b1m controls the reliability of cue m [5]. The exact form of the feedforward input is not crucial, as\nlong as it has a uni-modal shape.\n\n4.3 Results\n\nWe \ufb01rst verify that our model reproduces the characteristics of congruent and opposite neurons.\nFigs. 4A&B show the tuning curves of a congruent and an opposite neuron with respect to either\nvisual or vestibular cues, which demonstrate that neurons in our model indeed exhibit the congruent\nor opposite direction selectivity similar to Fig. 1.\nWe then investigate the mean population activities of our model under different cuing conditions.\nWhen only cue x1 is applied to module 1, both the congruent and opposite neuronal networks in\n\n7\n\nCombinedCue 1 (0\u00b0)Cue 2 (-60\u00b0)ABCDEFGHIJ090180090180Predicted mean of disparity D(sm|xm;sm|x\u2113) (\u00b0)Measured mean (\u00b0)Opposite (Module 1)Opposite (Module 2)R2=0.99090180090180Predicted mean of posterior p(sm|x1,x2) (\u00b0)Measured mean (\u00b0)Congruent (Module 1)Congruent (Module 2)R2=0.99123123Predicted concentration ofdisparity D(sm|xm;sm|x\u2113)Measured concentrationx103x103R2=0.89123123Predicted concentration ofposterior p(sm|x1,x2)Measured concentrationx103x103R2=0.91Estimates of congruent neuronsEstimates of opposite neurons\u2212180018001020Neuron index \u03b8\u22129090\u2212180018001020Neuron index \u03b8\u22129090\u2212180018001020Firing rate (Hz)Neuron index \u03b8Congruent Opposite\u22129090Cue 1Cue 2Firing rate (Hz)01020\u2212180\u221290018090Cue direction xm (\u00b0)Congruent neuron01020Opposite neuron\u22121800180Cue direction xm (\u00b0)\u22129090Tuning curvesBump heightPopulation activities in module 1Cue disparity x2-x1 (\u00b0)090180161820Firing rate (Hz)Congruent Opposite\fmodule 1 receive the feedforward input and generate bumps at x1 (Fig. 4C). When only cue x2\nis applied to module 2, the congruent neuronal network at module 1 receives a reciprocal input\nand generates a bump at x2, whereas the opposite neuronal network receives an offset reciprocal\ninput and generates a bump at x2 + \u03c0 (Fig. 4D). For the indirect cue x2, the neural activities it\ninduces at module 1 is lower than that induced by the direct cue x1 (Fig. 4C). When both cues are\npresented, the congruent neuronal network integrates the feedforward and reciprocal inputs, whereas\nthe opposite neuronal network computes their disparity by integrating the feedforward inputs and the\noffset reciprocal inputs shifted by \u03c0 (Fig. 4E). The two networks compete with each other via divisive\nnormalization. Fig. 4F shows that when the disparity between cues is small, the activity of congruent\nneurons is higher than that of opposite neurons. With the increase of cue disparity, the activity of the\ncongruent neuronal network decreases, whereas the activity of the opposite neurons increases. These\ncomplementary changes in activities of congruent and opposite neurons provide a clue for other parts\nof the brain to evaluate whether the cues are from the same or different objects [24].\nFinally, to verify whether Bayes-optimal multisensory information processing is achieved in our\nmodel, we check the validity of Eqs. (7-8) for multisensory integration p(sm|x1, x2) by congruent\nneurons in module m, and Eqs. (12-13) for multisensory segregation D(sm|xm; sm|xl) (l (cid:54)= m) by\nopposite neurons in module m. Take the veri\ufb01cation of the congruent neuronal network in module m\nas an example. When a pair of cues are simultaneously applied, the actual mean and concentration\nof the networks\u2019s estimates (bump position) are measured through population vector [25] (SI Sec.\n4.2). To obtain the Bayesian predictions for the network\u2019s estimate under combined cue condition\n(details in SI Sec. 4.3), the mean and concentration of that network\u2019s estimates under either single\ncue conditions are also measured, and then are substituted into Eqs. (7-8). Comparisons between\nthe measured mean and concentration of congruent networks in two modules and the corresponding\ntheoretical predictions are shown in Fig. 4G&H, indicating an excellent \ufb01t, where each dot is the\nresult under a particular set of parameters. Similarly, comparisons between the measured mean\nand concentration of opposite networks and the theoretical predictions (SI Sec. 4.3) are shown in\nFig. 4I&J, indicating opposite neurons indeed implement multisensory segregation.\n\n5 Conclusion and Discussion\n\nOver the past years, multisensory integration has received large attention in modelling studies, but\nthe equally important issue of multisensory segregation has been rarely explored. The present study\nproposes that opposite neurons, which is widely observed at MSTd and VIP, encode the disparity\ninformation between sensory cues. We built a computational model composed of reciprocally\ncoupled MSTd and VIP, and demonstrated that the characteristics of congruent and opposite cells\nnaturally emerge from the congruent and opposite connection patterns between modules, respectively.\nUsing the von Mises distribution, we derived the optimal criteria for integration and segregation\nof circular variables and found they have clear geometrical meanings: integration corresponds to\nvector summation while segregation corresponds to vector subtraction. We further showed that such a\ndecentralized system can realize optimal cue integration and segregation at each module distributively.\nTo our best knowledge, this work is the \ufb01rst modelling study unveiling the functional role of opposite\ncells. It has a far-reaching implication on multisensory information processing, that is, the brain\ncan exploit sisters of congruent and opposite neurons to implement cue integration and segregation\nconcurrently.\nFor simplicity, only perfectly congruent or perfectly opposite neurons are considered, but in reality,\nthere are some portions of neurons whose differences of preferred visual and vestibular heading\ndirections are in between 0\u25e6 and 180\u25e6 (Fig. 1C). We checked that those neurons can arise from adding\nnoises in the reciprocal connections. As long as the distribution in Fig. 1C is peaked at 0\u25e6 and 180\u25e6,\nthe model can implement concurrent integration and segregation. Also, we have only pointed out\nthat the competition between congruent and opposite neurons provides a clue for the brain to judge\nwhether the cues are likely to originate from the same or different objects, without exploring how the\nbrain actually does this. These issues will be investigated in our future work.\n\nAcknowledgments\n\nThis work is supported by the Research Grants Council of Hong Kong (N_HKUST606/12 and 605813) and\nNational Basic Research Program of China (2014CB846101) and the Natural Science Foundation of China\n(31261160495).\n\n8\n\n\fReferences\n[1] Yong Gu, Dora E Angelaki, and Gregory C DeAngelis. Neural correlates of multisensory cue integration\n\nin macaque mstd. Nature Neuroscience, 11(10):1201\u20131210, 2008.\n\n[2] Marc O Ernst and Heinrich H B\u00fclthoff. Merging the senses into a robust percept. Trends in Cognitive\n\nSciences, 8(4):162\u2013169, 2004.\n\n[3] David Alais and David Burr. The ventriloquist effect results from near-optimal bimodal integration. Current\n\nBiology, 14(3):257\u2013262, 2004.\n\n[4] Marc O Ernst and Martin S Banks. Humans integrate visual and haptic information in a statistically optimal\n\nfashion. Nature, 415(6870):429\u2013433, 2002.\n\n[5] Wei Ji Ma, Jeffrey M Beck, Peter E Latham, and Alexandre Pouget. Bayesian inference with probabilistic\n\npopulation codes. Nature Neuroscience, 9(11):1432\u20131438, 2006.\n\n[6] Tomokazu Ohshiro, Dora E Angelaki, and Gregory C DeAngelis. A normalization model of multisensory\n\nintegration. Nature Neuroscience, 14(6):775\u2013782, 2011.\n\n[7] Wen-Hao Zhang, Aihua Chen, Malte J Rasch, and Si Wu. Decentralized multisensory information\n\nintegration in neural systems. The Journal of Neuroscience, 36(2):532\u2013547, 2016.\n\n[8] Mark T Wallace, GE Roberson, W David Hairston, Barry E Stein, J William Vaughan, and Jim A Schirillo.\nUnifying multisensory signals across time and space. Experimental Brain Research, 158(2):252\u2013258,\n2004.\n\n[9] Ahna R Girshick and Martin S Banks. Probabilistic combination of slant information: weighted averaging\n\nand robustness as optimal percepts. Journal of Vision, 9(9):8\u20138, 2009.\n\n[10] Aihua Chen, Gregory C DeAngelis, and Dora E Angelaki. Functional specializations of the ventral\nintraparietal area for multisensory heading discrimination. The Journal of Neuroscience, 33(8):3567\u20133581,\n2013.\n\n[11] Michael L Morgan, Gregory C DeAngelis, and Dora E Angelaki. Multisensory integration in macaque\n\nvisual cortex depends on cue reliability. Neuron, 59(4):662\u2013673, 2008.\n\n[12] Aihua Chen, Gregory C DeAngelis, and Dora E Angelaki. Representation of vestibular and visual cues to\n\nself-motion in ventral intraparietal cortex. The Journal of Neuroscience, 31(33):12036\u201312052, 2011.\n\n[13] Yong Gu, Paul V Watkins, Dora E Angelaki, and Gregory C DeAngelis. Visual and nonvisual contributions\nto three-dimensional heading selectivity in the medial superior temporal area. The Journal of Neuroscience,\n26(1):73\u201385, 2006.\n\n[14] Richard F Murray and Yaniv Morgenstern. Cue combination on the circle and the sphere. Journal of vision,\n\n10(11):15\u201315, 2010.\n\n[15] Jean-Pierre Bresciani, Franziska Dammeier, and Marc O Ernst. Vision and touch are automatically\n\nintegrated for the perception of sequences of events. Journal of Vision, 6(5):2, 2006.\n\n[16] Neil W Roach, James Heron, and Paul V McGraw. Resolving multisensory con\ufb02ict: a strategy for balancing\nthe costs and bene\ufb01ts of audio-visual integration. Proceedings of the Royal Society of London B: Biological\nSciences, 273(1598):2159\u20132168, 2006.\n\n[17] Yoshiyuki Sato, Taro Toyoizumi, and Kazuyuki Aihara. Bayesian inference explains perception of unity and\nventriloquism aftereffect: identi\ufb01cation of common sources of audiovisual stimuli. Neural Computation,\n19(12):3335\u20133355, 2007.\n\n[18] Mehrdad Jazayeri and J Anthony Movshon. Optimal representation of sensory information by neural\n\npopulations. Nature Neuroscience, 9(5):690\u2013696, 2006.\n\n[19] Wen-Hao Zhang and Si Wu. Reciprocally coupled local estimators implement bayesian information\n\nintegration distributively. In Advances in Neural Information Processing Systems, pages 19\u201327, 2013.\n\n[20] Driss Boussaoud, Leslie G Ungerleider, and Robert Desimone. Pathways for motion analysis: cortical\nconnections of the medial superior temporal and fundus of the superior temporal visual areas in the\nmacaque. Journal of Comparative Neurology, 296(3):462\u2013495, 1990.\n\n[21] Joan S Baizer, Leslie G Ungerleider, and Robert Desimone. Organization of visual inputs to the inferior\ntemporal and posterior parietal cortex in macaques. The Journal of Neuroscience, 11(1):168\u2013190, 1991.\n[22] Yong Gu, Gregory C DeAngelis, and Dora E Angelaki. Causal links between dorsal medial superior\ntemporal area neurons and multisensory heading perception. The Journal of Neuroscience, 32(7):2299\u2013\n2313, 2012.\n\n[23] Matteo Carandini and David J Heeger. Normalization as a canonical neural computation. Nature Reviews\n\nNeuroscience, 13(1):51\u201362, 2012.\n\n[24] Tatiana A Engel and Xiao-Jing Wang. Same or different? a neural circuit mechanism of similarity-based\n\npattern match decision making. The Journal of Neuroscience, 31(19):6982\u20136996, 2011.\n\n[25] Apostolos P Georgopoulos, Andrew B Schwartz, and Ronald E Kettner. Neuronal population coding of\n\nmovement direction. Science, 233(4771):1416\u20131419, 1986.\n\n9\n\n\f", "award": [], "sourceid": 1581, "authors": [{"given_name": "Wen-Hao", "family_name": "Zhang", "institution": "Institute of Neuroscience"}, {"given_name": "He", "family_name": "Wang", "institution": "HKUST"}, {"given_name": "K. Y. Michael", "family_name": "Wong", "institution": "HKUST"}, {"given_name": "Si", "family_name": "Wu", "institution": "Beijing Normal University"}]}