{"title": "A state-space model for inferring effective connectivity of latent neural dynamics from simultaneous EEG/fMRI", "book": "Advances in Neural Information Processing Systems", "page_first": 4662, "page_last": 4671, "abstract": "Inferring effective connectivity between spatially segregated brain regions is important for understanding human brain dynamics in health and disease. Non-invasive neuroimaging modalities, such as electroencephalography (EEG) and functional magnetic resonance imaging (fMRI), are often used to make measurements and infer connectivity. However most studies do not consider integrating the two modalities even though each is an indirect measure of the latent neural dynamics and each has its own spatial and/or temporal limitations. In this study, we develop a linear state-space model to infer the effective connectivity in a distributed brain network based on simultaneously recorded EEG and fMRI data. Our method first identifies task-dependent and subject-dependent regions of interest (ROI) based on the analysis of fMRI data. Directed influences between the latent neural states at these ROIs are then modeled as a multivariate autogressive (MVAR) process driven by various exogenous inputs. The latent neural dynamics give rise to the observed scalp EEG measurements via a biophysically informed linear EEG forward model. We use a mean-field variational Bayesian approach to infer the posterior distribution of latent states and model parameters. The performance of the model was evaluated on two sets of simulations. Our results emphasize the importance of obtaining accurate spatial localization of ROIs from fMRI. Finally, we applied the model to simultaneously recorded EEG-fMRI data from 10 subjects during a Face-Car-House visual categorization task and compared the change in connectivity induced by different stimulus categories.", "full_text": "A state-space model for inferring effective\nconnectivity of latent neural dynamics from\n\nsimultaneous EEG/fMRI\n\nTao Tu\n\nColumbia University\n\ntt2531@columbia.edu\n\nJohn Paisley\n\nColumbia University\n\njpaisley@columbia.edu\n\nStefan Haufe\n\nCharit\u00e9 \u2013 Universit\u00e4tsmedizin Berlin\n\nstefan.haufe@charite.de\n\nPaul Sajda\n\nColumbia University\n\npsajda@columbia.edu\n\nAbstract\n\nInferring effective connectivity between spatially segregated brain regions is impor-\ntant for understanding human brain dynamics in health and disease. Non-invasive\nneuroimaging modalities, such as electroencephalography (EEG) and functional\nmagnetic resonance imaging (fMRI), are often used to make measurements and\ninfer connectivity. However most studies do not consider integrating the two modal-\nities even though each is an indirect measure of the latent neural dynamics and\neach has its own spatial and/or temporal limitations. In this study, we develop a\nlinear state-space model to infer the effective connectivity in a distributed brain\nnetwork based on simultaneously recorded EEG and fMRI data. Our method \ufb01rst\nidenti\ufb01es task-dependent and subject-dependent regions of interest (ROI) based\non the analysis of fMRI data. Directed in\ufb02uences between the latent neural states\nat these ROIs are then modeled as a multivariate autogressive (MVAR) process\ndriven by various exogenous inputs. The latent neural dynamics give rise to the ob-\nserved scalp EEG measurements via a biophysically informed linear EEG forward\nmodel. We use a mean-\ufb01eld variational Bayesian approach to infer the posterior\ndistribution of latent states and model parameters. The performance of the model\nwas evaluated on two sets of simulations. Our results emphasize the importance\nof obtaining accurate spatial localization of ROIs from fMRI. Finally, we applied\nthe model to simultaneously recorded EEG-fMRI data from 10 subjects during a\nFace-Car-House visual categorization task and compared the change in connectivity\ninduced by different stimulus categories.\n\n1\n\nIntroduction\n\nIdentifying the spatiotemporal dependence among distributed cortical regions is often seen as crucial\nfor understanding the macro-scale neural dynamics underlying human cognition. Such spatiotemporal\ndependencies can be quanti\ufb01ed statistically by the modeling of effective connectivity, which is de\ufb01ned\nas the time-lagged in\ufb02uence of one brain region over another [1]. Effective connectivity has been\nintroduced in the framework of dynamic causal modeling (DCM). DCM uses a state-space model with\nhidden state variables to describe task-dependent \"causal\" interactions between latent neural states\nand how the activity of regional neural states translates into observed neural measurements [2, 3].\nEstimating effective connectivity between anatomically segregated brain regions is a challenging\nproblem for several reasons: 1) the inference is made on unobserved latent states rather than directly\non the observations; 2) latent neural dynamics often evolve on a fast time scale so it requires the\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fobservation time series to be measured on a similar temporal scale; 3) accurate spatial localization of\nthe activated brain regions is often a prerequisite for the speci\ufb01cation of a meaningful dynamic causal\nmodel.\nTo address these challenges, a number of state-space based modeling techniques have been developed\nand applied to a variety of non-invasive neuroimage modalities such as electroencephalography (EEG)/\nmagnetoencephalography (MEG), functional magnetic resonance imaging (fMRI), and functional\nnear-infrared spectroscopy (fNIRS). Modalities like EEG and MEG with high temporal resolution\noffer advantages in terms of measuring and inferring effective connectivity. Cheung et al. [4]\nproposed a state-space model where the latent neural dynamics at pre-de\ufb01ned ROIs were modeled as\na multivariate autoregressive (MVAR) process. They assumed a known EEG forward model with\nunknown spatial distribution of the EEG sources within each ROI. Haufe et al. [5] used a similar\nMVAR approach to model the connectivity in EEG source space where the spatial source demixing\nwas optimized jointly with the connectivity estimation. David et al. [6] used a nonlinear hierarchical\nneural mass model for the \"casual\" modeling of evoked responses in EEG/MEG. Another model\nfor evoked responses in MEG/EEG was proposed by Yang et al. [7] where a time-varying MVAR\nmodel was used to estimate the dynamic connectivity among multiple ROIs. In contrast to the work\nby Cheung et al. [4], they also used a known MEG/EEG forward model for the evoked responses,\nbut sources within the same ROI were modeled as independent Gaussian variables. Other dynamical\nmodels leveraging the relatively high spatial resolution of fMRI [3, 8, 9] and fNIRs [10, 11] have\nalso been developed for brain connectivity analysis.\nAll of these inference methods are based on neural measurements from a single modality, and therefore\nsuffer from potentially suboptimal estimates of the true latent neural dynamics due to the limitation in\nspatial or temporal resolution of the modality. Simultaneous EEG-fMRI is a neuroimaging technique\nthat leverages the complementary strengths of both modalities, namely 3D spatial resolution of fMRI\nand temporal resolution of EEG. Given that the data from two modalities are recorded under identical\nexperimental conditions, one can use fMRI activations as a spatial prior to improve the accuracy of\nEEG source localization [12, 13, 14].\nIn this paper we propose a linear state-space model for estimating the effective connectivity using, as\nobservations, data from simultaneously recorded EEG and fMRI. Our goal is to combine EEG with\nfMRI to arrive at estimates of the latent neural dynamics with high spatiotemporal resolution. Since\nfMRI offers signi\ufb01cant advantage over EEG in terms of spatially localizing potential source activity,\nwe \ufb01rst identify task-speci\ufb01c ROIs from the analysis of fMRI data on each individual subject. The\nlocations of these ROIs are used as spatial constraints to inform the effective connectivity modeling\nof EEG. Similar to the ROI source model proposed by Yang et al. [7], we also model the latent state\nvariables as the mean source activity at each ROI. Each source inside one ROI follows a Gaussian\ndistribution with the ROI mean and a shared unknown variance parameter. In contrast to [7], we model\nthe state equation as an MVAR process, which describes the directed interactions between latent states\ndriven by deterministic inputs speci\ufb01c to an experiment. Inputs can directly in\ufb02uence the activity at a\nparticular region (external input) or they can modulate the connectivity between regions. Finally, an\nEEG forward model based on a pre-estimated lead \ufb01eld matrix was constructed together with the\nROI source model to generate scalp EEG observations. We use a mean-\ufb01eld variational Bayesian\napproach to infer the posterior distribution of latent variables and model parameters. The posterior\nestimates are updated ef\ufb01ciently via a sequential Kalman \ufb01lter and the use of conjugate priors. We\nevaluated the model performance on two sets of simulations and demonstrated the importance of\nthe spatial speci\ufb01city provided by fMRI. We then applied the state-space model to simultaneously\nEEG-fMRI recordings from 10 subjects during a face-car-house rapid decision-making task.\n\n2 Model\n\nModel description Our linear state-space model for inferring the latent neural dynamics consists of\na state equation and two observation equations for EEG. In the state equation, we model the temporal\ndependence between latent state variables as a \ufb01rst-order MVAR process in the presence of external\nand context-dependent inputs:\n\nst = Ast\u22121 +\n\nBkmk\n\nt st\u22121 + Dut + \u03c9t\n\n(1)\n\nK(cid:88)\n\nk=1\n\n2\n\n\fwhere st is an S \u00d7 1 vector of latent state variables at time t. Each element in st represents the\nmean activity of all EEG sources within one of S ROIs. A is an S \u00d7 S intrinsic connectivity matrix\nwherein each entry denotes the connection strength between a pair of latent variables in the absence\nt , where k = 1, 2, ...K, represents the kth modulatory input at time t. Bk \u2208 RS\u00d7S is the\nof input. mk\nkth modulatory connectivity matrix where each element denotes the change in connectivity induced\nt . ut is an S \u00d7 1 vector that denotes the external input at each ROI and\nby the modulatory input mk\nD is an S \u00d7 S diagonal matrix whose diagonal element denotes the strength of ut. \u03c9t \u2208 RS\u00d71 is a\nGaussian state noise vector at time t with a zero mean and a diagonal covariance matrix Qs. This\nbilinear model used to approximate the latent state dynamics modulated by task demand is similar to\nthat in [3].\nThe observation model for EEG consists of two equations. We used a volumetric source model which\nassumes that EEG sources are uniformly distributed on a 3-D grid inside the brain. The position and\nthe orientation of each source (dipole) is \ufb01xed and pre-estimated from real data in this model. Source\nactivity xt propagates through brain tissues and generates EEG potentials yt measured by electrodes\nplaced on the scalp via the following linear forward model:\n\nyt = Lxt + et,\n\n(2)\nwhere yt \u2208 RM\u00d71 is the EEG observations measured from M channels at time t. xt \u2208 RU\u00d71is the\nactivity of U EEG sources at time t. L \u2208 RM\u00d7U is the lead \ufb01eld matrix that describes the mapping\nfrom EEG source space to channel space. L in our model was pre-computed by solving the EEG\nforward modeling problem [15]. et is an M \u00d7 1 vector that models the noise at each channel as\na Gaussian with zero mean and covariance matrix Qy. Solving xt from yt is called EEG inverse\nmodeling or source localization. It is an ill-posed problem since U (cid:29) M. Many EEG source\nlocalization methods such as minimum-norm estimation (MNE) [16] require the estimate of Qy from\nbaseline data. The solution to EEG source localization is not unique and often not robust, especially\nbased on EEG data alone. Therefore, we did not model each single source in the whole brain as\na latent variable. Similar as in [7], the source activity xt in our model is composed of the latent\nvariables st by the following equation:\n\nxt = Gst + \u0001t\n\n(3)\nwhere G \u2208 RU\u00d7S is a binary indicator matrix. Each row of G is a one-hot vector that encodes the\nmembership of each source in one of the ROIs. \u0001t \u2208 RU\u00d71 is a Gaussian noise term with zero mean\nand U \u00d7 U diagonal covariance matrix Qx. Consequently, each source in xt is Gaussian distributed\naround its corresponding ROI mean in st with a variance \u03c32\nr , r = 1, . . . , S speci\ufb01ed in the diagonal\nelements of Qx . If a source in xt is not contained in any ROI, it is modeled as a Gaussian variable\nwith a zero mean and variance \u03c32\n0. This model assumes that all sources in the rthROI have the same\nr, while sources that do not belong to any ROI have the same variance parameter\nvariance parameter \u03c32\n0. Therefore, there are only S + 1 distinct elements in the diagonal of Qx.\n\u03c32\nSubstituting (3) into (2) and eliminating xt, the EEG observation model can be expressed as:\n\n(4)\nwhere C = LG is a known M \u00d7 S matrix and \u03c6t is the Gaussian noise term at time t with a zero\nmean and an M \u00d7 M covariance matrix R = Qy + LQxL(cid:48). L(cid:48) denotes the transpose of L.\nTaken together, our linear state-space model can be expressed as (see Figure 1 for illustration):\n\nyt = Cst + \u03c6t\n\nst|st\u22121 \u223c N (Ast\u22121 +\n\nBkmk\n\nt st\u22121 + Dut, Qs),\n\nyt|st \u223c N (Cst, R) .\n\n(5)\n\nK(cid:88)\n\nk=1\n\n(cid:90)\n\nL(q) =\n\nt=1 and the unknown model parameters \u03b8 =(cid:8)A,{Bk}K\n\nModel inference Given EEG observations Y = {yt}T\nt=1, we use the mean-\ufb01eld variational\nBayesian (VB) approximation to make inference on the posterior distributions of the latent state\nvariables S = {st}T\n1B shows the probabilistic graphical representation of our model. In VB, we make analytical approxi-\nmation to the joint posterior distribution p(S, \u03b8|Y) in order to maximize the evidence lower bound\n(ELBO)[17]:\n\nk=1, D, Qs, R(cid:9). Figure\n\nq(S, \u03b8) log\n\np(S, \u03b8, Y)\n\nq(S, \u03b8)\n\nd\u03b8dS\n\n(6)\n\n3\n\n\fFigure 1: Model overview. A, Illustration of the linear state-space model for simultaneous EEG/fMRI.\nB, Probabilistic graphical representation of the model.\n\nwhere q(S, \u03b8) is an arbitrary density from a family of variational distributions. It is easy to show that\nthe ELBO objective is maximized when q(S, \u03b8) = p(S, \u03b8|Y). Since p(S, \u03b8|Y) is often intractable,\nwe choose a density from the mean-\ufb01eld variational family having the form q(S, \u03b8) = q(S|Y)q(\u03b8|Y)\nto approximate p(S, \u03b8|Y). The solution that maximizes L(q) satis\ufb01es [17]:\n\nlog q(S|Y) \u221d E\u03b8(log p(S, \u03b8, Y))\nlog q(\u03b8|Y) \u221d ES(log p(S, \u03b8, Y))\n\n(7)\n(8)\n\nwhere the expectation is taken with respect to q(\u03b8|Y) and q(S|Y) respectively.\nEquation (7) is the VB-E step where we estimate the posterior distribution of latent variable q(S|Y)\ngiven the current estimate of q(\u03b8|Y). Since we assume Gaussian posterior on S, we use Kalman\n\ufb01ltering and smoothing to sequentially update the posterior mean \u00b5T\nt of the latent\nvariables at every time t. More details of the derivation are provided in Appendix.\nEquation (8) is the VB-M step where we update the posterior distribution of model parameters q(\u03b8|Y)\nk=1, D, Qs\n\ngiven the current estimate of q(S|Y). For the state model parameters \u03b8S =(cid:8)A,{Bk}K\n\nt and covariance \u03a3T\n\n(cid:9),\n\nwe choose a Gaussian-Gamma conjugate prior according to the principle of automatic relevance\ndetermination (ARD). ARD assigns a separate shrinkage prior to each element of the connectivity\nmatrices which in turn is adjusted by a hyper-prior [17, 18]. It encourages a sparse structure in the\nconnectivity matrices to enhance interpretability. The use of conjugate priors also allows one to\nobtain closed-form solution for the posterior updates of model parameters. Since we assume the state\nnoise covariance Qs to be diagonal, we can estimate each row in the model parameters \u03b8S separately.\nSpeci\ufb01cally, the rth row of the state equation can be expressed as:\n\u03c9t[r] \u223c N (0, \u03b2\u22121[r]),\n\n(9)\n(cid:48).\nand \u03b7[r] = [a[r], b1[r], ..., bK[r], d[r]]\nwhere \u02dcst[r] =\n\u03b2[r] is the precision of the state noise at the rth row; a[r] and bk[r] are the rth rows of A and Bk,\nrespectively; d[r] is the rth diagonal element of D.\nWe assume the following Gaussian-Gamma conjugate priors for \u03b7[r], \u03b2[r], and \u03b1 [19]:\n\n(cid:21)\n, \u02dcFt =(cid:2)IS m1\n\nst[r] = \u03b7(cid:48)[r]\u02dcst[r] + \u03c9t[r],\n\n(cid:20)\u02dcFtst\u22121\n\n\u03b2[r] = 1/Qs(r, r)\n\nut[r]\n\nt IS . . . mK\n\nt IS\n\n(cid:3)(cid:48)\n\np(\u03b7[r], \u03b2[r]|\u03b1) = N(cid:0)0, (\u03b2[r]\u039b\u03b1)\u22121(cid:1) Gamma(a0, b0),\n\n(K+1)S+1(cid:89)\n\np(\u03b1) =\n\nGamma(c0, d0) (10)\n\nwhere \u03b1 = [\u03b11, \u03b12, ..., \u03b1(K+1)S+1] is a vector of hyperparameters on each element of \u03b7[r] and \u039b\u03b1\nis a diagonal matrix with the vector \u03b1. Each hyperparameter in \u03b1 has a separate Gamma prior. The\nvariational joint posterior for \u03b7[r] and \u03b2[r] has the same form as their priors:\n\nq(\u03b7[r], \u03b2[r]|Y) = N (\u00af\u00b5[r], \u03b2\u22121[r] \u00af\u03a3[r])Gamma(\u00afa[r], \u00afb[r])\n\n(11)\n\ni=1\n\n4\n\n\fwhere\n\n\u00af\u00b5[r] = \u00af\u03a3[r]\n\n(cid:21)\n\n+ E\u03b1(\u039b\u03b1)\n\n\u00af\u03a3\u22121[r] =\n\n(cid:20)(cid:80)T\n(cid:80)T\n(cid:20)(cid:80)T\n(cid:80)T\n\nt=2\n\n(cid:21)\n\n\u02dcFtEs[st\u22121s(cid:48)\nt=2\nt=2 ut[r](\u00b5T\n\u02dcFtEs[st[r]st\u22121]\nt=2 ut[r]\u00b5T\nT \u2212 1\n\nt [r]\n\n,\n\nt\u22121]\u02dcF(cid:48)\nt\u22121)(cid:48) \u02dcF(cid:48)\n\nt\n\nt\n\n\u00afa[r] = a0 +\n\n, \u00afb[r] = b0 +\n\n2\n\n(cid:34) T(cid:88)\n\nt=2\n\n1\n2\n\n(cid:80)T\n(cid:80)T\n\nt=2\n\nt\u22121ut[r]\n\n\u02dcFt\u00b5T\nt=2 (ut[r])2\n\n(cid:18) \u00afc1\n\n\u00afd1\n\nE\u03b1(\u039b\u03b1) = diag\n\n,\n\n\u00afc2\n\u00afd2\n\n, ...,\n\n\u00afc(K+1)S+1\n\u00afd(K+1)S+1\n\nEs[(st[r])2] \u2212 \u00af\u00b5(cid:48)[r] \u00af\u03a3\u22121[r] \u00af\u00b5[r]\n\n(cid:19)\n\n(12)\n\n(13)\n\n(14)\n\n(cid:35)\n\nThe posterior for each hyperparameter \u03b1j, j = 1, 2, ..., (K + 1)S + 1 can be computed independently:\n\nwhere\n\n\u00afcj = c0 +\n\n1\n2\n\n,\n\n\u00afdj = d0 +\n\n1\n2\n\n(\u00af\u00b5[r, j])2 + \u00af\u03a3r[j, j]\n\nq(\u03b1j|Y) = Gamma(\u03b1j|\u00afcj, \u00afdj)\n\n(cid:20) \u00afa[r]\n\n\u00afb[r]\n\n(cid:21)\n\n\u00af\u00b5[r, j] is the jth element of \u00af\u00b5[r] and \u00af\u03a3r[j, j] is the jth diagonal element of \u00af\u03a3[r].\nThe noise covariance R comprises two unknown quantities Qy and Qx. Choosing a conjugate prior\nfor each of them individually is dif\ufb01cult. Since Qy and Qx are not of primary interest in our study,\nwe optimize R directly. We set the inverse Wishart prior IW (v0, V0) on R [20]:\n\n(15)\n\n(16)\n\n(17)\n\n(18)\n\nq(R|y) = IW (vn, Vn)\n\n(cid:32) T(cid:88)\n\n(cid:33)\n\nwhere\n\nvn = v0 + T, Vn = V0 +\n\n(yt \u2212 C\u00b5T\n\nt )(yt \u2212 C\u00b5T\n\nt )(cid:48) + C\u03a3T\n\nt C(cid:48)\n\nThe implementation of the algorithm in Matlab and the dataset are available at https://github.\ncom/taotu/VBLDS_Connectivity_EEG_fMRI.\n\nt=1\n\n3 Results\n\nWe \ufb01rst evaluated the performance of the state-space model on simulated datasets and then applied\nthe model to real simultaneously recorded EEG and fMRI data (see more details in Appendix). In\nthe simulation study, we assessed the performance of the model when spatial localization of ROIs\nis inaccurate, simulating the scenario when fMRI information is not available. We generated two\nsimulation scenarios corresponding to two different types of EEG-fMRI experiment designs: a block\ndesign and an event-related design. For analysis of the real simultaneous EEG-fMRI data, we applied\nthe state-space model on the EEG data recorded simultaneously with fMRI to infer the induced\nconnectivity change between brain regions activated during a Face-Car-House visual categorization\ntask. Combining the subject-speci\ufb01c fMRI activation maps and the EEG temporal dynamics enabled\nus to compare differences in modulatory connectivity induced by face stimuli vs. house stimuli.\n\n3.1 Simulations\n\nScenario 1: Block Design We simulated the latent dynamics in the brain network consisting of\nS = 5 ROIs using the structure shown in Figure 2A. The external input was modeled as a sequence of\nimpulse functions with an inter-stimulus interval (ISI) uniformly drawn between 2 s to 2.5 s (longer\nthan the fMRI repetition time TR=2 s). The modulatory input was modeled as alternating on-off\nblocks with a block duration of 20 s to simulate a block design fMRI experiment where change\nin the network connectivity could be induced by stimulus presentation or alternation of cognitive\nstates (such as attention and salience). The external input feeds into FFA with a strength of 0.9 and\nthe modulatory input changes the connection strength from SPL to PPA and from ACC to FEF. In\nparticular, the direction of the modulatory connection from SPL to PPA is opposite to that of the\nintrinsic connection between them. The state covariance Qs was set to be the identity matrix. The\nROI variance \u03c32\nr , r = 0, 1, . . . , S in Qx was drawn from a Gamma distribution \u0393(0.2, 1) whose\nshape and scale parameters were estimated from real data. The EEG measurement noise covariance\nQy was also estimated from real data during the baseline period. We simulated the latent ROI mean\n\n5\n\n\factivity and EEG data for a duration of T = 8 min with a sampling rate of 100 Hz. The unit of the\nsimulated EEG measurements was microvolt.\nScenario 2: Event-related Design To mimic a more realistic EEG-fMRI experiment design, we\nsimulated two modulatory inputs that induce different connectivity patterns shown in Figure 2B. The\nmodulatory inputs were modeled as a sequence of discrete events with a duration of 2 s. The ISI\nwas also drawn uniformly from 2 s to 2.5 s so that there was no overlap between the two inputs.\nOther parameters were the same as in scenario 1. The aim of this simulation was to test whether\nthe algorithm could correctly distinguish the modulatory connectivity matrices induced by different\nmodulatory inputs.\nTo illustrate the value of the high spatial speci\ufb01city provided by fMRI, we simulated an \u2019EEG-only\u2019\ncondition where fMRI data was not available. To achieve this, the anatomical region of each ROI\nwas dilated so that the number of sources erroneously included was approximately 30% of the total\nnumber of true sources across 5 ROIs. The direct outcome of this spatial smearing was that more\nrows in G would have nonzero entries. In the absence of fMRI data, one typically has to de\ufb01ne\nROIs based on atlases de\ufb01ned based on structural brain images, which may cause inaccurate spatial\nlocalization of ROIs. We compared the performance of the algorithm between the \u2019EEG-fMRI\u2019 and\nthe \u2019EEG-only\u2019 conditions to highlight the importance of the spatial resolution added by fMRI.\nTen independent simulation datasets were generated for each scenario. For each simulation dataset,\nwe applied the EEG-fMRI method and the EEG-only method separately. Since we simulated relatively\nlarge samples, we chose small non-informative priors for the model parameters. Two methods were\ninitialized with the same set of parameters (see Appendix). The performance of the algorithm in\nrecovering the intrinsic and modulatory connectivity matrices A and B as well as the noise covariance\nmatrices Qs and R was evaluated using the relative error between the true and estimated values\nde\ufb01ned as:\n\n||Xtrue||F\n\n(19)\nwhere || \u00b7 ||F is the Frobenius norm of a matrix. Statistical inference on the entries of the connectivity\nmatrices is straightforward since they have Gaussian posteriors. Prior to calculating the relative error\nof A and B, we thresholded each connection according to its posterior distribution at P < 0.05 with\nBonferroni correction to account for multiple comparison (N=108). Figure 2 shows the comparison\nof the relative error between the two methods. For both simulation scenarios, EEG-fMRI method\ngenerated more accurate estimation than EEG-only method. In our EEG-only simulation, even though\nonly a small number of sources (38) that did not contribute to the underlying dynamics was falsely\nassigned to all ROIs, the performance largely decreased. In practice, without the fMRI data, one\nwould only get more inaccurate spatial localization of ROIs.\n\n||Xtrue \u2212 \u02c6Xest||F\n\ne =\n\nFigure 2: Performance on two sets of simulations. A, 5-node network structure in scenario 1 and the\nrelative error of A, B1, Qs, R for the EEG-fMRI (orange) and the EEG-only (blue) conditions. B,\nSimilar comparison for scenario 2 where two modulatory matrices B2 and B3 were simulated. Blue\nline denotes intrinsic connection and blue dotted line denotes modulatory connection between two\nnodes. Error bar represents the standard error of the mean across 10 independent simulations.\n\n6\n\n\f3.2 Simultaneous EEG-fMRI Data\n\nWe then applied our state-space model method on simultaneously recorded EEG and fMRI data from\n10 subjects. The data were recorded when subjects performed an event-related three-choice visual\ncategorization task. On each trial, an image of a face, car, or house was presented at random for 100\nms. The ISI ranged uniformly between 2 s and 2.5 s. Subjects reported their choice of the image\ncategory by pressing one of the three buttons on an MR-compatible button response pad. Each subject\ncompleted 4 runs of the categorization task. In each run, there were 180 trials (60 per category) with\na total duration of 560 s. Previous studies [21, 22] have implicated two spatially and temporally\nseparate brain networks (which we term the \u2019early\u2019 and \u2019late\u2019 networks) during this rapid perceptual\ndecision task based on an EEG-informed fMRI analysis approach. However, the latent brain dynamics\nwere inferred from the fMRI data, which \ufb02uctuate on a much slower timescale than the latent neural\nprocesses. In this study, we leveraged the high temporal resolution of the EEG data in combination\nwith the high spatial speci\ufb01city of fMRI to estimate the latent brain dynamics underlying behavior in\nthis task. In particular, we selected 3 regions (FFA, PPA, SPL) from the early network and 3 regions\n(ACC, premotor cortex, FEF) from the late network that constituent a brain network of 6 ROIs (Figure\n3A). We added premotor cortex (PMC) in our analysis because the task involved motor planning\nand execution. FFA and PPA were determined based on a separate functional localizer task for each\nsubject and they were included because of their selectivity in the early sensory processing of faces\nand houses. SPL, ACC and FEF have all been shown to involve at different stages of the perceptual\ndecision-making. The ROIs were determined based on a group-level EEG-informed fMRI analysis in\nthe standard space but were then transformed back into each subject\u2019s native anatomical space.\nStatistical inference For each subject, we \ufb01tted the model to each of the 4 runs separately. The\nestimated modulatory connectivity matrices corresponding to face and house were z-scored and\nthresholded according to their posterior probability. We then performed a two-tailed z-test on the\nmean z-scored connectivity values across 40 runs from 10 subjects. Signi\ufb01cance was determined at\np < 0.05 with Bonferroni correction to account for multiple comparisons across three connectivity\nmatrices. Signi\ufb01cant differences between face and house networks was determined using a paired t-\ntest on the z-scored connections at p < 0.05. Figure 3B shows the mean network connectivity pattern\nfor face and house stimuli, respectively. Since both positive connections and negative connections are\nmeaningful, we showed the absolute value of all signi\ufb01cant connections.\nWe consider the effective connectivity we infer with respect to differences between face stimuli\nand house stimuli. Faces and houses are object types often used in fMRI and EEG studies to study\nobject recognition and decision-making. Each of these stimulus categories is known to preferentially\nactivate different regions of the brain (FFA for faces and PPA for houses/places). These stimuli are\nalso interesting in this context since they are selected so that the organization of the features making\nup the objects overlaps (eyes and windows in same relative positions as are nose and door) and thus\ncan be challenging to discriminate in the presence of visual noise and rapid stimulus presentation.\nOur results show that effective connectivity differences are apparent, speci\ufb01cally we see an increase\nin effective connectivity when a house is presented relative to when the stimulus is a face. The\nspeci\ufb01c connections contributing to this difference are shown in Figure 3C. Interestingly, these\ndifferences involve connections with the ACC as well as the FEF and FFA, which are areas implicated\nin cognitive control, decision monitoring, attention and object recognition, especially of faces. The\nfact that the connections are more engaged for house stimuli suggests that there is more of a need to\nlink these areas when a house is presented relative to a face\u2013i.e this additional connectivity is required\nfor recognizing a house relative to a face. Previous work [21] showed that network connectivity is\nlikely a source of how bias effects toward faces are manifested in our choices. This current, though\npreliminary result, suggests that overcoming this bias requires additional network connectivity.\n\n4 Discussion\n\nLeveraging the complementary strengths of EEG and fMRI, we proposed a linear state-space model\nto estimate the effective connectivity between spatially segregated but functionally integrated brain\nregions. Speci\ufb01cally, we focused on the analysis of effective connectivity driven by various context-\ndependent inputs. We modeled the latent state variables as the mean source activity in each ROI\nand assumed that all source points belonging to one ROI are independent Gaussian variables with a\nshared variance and common ROI mean, similar to the model proposed by [7]. However, our model\nalso exploits the simultaneously recorded fMRI data to generate task-dependent ROIs speci\ufb01c to\n\n7\n\n\fFigure 3: Network connectivity patterns estimated from simultaneous EEG/fMRI data. A, Illustration\nof ROI locations. B, Mean network connectivity induced by the face and house stimuli. C, Mean\ndifference in directional connections between face and house. Blue line represents unidirectional\nconnection and yellow line represents bidirectional connection.\n\neach individual subject. Since the ROIs identi\ufb01ed by fMRI are much smaller and more localized than\nthose de\ufb01ned by an atlas on a standard brain, it was more reasonable to assume that all sources within\none ROI have similar activity. Moreover, important ROIs activated in the task were less likely to be\nneglected when fMRI information was available. Our simulation study further demonstrated that the\nestimation error largely increased even when a small number of spurious sources were included in\neach ROI. Together our results show that the high spatial speci\ufb01city provided by fMRI is critical to\nROI based connectivity analysis.\nOur model substantially differs from [7] in that it is designed to explain continuously evolving EEG\nrecordings as opposed to epoched EEG responses. Yang et al. [7] modeled the dynamic connectivity\non stimulus-locked evoked responses, a reasonable approach when one is interested in the effect\nspeci\ufb01c to a single class of stimulus. On the other hand, our approach allows one to incorporate\nmultiple exogenous covariates either as external or modulatory inputs in the dynamical system so\nthat one can investigate the causal effects of multiple experimental manipulations simultaneously.\nIn essence, the learned latent neural dynamics become a low dimensional representation of the\nobserved EEG dynamics. Our simulation showed that the algorithm could separate the connectivity\nmatrices induced by different stimuli, even when the sign of the intrinsic and modulatory connectivity\nwas opposite to each other. Since the number of parameters is large in this case, we used sparsity\nregularization via ARD prior to yield more interpretable results. Nevertheless, our model can also\nbe easily modi\ufb01ed to analyze connectivity for resting state experiments. Furthermore, since it is\noften dif\ufb01cult to acquire large samples of simultaneous EEG-fMRI data and the interpretability of the\nmodel is important, we chose a biophysically informed linear EEG forward model as opposed to a\ndeep-learning based approach [23, 24].\nOne limitation of our model is that we assumed the state noise covariance Qs to be diagonal. This is\noften not true in practice. We also assumed that the sources in the same ROI are independent Gaussian\ndistributed, i.e. Qx is diagonal. But these sources could be both spatially and temporally dependent.\nWith the available fMRI information, we can potentially design a more complex spatiotemporal\nstructure for Qx. In this work, we chose to optimize R directly. An alternative approach is to keep\nQy \ufb01xed and only optimize with respect to Qx. We did not compare the difference between the two\napproaches, but optimizing over R can be easily done via conjugacy.\nSince our model solves the ill-posed EEG inverse problem implicitly, we used information from\nfMRI as spatial prior to solve the EEG source localization using MNE, and obtained a reasonable\ninitial guess for our algorithm. Another limitation is that we assumed a \ufb01xed dipole orientation in\nthe lead \ufb01eld matrix and this orientation was estimated based on MNE. In future work, we plan to\ntreat the dipole orientation as unknown parameter over which to optimize. Finally, the temporally\ncontinuous nature of our estimation scheme provides an easy framework to incorporate fMRI time\nseries at each ROI so that temporal information from both EEG and fMRI can be used to infer the\nlatent neural dynamics. Our future work will investigate different choices of generative models for\nfMRI signals to better integrate with EEG.\n\nAcknowledgement\n\nThis work was supported by the Army Research Laboratory (Cooperative Agreement W911NF-10-2-\n0022) and the Army Research Of\ufb01ce (Grant W911NF-16-1-0507).\n\n8\n\n\fReferences\n[1] Karl J Friston. Functional and effective connectivity: a review. Brain Connectivity, 1(1):13\u201336, 2011.\n\n[2] Klaas Enno Stephan and Karl J Friston. Analyzing effective connectivity with functional magnetic\n\nresonance imaging. Wiley Interdisciplinary Reviews: Cognitive Science, 1(3):446\u2013459, 2010.\n\n[3] Karl J Friston, Lee Harrison, and Will Penny. Dynamic causal modelling. NeuroImage, 19(4):1273\u20131302,\n\n2003.\n\n[4] Bing Leung Patrick Cheung, Brady Alexander Riedner, Giulio Tononi, and Barry D Van Veen. Estimation\nof cortical connectivity from EEG using state-space models. IEEE Transactions on Biomedical Engineering,\n57(9):2122\u20132134, 2010.\n\n[5] Stefan Haufe, Ryota Tomioka, Guido Nolte, Klaus-Robert M\u00fcller, and Motoaki Kawanabe. Modeling\nsparse connectivity between underlying brain sources for EEG/MEG. IEEE transactions on Biomedical\nEngineering, 57(8):1954\u20131963, 2010.\n\n[6] Olivier David, Stefan J Kiebel, Lee M Harrison, J\u00e9r\u00e9mie Mattout, James M Kilner, and Karl J Friston.\nDynamic causal modeling of evoked responses in EEG and MEG. NeuroImage, 30(4):1255\u20131272, 2006.\n\n[7] Ying Yang, Elissa Aminoff, Michael Tarr, and Kass E Robert. A state-space model of cross-region dynamic\nconnectivity in MEG/EEG. In Advances in Neural Information Processing Systems, pages 1234\u20131242,\n2016.\n\n[8] Srikanth Ryali, Kaustubh Supekar, Tianwen Chen, and Vinod Menon. Multivariate dynamical systems\n\nmodels for estimating causal interactions in fMRI. NeuroImage, 54(2):807\u2013823, 2011.\n\n[9] Firdaus Janoos, Shantanu Singh, Raghu Machiraju, William M Wells, and Istvan A M\u00f3rocz. State\u2013space\nmodels of mental processes from fMRI. In Biennial International Conference on Information Processing\nin Medical Imaging, pages 588\u2013599. Springer, 2011.\n\n[10] Keum-Shik Hong and Hoang-Dung Nguyen. State-space models of impulse hemodynamic responses over\n\nmotor, somatosensory, and visual cortices. Biomedical Optics Express, 5(6):1778\u20131798, 2014.\n\n[11] Sunghee Dong and Jichai Jeong.\n\nImprovement in recovery of hemodynamic responses by extended\nkalman \ufb01lter with non-linear state-space model and short separation measurement. IEEE Transactions on\nBiomedical Engineering, 66(8):2152\u20132162, 2019.\n\n[12] Arthur K Liu, John W Belliveau, and Anders M Dale. Spatiotemporal imaging of human brain activity\nusing functional MRI constrained magnetoencephalography data: Monte carlo simulations. Proceedings of\nthe National Academy of Sciences, 95(15):8945\u20138950, 1998.\n\n[13] Karl Friston, Lee Harrison, Jean Daunizeau, Stefan Kiebel, Christophe Phillips, Nelson Trujillo-Barreto,\nRichard Henson, Guillaume Flandin, and J\u00e9r\u00e9mie Mattout. Multiple sparse priors for the M/EEG inverse\nproblem. NeuroImage, 39(3):1104\u20131120, 2008.\n\n[14] Anders M Dale, Arthur K Liu, Bruce R Fischl, Randy L Buckner, John W Belliveau, Jeffrey D Lewine,\nand Eric Halgren. Dynamic statistical parametric mapping: combining fMRI and MEG for high-resolution\nimaging of cortical activity. Neuron, 26(1):55\u201367, 2000.\n\n[15] Hans Hallez, Bart Vanrumste, Roberta Grech, Joseph Muscat, Wim De Clercq, Anneleen Vergult, Yves\nD\u2019Asseler, Kenneth P Camilleri, Simon G Fabri, Sabine Van Huffel, et al. Review on solving the forward\nproblem in EEG source analysis. Journal of Neuroengineering and Rehabilitation, 4(1):46, 2007.\n\n[16] Matti S H\u00e4m\u00e4l\u00e4inen and Risto J Ilmoniemi. Interpreting magnetic \ufb01elds of the brain: minimum norm\n\nestimates. Medical & Biological Engineering & Computing, 32(1):35\u201342, 1994.\n\n[17] Christopher M Bishop. Pattern Recognition and Machine Learning. Springer, 2006.\n\n[18] David P Wipf and Srikantan S Nagarajan. A new view of automatic relevance determination. In Advances\n\nin Neural Information Processing Systems, pages 1625\u20131632, 2008.\n\n[19] Jim E Grif\ufb01n, Philip J Brown. Inference with normal-gamma prior distributions in regression problems.\n\nBayesian Analysis, 5(1):171\u2013188, 2010.\n\n[20] Tom Leonard, John SJ Hsu. Bayesian inference for a covariance matrix. The Annals of Statistics,\n\n20(4):1669\u20131696, 1992.\n\n9\n\n\f[21] Tao Tu, Noam Schneck, Jordan Muraskin, and Paul Sajda. Network con\ufb01gurations in the human brain\n\nre\ufb02ect choice bias during rapid face processing. Journal of Neuroscience, 37(50):12226\u201312237, 2017.\n\n[22] Jordan Muraskin, Truman R Brown, Jennifer M Walz, Tao Tu, Bryan Conroy, Robin I Goldman, and Paul\nSajda. A multimodal encoding model applied to imaging decision-related neural cascades in the human\nbrain. NeuroImage, 180:211\u2013222, 2018.\n\n[23] Heung-Il Suk, Chong-Yaw Wee, Seong-Whan Lee, and Dinggang Shen. State-space model with deep\n\nlearning for functional dynamics estimation in resting-state fMRI. NeuroImage, 129:292\u2013307, 2016.\n\n[24] Yitong Li, Kafui Dzirasa, Lawrence Carin, David E Carlson. Targeting EEG/LFP synchrony with neural\n\nnets. In Advances in Neural Information Processing Systems, pages 4620\u20134630, 2017.\n\n10\n\n\f", "award": [], "sourceid": 2613, "authors": [{"given_name": "Tao", "family_name": "Tu", "institution": "Columbia University"}, {"given_name": "John", "family_name": "Paisley", "institution": "Columbia University"}, {"given_name": "Stefan", "family_name": "Haufe", "institution": "Charit\u00e9 \u2013 Universit\u00e4tsmedizin Berlin"}, {"given_name": "Paul", "family_name": "Sajda", "institution": "Columbia University"}]}