{"title": "Incorporating Context into Language Encoding Models for fMRI", "book": "Advances in Neural Information Processing Systems", "page_first": 6628, "page_last": 6637, "abstract": "Language encoding models help explain language processing in the human brain by learning functions that predict brain responses from the language stimuli that elicited them. Current word embedding-based approaches treat each stimulus word independently and thus ignore the influence of context on language understanding. In this work we instead build encoding models using rich contextual representations derived from an LSTM language model. Our models show a significant improvement in encoding performance relative to state-of-the-art embeddings in nearly every brain area. By varying the amount of context used in the models and providing the models with distorted context, we show that this improvement is due to a combination of better word embeddings learned by the LSTM language model and contextual information. We are also able to use our models to map context sensitivity across the cortex. These results suggest that LSTM language models learn high-level representations that are related to representations in the human brain.", "full_text": "Incorporating Context into Language Encoding\n\nModels for fMRI\n\nShailee Jain1 Alexander G Huth1,2\n\nDepartments of 1Computer Science & 2Neuroscience\n\n{shailee, huth}@cs.utexas.edu\n\nThe University of Texas at Austin\n\nAustin, TX 78751\n\nAbstract\n\nLanguage encoding models help explain language processing in the human\nbrain by learning functions that predict brain responses from the language\nstimuli that elicited them. Current word embedding-based approaches treat\neach stimulus word independently and thus ignore the in\ufb02uence of context on\nlanguage understanding. In this work, we instead build encoding models using rich\ncontextual representations derived from an LSTM language model. Our models\nshow a signi\ufb01cant improvement in encoding performance relative to state-of-the-art\nembeddings in nearly every brain area. By varying the amount of context used\nin the models and providing the models with distorted context, we show that\nthis improvement is due to a combination of better word embeddings learned by\nthe LSTM language model and contextual information. We are also able to use\nour models to map context sensitivity across the cortex. These results suggest\nthat LSTM language models learn high-level representations that are related to\nrepresentations in the human brain.\n\n1\n\nIntroduction\n\nTo extract meaning from natural speech, the human brain combines information about each word\nwith previous words, or context. Without context, humans would be unable to understand homonyms,\nparse phrases, or resolve coreferences. Contextual information must thus be represented in the\nhuman cortex. One powerful tool for mapping cortical representations is encoding models, which use\nfeatures extracted from stimuli to predict brain responses recorded with fMRI. Previous language\nencoding studies have successfully mapped word-level semantic representations using embedding\nvectors (15; 22; 9; 8), but this approach neglects the effect of context by assuming that the response to\neach word is independent. Overcoming this limitation would require extracting language features that\nincorporate context. However, no existing hand-designed feature space is well-suited for this task.\nSeveral recent studies have overcome the limitations of hand-designed feature spaces by instead\nusing representations discovered by supervised deep neural networks. These networks are trained to\nperform tasks that the human brain performs: recognize objects in visual scenes (2; 7; 5) or words and\nmusical genres in sounds (11). Stimuli from fMRI experiments are fed into the pretrained networks\nand activations from each layer are treated as separate feature spaces for encoding models. These\nrepresentations prove highly effective at modeling brain responses in the visual and auditory cortex,\nsuggesting that the networks discover representations similar to those in the brain. However, in\nboth cases, representations at both the lowest-level (images, sound spectrograms) and highest-level\n(image categories, words, music genres) are known a priori, and the networks are used to discover\nintermediate ones. In contrast, for language the highest-level representation is unknown, making this\nsupervised approach inapplicable.\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\fFigure 1: Contextual language encoding model with narrative stimuli. Each word in the story is\n\ufb01rst projected into a 985-dimensional embedding space. Sequences of word representations are then\nfed into an LSTM network that was pretrained as a language model. In this example, we extract\nrepresentations for each word in the stimulus from layer 2 of the LSTM with context length 1 (i.e.\nconsidering only one word before the current word). A low-pass Lanczos \ufb01lter is used to resample\nthe contextual representations to the low temporal resolution of fMRI, and ridge regression is used to\nmap downsampled contextual representations to fMRI responses. The dotted line indicates the path\nfollowed by the traditional, non-contextual approach that relies solely on individual word embeddings.\n\nAn alternative is to use self-supervised networks like neural language models (LMs). These models\nare trained to predict the next word in a sequence based on the previous words (4), typically using\nlong short-term memory (LSTM) networks (21). LSTM LMs have been shown to discover high-level\nsemantic and contextual representations that are useful for many natural language processing tasks\n(18; 13; 19). Further, a previous study suggests that representations learned by an LM can be useful\nfor modeling MEG brain responses to reading (23). Thus, it seems plausible that LSTM LMs could\ndiscover contextual representations that would be effective for understanding language processing in\nthe brain.\nAnother useful property of an LSTM LM is that it can generate many different types of representations\nthat can be used to probe diverse representations in the brain. Firstly, different brain areas are known\nto have different \u2018temporal receptive \ufb01elds\u2019 that indicate the amount of context they are sensitive to\n(12). This effect can be emulated in an LSTM by varying the number of words, or \u2018context length\u2019,\nused to generate representations. Secondly, LSTM LMs have multiple layers that can each capture\ndistinct representations. Some previous natural language processing works use only the last layer\n(18; 13) while others use a linear combination across layers (19). However, previous fMRI studies\nhave found that different layers from supervised networks predicted different brain areas (2; 7; 5; 11),\nsuggesting that each layer should be tested separately.\nIn this work, we make two broad contributions. Firstly, we use representations discovered by an\nLSTM LM to incorporate context into encoding models that predict fMRI responses to natural,\nnarrative speech (Figure 1). These contextual models perform signi\ufb01cantly better at predicting\nbrain responses than previously published word embedding models. Secondly, we compare the\neffectiveness of models that use different LSTM layers and context lengths. This reveals a hierarchy\nof brain areas that are sensitive to both different types of contextual information and different temporal\nreceptive \ufb01eld sizes. Our LSTM-based contextual encoding models thus not only outperform the best\nhand-designed feature spaces for language encoding, but also provide insights into the representation\nof linguistic context in the cortex.\n\n2 Natural Language fMRI Experiment\n\nWe use data from an fMRI experiment where the stimulus consisted of 11 naturally spoken narrative\nstories from The Moth Radio Hour, totalling over 2 hours or roughly 23,800 words. These data were\nused in previous publications and are described in detail there (9; 8). Each story is transcribed and\nthe transcripts are aligned to the audio, revealing the exact time when each word is spoken. These\nrich and complex stimuli are highly representative of the language that humans perceive on a daily\nbasis, and understanding these stimuli relies heavily on contextual information.\n\n2\n\n\fMeasured brain responses consist of whole-brain blood-oxygen level dependent (BOLD) signals\nrecorded from 6 subjects (2 female) using functional magnetic resonance imaging (fMRI), while they\nlistened to stimuli. Images were obtained using gradient-echo EPI on a 3T Siemens TIM Trio scanner\nat the UC Berkeley Brain Imaging Center using a 32-channel volume coil, TR = 2.0045 seconds\n(yielding about 4,000 timepoints for each subject), TE = 31 ms, \ufb02ip angle = 70 degrees, voxel size =\n2.24 \u00d7 2.24 \u00d7 4.1 mm (slice thickness = 3.5 mm with 18% slice gap), matrix size = 100 \u00d7 100, and\n30 axial slices. All experiments were approved by the UC Berkeley Committee for the Protection of\nHuman Subjects.\n\n3 Voxelwise encoding models\nEncoding models aim to approximate the function f (S) \u2192 R that maps a stimulus S to observed brain\nresponses R (16; 24; 10). For natural language, S is a continuous string of words w1, w2, . . . , wT .\nIn fMRI experiments, a separate encoding model \u02c6fj is typically estimated for each voxel based\non a training dataset, {Strn, Rtrn}. To evaluate model performance, the estimated model is used\nto predict responses in a separate testing dataset, \u02c6Rtest,j = \u02c6fj(Stest). Model performance for a\nsingle voxel is computed as the Pearson correlation coef\ufb01cient between real and predicted responses,\nrj = corr(Rtest,j, \u02c6Rtest,j).\nIn practice, the limited amount and quality of fMRI data cause generic nonlinear function\napproximators to perform poorly as encoding models. Instead, it is common to use linearized\nmodels, which assume that f is a nonlinear transformation of the stimuli into some feature space\nfollowed by a linear projection, f (S) := g(S)(T,P )\u03b2(P,V ), where g nonlinearly transforms S into\na P -dimensional feature space, and \u03b2 contains a separate set of P linear weights for each of the\nV voxels. The function g is typically chosen to extract stimulus features that are thought to be\nrepresented in the brain, and the weights \u03b2 are learned using regularized linear regression. Here,\nwe use ridge regression (L2-norm) to estimate weights for all encoding models, as described in a\nprevious publication (9).\nBOLD responses are low-pass relative to the stimuli that elicited them. To account for this effect\nwe resample and low-pass \ufb01lter the stimulus representation g(S)(T,P ) to the same rate as the fMRI\nacquisition with the use of a Lanczos \ufb01lter, yielding g(cid:48)(S)(T (cid:48),P ). Then, to account for hemodynamic\ndelay we use a \ufb01nite impulse response (FIR) model with 4 delays (2, 4, 6, and 8 seconds), similar to\nprevious work (9; 8).\n\n3.1 Word embeddings for language encoding\n\nThe key question when constructing an encoding model is how to choose the nonlinear transformation\ng that is used to represent the stimuli. The state-of-the-art language encoding model (9) uses a word\nembedding space that represents each word wi as a 985-dimensional vector ei. These representations\nare based on word co-occurrence statistics from a large corpus of English text. (Although not identical,\nthis embedding performs similarly to popular methods such as word2vec (14).) Word embedding\nspaces capture the semantic similarity between words, such that words with similar lexical meanings\nare represented by similar embedding vectors. Encoding models that use word embeddings have\nproven quite successful at predicting brain responses to both single words (15; 25) and continuous\nlanguage (22; 9; 8; 17). However, one critical \ufb02aw is that these models assume that the brain response\nto each word is independent of the other words in the stimulus. Additionally, in most experiments the\nwords are presented in isolation as opposed to a naturalistic narrative. These models treat the stimulus\nas a bag-of-words, ignoring temporal order and dependencies that are known to be signi\ufb01cant for\nlanguage processing in the brain (12). Our goal in this work is to \ufb01nd representations that bridge this\ngap by explicitly considering the order and dependency between words in a narrative.\n\n4 Learning representations of context\n\nLanguage models are trained to predict the next word in a sequence given information about previous\nwords, i.e. for some sequence [wi\u2212m, wi\u2212m+1 . . . wi, wi+1], they aim to maximize the probability\nP (wi+1|wi\u2212m . . . wi). To achieve this in an LSTM LM, representations of the words [wi\u2212m . . . wi]\nare effectively combined in a hidden state hi that can be used to predict wi+1. As a consequence,\n\n3\n\n\fhi must capture relevant contextual information (19) and might serve as a plausible candidate for\nmodeling contextual representations in the brain.\nLSTM LMs comprise LSTM cells arranged as a hierarchy of L layers. For the word wi, an LSTM\ncell at layer l takes two inputs: a representation of the current word from the previous layer (hi,l\u22121),\nand the state of the current LSTM cell after the previous word (hi\u22121,l). The cell then performs a\nnonlinear transformation on these inputs to create a new word representation (hi,l) that is both fed to\nthe next layer and used to modify the state of the current cell for the next word. Thus, the hidden\nstate for word wi at layer l can be de\ufb01ned as\n\n(cid:26)LST M (ei, hi\u22121,1) for l = 1\n\nLST M (hi,l\u22121, hi\u22121,l) otherwise,\n\nhi,l =\n\n(1)\n\nwhere ei is a pretrained embedding vector that is used as the input representation for wi, and LST M\ndenotes the nonlinear function applied by the LSTM cell on the current and previous words. Therefore,\nlayer 1 makes direct use of the embedding representation while subsequent layers increasingly depend\non recurrent states.\nBefore the LSTM can be used to extract contextual representations, it must be trained on the language\nmodeling task. Our LSTM LM is pretrained on a corpus of comments scraped from reddit.com\n(9), comprising over 20 million words. We chose this particular corpus because the informal and\nconversational nature of the text is more similar to our stimulus stories than most conventional corpora.\nThe vocabulary is restricted to include all the words in S as well as the top 10,000 most frequent\nwords from the corpus. This ensures that the network is aware of every word in S and can assign\nsome prediction probability to each. For the input representations ei, we use the 985-dimensional\nword embedding from the state-of-the-art encoding model (9). Our selected LSTM LM architecture\nhas L = 3 layers, each with 985-dimensional hidden states. This dimensionality was chosen to match\nthat of the input embedding so that every encoding model would have the same number of parameters.\nThe LM is trained on word sequences of length M = 20, as we observe little gain in perplexity with\nlonger sequences. The model is implemented in TensorFlow (1) with cross-entropy loss. We use the\nRMSProp optimizer (20) with an initial learning rate of 0.0005 and inverse time decay at 0.7 decay\nrate. Additionally, we implement early-stopping when the loss starts to plateau and no dropout. After\ntraining the LSTM LM, we evaluate its performance using S as a test set. Due to the limited amount\nof stories and their high diversity, we observe no bene\ufb01t of \ufb01ne-tuning the LM on a part of S and\nevaluating on the rest.\n\n4.1 Extracting contextual information\n\nOur LSTM LM uses information from up to M = 20 previous words to construct L = 3 separate\ncontext representations. By varying the number of words the LSTM has seen before the representation\nis extracted, we can produce representations with different \u2018temporal receptive \ufb01eld\u2019 sizes (12). And\nby extracting representations from different layers we may be able to capture different types of\nhigh-level contextual representations (19).\nWe extract M \u00d7 L = 60 separate representations of the stimuli, one for each context length and at\neach layer, by forward-propagating the stimuli through the pretrained LSTM LM. To obtain context\nrepresentations for a stimulus story S = [w1, . . . , wn] with a desired context length m \u2208 [0..M \u2212 1]\nand layer l \u2208 [1..L], we use the following procedure. First, for each word wi we extract the previous\nm words to form a length m + 1 sequence [wi\u2212m, . . . , wi]. The m context words and word wi are\nsequentially fed into the pretrained LSTM LM, and then the activations from layer l of the LSTM\nLM are extracted. The resulting 985-dimensional vector combines the information in wi with the\nprevious m words through recurrent connections, building an effective contextual representation for\nwi. This procedure thus allows us to incorporate and experiment with varying temporal \ufb01elds and\ncontextual information in language encoding.\n\n5 Results\n\nHere, we evaluate the performance of our contextual encoding models against the state-of-the-art\nembedding-based model and examine the effect of using different context layers and lengths. We test\nthe importance of context by distorting it and observing the effect on encoding performance. Finally\n\n4\n\n\fFigure 2: (a) Contextual encoding model performance with different context lengths and layers vs.\nstate-of-the-art embedding. Results are averaged across 6 subjects \u00b1 adjusted standard error of the\nmean. Contextual models from all layers outperform the embedding. Increasing context length\nuniformly improves performance in every layer, and different performance across layers suggests that\neach represents different information. Best performance is obtained using layer 2 with long context.\n(b) To show that contextual models do indeed rely on context, we re-\ufb01t the encoding models with\nrepresentations that used random words as context. Random-context models perform worse than the\noriginal models, and most perform worse than the embedding, except in layer 1. (c) Further, we swap\ncontext between different occurrences of each word and re-\ufb01t the encoding models. Swapped-context\nmodels also perform worse than the original, showing that linguistically plausible but wrong context\nalso hurts the model. (d) Performance of the LSTM network on the language modeling task, as a\nfunction of context length. LM performance is measured by inverse perplexity (higher is better).\nLonger context leads to better LM performance.\n\nwe examine which brain areas are best \ufb01t by different context lengths and layers, revealing a temporal\nhierarchy in the human language processing system. All brain maps are made using pycortex (6).\n\n5.1 Language encoding performance for context representations\n\nAfter \ufb01tting the 60 contextual encoding models (with M = 20 different context lengths and L = 3\ndifferent layers) and a baseline embedding model from (9), we evaluate model performance by\npredicting fMRI responses on a test dataset comprising one 10-minute story. Model performance in\nsv = rsv\u00d7|rsv|, the fraction of variance explained by the\nsubject s is summarized by \ufb01rst computing r2\nsv). Finally, to\nmodel in the vth voxel. Then, we sum across all voxels in subject s, giving r2\nsummarize model performance across the 6 subjects we average all the r2 values, r2\ns)/6.\nStandard error of the mean (SEM) is used to compute error bars.\nFigure 2(A) shows encoding model performance for each model, averaged across subjects. Contextual\nmodels signi\ufb01cantly beat the embedding in nearly every model variation. This supports our hypothesis\nthat contextual representations are a better match to the human brain than pure word embeddings.\nThis effect is also seen in Figure 3, which shows the difference in performance between the contextual\nand embedding models for every voxel in one subject. While some voxels show little difference, most\nregions are considerably improved with the contextual model.\n\nmean = \u03a3s(r2\n\ns = \u03a3v(r2\n\nImpact of context length. Figure 2(A) shows that model performance increases monotonically\nwith context length, for all layers. This suggests the encoding models are successfully exploiting\ncontextual information that the LSTM LM has learned to extract. However, model performance\nplateaus after 10-15 words, suggesting that the LSTM LM was unable to successfully incorporate\ncontextual information beyond this timescale.\nThis result is mirrored by the performance of the LSTM LM on the language modeling task,\nshown in Figure 2(C). Here, we test how well the LSTM LM predicts the next word from context\non the stimulus S. For each context length m we compute inverse perplexity, P P X\u22121(m) =\ni=1 log( P (wi|wi\u2212m\u22121,\u00b7\u00b7\u00b7 ,wi\u22121) ) / n, where higher values mean better LM performance. The\ne\u03a3n\nsimilarity between LM and encoding performance suggests that the ability of an LM to generate\n\n5\n\n012345678910111213141516171819Context Length500550600650700750800850900Encoding Model PerformanceLayer 1EmbeddingLayer 2Layer 3012345678910111213141516171819Context LengthLayer 1RandomLayer 2RandomLayer 3Random012345678910111213141516171819Context LengthLayer 1SwappedLayer 2SwappedLayer 3Swapped051015(a)(b)(c)(d)0.00400.00450.00500.00550.00600.00651/ppx\fFigure 3: Difference between contextual and embedding model performance across cortex. The\ndifference in r values is computed for each voxel in one subject (S1) and projected onto the subject\u2019s\n\ufb02attened cortical surface. White outlines show ROIs identi\ufb01ed using other experiments. Dashed lines\nindicate cuts made to the cortical surface during the \ufb02attening procedure and the green stars indicate\nthe apex of each cut. Red voxels are best modeled by the contextual model, blue are best modeled\nby the embedding, and white show no difference. Voxels poorly predicted by both models appear\nblack. Overall, the contextual model performs better, although some voxels have approximately equal\nperformance with both models.\n\nuseful encoding representations is tied to its language modeling ability. This suggests that better LMs\nyield better encoding models.\nThe encoding model results in Figure 2(A) also suggest that the improvement of our model can be\npartially explained by the LSTM LM simply learning a better word embedding than the baseline.\nModels with zero context length use no contextual information and thus are simply nonlinear\ntransformations of the input embedding. The model for layer 1 with zero context is substantially\nbetter than the embedding, suggesting that layer 1 learns to generate a new embedding that is a better\nmatch to the brain. However, the best model for each layer uses the longest context, suggesting that\ncontextual information is important above and beyond the improved embedding. We explore this\nissue further in Section 5.2.\n\nImpact of LSTM layers. Figure 2(A) also shows that there are differences between models that\nuse different LSTM layers. Further, the effect of context length is different for each layer. Layer 1\npredicts substantially better than baseline even with short context, but only improves modestly with\ncontext length. Layer 2 predicts roughly the same as baseline with zero context, but jumps sharply\nwith even one word of context and is the best model thereafter. Layer 3 predicts worse than the\nembedding with short context, but sees the largest improvement with context length. Investigating\nthe differences between representations in different layers is an important direction for future work.\n\n5.2 Distorting context representations\n\nFigure 2(A) shows that the LSTM LM produces useful representations for encoding. However, it also\nshows that part of the improvement is due to the LM learning better embeddings, as can be seen by\n\n6\n\n\fthe good performance of layer 1 with zero context. To directly test the importance of context in these\nmodels, we conduct two experiments where we degrade the quality of the context.\nIn the \u2018random\u2018 experiment, we modify the representation extraction procedure so that the context for\nword wi consists of a randomly chosen, nonsensical sequence of words rather than the words that\nactually precede wi in the stimulus. This yields a distorted representation for wi, where the amount\nof distortion scales with the context length. Encoding model performance using these distorted\nrepresentations is shown in Figure 2(B). It is obvious that the performance greatly suffers from the\ncontext distortion. Further, the impact on performance progressively increases across LSTM layers\nwith layer 1 being affected the least, and still beating the baseline. Since the representations beyond\nlayer 1 do not directly make use of the embedding and increasingly rely on contextual information\npassed through recurrence, the progressive impact across layers is justi\ufb01ed. This demonstrates that\nthe LSTM successfully incorporates context, although there are differences across layers.\nIn the \u2018swapping\u2019 experiment, for each stimulus word and context length we swap the actual context\nwith the context of another occurrence of the same word (e.g. \u201cthis is my dog\" \u2192 \u201cI saw the dog\").\nSwapping is done after inferring context vectors using the LSTM but before regression on the fMRI\ndata. For words that only appear once in the stimulus (< 6%), we leave the original context in place.\nFor each context length, we pick the new context that has the fewest words in common with the\noriginal. Thus, the swapped contexts picked for each successive length are sometimes different,\nleading to the increased variance across context lengths. The results in Figure 2(C) show that\nlinguistically plausible but wrong contexts also greatly impact encoding performance. Additionally,\nwe observe the same patterns of progressively greater impact across layers as in the random context\nmodels. This demonstrates even more effectively than before that the context model is picking up\ninformation relevant for cortical representations.\n\nFigure 4: Context length preference across cortex. An index of context length preference is computed\nfor each voxel in one subject (S1) and projected onto that subject\u2019s cortical surface. Voxels shown\nin blue are best modeled using short context, while red voxels are best modeled with long context.\nNon-signi\ufb01cantly predicted voxels (mean r < 0.11) are gray. Insets show model performance with\neach context length for two representative voxels, one that prefers short context (right) and one\nlong (left). Generally, voxels in low-level language areas (AC) prefer short context, while voxels in\nhigher-level language areas prefer long.\n\n5.3 Model preference across cortex\n\nFinally, we use our models to investigate the importance of contextual information for representations\nin each area of the cortex.\n\n7\n\n\fContext length preference across cortex. Although longer context produces better encoding\nmodels overall, the same is not true for every brain area. To investigate this issue we compute a\ncontext length preference index separately for each voxel. This index is de\ufb01ned as the projection\nof each voxel\u2019s \u2018context pro\ufb01le\u2019, a length-M vector containing prediction performance for each\ncontext length with LSTM layer 2, onto the \ufb01rst principal component of context pro\ufb01les across all\nwell-predicted voxels (r > 0.15). Layer 2 is used because it has the best performance overall; this\nanalysis was repeated for the other layers and yielded similar results. This index was selected because\nit accounts for the fact that performance plateaus for context lengths between 10 and 20, and it avoids\ncomputing a noisy argmax across context lengths.\nFigure 4 shows the context length preference for each voxel in one subject. Although most voxels\nprefer long context, clusters of short-context-preferring voxels appear in auditory cortex (AC),\nBroca\u2019s area, and left temporo-parietal junction (TPJ). These areas thus seem to selectively represent\ninformation about the current word rather than information that is built up with context. This supports\nthe \ufb01ndings of (12), which showed that similar brain areas respond strongly even when a story is\nscrambled to remove context. In a previous study, Baldassano et al. (3) report \ufb01ndings that suggest\ndifferent temporal scale selectivity across the cortex. Our contextual encoding model captures this\ntimescale variance, but goes further by explicitly modeling the transformation from stimulus to\nresponse.\n\nFigure 5: LSTM Layer preference across cortex. An index of layer preference is computed for each\nvoxel in one subject (S1) and then projected onto that subject\u2019s cortical surface. Voxels that are much\nbetter-predicted by layer 1 than the others appear red; layer 2, green; and layer 3, blue. Voxels equally\nwell-predicted by all layers appear white. Non-signi\ufb01cantly predicted voxels (mean r < 0.11) are\ngray. Insets show model performance with each layer (averaged across context lengths) for two\nrepresentative voxels, one that slightly prefers layers 1 & 3 (left), and one that slightly prefers layer 2\n(right). For most voxels there is little difference in performance from different layers. However, there\nis a slight preference in AC for layers 1 & 3, and in higher semantic regions for layer 2.\n\nLSTM layer preference across cortex. Earlier experiments found strong correspondence between\nlayers of deep supervised networks and brain areas in visual (7; 5) and auditory (11) cortex. It is\nunclear whether a similar pattern should be expected for self-supervised networks such as the LSTM\nLM used here, where there is no clear hierarchy of layers. To examine this issue we visualized\nlayer preference across cortex in Figure 5. Here a color is assigned to each signi\ufb01cantly predicted\nvoxel according to the relative encoding performance of each layer. Performance of the layer 1\nmodel is shown using the red component of the color, layer 2 using green, and layer 3 using blue.\nNon-signi\ufb01cantly predicted voxels appear gray, and voxels that are predicted equally well by each\nlayer appear white.\n\n8\n\n\fOverall the performance of all layers is highly similar, rendering most voxels nearly white. Still, layer\n2 provides the best predictions by a small margin, giving many voxels a green tint. However, auditory\ncortex (AC) shows a clear preference for layers 1 and 3, giving it a purple tint. Thus it seems that\nlow-level speech processing (AC) is better modeled by layers 1 and 3, while high-level processing\nis better modeled by layer 2. This suggests that the middle layer of the LSTM LM is learning the\nhighest-level representations, while layers 1 and 3, which are \u201ccloser\" to the word embeddings,\nare learning lower-level representations. This \ufb01nding is in stark contrast to experiments that used\nsupervised networks (7; 5; 11). Those experiments found that high-level cortical areas were best\nmodeled by higher layers of the network. Our results suggest that the highest-level representations in\nself-supervised models might emerge at the layer which is farthest from the input.\n\n6 Conclusions\n\nIn this work, we effectively incorporate context representations for language using an LSTM LM.\nWe observe that representations outperform state-of-the-art embedding based models and also show\ndistinct behavior across different context lengths and LSTM layers. Our \ufb01ndings suggest that these\nmodels do indeed incorporate context and temporal order, albeit differently across layers. Finally, we\nshow how our models explain the differences in language processing across different cortical regions,\nfrom low-level to high-level language areas.\nAs future work, we would like to explore and understand the information captured in the contextual\nrepresentations that helps to model language processing in the cortex. Additionally, it would be\nworth exploring the roles played by different LSTM gates in developing these representations while\nincorporating context.\n\nAcknowledgments\n\nWe thank Jack Gallant, Wendy de Heer, Frederic Theunissen, and Thomas Grif\ufb01ths for helping\ndesign the fMRI experiment and collecting the data used here; Brittany Grif\ufb01n and Anwar Nu\u00f1ez\nfor segmenting and \ufb02attening cortical surfaces; and Niko Kriegeskorte for useful discussions. Data\ncollection was supported by NSF grant IIS-1208203 and NIH NEI grant EY019684-01A1. This work\nwas supported by grants from the Burroughs-Wellcome Fund and NVIDIA. We also acknowledge the\nTexas Advanced Computing Center (TACC) at The University of Texas at Austin for providing HPC\nresources that have contributed to the research results reported within this paper.\n\nReferences\n[1] Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard,\nM., Kudlur, M., Levenberg, J., Monga, R., Moore, S., Murray, D.G., Steiner, B., Tucker, P., Vasudevan,\nV., Warden, P., Wicke, M., Yu, Y., Zheng, X.: Tensor\ufb02ow: A system for large-scale machine learning.\nIn: Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation. pp.\n265\u2013283. OSDI\u201916, USENIX Association, Berkeley, CA, USA (2016), http://dl.acm.org/citation.\ncfm?id=3026877.3026899\n\n[2] Agrawal, P., Stansbury, D., Malik, J., Gallant, J.L.: Pixels to voxels: Modeling visual representation in the\n\nhuman brain. CoRR abs/1407.5104 (2014)\n\n[3] Baldassano, C., Chen, J., Zadbood, A., Pillow, J.W., Hasson, U., Norman, K.A.: Discovering event\nstructure in continuous narrative perception and memory. Neuron 95(3), 709 \u2013 721.e5 (2017), http:\n//www.sciencedirect.com/science/article/pii/S0896627317305937\n\n[4] Bengio, Y., Ducharme, R., Vincent, P., Janvin, C.: A neural probabilistic language model. J. Mach. Learn.\n\nRes. 3, 1137\u20131155 (Mar 2003), http://dl.acm.org/citation.cfm?id=944919.944966\n\n[5] Eickenberg, M., Gramfort, A., Varoquaux, G., Thirion, B.: Seeing it all: Convolutional network layers\nmap the function of the human visual system. NeuroImage 152(Supplement C), 184 \u2013 194 (2017), http:\n//www.sciencedirect.com/science/article/pii/S1053811916305481\n\n[6] Gao, J.S., Huth, A.G., Lescroart, M.D., Gallant, J.L.: Pycortex: an interactive surface visualizer for fmri.\nFrontiers in Neuroinformatics 9, 23 (2015), https://www.frontiersin.org/article/10.3389/\nfninf.2015.00023\n\n9\n\n\f[7] G\u00fc\u00e7l\u00fc, U., van Gerven, M.A.J.: Deep neural networks reveal a gradient in the complexity of neural\nrepresentations across the ventral stream. Journal of Neuroscience 35(27), 10005\u201310014 (2015), http:\n//www.jneurosci.org/content/35/27/10005\n\n[8] de Heer, W.A., Huth, A.G., Grif\ufb01ths, T.L., Gallant, J.L., Theunissen, F.E.: The hierarchical cortical\norganization of human speech processing. Journal of Neuroscience (2017), http://www.jneurosci.\norg/content/early/2017/06/06/JNEUROSCI.3267-16.2017\n\n[9] Huth, A.G., de Heer, W.A., Grif\ufb01ths, T.L., Theunissen, F.E., Gallant, J.L.: Natural speech reveals the\nsemantic maps that tile human cerebral cortex. Nature 532(7600), 453\u2013458 (2016), https://doi.org/\n10.1038/nature17637\n\n[10] Kay, K., Naselaris, T., Prenger, R., Gallant, J.: Identifying natural images from human brain activity.\n\nNature 452(7185), 352\u2013355 (3 2008)\n\n[11] Kell, A.J., Yamins, D.L., Shook, E.N., Norman-Haignere, S.V., McDermott, J.H.: A task-optimized\nneural network replicates human auditory behavior, predicts brain responses, and reveals a cortical\nprocessing hierarchy. Neuron 98(3), 630 \u2013 644.e16 (2018), http://www.sciencedirect.com/\nscience/article/pii/S0896627318302502\n\n[12] Lerner, Y., Honey, C.J., Silbert, L.J., Hasson, U.: Topographic mapping of a hierarchy of temporal\nreceptive windows using a narrated story. Journal of Neuroscience 31(8), 2906\u20132915 (2011), http:\n//www.jneurosci.org/content/31/8/2906\n\n[13] McCann, B., Bradbury, J., Xiong, C., Socher, R.: Learned in translation: Contextualized word vectors.\n\nCoRR abs/1708.00107 (2017)\n\n[14] Mikolov, T., Sutskever,\n\nI., Chen, K., Corrado, G.S., Dean, J.: Distributed representations\nIn: Burges, C.J.C., Bottou, L., Welling,\nof words and phrases and their compositionality.\nM., Ghahramani, Z., Weinberger, K.Q.\nInformation Processing\nSystems 26, pp. 3111\u20133119. Curran Associates, Inc. (2013), http://papers.nips.cc/paper/\n5021-distributed-representations-of-words-and-phrases-and-their-compositionality.\npdf\n\n(eds.) Advances in Neural\n\n[15] Mitchell, T.M., Shinkareva, S.V., Carlson, A., Chang, K.M., Malave, V.L., Mason, R.A., Just, M.A.:\nPredicting human brain activity associated with the meanings of nouns. Science 320(5880), 1191\u20131195\n(2008), http://science.sciencemag.org/content/320/5880/1191\n\n[16] Naselaris, T., Kay, K., Nishimoto, S., Gallant, J.: Encoding and decoding in fmri 56, 400\u201310 (05 2011)\n\n[17] Pereira, F., Lou, B., Pritchett, B., Ritter, S., Gershman, S.J., Kanwisher, N., Botvinick, M., Fedorenko, E.:\nToward a universal decoder of linguistic meaning from brain activation. Nature communications 9(1), 963\n(2018)\n\n[18] Peters, M.E., Ammar, W., Bhagavatula, C., Power, R.: Semi-supervised sequence tagging with bidirectional\n\nlanguage models. In: ACL (1). pp. 1756\u20131765. Association for Computational Linguistics (2017)\n\n[19] Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., Zettlemoyer, L.: Deep\ncontextualized word representations. CoRR abs/1802.05365 (2018), http://arxiv.org/abs/1802.\n05365\n\n[20] Ruder, S.: An overview of gradient descent optimization algorithms. CoRR abs/1609.04747 (2016)\n\n[21] Sundermeyer, M., Schl\u00fcter, R., Ney, H.: Lstm neural networks for language modeling. In: Thirteenth\n\nAnnual Conference of the International Speech Communication Association (2012)\n\n[22] Wehbe, L., Murphy, B., Talukdar, P., Fyshe, A., Ramdas, A., Mitchell, T.: Simultaneously uncovering the\npatterns of brain regions involved in different story reading subprocesses. PLOS ONE 9(11), 1\u201319 (11\n2014), https://doi.org/10.1371/journal.pone.0112575\n\n[23] Wehbe, L., Vaswani, A., Knight, K., Mitchell, T.M.: Aligning context-based statistical models of language\n\nwith brain activity during reading. In: EMNLP (2014)\n\n[24] Wu, M.C.K., David, S.V., Gallant, J.L.: Complete functional characterization of sensory neurons by system\nidenti\ufb01cation. Annual Review of Neuroscience 29(1), 477\u2013505 (2006), https://doi.org/10.1146/\nannurev.neuro.29.051605.113024, pMID: 16776594\n\n[25] Xu, H., Murphy, B., Fyshe, A.: Brainbench: A brain-image test suite for distributional semantic models. In:\nProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. pp. 2017\u20132021\n(2016)\n\n10\n\n\f", "award": [], "sourceid": 3340, "authors": [{"given_name": "Shailee", "family_name": "Jain", "institution": "The University of Texas at Austin"}, {"given_name": "Alexander", "family_name": "Huth", "institution": "The University of Texas at Austin"}]}