{"title": "Visual Reference Resolution using Attention Memory for Visual Dialog", "book": "Advances in Neural Information Processing Systems", "page_first": 3719, "page_last": 3729, "abstract": "Visual dialog is a task of answering a series of inter-dependent questions given an input image, and often requires to resolve visual references among the questions. This problem is different from visual question answering (VQA), which relies on spatial attention ({\\em a.k.a. visual grounding}) estimated from an image and question pair. We propose a novel attention mechanism that exploits visual attentions in the past to resolve the current reference in the visual dialog scenario. The proposed model is equipped with an associative attention memory storing a sequence of previous (attention, key) pairs. From this memory, the model retrieves previous attention, taking into account recency, that is most relevant for the current question, in order to resolve potentially ambiguous reference(s). The model then merges the retrieved attention with the tentative one to obtain the final attention for the current question; specifically, we use dynamic parameter prediction to combine the two attentions conditioned on the question. Through extensive experiments on a new synthetic visual dialog dataset, we show that our model significantly outperforms the state-of-the-art (by ~16 % points) in the situation where the visual reference resolution plays an important role. Moreover, the proposed model presents superior performance (~2 % points improvement) in the Visual Dialog dataset, despite having significantly fewer parameters than the baselines.", "full_text": "Visual Reference Resolution using Attention Memory\n\nfor Visual Dialog\n\nPaul Hongsuck Seo\u2020\n\u2020POSTECH\n\nAndreas Lehrmann\u00a7\n\n{hsseo, bhhan}@postech.ac.kr {andreas.lehrmann, lsigal}@disneyresearch.com\n\nBohyung Han\u2020\n\u00a7Disney Research\n\nLeonid Sigal\u00a7\n\nAbstract\n\nVisual dialog is a task of answering a series of inter-dependent questions given an\ninput image, and often requires to resolve visual references among the questions.\nThis problem is different from visual question answering (VQA), which relies on\nspatial attention (a.k.a. visual grounding) estimated from an image and question\npair. We propose a novel attention mechanism that exploits visual attentions in the\npast to resolve the current reference in the visual dialog scenario. The proposed\nmodel is equipped with an associative attention memory storing a sequence of\nprevious (attention, key) pairs. From this memory, the model retrieves the previ-\nous attention, taking into account recency, which is most relevant for the current\nquestion, in order to resolve potentially ambiguous references. The model then\nmerges the retrieved attention with a tentative one to obtain the \ufb01nal attention for\nthe current question; speci\ufb01cally, we use dynamic parameter prediction to combine\nthe two attentions conditioned on the question. Through extensive experiments\non a new synthetic visual dialog dataset, we show that our model signi\ufb01cantly\noutperforms the state-of-the-art (by \u2248 16 % points) in situations, where visual ref-\nerence resolution plays an important role. Moreover, the proposed model achieves\nsuperior performance (\u2248 2 % points improvement) in the Visual Dialog dataset [1],\ndespite having signi\ufb01cantly fewer parameters than the baselines.\n\n1\n\nIntroduction\n\nIn recent years, advances in the design and optimization of deep neural network architectures have led\nto tremendous progress across many areas of computer vision (CV) and natural language processing\n(NLP). This progress, in turn, has enabled a variety of multi-modal applications spanning both\ndomains, including image captioning [2\u20134], language grounding [5, 6], image generation from\ncaptions [7, 8], and visual question answering (VQA) on images [9\u201321] and videos [22\u201324].\nThe VQA task, in particular, has received broad attention because its formulation requires a universal\nunderstanding of image content. Most state-of-the-art methods [10, 13, 15] address this inherently\nchallenging problem through an attention mechanism [3] that allows to visually ground linguistic\nexpressions; they identify the region of visual interest referred to by the question and predict the\nanswer based on the visual information in that region.\nMore recently, Visual Dialog [1] has been introduced as a generalization of the VQA task. Unlike\nVQA, where every question is asked independently, a visual dialog system needs to answer a sequence\nof questions about an input image. The sequential and inter-dependent property of questions in a\ndialog presents additional challenges. Consider the simple image and partial dialog in Figure 1. Some\nquestions (e.g., #1: \u2018How many 9\u2019s are there in the image?\u2019) contain the full information needed to\nattend to the regions within the image and answer the question accurately. Other questions (e.g., #6:\n\u2018What is the number of the blue digit?\u2019) are ambiguous on their own and require knowledge obtained\nfrom the prior questions (1, 2, 3, and 5 in particular) in order to resolve attention to the speci\ufb01c region\n\n31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.\n\n\f# Question\n1 How many 9\u2019s are there in the image?\n2 How many brown digits are there among them?\n3 What is the background color of the digit at the left of it?\n4 What is the style of the digit?\n5 What is the color of the digit at the left of it?\n6 What is the number of the blue digit?\n7 Are there other blue digits?\n\nAnswer\n\nfour\none\nwhite\n\ufb02at\nblue\n\n4\ntwo\n\nFigure 1: Example from MNIST Dialog. Each pair consists of an image (left) and a set of sequential\nquestions with answers (right).\n\nthe expression (\u2018the blue digit\u2019) is referring to. This process of visual reference resolution1 is the key\ncomponent required to localize attention accurately in the presence of ambiguous expressions and\nthus plays a crucial role in extending VQA approaches to the visual dialog task.\nWe perform visual reference resolution relying on a novel attention mechanism that employs an\nassociative memory to obtain a visual reference for an ambiguous expression. The proposed model\nutilizes two types of intermediate attentions: tentative and retrieved ones. The tentative attention is\ncalculated solely based on the current question (and, optionally, the dialog history), and is capable of\nfocusing on an appropriate region when the question is unambiguous. The retrieved attention, used\nfor visual reference resolution, is the most relevant previous attention available in the associative\nmemory. The \ufb01nal attention for the current question is obtained by combining the two attention maps\nconditioned on the question; this is similar to neural module networks [12, 14], which dynamically\ncombine discrete attention modules, based on a question, to produce the \ufb01nal attention. For this task,\nour model adopts a dynamic parameter layer [9] that allows us to work with continuous space of\ndynamic parametrizations, as opposed to the discrete set of parametrizations in [12, 14].\n\nContributions We make the following contributions. (1) We introduce a novel attention process\nthat, in addition to direct attention, resolves visual references by modeling the sequential dependency\nof the current question on previous attentions through an associative attention memory; (2) We\nperform a comprehensive analysis of the capacity of our model for the visual reference resolution task\nusing a synthetic visual dialog dataset (MNIST dialog) and obtain superior performance compared to\nall baseline models. (3) We test the proposed model in a visual dialog benchmark (VisDial [1]) and\nshow state-of-the-art performance with signi\ufb01cantly fewer parameters.\n\n2 Related Work\n\nVisual Dialog Visual dialogs were recently proposed in [1] and [25], focusing on different aspects\nof a dialog. While the conversations in the former contain free-form questions about arbitrary objects,\nthe dialogs in the latter aim at object discovery through a series of yes/no questions. Reinforcement\nlearning (RL) techniques were built upon those works in [26] and [27]. Das et al. [26] train two agents\nby playing image guessing games and show that they establish their own communication protocol\nand style of speech. In [27], RL is directly used to improve the performance of agents in terms of the\ntask completion rate of goal-oriented dialogs. However, the importance of previous references has\nnot yet been explored in the visual dialog task.\n\nAttention for Visual Reference Resolution While visual dialog is a recent task, VQA has been\nstudied extensively and attention models have been known to be bene\ufb01cial for answering independent\nquestions [10\u201316]. However, none of those methods incorporate visual reference resolution, which is\nneither necessary nor possible in VQA but essential in visual dialog. Beyond VQA, attention models\nare used to \ufb01nd visual groundings of linguistic expressions in a variety of other multi-modal tasks,\nsuch as image captioning [3, 4], VQA in videos [22], and visual attributes prediction [28]. Common\nto most of these works, an attention is obtained from a single embedding of all linguistic inputs.\nInstead, we propose a model that embeds each question in a dialog separately and calculates the\ncurrent question\u2019s attention by resolving its sequential dependencies through an attention memory and\na dynamic attention combination process. We calculate an attention through a dynamic composition\n\n1We coin this term by borrowing nomenclature, partially, from NLP, where coreference resolution attempts\nto solve the corresponding problem in language; the visual in visual reference resolution implies that we want to\ndo both resolve and visually ground the reference used in the question.\n\n2\n\n\fFigure 2: Architecture of the proposed network. The gray box represents the proposed attention\nprocess. Refer to Section 3 for the detailed description about individual modules (a)-(f).\n\nprocess taking advantage of a question\u2019s semantic structure, which is similar to [12] and [14].\nHowever, the proposed method still differs in that our attention process is designed to deal with\nambiguous expressions in dialogs by dynamically analyzing the dependencies of questions at each\ntime step. In contrast, [12] and [14] obtain the attention for a question based on its compositional\nsemantics that is completely given at the time of the network structure prediction.\n\nMemory for Question Answering Another line of closely related works is the use of a memory\ncomponent to question answering models. Memory networks with end-to-end training are \ufb01rst\nintroduced in [29], extending the original memory network [30]. The memories in these works are\nused to store some factoids in a given story and the supporting facts for answering questions are\nselectively retrieved through memory addressing. A memory network with an episodic memory was\nproposed in [31] and applied to VQA by storing the features at different locations of the memory [32].\nWhile these memories use the contents themselves for addressing, [33] proposes associative memories\nthat have a key-value pair at each entry and use the keys for addressing the value to be retrieved.\nFinally, the memory component is also utilized for visual dialog in [1] to actively select the previous\nquestion in the history. Memories in these previous memory networks store given factoids to retrieve\na supporting fact. In contrast, our attention memory stores previous attentions, which represent\ngrounded references for previous questions, to resolve the current reference based on the sequential\ndependency of the referring expressions. Moreover, we adopt an associative memory to use the\nsemantics of QA pairs for addressing.\n\n3 Visual Dialog Model with Attention Memory-based Reference Resolution\n\nVisual dialog is the task of building an agent capable of answering a sequence of questions presented\nin the form of a dialog. Formally, we need to predict an answer yt \u2208 Y, where Y is a set of discrete\nanswers or a set of natural language phrases/sentences, at time t given input image I, current question\nqt, and dialog history H = {h\u03c4| h\u03c4 = (q\u03c4 , y\u03c4 ) , 0 \u2264 \u03c4 < t}.\nWe utilize the encoder-decoder architecture recently introduced in [1], which is illustrated in Figure 2.\nSpeci\ufb01cally, we represent a triplet (q, H, I) with et by applying three different encoders, based on\nrecurrent (RNN with long-short term memory units), hierarchical recurrent (HRNN)2 and convolu-\ntional (CNN) neural networks, followed by attention and fusion units (Figure 2 (a)-(f)). Our model\nthen decodes the answer yt from the encoded representation et (Figure 2 (g)). Note that, to obtain the\nencoded representation et, the CNN image feature map f computed from I undergoes a soft spatial\nattention process guided by the combination of qt and H as follows:\n\nct = fc(RNN(qt), HRNN(H))\nt = [\u03b1t(ct)](cid:62) \u00b7 f =\nf att\n\nN(cid:88)\n\n\u03b1t,n(ct) \u00b7 fn,\n\n(1)\n\n(2)\n\nn=1\n\nwhere fc (Figure 2 (d)) denotes a fully connected layer, \u03b1n(ct) is the attention map conditioned on a\nfused encoding of qt and H, n is the location index in the feature map, and N is the size of the spatial\ngrid of the feature map. This attention mechanism is the critical component that allows the decoder\nto focus on relevant regions of the input image; it is also the main focus of this paper.\n\n2The questions and the answers of a history are independently embedded using LSTMs and then fused by a\nfc layer with concatenation to form QA encodings. The fused QA embedding at each time step is \ufb01nally fed to\nanother LSTM and the \ufb01nal output is used for the history encoding.\n\n3\n\n(a) RNNquestion\ud835\udc5e(d) fc\ud835\udc84\ud835\udc61(b) HRNNhistory\ud835\udc3b(c) CNNimage \ud835\udc3c(e) attentionprocess\ud835\udc87attention memory\ud835\udc87\ud835\udc61att\ud835\udc8c\ud835\udc61mem\ud835\udf36\ud835\udc61\ud835\udc84\ud835\udc61(f) fc(g) answer decoder\ud835\udc86\ud835\udc61(h) keygeneration\ud835\udc66\ud835\udc61\ud835\udc84\ud835\udc61\ud835\udc8c\ud835\udc61\f(a) Dynamic combination of attentions\n\n(b) Attention retrieval from memory\nFigure 3: Attention process for visual dialog task. (a) The tentative and relevant attentions are \ufb01rst\nobtained independently and then dynamically combined depending on the question embedding. (b)\nTwo boxes represent memory containing attentions and corresponding keys. Question embedding ct\nis projected by W mem and compared with keys using inner products, denoted by crossed circles, to\ngenerate address vector \u03b2t. The address vector is then used as weights for computing a weighted\n).\naverage of all memory entries, denoted by \u03a3 within circle, to retrieve memory entry (\u03b1mem\n\n, kmem\n\nt\n\nt\n\nt\n\nWe make the observation that, for certain questions, attention can be resolved directly from ct. This\nis called tentative attention and denoted by \u03b1tent\n. This works well for questions like #1 in Figure 1,\nwhich are free from dialog referencing. For other questions like #6, resolving reference linguistically\nwould be dif\ufb01cult (e.g., linguistic resolution may look like: \u2018What number of the digit to the left\nto the left of the brown 9\u2019). That said, #6 is straightforward to answer if the attention utilized to\nanswer #5 is retrieved. This process of visual reference resolution gives rise to attention retrieval\nfrom the memory. The \ufb01nal attention \u03b1t(ct) is computed using dynamic parameter layer,\n\u03b1mem\nwhere the parameters are conditioned on ct. To summarize, an attention is composed of three steps in\nthe proposed model: tentative attention, relevant attention retrieval, and dynamic attention fusion as\nillustrated in Figure 3a. We describe the details of each step below.\n\nt\n\n3.1 Tentative Attention\n\nst,n =(cid:0)Wtent\n\n(cid:1)(cid:62)(cid:0)Wtent\n\n(cid:1)\n\nWe calculate the tentative attention by computing similarity, in the joint embedding space, of the\nencoding of the question and history, ct, and each feature vector, fn, in the image feature grid f:\n\nc\n\nct\n\n(3)\n(4)\nare projection matrices for the question and history encoding and the image\n\nt = softmax ({st,n, 1 < n < N}) ,\n\u03b1tent\n\nf fn\n\nwhere Wtent\nfeature vector, respectively, and st,n is an attention score for a feature at the spatial location n.\n\nand Wtent\n\nf\n\nc\n\n3.2 Relevant Attention Retrieval from Attention Memory\n\nAs a reminder, in addition to the tentative attention, our model obtains the most relevant previous\nattention using an attention memory for visual reference resolution.\n\nAssociative Attention Memory The proposed model is equipped with an associative mem-\nory, called an attention memory, to store previous attentions. The attention memory Mt =\n{(\u03b10, k0) , (\u03b11, k1) , . . . , (\u03b1t\u22121, kt\u22121)} stores all the previous attention maps \u03b1\u03c4 with their corre-\nsponding keys k\u03c4 for associative addressing. Note that \u03b10 is NULL attention and set to all zeros. The\nNULL attention can be used when no previous attention reference is required for the current reference\nresolution.\n\n4\n\n\u2026\u2026attention retrieval\ud835\udf36\ud835\udf0f\ud835\udc8c\ud835\udf0f\ud835\udc84\ud835\udc61\ud835\udf36\ud835\udc61mem\ud835\udc8c\ud835\udc61memtentativeattention\ud835\udc87\ud835\udf36\ud835\udc61tent\ud835\udc7eDPL\ud835\udc84\ud835\udc61\ud835\udf36\ud835\udc61dynamic combination\ud835\udc84\ud835\udc61\u2026\ud835\udc8c0\ud835\udc8c1\ud835\udc8c\ud835\udc61\u22121\u2026\u2026\ud835\udf360\ud835\udf361\ud835\udf36\ud835\udc61\u22121\ud835\udf37\ud835\udc61\u2211\u2211\ud835\udc7emem\ud835\udc84\ud835\udc61\ud835\udf36\ud835\udc61mem\ud835\udc8c\ud835\udc61memkeysattentions\fThe most relevant previous attention is retrieved based on the key comparison as illustrated in\nFigure 3b. Formally, the proposed model addresses the memory given the embedding of the current\nquestion and history ct using\n\n(cid:62)\nmt,\u03c4 = (W memct)\n\n(5)\nwhere W mem projects the question and history encoding onto the semantic space of the memory\nkeys. The relevant attention \u03b1mem\nare then retrieved from the attention memory using\nthe computed addressing vector \u03b2t by\n\nand \u03b2t = softmax ({mt,\u03c4 , 0 < \u03c4 < t \u2212 1}) ,\n\nand key kmem\n\nk\u03c4\n\nt\n\nt\n\nt\u22121(cid:88)\n\nt\u22121(cid:88)\n\n\u03b1mem\n\nt\n\n=\n\n\u03b2t,\u03c4 \u03b1\u03c4\n\nand\n\nkmem\n\nt\n\n=\n\n\u03b2t,\u03c4 k\u03c4 .\n\n(6)\n\n\u03c4 =0\n\n\u03c4 =0\n\nThis relevant attention retrieval allows the proposed model to resolve the visual reference by indirectly\nresolving coreferences [34\u201336] through the memory addressing process.\nIncorporating Sequential Dialog Structure While the associative addressing is effective in re-\ntrieving the most relative attention based on the question semantics, we can improve the performance\nby incorporating sequential structure of the questions in a dialog. Considering that more recent\nattentions are more likely to be referred again, we add an extra term to Eq. (5) that allows preference\nk\u03c4 + \u03b8 (t \u2212 \u03c4 ) where \u03b8 is a learnable parameter\nfor sequential addressing, i.e., m(cid:48)\nweighting the relative time distance (t \u2212 \u03c4 ) from the current time step.\n\n(cid:62)\nt,\u03c4 = (W memct)\n\n3.3 Dynamic Attention Combination\n\nt\n\nAfter obtaining both attentions, the proposed model combines them. The two attention maps \u03b1tent\nand \u03b1mem\nare \ufb01rst stacked and fed to a convolution layer to locally combine the attentions. After\ngenerating the locally combined attention features, it is \ufb02attened and fed to a fully connected (fc)\nlayer with softmax generating the \ufb01nal attention map. However, a fc layer with \ufb01xed weights would\nalways result in the same type of combination although the merging process should, as we argued\npreviously, depend on the question. Therefore, we adopt the dynamic parameter layer introduced\nin [9] to adapt the weights of the fc layer conditioned on the question at test time. Formally, the \ufb01nal\nattention map \u03b1t(ct) for time t is obtained by\n\nt\n\n\u03b1t(ct) = softmax(cid:0)W DPL (ct) \u00b7 \u03b3(\u03b1tent\n\nt\nwhere W DPL(ct) are the dynamically determined weights and \u03b3(\u03b1tent\n) is the \ufb02attened output\nof the convolution obtained from the stacked attention maps. As in [9], we use a hashing technique to\npredict the dynamic parameters without explosive increase of network size.\n\n, \u03b1mem\n\nt\n\nt\n\nt\n\n)(cid:1) ,\n\n, \u03b1mem\n\n(7)\n\n3.4 Additional Components and Implementation\n\nIn addition to the attended image feature, we \ufb01nd other information useful for answering the question.\nTherefore, for the \ufb01nal encoding et at time step t, we fuse the attended image feature embedding f att\nwith the context embedding ct, the attention map \u03b1t and the retrieved key kmem\nfrom the memory,\nby a fc layer after concatenation (Figure 2f).\nFinally, when we described the associative memory in Section 3, we did not specify the memory key\ngeneration procedure. In particular, after answering the current question, we append the computed\nattention map to the memory. When storing the current attention into memory, the proposed model\ngenerates a key kt by fusing the context embedding ct with the current answer embedding at through\na fc layer (Figure 2h). Note that an answer embedding at is obtained using LSTM.\nLearning Since all the modules of the proposed network are fully differentiable, the entire network\ncan be trained end-to-end by standard gradient-based learning algorithms.\n\nt\n\nt\n\n4 Experiments\nWe conduct two sets of experiments to verify the proposed model. To highlight the model\u2019s ability\nto resolve visual references, we \ufb01rst perform experiment with a synthetic dataset that is explicitly\ndesigned to contain ambiguous expressions and strong inter-dependency among questions in the\nvisual dialog. We then show that the model also works well in the real VisDial [1] benchmark.\n\n5\n\n\fBasemodel\n\nI\nQ\n\nLF [1]\nHRE [1]\nMN [1]\n\nATT\n\nAMEM\n\n+H +SEQ\n\u2013\n\u2013\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n\u2013\n(cid:88)\n\u2013\n(cid:88)\n\u2013\n(cid:88)\n\n\u2013\n\u2013\n\u2013\n\u2013\n\u2013\n\u2013\n\u2013\n\u2013\n\u2013\n\u2013\n(cid:88)\n(cid:88)\n\nAccuracy\n\n20.18\n36.58\n37.58\n45.06\n49.10\n48.51\n62.62\n79.72\n87.53\n89.20\n90.05\n96.39\n\nFigure 4: Results on MNIST Dialog. Answer prediction accuracy [%] of all models for all questions\n(left) and accuracy curves of four models at different dialog steps (right). +H and +SEQ represent the\nuse of history embeddings in models and addressing with sequential preference, respectively.\n\n4.1 MNIST Dialog Dataset\n\nExperimental Setting We create a synthetic dataset, called MNIST Dialog3, which is designed for\nthe analysis of models in the task of visual reference resolution with ambiguous expressions. Each\nimage in MNIST Dialog contains a 4 \u00d7 4 grid of MNIST digits and each MNIST digit in the grid\nhas four randomly sampled attributes, i.e., color = {red, blue, green, purple, brown}, bgcolor =\n{cyan, yellow, white, silver, salmon}, number = {x|0 \u2264 x \u2264 9} and style = {\ufb02at, stroke}, as\nillustrated in Figure 1. Given the generated image from MNIST Dialog, we automatically generate\nquestions and answers about a subset of the digits in the grid that focus on visual reference resolution.\nThere are two types of questions: (i) counting questions and (ii) attribute questions that refer to a\nsingle target digit. During question generation, the target digits for a question is selected based on a\nsubset of the previous targets referred to by ambiguous expressions, as shown in Figure 1. For ease\nof evaluation, we generate a single word answer rather than a sentence for each question and there\nare a total of 38 possible answers ( 1\n38 chance performance). We generated 30K / 10K / 10K images\nfor training / validation / testing, respectively, and three ten-question dialogs for each image.\nThe dimensionality of the word embedding and the hidden state in the LSTMs are set to 32 and 64,\nrespectively. All LSTMs are single-layered. Since answers are single words, the answer embedding\nRNN is replaced with a word embedding layer in both the history embedding module and the memory\nkey generation module. The image feature extraction module is formed by stacking four 3 \u00d7 3\nconvolutional layers with a subsequent 2\u00d7 2 pooling layer. The \ufb01rst two convolutional layers have 32\nchannels, while there are 64 channels in the last two. Finally, we use 512 weight candidates to hash the\ndynamic parameters of the attention combination process. The entire network is trained end-to-end\nby minimizing the cross entropy of the predicted answer distribution at every step of the dialogs.\nWe compare our model (AMEM) with three different groups of baselines. The simple baselines show\nthe results of using statistical priors, where answers are obtained using image (I) or question (Q)\nonly. We also implement the late fusion model (LF), the hierarchical recurrent encoder with attention\n(HREA) and the memory network encoder (MN) introduced in [1]. Additionally, an attention-based\nmodel (ATT), which directly uses tentative attention, without memory access, is implemented as a\nstrong baseline. For some models, two variants are implemented: one using history embeddings and\nthe other one not. These variations give us insights on the effect of using history contexts and are\ndistinguished by +H. Finally, another two versions of the proposed model, orthogonal to the previous\nones, are implemented with and without the sequential preference in memory addressing (see above),\nwhich is denoted by +SEQ.\nResults Figure 4 shows the results on MNIST Dialog. The answer prediction accuracy over all\nquestions of dialogs is presented in the table on the left. It is noticeable that the models using attention\nmechanisms (AMEM and ATT) signi\ufb01cantly outperform the previous baseline models (LF, HRE and\nMN) introduced in [1], while these baselines still perform better than the simple baseline models. This\nsigni\ufb01es the importance of attention in answering questions, consistent with previous works [10\u201314].\n\n3The dataset is available at http://cvlab.postech.ac.kr/research/attmem\n\n6\n\n246810dialog step ID0.60.70.80.91.0accuracyATTATT+HAMEMAMEM+H+SEQ\fFigure 5: Memory addressing coef\ufb01cients with and without sequential preference. Both models\nput large weights on recent elements (smaller relative time difference) to deal with the sequential\nstructure of dialogs.\n\nFigure 6: Characteristics of dynamically predicted weights for attention combination. Dynamic\nweights are computed from 1,500 random samples at dialog step 3 and plotted by t-SNE. Each \ufb01gure\npresents clusters formed by different semantics of questions. (left) Clusters generated by different\nquestion types. (middle) Subclusters formed by types of spatial relationships in attribute questions.\n(right) Subclusters formed by ways of specifying targets in counting questions; cluster sub_targets\ncontains questions whose current target digits are included in the targets of the previous question.\n\nExtending ATT to incorporate history embeddings during attention map estimation increases the\naccuracy by about 17%, resulting in a strong baseline model.\nHowever, even the simplest version of the proposed model, which does not use history embeddings or\naddressing with sequential preference, already outperforms the strong baseline by a large margin. Note\nthat this model still has indirect access to the history through the attention memory, although it does\nnot have direct access to the encodings of past question/answer pairs when computing the attention.\nThis signi\ufb01es that the use of the attention memory is more helpful in resolving the current reference\n(and computing attention), compared to a method that uses more traditional tentative attention\ninformed by the history encoding. Moreover, the proposed model with history embeddings further\nincreases the accuracy by 1.7%. The proposed model reaches >96% accuracy when the sequential\nstructure of dialogs is taken into account by the sequential preference in memory addressing.\nWe also present the accuracies of the answers at each dialog step for four models that use attentions in\nFigure 4 (right). Notably, the accuracy of ATT drops very fast as the dialog progresses and reference\nresolution is needed. Adding history embeddings to the tentative attention calculation somewhat\nreduces the degradation. The use of the attention memory gives a very signi\ufb01cant improvement,\nparticularly at later steps in the dialog when complex reference resolution is needed.\n\nParameter Analysis When we observed the learned parameter \u03b8 for the sequential preference, it is\nconsistently negative in all experiments; it means that all models prefer recent elements. A closer\nlook at the addressing coef\ufb01cients \u03b2t with and without the sequential preference reveals that both\nvariants have a clear preferences for recent elements, as depicted in Figure 5. It is interesting that the\ncase without the bias term shows a stronger preference for recent information, but its \ufb01nal accuracy is\nlower than the version with the bias term. It seems that W mem without bias puts too much weight on\nrecent elements, resulting in worse performance. Based on this observation, we learn W mem and \u03b8\njointly to \ufb01nd better coef\ufb01cients than W mem alone.\nThe dynamically predicted weights form clusters with respect to the semantics of the input questions\nas illustrated in Figure 6, where 1,500 random samples at step 3 of dialogs are visualized using t-SNE.\nIn Figure 6 (left), the two question types (attribute and counting) create distinct clusters. Each of\n\n7\n\n246810relative time distance (t)0.00.10.20.30.4coefficient tAMEMAMEM+SEQ402002040402002040attributecounting402002040402002040no_relationleftrightbelowabove402002040402002040targetssub_targets\fFigure 7: Qualitative analysis on MNIST Dialog. Given an input image and a series of questions\nwith their visual grounding history, we present the memory retrieved and \ufb01nal attentions for the current\nquestion in the second and third columns, respectively. The proposed network correctly attends to\ntarget reference and predicts correct answer. The last two columns present the manually modi\ufb01ed\nattention and the \ufb01nal attention obtained from the modi\ufb01ed attention, respectively. Experiment shows\nconsistency of transformation between attentions and semantic interpretability of our model.\n\nthese, in turn, contains multiple sub-clusters formed by other semantics, as presented in Figure 6\n(middle) and (right). In the cluster of attribute questions, sub-clusters are mainly made by types of\nspatial relationship used to specify the target digit (e.g., #3 in Figure 1), whereas sub-clusters in\ncounting questions are based on whether the target digits of the question are selected from the targets\nof the previous question or not (e.g., #1 vs. #2 in Figure 1).\nFigure 7 illustrates qualitative results. Based on the history of attentions stored in the attention\nmemory, the proposed model retrieves the previous reference as presented in the second column.\nThe \ufb01nal attention for the current question is then calculated by manipulating the retrieved attention\nbased on the current question. For example, the current question in Figure 7 refers to the right\ndigit of the previous reference, and the model identi\ufb01es the target reference successfully (column 3)\nas the previous reference (column 2) is given accurately by the retrieved attention. To investigate\nconsistency with respect to attention manipulation, we move the region of the retrieved attention\nmanually (column 4) and observe the \ufb01nal attention map calculated from the modi\ufb01ed attention\n(column 5). It is clear that our reference resolution procedure works consistently even with the\nmanipulated attention and responds to the question accordingly. This shows a level of semantic\ninterpretability of our model. See more qualitative results in Section A of our supplementary material.\n\n4.2 Visual Dialog (VisDial) Dataset\nExperimental Setting\nIn the VisDial [1] dataset4, the dialogs are collected from MS-COCO [37]\nimages and their captions. Each dialog is composed of an image, a caption, and a sequence of ten\nQA pairs. Unlike in MNIST Dialog, answers to questions in VisDial are in free form text. Since\neach dialog always starts with an initial caption annotated in MS-COCO, the initial history is always\nconstructed using the caption. The dataset provides 100 answer candidates for each question and\naccuracy of a question is measured by the rank of the matching ground-truth answer. Note that this\ndataset is less focused on visual reference resolution and contains fewer ambiguous expressions\ncompared to MNIST Dialog. We estimate the portion of questions containing ambiguous expressions\nto be 94% and 52% in MNIST Dial and VisDial, respectively5.\nWhile we compare our model with various encoders introduced in [1], we \ufb01x the decoder to a\ndiscriminative decoder that directly ranks the answer candidates through their embeddings. Our\nbaselines include three visual dialog models, i.e., late fusion model (LF), hierarchical recurrent\nencoder (HRE) and memory network encoder (MN), and two attention based VQA models (SAN and\n\n4We use recently released VisDial v0.9 with the benchmark splits [1].\n5We consider pronouns and de\ufb01nite noun phrases as ambiguous expressions and count them using a POS\n\ntagger in NLTK (http://www.nltk.org/).\n\n8\n\nHistory:Are there any 9's in the image ?threeHow many digits in a yellow background are there among them ?oneWhat is the color of the digit ?redWhat is the color of the digit at the right of it ?blueWhat is the style of the blue digit ?flatCurrent QA:What is the color of the digit at the right of it ?violetRetrieved attention from memoryFinal attentionManually modifiedretrieved attentionFinal attentionPredicted answer: violetInput imagePredicted answer: green\fTable 1: Experimental results on VisDial. We show the number of parameters, mean reciprocal\nrank (MRR), recall@k and mean rank (MR). +H and ATT indicate use of history embeddings in\nprediction and attention mechanism, respectively.\n# of params\n\nMRR\n0.3735\n0.5508\n0.5578\n0.5759\n0.5807\n0.5695\n0.5846\n0.5868\n0.5849\n0.5965\n0.5764\n0.5788\n0.6196\n0.6192\n0.6227\n0.6210\n\nR@1\n23.55\n41.24\n41.75\n43.33\n43.82\n42.70\n44.67\n44.82\n44.03\n45.55\n43.44\n43.51\n48.24\n48.05\n48.53\n48.40\n\nR@5\n48.52\n70.45\n71.45\n74.27\n74.68\n73.25\n74.50\n74.81\n75.26\n76.22\n74.26\n74.49\n78.33\n78.39\n78.66\n78.39\n\nR@10 MR\n53.23\n26.50\n7.08\n79.83\n6.74\n80.94\n5.87\n83.68\n84.07\n5.78\n6.11\n82.97\n5.72\n84.22\n5.66\n84.36\n5.68\n84.49\n85.37\n5.46\n5.88\n83.72\n5.84\n83.96\n4.92\n87.11\n4.88\n87.12\n87.43\n4.86\n4.92\n87.12\n\nModel\nAnswer prior [1]\nLF-Q [1]\nLF-QH [1]\nLF-QI [1]\nLF-QIH [1]\nHRE-QH [1]\nHRE-QIH [1]\nHREA-QIH [1]\nMN-QH [1]\nMN-QIH [1]\nSAN-QI [10]\nHieCoAtt-QI [15]\nAMEM-QI\nAMEM-QIH\nAMEM+SEQ-QI\nAMEM+SEQ-QIH\n\n+H ATT\n\u2013\n\u2013\n(cid:88)\n\u2013\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n\u2013\n\u2013\n\u2013\n(cid:88)\n\u2013\n(cid:88)\n\n\u2013\n\u2013\n\u2013\n\u2013\n\u2013\n\u2013\n\u2013\n\u2013\n\u2013\n\u2013\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n\nn/a\n\n8.3 M (3.6x)\n12.4 M (5.4x)\n10.4 M (4.6x)\n14.5 M (6.3x)\n15.0 M (6.5x)\n16.8 M (7.3x)\n16.8 M (7.3x)\n12.4 M (5.4x)\n14.7 M (6.4x)\n\nn/a\nn/a\n\n1.7 M (0.7x)\n2.3 M (1.0x)\n1.7 M (0.7x)\n2.3 M (1.0x)\n\nHieCoAtt) with the same decoder. The three visual dialog baselines are trained with different valid\ncombinations of inputs, which are denoted by Q, I and H in the model names.\nWe perform the same ablation study of our model with the one for MNIST Dialog dataset. The conv5\nlayer in VGG-16 [38] trained on ImageNet [39] is used to extract the image feature map. Similar\nto [1], all word embedding layers share their weights and an LSTM is used for embedding the current\nquestion. For the models with history embedding, we use additional LSTMs for the questions, the\nanswers, and the captions in the history. Based on our empirical observation, we share the parameters\nof the question and caption LSTMs while having a separate set of weights for the answer LSTM.\nEvery LSTM embedding sentences is two-layered, but the history LSTM of HRNN has a single layer.\nWe employ 64 dimensional word embedding vectors and 128 dimensional hidden state for every\nLSTM. Note that the the dimensionality of our word embeddings and hidden state representations in\nLSTMs are signi\ufb01cantly lower than the baselines (300 and 512 respectively). We train the network\nusing Adam [40] with the initial learning rate of 0.001 and weight decaying factor 0.0001. Note that\nwe do not update the feature extraction network based on VGG-16.\nResults Table 1 presents mean reciprocal rank (MRR), mean rank (MR), and recall@k of the\nmodels. Note that lower is better for MRs but higher is better for all other evaluation metrics. All\nvariants of the proposed model outperform the baselines in all metrics, achieving the state-of-the-\nart performance. As observed in the experiments on MNIST Dialog, the models with sequential\npreference (+SEQ) show better performances compared to the ones without it. However, we do not\nsee additional bene\ufb01ts from using a history embedding on VisDial, in contrast to MNIST Dialog. The\nproposed algorithm also has advantage over existing methods in terms of the number of parameters.\nOur full model only requires approximately 15% of parameters compared to the best baseline model\nwithout counting the parameters in the common feature extraction module based on VGG-16. In\nVisDial, the attention based VQA techniques with (near) state-of-the-art performances are not as\ngood as the baseline models of [1] because they treat each question independently. The proposed\nmodel improves the performance on VisDial by facilitating the visual reference resolution process.\nQualitative results for VisDial dataset are presented in Section B of the supplementary material.\n\n5 Conclusion\nWe proposed a novel algorithm for answering questions in visual dialog. Our algorithm resolves\nvisual references in dialog questions based on a new attention mechanism with an attention memory,\nwhere the model indirectly resolves coreferences of expressions through the attention retrieval process.\nWe employ the dynamic parameter prediction technique to adaptively combine the tentative and\nretrieved attentions based on the question. We tested on both synthetic and real datasets and illustrated\nimprovements.\n\n9\n\n\fAcknowledgments\n\nThis work was supported in part by the IITP grant funded by the Korea government (MSIT) [2017-0-\n01778, Development of Explainable Human-level Deep Machine Learning Inference Framework;\n2017-0-01780, The Technology Development for Event Recognition/Relational Reasoning and\nLearning Knowledge based System for Video Understanding; 2016-0-00563, Research on Adaptive\nMachine Learning Technology Development for Intelligent Autonomous Digital Companion].\n\nReferences\n[1] Das, A., Kottur, S., Gupta, K., Singh, A., Yadav, D., Moura, J.M., Parikh, D., Batra, D.: Visual\n\nDialog. In CVPR. (2017)\n\n[2] Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: A neural image caption generator.\n\nIn CVPR. (2015)\n\n[3] Xu, K., Ba, J., Kiros, R., Courville, A., Salakhutdinov, R., Zemel, R., Bengio, Y.: Show, attend\n\nand tell: Neural image caption generation with visual attention. In ICML. (2015)\n\n[4] Mun, J., Cho, M., Han, B.: Text-guided attention model for image captioning. AAAI (2016)\n[5] Huang, D.A., Lim, J.J., Fei-Fei, L., Niebles, J.C.: Unsupervised visual-linguistic reference\n\nresolution in instructional videos. In CVPR. (2017)\n\n[6] Rohrbach, A., Rohrbach, M., Hu, R., Darrell, T., Schiele, B.: Grounding of textual phrases in\n\nimages by reconstruction. In ECCV. (2016)\n\n[7] Mansimov, E., Parisotto, E., Ba, J., Salakhutdinov, R.: Generating images from captions with\n\nattention. In ICLR. (2016)\n\n[8] Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., Lee, H.: Generative adversarial text\n\nto image synthesis. In ICML. (2016)\n\n[9] Noh, H., Seo, P.H., Han, B.: Image question answering using convolutional neural network\n\nwith dynamic parameter prediction. In CVPR. (2016)\n\n[10] Yang, Z., He, X., Gao, J., Deng, L., Smola, A.: Stacked attention networks for image question\n\nanswering. In CVPR. (2016)\n\n[11] Xu, H., Saenko, K.: Ask, attend and answer: Exploring question-guided spatial attention for\n\nvisual question answering. In ECCV. (2016)\n\n[12] Andreas, J., Rohrbach, M., Darrell, T., Klein, D.: Deep compositional question answering with\n\nneural module networks. In CVPR. (2016)\n\n[13] Kim, J.H., On, K.W., Lim, W., Kim, J., Ha, J.W., Zhang, B.T.: Hadamard Product for Low-rank\n\nBilinear Pooling. In ICLR. (2017)\n\n[14] Andreas, J., Rohrbach, M., Darrell, T., Klein, D.: Neural module networks. In CVPR. (2016)\n[15] Lu, J., Yang, J., Batra, D., Parikh, D.: Hierarchical question-image co-attention for visual\n\nquestion answering. In NIPS. (2016)\n\n[16] Fukui, A., Park, D.H., Yang, D., Rohrbach, A., Darrell, T., Rohrbach, M.: Multimodal compact\n\nbilinear pooling for visual question answering and visual grounding. In EMNLP. (2016)\n\n[17] Noh, H., Han, B.: Training recurrent answering units with joint loss minimization for vqa.\n\narXiv preprint arXiv:1606.03647 (2016)\n\n[18] Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C.L., Parikh, D.: VQA: Visual\n\nQuestion Answering. In ICCV. (2015)\n\n[19] Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., Parikh, D.: Making the V in VQA matter:\n\nElevating the role of image understanding in Visual Question Answering. In CVPR. (2017)\n\n[20] Zhang, P., Goyal, Y., Summers-Stay, D., Batra, D., Parikh, D.: Yin and Yang: Balancing and\n\nanswering binary visual questions. In CVPR. (2016)\n\n[21] Malinowski, M., Rohrbach, M., Fritz, M.: Ask your neurons: A neural-based approach to\n\nanswering questions about images. In ICCV. (2015)\n\n[22] Mun, J., Seo, P.H., Jung, I., Han, B.: MarioQA: Answering questions by watching gameplay\n\nvideos. arXiv preprint arXiv:1612.01669 (2016)\n\n10\n\n\f[23] Zhu, L., Xu, Z., Yang, Y., Hauptmann, A.G.: Uncovering temporal context for video question\n\nand answering. arXiv preprint arXiv:1511.04670 (2015)\n\n[24] Tapaswi, M., Zhu, Y., Stiefelhagen, R., Torralba, A., Urtasun, R., Fidler, S.: Movieqa:\n\nUnderstanding stories in movies through question-answering. In CVPR. (2016)\n\n[25] de Vries, H., Strub, F., Chandar, S., Pietquin, O., Larochelle, H., Courville, A.: Guesswhat?!\n\nvisual object discovery through multi-modal dialogue. In CVPR. (2017)\n\n[26] Das, A., Kottur, S., Moura, J.M., Lee, S., Batra, D.: Learning cooperative visual dialog agents\n\nwith deep reinforcement learning. arXiv preprint arXiv:1703.06585 (2017)\n\n[27] Strub, F., de Vries, H., Mary, J., Piot, B., Courville, A., Pietquin, O.: End-to-end optimization of\ngoal-driven and visually grounded dialogue systems. arXiv preprint arXiv:1703.05423 (2017)\n[28] Seo, P.H., Lin, Z., Cohen, S., Shen, X., Han, B.: Progressive attention networks for visual\n\nattribute prediction. arXiv preprint arXiv:1606.02393 (2016)\n\n[29] Sukhbaatar, S., Weston, J., Fergus, R., et al.: End-to-end memory networks. In NIPS. (2015)\n[30] Weston, J., Chopra, S., Bordes, A.: Memory networks. In ICLR. (2015)\n[31] Kumar, A., Irsoy, O., Ondruska, P., Iyyer, M., Bradbury, J., Gulrajani, I., Zhong, V., Paulus, R.,\nSocher, R.: Ask me anything: Dynamic memory networks for natural language processing. In\nICML. (2016)\n\n[32] Xiong, C., Merity, S., Socher, R.: Dynamic memory networks for visual and textual question\n\nanswering. In ICML. (2016)\n\n[33] Miller, A., Fisch, A., Dodge, J., Karimi, A.H., Bordes, A., Weston, J.: Key-value memory\n\nnetworks for directly reading documents. In EMNLP. (2016)\n\n[34] Clark, K., Manning, C.D.: Deep reinforcement learning for mention-ranking coreference\n\nmodels. In EMNLP. (2016)\n\n[35] Clark, K., Manning, C.D.: Improving coreference resolution by learning entity-level distributed\n\nrepresentations. In ACL. (2016)\n\n[36] Clark, K., Manning, C.D.: Entity-centric coreference resolution with model stacking. In ACL.\n\n(2015)\n\n[37] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll\u00e1r, P., Zitnick, C.L.:\n\nMicrosoft coco: Common objects in context. In ECCV. (2014)\n\n[38] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recogni-\n\ntion. ICLR (2015)\n\n[39] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical\n\nimage database. In CVPR. (2009)\n\n[40] Kingma, D., Ba, J.:\n\narXiv:1412.6980 (2014)\n\nAdam: A method for stochastic optimization.\n\narXiv preprint\n\n11\n\n\f", "award": [], "sourceid": 2079, "authors": [{"given_name": "Paul Hongsuck", "family_name": "Seo", "institution": "POSTECH"}, {"given_name": "Andreas", "family_name": "Lehrmann", "institution": "Disney Research"}, {"given_name": "Bohyung", "family_name": "Han", "institution": "POSTECH"}, {"given_name": "Leonid", "family_name": "Sigal", "institution": "Disney Research / University of British Columbia"}]}