{"title": "Middle-Out Decoding", "book": "Advances in Neural Information Processing Systems", "page_first": 5518, "page_last": 5529, "abstract": "Despite being virtually ubiquitous, sequence-to-sequence models are challenged by their lack of diversity and inability to be externally controlled. In this paper, we speculate that a fundamental shortcoming of sequence generation models is that the decoding is done strictly from left-to-right, meaning that outputs values generated earlier have a profound effect on those generated later. To address this issue, we propose a novel middle-out decoder architecture that begins from an initial middle-word and simultaneously expands the sequence in both directions. To facilitate information flow and maintain consistent decoding, we introduce a dual self-attention mechanism that allows us to model complex dependencies between the outputs. We illustrate the performance of our model on the task of video captioning, as well as a synthetic sequence de-noising task. Our middle-out decoder achieves significant improvements on de-noising and competitive performance in the task of video captioning, while quantifiably improving the caption diversity. Furthermore, we perform a qualitative analysis that demonstrates our ability to effectively control the generation process of our decoder.", "full_text": "Middle-Out Decoding\n\nShikib Mehri\n\nDepartment of Computer Science\nUniversity of British Columbia\n\namehri@cs.cmu.edu\n\nLeonid Sigal\n\nDepartment of Computer Science\nUniversity of British Columbia\n\nlsigal@cs.ubc.ca\n\nAbstract\n\nDespite being virtually ubiquitous, sequence-to-sequence models are challenged\nby their lack of diversity and inability to be externally controlled. In this paper,\nwe speculate that a fundamental shortcoming of sequence generation models is\nthat the decoding is done strictly from left-to-right, meaning that outputs values\ngenerated earlier have a profound effect on those generated later. To address this\nissue, we propose a novel middle-out decoder architecture that begins from an\ninitial middle-word and simultaneously expands the sequence in both directions.\nTo facilitate information \ufb02ow and maintain consistent decoding, we introduce\na dual self-attention mechanism that allows us to model complex dependencies\nbetween the outputs. We illustrate the performance of our model on the task of\nvideo captioning, as well as a synthetic sequence de-noising task. Our middle-\nout decoder achieves signi\ufb01cant improvements on de-noising and competitive\nperformance in the task of video captioning, while quanti\ufb01ably improving the\ncaption diversity. Furthermore, we perform a qualitative analysis that demonstrates\nour ability to effectively control the generation process of our decoder.\n\nIntroduction\n\n1\nNeural encoder-decoder architectures have gained signi\ufb01cant popularity in tasks such as language\ntranslation (Sutskever et al., 2014; Bahdanau et al., 2015; Wu et al., 2016), image captioning (Vinyals\net al., 2015; Xu et al., 2015), video captioning (Venugopalan et al., 2015a), visual question answering\n(Antol et al., 2015; Malinowski et al., 2015) and many others. In such architectures, the input is\ntypically encoded through the use of an encoder (e.g., a CNN for an image or RNN, with LSTM or\nGRU cells, for a sentence or a variable length input) into a \ufb01xed-length vector hidden representation.\nThe decoder, for most problems, is then tasked with generating a sequence of words, or, more\ngenerally, vector values, conditioned on the hidden representation of the encoder.\nOne issue with such encoder-decoder architectures is that conditioning is done through the initial\nhidden state of the decoder (Vinyals et al., 2015) and has to be propagated forward to generate a\nrelevant output sequence. This design makes it dif\ufb01cult, for example, to generate relevant image\ncaptions as the encoded image information is often overwhelmed by the language prior implicitly\nlearned by the decoder (Devlin et al., 2015; Heuer et al., 2016). Among potential solutions is the\nuse of an encoded feature vector as a recurrent input to each decoder cell (Mao et al., 2015) or the\nuse of attention over the encoder (Xu et al., 2015) to focus generation of each subsequent output.\nModeling of long dependencies among the outputs in the decoded sequence is another challenge,\noften addressed by self-attention (Werlen et al., 2018; Daniluk et al., 2017; Vaswani et al., 2017).\nIn the existing architectures, however, the decoding is done strictly from left-to-right meaning that\nearlier generated output values have a profound effect on the later generated tokens. This makes the\ndecoding (linguistically) consistent, but at the same time limited in diversity. Left-to-right decoding\nalso presents challenges when one wants to have some control over a word or value that should appear\nlater in the sequence. Ensuring that a particular important word explicitly appears in the caption is\nnot possible with any of the left-to-right decoding mechanisms. Similarly, for decoding of symmetric\nsequences, a left-to-right decoding may introduce biases which break natural symmetries.\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\fIn this paper, we propose a novel middle-out decoder architecture that addresses these challenges.\nThe proposed architecture consists of two RNN decoders, implemented using LSTMs, one which\ndecodes to the right and another to the left; both begin from the same important (middle) output\nword or value automatically predicted by a classi\ufb01er. Output can also be directly controlled by\nspecifying an important (middle) word or value (e.g., allowing controlled and more diverse caption\ngeneration focused on a speci\ufb01c verb, adjective, or object). As we show in experiments such\nability is dif\ufb01cult to achieve using traditional left-to-right decoder architectures. During training\nand inference of our middle-out decoder the left and right LSTMs alternate between generating\nwords/values in the two directions until they reach their respective STOP tokens. Further, a novel\ndual self-attention mechanism, that attends over both the generated output and hidden states, is used\nto ensure information \ufb02ow and consistent decoding in both directions. The proposed architecture\nis a generalization of a family of traditional decoders that generate sequences left-to-right, where\nthe important (middle) output is deterministically taken to be a sentence START token. As such, our\nmiddle-out decoder can be used in any modern architecture that requires sequential decoding.\nContributions: Our main contribution is the middle-out decoder architecture, which is able to\ndecode a sequence starting from an important word or value in both directions. In order to ensure the\nmiddle-out generation is consistent, we also introduce a novel dual self-attention mechanism. We\nillustrate the effectiveness of our model through qualitative and quantitative experiments where we\nshow that it can achieve state-of-the-art results. Further, we show vastly improved performance in\ncases where an important word or value can be predicted well (e.g., decoding of sequences that have\nsymmetries, or video captioning where an action verb can be estimated reliably).\n\n2 Related Work\n\nNeural Encoder-Decoder Architectures: Early neural architectures for language generation\n(Sutskever et al., 2014) produced sequential outputs, conditioned on a \ufb01xed-length hidden vec-\ntor representation. The introduction of the attention mechanism (Bahdanau et al., 2015), allowed the\nlanguage decoder to attend to the encoded input rather than necessitating that it be compressed into a\n\ufb01xed-length vector. Similar encoder-decoder architectures, with CNN encoding (Vinyals et al., 2015)\nfor images and CNN+RNN encoding (Venugopalan et al., 2015a) for videos, were employed for the\ntasks of captioning and visual question answering (Antol et al., 2015; Malinowski et al., 2015). The\nproposed middle-out decoding focuses speci\ufb01cally on the decoder component of such models.\nSelf-attention: Recently, there have been a number of studies exploring the use of self-attention\nto improve language decoding (Werlen et al., 2018; Daniluk et al., 2017; Cheng et al., 2016; Liu\nand Lapata, 2017; Vaswani et al., 2017). Werlen et al. (2018) attend over the embedded outputs\nof the decoder, allowing for non-sequential dependencies between outputs. Daniluk et al. (2017)\npropose the self-attentive RNN, which attends over its own past hidden states. Vaswani et al. (2017)\npresent the Transformer, an architecture that produces language, without a sequential model (e.g. an\nLSTM) and instead relies entirely on a number of attention mechanisms, including self-attention in\nthe decoder. We utilize a novel combination of these self-attention mechanisms in our generation,\nwhich we refer to as dual self-attention.\nNon-traditional Decoders: Most decoder architectures are sequential and generate sequences left-\nto-right (Sutskever et al., 2014), which leads to challenges such as lack of control and diversity\n(Devlin et al., 2015). Very recently tree structured decoders (Polosukhin and Skidanov, 2018) have\nbeen employed for tasks where outputs form well-de\ufb01ned structured representations (e.g., programs).\nOur model makes no assumptions about the structure of the output sequence, beyond the notion that\na certain element of the sequence may be more important than others.\nControlled Decoding: Hokamp and Liu (2017) introduced Grid Beam Search, an algorithm that\nenables controlled language generation, which they employ for the task of machine translation. Their\nmethod makes no architectural changes, and instead modi\ufb01es beam search decoding to force the\ninclusion of certain phrases. Post and Vilar (2018) introduces a faster algorithm with the same effects.\nWhile our controlled generation is not as \ufb02exible as Grid Beam Search (Hokamp and Liu, 2017), we\ndemonstrate that our model outperforms it in a con\ufb01ned, albeit realistic, setting.\nDiversi\ufb01ed Decoding: There has been some exploration into diverse language generation, particu-\nlarly for the task of image captioning. Dai et al. (2017) uses a Conditional GAN, along with a policy\n\n2\n\n\fgradient, to generate diverse captions. Wang et al. (2016) presents GroupTalk, a framework which\nsimultaneously learns multiple image caption distributions, to mimic human diversity.\nVideo Captioning: While our models are not speci\ufb01c to any particular task, we illustrate their\nperformance on video captioning (Chen and Dolan, 2011). As such, it is related to work done in this\nproblem space (Venugopalan et al., 2015b,a; Yao et al., 2015; Pan et al., 2016a,b; Yu et al., 2016;\nSong et al., 2017; Pasunuru and Bansal, 2017; Yu et al., 2017) to which we compare our results.\n\n3 Approach\n\nWe now introduce our middle-out decoder, a novel architecture for the generation of sequential outputs\nfrom the middle outwards. Unlike standard decoders which generate sequences from left-to-right,\nour middle-out decoder simultaneously expands the sequence in both directions. Intuitively, doing so\nallows us to begin decoding from the most important component of a sequence, thereby allowing\ngreater control over the resulting sequence generation.\nWe begin by discussing a baseline attention sequence-to-sequence model (Bahdanau et al., 2015).\nNext, we discuss the self-attention mechanism (Daniluk et al., 2017; Werlen et al., 2018), and present\nour novel dual self-attention applied to our baseline model. We then introduce our middle-out decoder,\nalong with a classi\ufb01cation model to predict the initial middle word.\n\n3.1 Baseline Attention Sequence-to-Sequence Model\n\nOur baseline model architecture, visualized in Figure 1, is that of Bahdanau et al. (2015): a bidirec-\ntional LSTM encoder and a unidirectional LSTM decoder. The input sequence is \ufb01rst passed through\nthe encoder to produce a hidden state, he\ni , for each time-step i of the input sequence. To produce the\noutput sequence, we compute the hidden state at time t as a function of the previous decoder hidden\nstate hd\n\nt\u22121, the embedding of the previously generated word et\u22121 and the context vector ct:\n\nhd\nt = S(hd\n\nt\u22121, et\u22121, ct).\n\nThe context vector ct, is a weighted sum over the encoder hidden states:\n\nn(cid:88)\n\ni=1\n\nct =\n\n\u03b1t,ihe\ni .\n\nThe alignment weights, \u03b1t,i, allow our model to attend to different time-steps of the encoder\ndependant on the current hidden state of the decoder. We compute the weights as follows:\n\n(cid:80)n\n\n\u03b1t,i =\n\nexp(score(hd\nk=1 exp(score(hd\n\ni ))\nt\u22121, he\n\nt\u22121, he\n\nk))\n\n.\n\n(1)\n\n(2)\n\n(3)\n\n(4)\n\n(5)\n\nThe score function used is the general attention mechanism described in Luong et al. (2015):\n\nscore(hd\n\nt\u22121Wahe\ni .\nt , we determine the output distribution, p(wt|hd\n\ni ) = hd\n\nt\u22121, he\n\nt ) through a softmax over all of\n\nUpon computing hd\nthe words in the vocabulary.\n\n3.2 Self-Attention Mechanism\n\nIn our baseline architecture, non-adjacent time-steps of the decoder may only communicate through\nthe propagation of a \ufb01xed-length vector, namely the hidden state of the LSTM. This hinders the model\nfrom learning long-term dependencies, which leads to a degraded quality for long output sequences.\nBahdanau et al. (2015) observed a similar issue on the encoder side and subsequently introduced the\nattention mechanism. Recent work (Daniluk et al., 2017; Werlen et al., 2018; Vaswani et al., 2017)\nhas proposed a self-attention mechanism to better capture the context of the decoder.\nWerlen et al. (2018) attend over the embedded outputs of the decoder, thereby modifying the\nrecurrence function to be:\n\nhd\nt = S(hd\n\nt\u22121, dt, ct)\n\n3\n\n\fFigure 1: Left: Our baseline sequence-to-sequence model. Right: Our baseline model complemented\nwith our dual self-attention mechanism. Note that the model visualized on the right, consists of two\nadditional attention mechanisms: over the embedded outputs (yellow) and over the decoder\u2019s hidden\nstates (light purple). Full sized diagrams may be found in the supplementary materials.\n\nwhere dt is de\ufb01ned as a weighted sum over the embedded outputs, {e1, e2, . . . et\u22121}. On the other\nhand, Daniluk et al. (2017) perform the attention over the decoder\u2019s hidden states rather than the\noutputs, resulting in the following recurrence function:\n\nwhere(cid:101)ht is a weighted sum over the decoder\u2019s past hidden states, {hd\n\n(6)\nt\u22121}. We propose\nto use dual self-attention, a novel combination of these two forms of self-attention, due to the\ncomplementary bene\ufb01ts that they provide. Our recurrence function therefore becomes:\n\nhd\nt = S(hd\n\n2, . . . hd\n\n1, hd\n\nt\u22121, et\u22121, ct,(cid:101)ht)\n\nt\u22121, et\u22121, dt, ct,(cid:101)ht)\n\nhd\nt = S(hd\n\nwhere dt, ct and(cid:101)ht are weighted sums of embedded past outputs, encoder hidden states and decoder\n\nhidden states, respectively.\nThe attention over the embedded outputs provides the decoder with direct information about the\npreviously generated terms, allowing it to model non-sequential dependencies between output words\n(Werlen et al., 2018). On the other hand, the attention over the decoder\u2019s hidden states best deals with\nlong-term dependencies, as it counteracts the problem of vanishing gradients (Bengio et al., 1994;\nHochreiter, 1998) by allowing us to directly backpropagate time-steps other than the previous one.\n\n(7)\n\n3.3 Middle-Out Decoder\n\nUnlike standard decoding, which typically begins from a START token and expands the sequence\nrightwards, middle-out decoding begins with an initial middle word and simultaneously expands the\nsequence in both directions.\nThe procedure by which we determine the value of the initial middle word is highly dependant on the\napplication. For example, for action-focused video caption generation, a natural choice for the middle\nword would be a verb; for object-centric image captioning a better choice might be a prominent noun.\nIn both cases, such words, which constitute a sub-set of the vocabulary, would need to be estimated\nfrom the encoded input itself. To accomplish this, we \ufb01rst utilize a classi\ufb01er to predict the initial\nword and then use a middle-out decoder that begins with the initial word and expands the sequence\noutwards. Notably, in some scenarios a user may want to supply the initial word directly to the\nalgorithm, resulting in a user-centric caption generation; this is something we explore in experiments.\nMiddle Word Classi\ufb01er: The middle word classi\ufb01er passes the input sequence through an LSTM\nencoder, to obtain the \ufb01nal hidden state he\nt . This hidden state is passed through a softmax layer, to\nobtain a probability distribution over the vocabulary. Note, other classi\ufb01ers may also be used.\n\n4\n\n\fFigure 2: A diagram visualizing our middle-out decoder, complemented with our dual self-attention\nmechanism. Note that the two self-attention mechanisms combine the otherwise detached decoders.\nThe middle-word shown in red is also attended over, allowing the two decoders to effectively depend\non the output of the classi\ufb01er (not pictured; can be found in the supplementary materials).\n\nMiddle-Out Architecture: The middle-out decoder consists of two LSTM decoders, one which\ndecodes to the right and another which decodes to the left. They both begin from the same initial\nword, outputted by the middle word classi\ufb01er, and the same initial hidden state, the \ufb01nal hidden\nstate of the encoder. Both during training and inference, the middle-out decoder alternates between\ngenerating a word on the left and a word on the right, with both decoders continuing until they hit\ntheir respective STOP tokens.\nIn our vanilla middle-out model, there is no interaction between the left and right decoders and it is\nlikely the case that the two decoders may generate sequences that are not jointly coherent. To combat\nthis issue, we utilize self-attention as described in the next section.\nMiddle-Out with Self-Attention: To ensure coherence in the output of the middle-out decoder, we\nutilize the previously described dual self-attention as a communication mechanism between the left\nand right decoders. Our recurrence function, after the addition of the self-attention mechanism, is\nshown in Eq. (7). The model is visualized in Figure 2.\n\nOur values for dt and(cid:101)ht are computed over the outputs of both the left and the right decoder without\nthe outputs of the other decoder. The value of(cid:101)ht, the weighted sum over all the decoder hidden\n\nany distinction, providing the two, otherwise disjoint, decoders the ability to communicate and\nensure a logically consistent output. The value of dt, the weighted sum over all the embedded\noutputs, provides the decoders with information that wouldn\u2019t have otherwise been observed, namely\n\nstates, also accomplishes this in addition to providing each decoder with the ability to in\ufb02uence the\ngradients of the other decoder. As described earlier, self-attention allows a decoder to backpropagate\nto non-adjacent time-steps, however, in the case of the middle-out decoder, it also allows two disjoint\ndecoders to backpropagate to entirely disconnected time-steps.\nEffectively, self-attention allows the middle-out decoder to maintain some notion of dependence\nbetween all of the outputs, without necessitating the use of a single sequential decoder.\n\n4 Experiments\n\nWe evaluate the aforementioned models on two sequence generation tasks. First, we evaluate middle-\nout decoders on the synthetic problem of de-noising a symmetric sequence. Next, we explore the\nproblem of video captioning on the MSVD dataset (Chen and Dolan, 2011), evaluating our models\nfor quality, diversity, and control.\n\n5\n\n\fFigure 3: A visualization of the synthetic de-noising task. On the left, we have a noisy input sequence\nwith each value being adjusted accordingly by an error sampled from a uniform distribution. On the\nright, we have the target output sequence, which has been de-noised.\n\n4.1 Synthetic De-noising Task\n\nDataset: We \ufb01rst generate a symmetric 1-dimensional sequence of numbers, the output sequence,\nand proceed to add noise to it, forming the 1-dimensional input sequence. To generate our symmetric\nsequence of numbers, we \ufb01rst sample \u00b5 from U nif orm(\u22121, 1) to serve as the middle number in our\nsequence. De\ufb01ning \u03c3 = \u00b52, we proceed to construct our sequence, y, of length 2N + 1 as follows:\n\ny = \u00b5 \u2212 N \u03c3\nN\n\n, ..., \u00b5 \u2212 2\u03c3\nN\n\n, \u00b5 \u2212 \u03c3\nN\n\n, \u00b5, \u00b5 \u2212 \u03c3\nN\n\n, \u00b5 \u2212 2\u03c3\nN\n\n, ..., \u00b5 \u2212 N \u03c3\nN\n\n(8)\n\nWe generate 1000 training samples and 100 testing samples, with the value of N being randomly\nselected such that 5 \u2264 N \u2264 10, resulting in sequences with lengths between 11 and 21.\nTo generate the noisy input sequence, we add noise sampled from U nif orm(\u22120.0035, 0.0035) to\neach element in the sequence ultimately obtaining our noisy input sequence, x, such that xi =\nyi + U nif orm(\u22120.0035, 0.0035). An example of an input sequence and corresponding output\nsequence is shown in Figure 3.\nExperimental Setup: We modify our models in order to best suit the nature of this task. Rather than\ntreating this as a classi\ufb01cation problem at each time-step, as is typically done for text generation, we\ninstead formulate it as a regression. As a result of our model outputting a real-valued number, we\nremove the softmax layer, the embeddings layer, and the self-attention mechanism.\nWe opt to utilize a classi\ufb01er to predict the initial value of the sequence for both the baseline model\nand the middle-out decoder. In both cases, the classi\ufb01er is a single linear layer that operates on the\n\ufb01nal hidden state of a bidirectional LSTM encoder and ultimately outputs a single number. All of our\nmodi\ufb01ed models are visualized and further described in the supplementary materials.\nWe utilize 100-dimensional LSTMs and the Adam optimizer (Kingma and Ba, 2015) with a learning\nrate of 1e\u22124. We train the models for 20, 000 steps with a batch size of 32, which is equivalent to\n640 passes over the data. During training we use teacher forcing, meaning we feed the ground-truth\noutput as input into the next time-step. During evaluation, we instead provide the model\u2019s own output\nas input into the next time-step.\nResults: We evaluate the de-noising of the sequences with two metrics, MSE and symmetric MSE.\nMean squared error (MSE) indicates the quality with which the model de-noised the sequence.\nSymmetric MSE measures the error between the two halves of the output sequence. Since the\nsequence is symmetric, theoretically the outputted sequence should have a very low error between its\ntwo halves. A higher error is indicative of a lack of coherence in the generation.\nThe results in Table 1 show that our middle-out decoder signi\ufb01cantly improves upon the performance\nof the sequence-to-sequence baseline, with a 75% decrease in the MSE value. There is an even larger\ngain in terms of the output symmetry, with an MSE value 570 times smaller. This suggests that the\nmiddle-out decoder better models non-trivial dependencies in its outputs both through the use of its\ndecoding order and self-attention mechanism.\n\n6\n\n\fModels\nSequence-to-Sequence with Attention\nMiddle-Out with Self-Attention\n\nMSE\n\n1.52 \u00d7 1e\u22123\n3.51 \u00d7 1e\u22124\n\nSymmetric MSE\n1.32 \u00d7 1e\u22124\n\n0.0780\n\nTable 1: Sequence generation results on a synthetic de-noising task. Middle-out decoding is shown to\nperform better on this task, both in terms of the quality and the symmetry of the de-noised sequence.\nMETEOR CIDEr-D ROUGE-L BLEU-4\nModels\n\nLSTM-YT (V) (Venugopalan et al., 2015b)\nS2VT (V + A) (Venugopalan et al., 2015a)\nTemporal Attention (G + C) (Yao et al., 2015)\nLSTM-E (V + C) (Pan et al., 2016b)\np-RNN (V + C) (Yu et al., 2016)\nHNRE + Attention (G + C) (Pan et al., 2016a)\nhLSTMat (R) (Song et al., 2017)\n\nPrevious Work\n26.9\n29.8\n29.6\n31.0\n32.6\n33.9\n33.6\n\nOur Models\n\n51.7\n\n65.8\n\n-\n-\n\n-\n\n-\n\n73.8\n\nSeq2Seq + Attention (I)\nSeq2Seq + Attention + Self-Attention (I)\nMiddle-Out + Attention (I)\nMiddle-Out + Attention + Self-Attention (I)\n\n34.0\n34.4\n30.9\n34.1\n\nSeq2Seq + Attention (I)\nMiddle-Out + Attention + Self-Attention (I)\n\n34.5\n40.9\n\nOracle Experiments\n\n78.9\n81.9\n68.6\n79.5\n\n77.8\n124.4\n\n-\n-\n-\n-\n-\n-\n-\n\n70.0\n70.5\n66.9\n69.8\n\n70.4\n78.6\n\n31.2\n\n-\n\n41.9\n45.3\n49.9\n46.7\n53.0\n\n47.4\n48.3\n40.8\n47.0\n\n47.4\n62.5\n\nTable 2: Video captioning results on the MSVD (Youtube2Text) dataset. We show previous work,\nour models as well as our oracle experiments. In this table, V, A, G, C, R, I respectively refer to\nvisual features obtained from VGGNet, AlexNet, GoogLeNet, C3D, Resnet-152 and Inception-v4.\nIt is worth noting that we only compare to single-model results. For reference, the state-of-the-art\nensemble model (Pasunuru and Bansal, 2017) obtains METEOR: 36.0, CIDEr-D: 92.4, ROUGE:\n92.4, BLEU-4: 54.5.\n\n4.2 Video Captioning\n\nVideo captioning is the task of generating a natural language description of the content within a video.\nFor this task, we utilize the MSVD (Youtube2Text) dataset (Chen and Dolan, 2011) to evaluate the\noutput quality, diversity, and control of the models described in Section 3.\nDataset: The MSVD (Youtube2Text) dataset (Chen and Dolan, 2011) consists of 1970 YouTube\nvideos with an average video length of 10 seconds and an average of 40 captions per video. We\nuse the standard splits provided by Venugopalan et al. (2015a) with 1200 training videos, 100 for\nvalidation and 670 for testing. We utilize frame-level features provided by Pasunuru and Bansal\n(2017). The videos were sampled at 3fps and passed through an Inception-v4 model (Szegedy et al.,\n2017), pretrained on ImageNet (Deng et al., 2009), to obtain 1536-dim feature vector for each frame.\nExperimental Setup: For all of our models, we use a 1024-dimensional LSTMs, 512-dimensional\nembeddings pretrained with word2vec (Mikolov et al., 2013), and the Adam optimizer with a learning\nrate of 1e\u22124. We utilize a batch size of 32 and train for 15 epochs. We train our models with a cross\nentropy loss function. We employ a scheduled sampling training strategy (Bengio et al., 2015), which\nhas greatly improved results in image captioning. We begin with a sampling rate of 0 and increase\nthe sampling rate every epoch by 0.05, with a maximum sampling rate of 0.25.\nDuring evaluation, we use beam search, with a beam size of 8 and a modi\ufb01ed beam scoring function\n(Huang et al., 2017) which adds the length of the output sequence to the beam\u2019s score.\nThe middle-word classi\ufb01er is trained to predict the most-common verb among all of the captions for\na particular video. To avoid potential redundancies in our classi\ufb01er\u2019s vocabulary (of size 40), we only\npredict words that end in \"ing\". We utilize a cross-entropy loss function to train the model.\nOutput Quality: To evaluate quality of the captions produced by our models, we use four evaluation\nmetrics popular for language generation tasks: METEOR (Denkowski and Lavie, 2014), BLEU-\n\n7\n\n\fClassi\ufb01er Accuracy METEOR CIDEr-D ROUGE-L BLEU-4\n31.64% Accuracy\n50% Accuracy\n75% Accuracy\n100% Accuracy\n\n79.5\n90.4\n105.6\n124.4\n\n69.8\n71.9\n75.1\n78.6\n\n34.1\n35.6\n38.4\n40.9\n\n47.0\n50.3\n56.2\n62.5\n\nSampling Ratio\n\n100%\n73.14%\n36.57%\n\n0%\n\nTable 3: We simulate various classi\ufb01cation accuracies by sampling from the output of our middle-word\nclassi\ufb01er and the oracle middle-words at different rates. We observe that we generate progressively\nbetter captions with improvements in classi\ufb01cation accuracy.\n\n4 (Papineni et al., 2002), CIDEr-D (Vedantam et al., 2015) and ROUGE-L (Lin, 2004). We use\nevaluation code from Microsoft COCO server (Chen et al., 2015).\nThe results shown in Table 2 demonstrate that the quality of the middle-out decoding is comparable to\nthat of our sequence-to-sequence baseline. Our dual self-attention mechanism applied to the baseline\noutperforms previous work, and signi\ufb01cantly improves the results of the middle-out decoder.\nWe note that the quality of our model depends on the quality of the middle word (verb) classi\ufb01er. Our\nclassi\ufb01er for illustrative purposes is simple, but could be substantially improved using specialized\naction classi\ufb01cation architectures, e.g., dual-stream C3D networks (Carreira and Zisserman, 2017).\nTo this end we perform an experiment to evaluate the performance of our model when provided with\nexternal input from an oracle. Rather than predicting the middle-word with a classi\ufb01er, we instead\nprovide the ground-truth to the model during inference. As our baseline sequence-to-sequence model\nis incapable of receiving an external input, we re-train a modi\ufb01ed model in which the decoder receives\nthe embedded ground-truth middle-word as input every time-step; i.e. both approaches use same data.\nThe results shown in Table 2, demonstrate that despite the fact that the baseline in our oracle\nexperiment was speci\ufb01cally trained for the task at hand, our middle-out decoder performs signi\ufb01cantly\nbetter, achieving a METEOR score of 40.9. This indicates that the middle-out decoder effectively\nutilizes external input to generate optimal captions.\nIn Table 3 we depict a series of experiments with various simulated accuracies for the middle-word\nclassi\ufb01er, by sampling from our classi\ufb01er (accuracy of 31.64%) and the oracle middle-words at\ndifferent ratios. This experiment demonstrates that reasonable improvements to the middle-word\nclassi\ufb01er would lead to strong performance gains in sequence generation.\nOutput Diversity: We quantify the diversity of the models by running a beam search, with a beam\nsize of 8, and evaluating the diversity between the generated captions for each video. To evaluate\ndiversity between the 8 captions, we employ Self-BLEU (Zhu et al., 2018) and Self-METEOR to\nassess similarity between the different captions in a generated set. The results in Table 4 demonstrate\nthat the middle-out decoder signi\ufb01cantly improves on the diversity of the generation caption set.\n\nModels\nSeq2Seq + Attention\nMiddle-Out + Attention + Self-Attention\n\nSelf METEOR Self BLEU-4\n\n49.6\n44.8\n\n77.8\n70.1\n\nTable 4: Diversity evaluation (lower is better) our for a generated set of captions. We compare our\nbaseline model as well as the middle-out decoder with self-attention.\n\nOutput Control: We de\ufb01ne the output control to be a measure of the model\u2019s susceptibility to\nexternal input. A model which is able to effectively use external input to improve its generated\ncaption can be said to be well controllable. The results of our oracle experiment, shown in Table 2,\ndemonstrate our middle-out decoder to effectively utilize external input.\nWe expand on the oracle experiment by comparing to alternative mechanisms for controlled sequence\ngeneration, namely Grid Beam Search (Hokamp and Liu, 2017). The results shown in Table 5\ndemonstrate that our middle-out decoder is better at effectively leveraging the oracle information to\nimprove the quality of the generated caption.\nTo further qualify the controllable nature of the middle-out decoder, we perform an experiment\nthat exempli\ufb01es the decoder\u2019s ability to attend to varying parts of the video dependant on the input\nmiddle-word. We concatenate two videos together, without any explicit distinction about the boundary\n\n8\n\n\fModels\nSeq2Seq + Attention (M)\nSeq2Seq + Attention (G)\nSeq2Seq + Attention (M) (G)\nMiddle-Out + Attention + Self-Attention\n\nMETEOR CIDEr-D ROUGE-L BLEU-4\n\n34.5\n40.4\n40.0\n40.9\n\n77.8\n121.6\n120.0\n124.4\n\n70.4\n77.7\n77.8\n78.6\n\n47.4\n61.0\n60.5\n62.5\n\nTable 5: An expansion of the oracle experiment, where we evaluate the ability of each model to\neffectively utilize external input, in this case, a provided middle-word. Here (M) indicates that the\nmodel was trained with middle-word embedding concatenated to its input at each time step. (G)\nrepresents the usage of Grid Beam Search (Hokamp and Liu, 2017) during inference.\n\nProvided Middle-Word\nSeq2Seq + Attention\nMiddle-Out + Self-Attention\n\na magician is performing magic\n\na man is playing piano\n\nperforming\n\nplaying\n\na man is playing the piano\n\na man is performing on a stage\n\na man is playing the piano\na man is playing the piano\n\nTable 6: The \ufb01rst row contains the sampled frames from the concatenated videos, as well as the\nground-truth captions for each video. The row below indicates the ground-truth word provided to\nthe model, as done in the oracle experiment. The proceeding rows are the outputs generated by our\nbaseline model (specially trained for the oracle experiment) and our middle-out decoder, when given\nthe speci\ufb01ed input word. More qualitative examples are provided in the supplementary materials.\n\nbetween the two videos, and demonstrate that by providing different initial middle-words to the\nmiddle-out decoder, we can effectively force it to generate captions pertaining to one of the videos.\nThe qualitative results of this experiment, shown in Table 6, demonstrate that the middle-out decoder\nis more susceptible to external input and therefore easier to control. Despite being trained to\nreceive a ground-truth middle-word, the baseline model is incapable of effectively utilizing provided\ninformation, often choosing to ignore it entirely.\nDual Self-Attention Ablation: We perform an ablation study in order to demonstrate the effective-\nness of our novel dual self-attention mechanism. The experiments shown in Table 7 demonstrate that\ndual self-attention outperforms the alternate methods for self-attention (Werlen et al., 2018; Daniluk\net al., 2017), when utilized with our self-attention sequence-to-sequence architecture.\n\nModels\nOutput Embedding Self-Attention\nHidden Self-Attention\nDual Self-Attention\n\nMETEOR CIDEr-D ROUGE-L BLEU-4\n\n34.2\n34.4\n34.4\n\n82.8\n80.7\n81.9\n\n70.3\n70.5\n70.5\n\n47.8\n47.2\n48.3\n\nTable 7: An ablation study demonstrating the effectiveness of our novel dual self-attention mechanism.\n\n5 Conclusion\n\nIn this paper, we present a novel middle-out decoder architecture which begins from an important\ntoken and expands the sequence in both directions. In order to ensure the coherence of the middle-out\ngeneration, we introduce a novel dual self-attention mechanism that allows us to model complex\ndependencies between the outputs. We illustrate the effectiveness of the proposed model both\nthrough quantitative and qualitative experimentation, where we demonstrate its capability to achieve\nstate-of-the-art results while also producing diverse outputs and exhibiting controlability.\n\n9\n\n\fReferences\nS. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. Lawrence Zitnick, and D. Parikh. Vqa: Visual question\nanswering. In Proceedings of the IEEE International Conference on Computer Vision, pages 2425\u20132433.\nIEEE, 2015.\n\nD. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. In\n\nProceedings of ICLR, 2015.\n\nS. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer. Scheduled sampling for sequence prediction with recurrent\n\nneural networks. In Advances in Neural Information Processing Systems, pages 1171\u20131179, 2015.\n\nY. Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is dif\ufb01cult. IEEE\n\nTransactions on Neural Networks, 5(2):157\u2013166, 1994.\n\nJ. Carreira and A. Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset.\n\nIn\nProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299\u20136308. IEEE,\n2017.\n\nD. Chen and W. Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th\nAnnual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages\n190\u2013200. Association for Computational Linguistics, 2011.\n\nX. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Doll\u00e1r, and C. L. Zitnick. Microsoft coco captions: Data\n\ncollection and evaluation server. arXiv preprint arXiv:1504.00325, 2015.\n\nJ. Cheng, L. Dong, and M. Lapata. Long short-term memory-networks for machine reading. In Proceedings of\n\nthe 2016 Conference on Empirical Methods in Natural Language Processing, pages 551\u2013561, 2016.\n\nB. Dai, S. Fidler, R. Urtasun, and D. Lin. Towards diverse and natural image descriptions via a conditional\ngan. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2970\u20132979.\nIEEE, 2017.\n\nM. Daniluk, T. Rockt\u00e4schel, J. Welbl, and S. Riedel. Frustratingly short attention spans in neural language\n\nmodeling. In Proceedings of ICLR, 2017.\n\nJ. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database.\nIn Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 248\u2013255. IEEE,\n2009.\n\nM. Denkowski and A. Lavie. Meteor universal: Language speci\ufb01c translation evaluation for any target language.\n\nIn Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 376\u2013380, 2014.\n\nJ. Devlin, H. Cheng, H. Fang, S. Gupta, L. Deng, X. He, G. Zweig, and M. Mitchell. Language models for\nimage captioning: The quirks and what works. In Proceedings of the 53rd Annual Meeting of the Association\nfor Computational Linguistics and the 7th International Joint Conference on Natural Language Processing\n(Volume 2: Short Papers), volume 2, pages 100\u2013105, 2015.\n\nH. Heuer, C. Monz, and A. W. Smeulders. Generating captions without looking beyond objects. In Workshop on\n\nStorytelling with Images and Videos, 2016.\n\nS. Hochreiter. The vanishing gradient problem during learning recurrent neural nets and problem solutions.\n\nInternational Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 6(02):107\u2013116, 1998.\n\nC. Hokamp and Q. Liu. Lexically constrained decoding for sequence generation using grid beam search. In\nProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long\nPapers), volume 1, pages 1535\u20131546, 2017.\n\nL. Huang, K. Zhao, and M. Ma. When to \ufb01nish? optimal beam search for neural text generation (modulo beam\nsize). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages\n2134\u20132139, 2017.\n\nD. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Proceedings of ICLR, 2015.\n\nC.-Y. Lin. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out, 2004.\n\nY. Liu and M. Lapata. Learning structured text representations. arXiv preprint arXiv:1705.09207, 2017.\n\n10\n\n\fT. Luong, H. Pham, and C. D. Manning. Effective approaches to attention-based neural machine translation.\nIn Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages\n1412\u20131421, 2015.\n\nM. Malinowski, M. Rohrbach, and M. Fritz. Ask your neurons: A neural-based approach to answering questions\nabout images. In Proceedings of the IEEE International Conference on Computer Vision, pages 1\u20139. IEEE,\n2015.\n\nJ. Mao, W. Xu, Y. Yang, J. Wang, Z. Huang, and A. Yuille. Deep captioning with multimocal recurrent neural\n\nnetworks. In Proceedings of ICLR, 2015.\n\nT. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. Distributed representations of words and phrases\nand their compositionality. In Advances in Neural Information Processing Systems, pages 3111\u20133119, 2013.\n\nP. Pan, Z. Xu, Y. Yang, F. Wu, and Y. Zhuang. Hierarchical recurrent neural encoder for video representation\nwith application to captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern\nRecognition, pages 1029\u20131038. IEEE, 2016a.\n\nY. Pan, T. Mei, T. Yao, H. Li, and Y. Rui. Jointly modeling embedding and translation to bridge video and\nlanguage. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages\n4594\u20134602. IEEE, 2016b.\n\nK. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of machine translation.\nIn Proceedings of 40th Annual Meeting of the Association for Computational Linguistics, pages 311\u2013318.\nAssociation for Computational Linguistics, 2002.\n\nR. Pasunuru and M. Bansal. Multi-task video captioning with video and entailment generation. In Proceedings\nof the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),\nvolume 1, pages 1273\u20131283. Association for Computational Linguistics, 2017.\n\nI. Polosukhin and A. Skidanov. Neural program search: Solving programming tasks from description and\n\nexamples. arXiv preprint arXiv:1802.04335, 2018.\n\nM. Post and D. Vilar. Fast lexically constrained decoding with dynamic beam allocation for neural machine\ntranslation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for\nComputational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages\n1314\u20131324, 2018.\n\nJ. Song, Z. Guo, L. Gao, W. Liu, D. Zhang, and H. T. Shen. Hierarchical lstm with adjusted temporal attention for\nvideo captioning. In Proceedings of the Twenty-Sixth International Joint Conference on Arti\ufb01cial Intelligence.\nAAAI Press, 2017.\n\nI. Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks. In Advances in\n\nneural information processing systems, pages 3104\u20133112, 2014.\n\nC. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi. Inception-v4, inception-resnet and the impact of residual\n\nconnections on learning. In AAAI, volume 4, page 12, 2017.\n\nA. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, \u0141. Kaiser, and I. Polosukhin. Attention\n\nis all you need. In Advances in Neural Information Processing Systems, pages 6000\u20136010, 2017.\n\nR. Vedantam, C. Lawrence Zitnick, and D. Parikh. Cider: Consensus-based image description evaluation. In\nProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4566\u20134575. IEEE,\n2015.\n\nS. Venugopalan, M. Rohrbach, J. Donahue, R. Mooney, T. Darrell, and K. Saenko. Sequence to sequence-video\nto text. In Proceedings of the IEEE International Conference on Computer Vision, pages 4534\u20134542. IEEE,\n2015a.\n\nS. Venugopalan, H. Xu, J. Donahue, M. Rohrbach, R. Mooney, and K. Saenko. Translating videos to natural\nlanguage using deep recurrent neural networks. In Proceedings of the 2015 Conference of the North American\nChapter of the Association for Computational Linguistics: Human Language Technologies, pages 1494\u20131504,\n2015b.\n\nO. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and tell: A neural image caption generator. In Proceedings\n\nof the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2015.\n\nZ. Wang, F. Wu, W. Lu, J. Xiao, X. Li, Z. Zhang, and Y. Zhuang. Diverse image captioning via grouptalk. In\nProceedings of the Twenty-Fifth International Joint Conference on Arti\ufb01cial Intelligence, pages 2957\u20132964.\nAAAI Press, 2016.\n\n11\n\n\fL. M. Werlen, N. Pappas, D. Ram, and A. Popescu-Belis. Self-attentive residual decoder for neural machine\ntranslation. Annual Conference of the North American Chapter of the Association for Computational\nLinguistics: Human Language Technologies, 2018.\n\nY. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey,\net al. Google\u2019s neural machine translation system: Bridging the gap between human and machine translation.\narXiv preprint arXiv:1609.08144, 2016.\n\nK. Xu, J. L. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhutdinov, R. S. Zemel, and Y. Bengio. Show, attend\nand tell: Neural image caption generation with visual attention. In Proceedings of the 32nd International\nConference on Machine Learning, 2015.\n\nL. Yao, A. Torabi, K. Cho, N. Ballas, C. Pal, H. Larochelle, and A. Courville. Describing videos by exploiting\ntemporal structure. In Proceedings of the IEEE International Conference on Computer Vision, pages 4507\u2013\n4515. IEEE, 2015.\n\nH. Yu, J. Wang, Z. Huang, Y. Yang, and W. Xu. Video paragraph captioning using hierarchical recurrent neural\nnetworks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages\n4584\u20134593. IEEE, 2016.\n\nY. Yu, H. Ko, J. Choi, and G. Kim. End-to-end concept word detection for video captioning, retrieval, and\nquestion answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,\npages 3165\u20133173. IEEE, 2017.\n\nY. Zhu, S. Lu, L. Zheng, J. Guo, W. Zhang, J. Wang, and Y. Yu. Texygen: A benchmarking platform for text\n\ngeneration models. arXiv preprint arXiv:1802.01886, 2018.\n\n12\n\n\f", "award": [], "sourceid": 2652, "authors": [{"given_name": "Shikib", "family_name": "Mehri", "institution": "Carnegie Mellon University"}, {"given_name": "Leonid", "family_name": "Sigal", "institution": "University of British Columbia"}]}