{"title": "Dialog-based Interactive Image Retrieval", "book": "Advances in Neural Information Processing Systems", "page_first": 678, "page_last": 688, "abstract": "Existing methods for interactive image retrieval have demonstrated the merit of integrating user feedback, improving retrieval results. However, most current systems rely on restricted forms of user feedback, such as binary relevance responses, or feedback based on a fixed set of relative attributes, which limits their impact. In this paper, we introduce a new approach to interactive image search that enables users to provide feedback via natural language, allowing for more natural and effective interaction. We formulate the task of dialog-based interactive image retrieval as a reinforcement learning problem, and reward the dialog system for improving the rank of the target image during each dialog turn. To mitigate the cumbersome and costly process of collecting human-machine conversations as the dialog system learns, we train our system with a user simulator, which is itself trained to describe the differences between target and candidate images. The efficacy of our approach is demonstrated in a footwear retrieval application. Experiments on both simulated and real-world data show that 1) our proposed learning framework achieves better accuracy than other supervised and reinforcement learning baselines and 2) user feedback based on natural language rather than pre-specified attributes leads to more effective retrieval results, and a more natural and expressive communication interface.", "full_text": "Dialog-based Interactive Image Retrieval\n\nXiaoxiao Guo\u2020\nIBM Research AI\n\nxiaoxiao.guo@ibm.com\n\nHui Wu\u2020\n\nIBM Research AI\nwuhu@us.ibm.com\n\nSteven Rennie\n\nFusemachines Inc.\n\nsrennie@gmail.com\n\nGerald Tesauro\nIBM Research AI\n\ngtesauro@us.ibm.com\n\nAbstract\n\nYu Cheng\n\nIBM Research AI\n\nchengyu@us.ibm.com\n\nRogerio Schmidt Feris\n\nIBM Research AI\n\nrsferis@us.ibm.com\n\nExisting methods for interactive image retrieval have demonstrated the merit of in-\ntegrating user feedback, improving retrieval results. However, most current systems\nrely on restricted forms of user feedback, such as binary relevance responses, or\nfeedback based on a \ufb01xed set of relative attributes, which limits their impact. In this\npaper, we introduce a new approach to interactive image search that enables users\nto provide feedback via natural language, allowing for more natural and effective\ninteraction. We formulate the task of dialog-based interactive image retrieval as a\nreinforcement learning problem, and reward the dialog system for improving the\nrank of the target image during each dialog turn. To mitigate the cumbersome and\ncostly process of collecting human-machine conversations as the dialog system\nlearns, we train our system with a user simulator, which is itself trained to describe\nthe differences between target and candidate images. The ef\ufb01cacy of our approach\nis demonstrated in a footwear retrieval application. Experiments on both simulated\nand real-world data show that 1) our proposed learning framework achieves better\naccuracy than other supervised and reinforcement learning baselines and 2) user\nfeedback based on natural language rather than pre-speci\ufb01ed attributes leads to\nmore effective retrieval results, and a more natural and expressive communication\ninterface.\n\n1\n\nIntroduction\n\nThe volume of searchable visual media has grown tremendously in recent years, and has intensi\ufb01ed the\nneed for retrieval systems that can more effectively identify relevant information, with applications in\ndomains such as e-commerce [1, 2], surveillance [3, 4], and Internet search [5, 6]. Despite signi\ufb01cant\nprogress made with deep learning based methods [7, 8], achieving high performance in such retrieval\nsystems remains a challenge, due to the well-known semantic gap between feature representations\nand high-level semantic concepts, as well as the dif\ufb01culty of fully understanding the user\u2019s search\nintent.\nA typical approach to improve search ef\ufb01cacy is to allow the user a constrained set of possible\ninteractions with the system [9, 10]. In particular, the user provides iterative feedback about retrieved\nobjects, so that the system can re\ufb01ne the results, allowing the user and system to engage in a\n\u201cconversation\u201d to resolve what the user wants to retrieve. For example, as shown in Figure 1, feedback\nabout relevance [11] allows users to indicate which images are \u201csimilar\u201d or \u201cdissimilar\u201d to the desired\nimage, and relative attribute feedback [12] allows the comparison of the desired image with candidate\nimages based on a \ufb01xed set of attributes. While these feedback paradigms are effective, the restrictions\n\n\u2020These two authors contributed equally to this work.\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\fFigure 1: In the context of interactive image retrieval, the agent incorporates the user\u2019s feedback to\niteratively re\ufb01ne retrieval results. Unlike existing work which are based on relevance feedback or\nrelative attribute feedback, our approach allows the user to provide feedback in natural language.\n\non the speci\ufb01c form of user interaction largely constrain the information that a user can convey to\nbene\ufb01t the retrieval process.\nIn this work, we propose a new approach to interactive visual content retrieval by introducing a novel\nform of user feedback based on natural language. This enables users to directly express, in natural\nlanguage, the most prominent conceptual differences between the preferred search object and the\nalready retrieved content, which permits a more natural human-computer interaction. We formulate\nthe task as a reinforcement learning (RL) problem, where the system directly optimizes the rank of\nthe target object, which is a non-differentiable objective.\nWe apply this RL based interactive retrieval framework to the task of image retrieval, which we\ncall dialog-based interactive image retrieval to emphasize its capability in aggregating history\ninformation compared to existing single turn approaches [13, 14, 15, 16]. In particular, a novel\nend-to-end dialog manager architecture is proposed, which takes natural language responses as user\ninput, and delivers retrieved images as output. To mitigate the cumbersome and costly process of\ncollecting and annotating human-machine dialogs as the system learns, we utilize a model-based\nRL approach by training a user simulator based on a corpus of human-written relative descriptions.\nSpeci\ufb01cally, to emulate a single dialog turn, where the user provides feedback regarding a candidate\nimage relative to what the user has in mind, the user simulator generates a relative caption describing\nthe differences between the candidate image and the user\u2019s desired image.1 Whereas there is a lot of\nprior work in image captioning [17, 18, 19], we explore the problem of relative image captioning, a\ngeneral approach to more expressive and natural communication of relative preferences to machines,\nand to use it as part of a user simulator to train a dialog system.\nThe ef\ufb01cacy of our approach is evaluated in a real-world application scenario of interactive footwear\nretrieval. Experimental results with both real and simulated users show that the proposed reinforce-\nment learning framework achieves better retrieval performance than existing techniques. Particularly,\nwe observe that feedback based on natural language is more effective than feedback based on pre-\nde\ufb01ned relative attributes by a large margin. Furthermore, the proposed RL training framework\nof directly optimizing the rank of the target image shows promising results and outperforms the\nsupervised learning approach which is based on the triplet loss objective. The main contributions of\nthis work are as follows:\n\n\u2022 A new vision/NLP task and machine learning problem setting for interactive visual content\nsearch, where the dialog agent learns to interact with a human user over the course of several\ndialog turns, and the user gives feedback in natural language.\n\n\u2022 A novel end-to-end deep dialog manager architecture, which addresses the above problem\nsetting in the context of image retrieval. The network is trained based on an ef\ufb01cient policy\noptimization strategy, employing triplet loss and model-based policy improvement [20].\n\n1In this work, the user simulator is trained on single-turn data and does not consider the dialog history. This\nreduces the sequence of responses to a \u201cbag\u201d of responses and implies that all sequences of a given set of actions\n(candidate images) are equivalent. Nevertheless, while the set of candidate images that maximize future reward\n(target image rank) are a set, selecting the image for the next turn naturally hinges on all previous feedback from\nthe user. Therefore, the entire set of candidate images can be ef\ufb01ciently constructed sequentially.\n\n2\n\nDesired ItemRelevance Feedback: NegativeRelative Attribute: MoreopenCandidate ADialog Feedback: Unlike the provided image, the one I want has an open back design with suede texture.Candidate BDialog Feedback: Unlike the provided image, the one I want has fur on the back and no sequin on top.Relevance Feedback: PositiveRelative Attribute: Less ornamental\f\u2022 The introduction of a computer vision task, relative image captioning, where the generated\ncaptions describe the salient visual differences between two images, which is distinct from\nand complementary to context-aware discriminative image captioning, where the absolute\nattributes of one image that discriminate it from another are described [21].\n\n\u2022 The contribution of a new dataset, which supports further research on the task of relative\n\nimage captioning. 2\n\n2 Related Work\n\nInteractive Image Retrieval. Methods for improving image search results based on user interaction\nhave been studied for more than 20 years [22, 10, 23]. Relevance feedback is perhaps the most\npopular approach, with user input speci\ufb01ed either as binary feedback (\u201crelevant\u201d or \u201cirrelevant\u201d) [11]\nor based on multiple relevance levels [24]. More recently, relative attributes (e.g., \u201cmore formal\nthan these,\u201d \u201cshinier than these\u201d) have been exploited as a richer form of feedback for interactive\nimage retrieval [12, 25, 26, 27, 28]. All these methods rely on a \ufb01xed, pre-de\ufb01ned set of attributes,\nwhereas our method relies on feedback based on natural language, providing more \ufb02exible and\nmore precise descriptions of the items to be searched. Further, our approach offers an end-to-end\ntraining mechanism which facilitates deployment of the system in other domains, without requiring\nthe explicit effort of building a new vocabulary of attributes.\n\nImage Retrieval with Natural Language Queries. Signi\ufb01cant progress has been recently made\non methods that lie in the intersection of computer vision and natural language processing, such as\nimage captioning [18, 19], visual question-answering [29, 30], visual-semantic embeddings [31, 32],\nand grounding phrases in image regions [33, 34]. In particular, our work is related to image or\nvideo retrieval methods based on natural language queries [13, 14, 15, 16]. These methods, however,\nretrieve images and videos in a single turn, whereas our proposed approach aggregates history\ninformation from dialog-based feedback and iteratively provides more re\ufb01ned results.\n\nVisual Dialog. Building conversational agents that can hold meaningful dialogs with humans has\nbeen a long-standing goal of Arti\ufb01cial Intelligence. Early systems were generally designed based on\nrule-based and slot-\ufb01lling techniques [35], whereas modern approaches have focused on end-to-end\ntraining, leveraging the success of encoder-decoder architectures and sequence-to-sequence learning\n[36, 37, 38]. Our work falls into the class of visually-grounded dialog systems [39, 40, 41, 42, 43].\nDas et al [39] proposed the task of visual dialog, where the system has to answer questions about\nimages based on a previous dialog history. De Vries et al. [40] introduced the GuessWhat game,\nwhere a series of questions are asked to pinpoint a speci\ufb01c object in an image, with restricted answers\nconsisting of yes/no/NA. The image guessing game [42] demonstrated emergence of grounded\nlanguage and communication among visual dialog agents with no human supervision, using RL to\ntrain the agents in a goal-driven dialog setting. However, these dialogs are purely text-based for both\nthe questioner and answerer agents, whereas we address the interactive image retrieval problem, with\nan agent presenting images to the user to seek feedback in natural language.\n\n3 Method\n\nOur framework , which we refer to as the dialog manager, considers a user interacting with a retrieval\nagent via iterative dialog turns. At the t-th dialog turn, the dialog manager presents a candidate image\nat selected from a retrieval database I = {Ii}N\ni=0 to the user. The user then provides a feedback\nsentence ot, describing the differences between the candidate image at and the desired image. Based\non the user feedback and the dialog history up to turn t, Ht = {a1, o1, ..., at, ot}, the dialog manager\nselects another candidate image at+1 from the database and presents it to the user. This process\ncontinues until the desired image is selected or the maximum number of dialog turns is reached.\nIn practice, the dialog manager could provide multiple images per turn to achieve better retrieval\nperformance. In this work, we focus on a simpli\ufb01ed scenario with a single image per interaction. We\nnote that the same framework could be extended to the multiple-image case by allowing the user to\nselect one image out of a list of candidate images to provide feedback on.\n\n2The project website is at: www.spacewu.com/posts/fashion-retrieval/\n\n3\n\n\fFigure 2: The proposed end-to-end framework for dialog-based interactive image retrieval.\n\n3.1 Dialog Manager: Model Architecture\n\nOur proposed dialog manager model consists of three main components: a Response Encoder,\na State Tracker, and a Candidate Generator, as shown in Figure 2. At the t-th dialog turn, the\nResponse Encoder embeds a candidate image and the corresponding user feedback {at, ot} into a\njoint visual-semantic representation xt \u2208 RD. The State Tracker then aggregates this representation\nwith the dialog history from previous turns, producing a new feature vector st \u2208 RD . The Candidate\nGenerator uses the aggregated representation st to select a new candidate image at+1 that is shown\nto the user. Below we provide details on the speci\ufb01c design of each of the three model components.\n\nResponse Encoder. The goal of the Response Encoder is to embed the information from the t-th\ndialog turn {at, ot} into a visual-semantic representation xt \u2208 RD. First, the candidate image is\nencoded using a deep convolutional neural network (CNN) followed by a linear transformation:\nt = ImgEnc(at) \u2208 RD. The CNN architecture in our implementation is an ImageNet pre-\nxim\ntrained ResNet-101 [44] and its parameters are \ufb01xed. Words in the user feedback sentence are\nrepresented with one-hot vectors and then embedded with a linear projection followed by a CNN as in\nt = TxtEnc(ot) \u2208 RD. Finally, the image feature vector and the sentence representation are\n[45]: xtxt\nconcatenated and embedded through a linear transformation to obtain the \ufb01nal response representation\nt ), where \u2295 is the concatenation operator and W \u2208 RD\u00d72D is the linear\nat time t: xt = W (xim\nprojection. The learnable parameters of the Response Encoder are denoted as \u03b8r.\n\nt \u2295 xtxt\n\nState Tracker. The State Tracker is based on a gated recurrent unit (GRU), which receives as input\nthe response representation xt, combines it with the history representation of previous dialog turns,\nand outputs the aggregated feature vector st. The forward dynamics of the State Tracker is written as:\ngt, ht = GRU(xt, ht\u22121), st = W sgt, where ht\u22121 \u2208 RD and gt \u2208 RD are the hidden state and the\noutput of the GRU respectively, ht is the updated hidden state, W s \u2208 RD\u00d7D is a linear projection\nand st \u2208 RD is the history representation updated with the information from the current dialog turn.\nThe learnable parameters of the State Tracker (GRU model) are denoted as \u03b8s. This memory-based\ndesign of the State Tracker allows our model to sequentially aggregate the information from user\nfeedback to localize the candidate image to be retrieved.\n\ni }N\n\ni=0, where xim\n\n\u03c0(j) = e\u2212dj /(cid:80)K\n\nCandidate Generator. Given the feature representation of all images from the retrieval database,\n{xim\ni = ImgEnc(Ii), we can compute a sampling probability based on the dis-\ntances between the history representation st to each image feature, xim\ni . Speci\ufb01cally, the sam-\npling probability is modeled using a softmax distribution over the top-K nearest neighbors of st:\nk=1 e\u2212dk , j = 1, 2, ..., K, where dk is the L2 distance of st to its k-th nearest\nneighbor in {xim\ni=0. Given the sampling distribution, two approaches can be taken to sample the\ncandidate image, denoted as at+1 = Ij(cid:48): (1) stochastic approach (used at training time), where\nj(cid:48) \u223c \u03c0, and (2) greedy approach (used at inference time), where j(cid:48) = arg maxj(\u03c0j). Combining the\nthree components of the model architecture, the overall learnable parameters of the dialog manager\nmodel is \u03b8 = {\u03b8r, \u03b8s}. Next, we explain how the network is trained end-to-end.\n\ni }N\n\n4\n\nGRUMLPResponse EncoderState TrackerRetrieval DatabaseCandidate GeneratorResponse Rep.Dialog ManagerText EncoderConcatenationtheoneIwanthasaclosed back and crystal buckle in the front.LinearMLPK-NNsStochastic SamplingGreedy SamplingTrainTestMLPImageRep.Candidate Image\ud835\udc4e\"\ud835\udc4e\"#$\ud835\udc5c\"Candidate ImageHistoryRep.\ud835\udc65\"\ud835\udc60\"Unlike the provided image, \f(a)\n\n(b)\n\nFigure 3: The learning framework: (a) The user simulator enables ef\ufb01cient exploration of the retrieval\ndialogs (Section 3.2.1); and (b) the policy network is learned using a combination of supervised\npre-training and model-based policy improvement (Section 3.2.2).\n\n3.2 Training the Dialog Manager\n\n(cid:3),\n\nthe goal of maximizing the expected sum of discounted rewards: max\u03c0 U \u03c0 = E(cid:2)(cid:80)T\n\nDirectly optimizing the ranking percentile metric in a supervised learning scheme is challenging since\nit is a non-differentiable function [46, 47]. Instead, we model the ranking percentile as the environment\nreward received by the agent and frame the learning process in a reinforcement learning setting with\nt=1 \u03b3t\u22121rt|\u03c0\u03b8\nwhere rt \u2208 R is the reward representing the ranking percentile of the target image at the t-th\ninteraction, \u03b3 is a discount factor determining the trade-off between short-term and long-term rewards,\nT is the maximum number of dialog turns, and \u03c0\u03b8 is the policy determined by network parameters \u03b8.3\nTraining an RL model for this problem requires extensive exploration of the action space, which\nis only feasible if a large amount of training data is available. However, collecting and annotating\nhuman-machine dialog data for our task is expensive. This problem is exacerbated in the situation of\nnatural language based user feedback, which incurs an even larger exploration space as compared to\napproaches based on a \ufb01xed set of attributes. In text-based dialog systems, it is common to circumvent\nthis issue by relying on user simulators [48]. We adopt a similar strategy, where a user simulator,\ntrained on human-written relative descriptions, substitutes the role of a real user in training the dialog\nmanager (illustrated in Figure 3a). Below we further describe our user simulator, as well as the\nreinforcement learning techniques that we used to optimize our dialog manager.\n\n3.2.1 User Simulator based on Relative Captioning\n\nHere we propose the use of a relative captioner to simulate the user. It acts as a surrogate for\nreal human users by automatically generating sentences that can describe the prominent visual\ndifferences between any pair of target and candidate images. We note that at each turn, our user\nsimulator generates feedback independent of previous user feedback, and previously retrieved images.\nWhile more sophisticated models that consider the dialog history could potentially be bene\ufb01cial,\ntraining such systems well may require orders of magnitude more annotated data. In addition, back-\nreferencing in dialogs can inherently be ambiguous and complex to resolve, even for humans. Based\non these considerations, we decided to \ufb01rst investigate the use of a single-turn simulator. While a\nfew related tasks have been studied previously, such as context-aware image captioning [21] and\nreferring expression generation [49], to the best of our knowledge, there is no existing dataset directly\nsupporting this task, so we introduce a new dataset as described in Section 4.\nWe experimented with several different ways of combining the visual features of the target and\nretrieved images. We include a comprehensive study of different models for the user simulator in\nAppendix C and show that the relative captioner based user model serves as a reasonable proxy for\nreal users. Speci\ufb01cally, we used feature concatenation to fuse the image features of the target and the\nreference image pair and applied the model of Show, Attend, and Tell [50] to generate the relative\ncaptions using a long short-term memory network (LSTM). For image feature extraction, we adopted\n\n3Strictly speaking, the optimal policy depends on the number of remaining dialog turns. We simplify the\n\npolicy to be a function independent of dialog turn numbers.\n\n5\n\nCandidate ImageTarget ImageRewardActionDialog ManagerText ResponseUserSimulator+Response EncoderState TrackerCandidate GeneratorAttentionLSTMSupervised Pre-trainingTriplet\tLossSelf-guided Policy ImprovementTop-k NNsHistoryRep.History Rep.Response EncoderState TrackerCandidate GeneratorActionDialog turn\ud835\udc60\"\ud835\udc4e\u2217Cross\tEntropyLoss\ud835\udc4e\"%&\fLsup = E(cid:104) T(cid:88)\n\nthe architecture of ResNet101 [44] pre-trained on ImageNet; and to better capture the localized visual\ndifferences, we added a visual attention mechanism; the loss function of the relative captioner is the\nsum of the negative log likelihood of the correct words [50].\n\n3.2.2 Policy Learning\nSupervised Pre-training. When the network parameters are randomly initialized at the beginning,\nthe history representations st are nearly random. To facilitate ef\ufb01cient exploration during RL training,\nwe \ufb01rst pre-train the policy using a supervised learning objective. While maximum likelihood-based\npre-training is more common, here we pre-train using the more discriminative triplet loss objective:\n\nmax(0,(cid:107)st \u2212 x+(cid:107)2 \u2212 (cid:107)st \u2212 x\u2212(cid:107)2 + m)\n\n(1)\n\nt=1\n\nwhere x+ and x\u2212 are the image features of the target image and a random image sampled from the\nretrieval database respectively, m is a constant for the margin and (cid:107).(cid:107)2 denotes L2-norm. Intuitively,\nby ensuring the proximity of the target image and the images returned by the system, the rank of\nthe target image can be improved without costly policy search from random initialization. However,\nentirely relying on this supervised learning objective deviates from our main learning objective, since\nthe triplet loss objective does not jointly optimize the set of candidate images to maximize expected\nfuture reward. 4\n\nModel-Based Policy Improvement. Given the known dynamics of the environment (in our case,\nthe user simulator), it is often advantageous to leverage its behavior for policy improvement. Here we\nadapt the policy improvement [20] to our model-based policy learning. Given the current policy \u03c0\nand the user simulator, the value of taking an action at using test-time con\ufb01guration can be ef\ufb01ciently\n\ncomputed by look-ahead policy value estimation Q\u03c0(ht, at) = E(cid:2)(cid:80)T\n\nt(cid:48)=t \u03b3t(cid:48)\u2212trt(cid:48)|\u03c0(cid:3). Because our\n\nuser simulator is essentially deterministic, one trajectory is suf\ufb01cient to estimate one action value.\nTherefore, an improved policy \u03c0(cid:48) can be derived from the current policy \u03c0 by selecting the best action\ngiven the value of the current policy, \u03c0(cid:48)(ht) \u2261 a\u2217\nt = arg maxa Q\u03c0(ht, a). Speci\ufb01cally, following\n[51], we minimize the cross entropy loss given the derived action, a\u2217,\n\n(cid:105)\n\nLimp = E(cid:104) \u2212 T(cid:88)\n\n(cid:16)\n\nlog\n\n\u03c0(a\u2217\n\nt|ht)\n\n(cid:17)(cid:105)\n\n(2)\n\nCompared to traditional policy gradient methods, the model-based policy improvement gradients\nhave lower variance, and converge faster. In Section 5, we further demonstrated the effectiveness of\nmodel-based policy improvement by comparing it with a recent policy gradient method. Figure 3b\nillustrates our policy learning method as described above.\n\nt=1\n\n4 Dataset: Relative Captioning\n\nOur user simulator aims to capture the rich and \ufb02exible language describing visual differences of\nany given image pair. The relative captioning dataset thus needs this property. We situated the data\ncollection procedure in a scenario of a shopping chatting session between a shopping assistant and a\ncustomer. The annotator was asked to take the role of the customer and provide a natural expression to\ninform the shopping assistant about the desired product item. To promote more regular, speci\ufb01c, and\nrelative user feedback, we provided a sentence pre\ufb01x for the annotator to complete when composing\ntheir response to a retrieved image. Otherwise the annotator response is completely free-form: no\nother constraints on the response were imposed. We used Amazon Mechanical Turk to crowdsource\nthe relative expressions. After manually removing erroneous annotations, we collected in total 10, 751\ncaptions, with one caption per pair of images.\nInterestingly, we observed that when the target image and the reference image are suf\ufb01ciently\ndifferent, users often directly describe the visual appearance of the target image, rather than using\nrelative expressions (c.f. fourth example in Figure 7(b), Appendix A). This behavior mirrors the\ndiscriminative captioning problem considered in [21], where a method must take in two images\n\n4More explanation on the difference between the two objectives is provided in Appendix E.\n\n6\n\n\fand produce a caption that refers only to one of them. Relative and discriminative captioning are\ncomplementary, and in practice, both strategies are used, and so we augmented our dataset by\npairing 3600 captions that were discriminative with additional dissimilar images. Our captioner and\ndialog-based interactive retriever are thus trained on both discriminative and relative captions, so as\nto be respectively more representative of and responsive to real users. Additional details about the\ndataset collection procedure and the analysis on dataset statistics are included in Appendix A and\nAppendix B.\n\n5 Experimental Results\n\nIn Section 5.1, we assess the contribution of each component of our pipeline for policy learning.\nTo evaluate the value of using free-form dialog feedback, we show experiments considering both\nsimulated user feedback (Section 5.2) and real-world user feedback (Section 5.3).\nAll experiments were performed on the Shoes dataset [53], with the same training and testing data\nsplit for all retrieval methods and for training the user simulator. 10, 000 database images were used\nduring training, and 4, 658 images for testing. The retrieval models are tested by retrieving images\non the testing set, starting from a randomly selected candidate image for the \ufb01rst dialog turn. Image\nretrieval performance is quanti\ufb01ed by the average rank percentile of the image returned by the dialog\nmanager on the test set. For details on architectural con\ufb01gurations, parameter settings, baseline\nimplementation, please refer to Appendix D.\n\n5.1 Analysis of the Learning Framework\n\nWe use our proposed user simulator to generate\ndata and provide extensive quantitative analysis\non the contribution of each model component.\n\nResults on Relative Captioner. Figure 5 pro-\nvides examples of simulator generated feedback\nand the collected user annotations. An interest-\ning observation is that even though the user sim-\nulator only occasionally generates descriptions\nthat exactly match the human annotations (the\nthird example in Figure 5), it can still summarize\nthe main visual differences between the images,\nsince inherently there are many ways to describe\ndifferences between two images. Qualitative ex-\namination of the generated relative expressions\nshowed that the user simulator can approximate\nfeedback of real users at a very low annotation\ncost (more analysis is included in Appendix C).\n\nFigure 4: Quantitative comparison of our method\nand two baselines and methods using feedback\nbased on a pre-de\ufb01ned set of relative attributes.\n\nPolicy Learning Results. To investigate how retrieval performance is affected by each component\nof the dialog manager, we compare our approach, denoted as Ours, against two variants: (1) SL:\nsupervised learning where the agent is trained only with triplet loss; (2) RL-SCST: policy learning\nusing Self-Critical Sequence Training (SCST) [19] after pre-training the network using the triplet\nloss objective. As shown in Figure 4 (solid lines), the average ranking percentile of the target image\nin all methods increases monotonically as the number of dialog turns increases. Both RL-based\nretrieval algorithms outperform the supervised pre-training, SL, which is expected since the triplet\nloss function does not directly optimize the retrieval ranking objective. Finally, Ours achieves\n98% average ranking percentile with only two dialog turns and consistently outperforms RL-SCST\nacross different dialog turns, which demonstrates the bene\ufb01t of the model-based policy improvement\ncomponent. We have observed similar results on the attribute-based baselines. Each of the SL based\nattribute model underperforms its RL version by \u223c 1% in retrieval ranking percentile.\n\n7\n\n\fFigure 5: Examples of human provided (green) and captioner generated relative descriptions (blue).\nWhile generated relative captions don\u2019t resemble human annotations in most cases, they can nonethe-\nless capture the main visual differences between the target image and the reference image.\n\n5.2 Effectiveness of Natural Language Feedback\n\nIn this section, we empirically evaluate the effect of natural language feedback, compared to pre-\nde\ufb01ned, relative attribute-based user feedback.\nGenerating Attribute Feedback. Each image in the dataset maps to a 10-D attribute vector, as\ndescribed in [12]. We adopted a rule-based feedback generator which concatenates the respective\nattribute words with \u201cmore\u201d or \u201cless\u201d, depending on the relative attribute values of a given image\npair. For example, if the \u201cshiny\u201d value of the candidate image and the target image are 0.9 and 1.0\nrespectively, then the rule-based feedback is \u201cmore shiny.\u201d Attributes are randomly sampled, similar\nto the relative attribute feedback generation in [12]. To simulate the scenario when users provide\nfeedbacks using multiple attributes, individual attribute phrases are combined. We adopted original\nattribute values in [12], which were predicted using hand-crafted image features, as well as attribute\nvalues predicted using deep neural networks in [54].\nResults. We trained the dialog manager using both dialog-based feedback and attribute-based\nfeedback (Attrn and Attrn (deep)), where the subscript number denotes the number of attributes used\nin the rule-based feedback generator and (deep) denote baselines using deep learning based attribute\nestimates as in [54]. The empirical result is summarized in Figure 4, including relative attribute\nfeedback using one, three and ten attribute phrases. The three attribute case matches the average\nlength of user feedback in free-form texts and the ten case uses all possible pre-de\ufb01ned attributes\nto provide feedback. Across different numbers of dialog turns, the natural language based agent\nproduced signi\ufb01cantly higher target image average ranking percentile than the attribute based methods.\nThe results suggest that feedback based on unrestricted natural language is more effective for retrieval\nthan the prede\ufb01ned set of relative attributes used in [12]. This is expected as the vocabulary of\nrelative attributes in [12] is limited. Even though deep learning based attribute estimates improve the\nattribute-based baselines signi\ufb01cantly, the performance gap between attribute based baseline and free\nform texts is still signi\ufb01cant. We conjecture that the main reason underlying the performance gap\nbetween attribute and free-form text based models is the effectively open domain for attribute use,\nwhich is dif\ufb01cult to realize in a practical user interface without natural language. In fact, free-form\ndialog feedback obviates constructing a reliable and comprehensive attribute taxonomy, which in\nitself is a non-trivial task [55].\n\n5.3 User Study of Dialog-based Image Retrieval\n\nIn this section, we demonstrate the practical use of our system with real users. We compare with\nan existing method, WhittleSearch [12], on the task of interactive footwear retrieval. WhittleSearch\nrepresents images as feature vectors in a pre-de\ufb01ned 10-D attribute space, and iteratively re\ufb01nes\nretrieval by incorporating the user\u2019s relative feedback on attributes to narrow down the search space\nof the target image. For each method, we collected 50 \ufb01ve-turn dialogs; at each turn, one image is\npresented to the user to seek feedback. For WhittleSearch, the user can choose to use any amount\nof attributes to provide relative feedback on during each interaction. The resulting average ranking\npercentile of the dialog manager and WhittleSearch are 89.9% and 70.3% respectively. In addition to\nimproved retrieval accuracy, users also reported that providing dialog-based feedback is more natural\ncompared to selecting the most relevant attributes from a pre-de\ufb01ned list.\nFigure 6 shows examples of retrieval dialogs from real users (please refer to Appendix E for more\nresults and discussions). We note that users often start the dialog with a coarse description of the\n\n8\n\nTargetReferenceTargetReferenceTargetReferenceTargetReferenceTargetReferencelight grey sneakers with Velcro(Unlike the provided image, the ones I want) are suedeare all blackare redis bolder with cow pattern and more ridged soleare whiteis darker in colorare suede with a closed toeare redhas a print with a strap\fFigure 6: Examples of the user interacting with the proposed dialog-based image retrieval framework.\n\nmain visual features (color, category) of the target. As the dialog progresses, users give more speci\ufb01c\nfeedback on \ufb01ne-grained visual differences. The bene\ufb01t of free-form dialog can be seen from the\n\ufb02exible usage of rich attribute words (\u201cleopard print on straps\u201d), as well as relative phrases (\u201cthinner\u201d,\n\u201chigher heel\u201d). Overall, these results show that the proposed framework for the dialog manager\nexhibits promising behavior on generalizing to real-world applications.\n\n6 Conclusions\n\nThis paper introduced a novel and practical task residing at the intersection of computer vision and\nlanguage understanding: dialog-based interactive image retrieval. Ultimately, techniques that are\nsuccessful on such tasks will form the basis for the high \ufb01delity, multi-modal, intelligent conversa-\ntional systems of the future, and thus represent important milestones in this quest. We demonstrated\nthe value of the proposed learning architecture on the application of interactive fashion footwear\nretrieval. Our approach, enabling users to provide natural language feedback, signi\ufb01cantly outper-\nforms traditional methods relying on a pre-de\ufb01ned vocabulary of relative attributes, while offering\nmore natural communication. As future work, we plan to leverage side information, such as textual\ndescriptions associated with images of product items, and to develop user models that are conditioned\non dialog histories, enabling more realistic interactions. We are also optimistic that our approach for\nimage retrieval can be extended to other media types such as audio, video, and e-books, given the\nperformance of deep learning on tasks such as speech recognition, machine translation, and activity\nrecognition.\nAcknowledgement. We would like to give special thanks to Professor Kristen Grauman for helpful\ndiscussions.\n\nReferences\n[1] Junshi Huang, Rogerio S Feris, Qiang Chen, and Shuicheng Yan. Cross-domain image retrieval with a\n\ndual attribute-aware ranking network. In ICCV, 2015.\n\n[2] Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes\n\nrecognition and retrieval with rich annotations. In CVPR, 2016.\n\n[3] Daniel A Vaquero, Rogerio S Feris, Duan Tran, Lisa Brown, Arun Hampapur, and Matthew Turk. Attribute-\n\nbased people search in surveillance environments. In WACV, 2009.\n\n[4] Zhiyuan Shi, Timothy M Hospedales, and Tao Xiang. Transferring a semantic representation for person\n\nre-identi\ufb01cation and search. In CVPR, 2015.\n\n[5] Albert Gordo, Jon Almaz\u00e1n, Jerome Revaud, and Diane Larlus. Deep image retrieval: Learning global\n\nrepresentations for image search. In ECCV, 2016.\n\n[6] Herve Jegou, Florent Perronnin, Matthijs Douze, Jorge S\u00e1nchez, Patrick Perez, and Cordelia Schmid.\nAggregating local image descriptors into compact codes. IEEE Transactions on Pattern Analysis and\nMachine Intelligence, 34(9):1704\u20131716, 2012.\n\n[7] Jiang Wang, Yang Song, Thomas Leung, Chuck Rosenberg, Jingbin Wang, James Philbin, Bo Chen, and\n\nYing Wu. Learning \ufb01ne-grained image similarity with deep ranking. In CVPR, 2014.\n\n[8] Albert Gordo, Jon Almazan, Jerome Revaud, and Diane Larlus. End-to-end learning of deep visual\n\nrepresentations for image retrieval. International Journal of Computer Vision, 124(2):237\u2013254, 2017.\n\n[9] Xiang Sean Zhou and Thomas S Huang. Relevance feedback in image retrieval: A comprehensive review.\n\nMultimedia systems, 8(6):536\u2013544, 2003.\n\n9\n\nAre strappy high heelsHas an animal printHas leopard print on strapsTargetAre white sneakersIs thinnerHas a higher heelTarget\f[10] Bart Thomee and Michael S Lew. Interactive search in image retrieval: a survey. International Journal of\n\nMultimedia Information Retrieval, 1(2):71\u201386, 2012.\n\n[11] Yong Rui, Thomas S Huang, Michael Ortega, and Sharad Mehrotra. Relevance feedback: a power tool for\ninteractive content-based image retrieval. IEEE Transactions on circuits and systems for video technology,\n8(5):644\u2013655, 1998.\n\n[12] Adriana Kovashka, Devi Parikh, and Kristen Grauman. Whittlesearch: Image search with relative attribute\n\nfeedback. In CVPR, 2012.\n\n[13] Stefanie Tellex and Deb Roy. Towards surveillance video search by natural language query. In ACM\n\nInternational Conference on Image and Video Retrieval, 2009.\n\n[14] Andrei Barbu, N Siddharth, and Jeffrey Mark Siskind. Saying what you\u2019re looking for: Linguistics meets\n\nvideo search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(10), 2016.\n\n[15] Shuang Li, Tong Xiao, Hongsheng Li, Bolei Zhou, Dayu Yue, and Xiaogang Wang. Person search with\n\nnatural language description. In CVPR, 2017.\n\n[16] Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jiashi Feng, Kate Saenko, and Trevor Darrell. Natural\n\nlanguage object retrieval. In CVPR, 2016.\n\n[17] Girish Kulkarni, Visruth Premraj, Sagnik Dhar, Siming Li, Yejin Choi, Alexander C Berg, and Tamara L\n\nBerg. Baby talk: Understanding and generating image descriptions. In CVPR, 2011.\n\n[18] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image\n\ncaption generator. In CVPR, 2015.\n\n[19] Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, and Vaibhava Goel. Self-critical\n\nsequence training for image captioning. In CVPR, 2017.\n\n[20] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT Press, 1998.\n\n[21] Ramakrishna Vedantam, Samy Bengio, Kevin Murphy, Devi Parikh, and Gal Chechik. Context-aware\n\ncaptions from context-agnostic supervision. In CVPR, 2017.\n\n[22] Myron Flickner, Harpreet Sawhney, et al. Query by image and video content: The qbic system. IEEE\n\nComputer, 28(9):23\u201332, 1995.\n\n[23] Yong Rui, Thomas S Huang, and Shih-Fu Chang. Image retrieval: Current techniques, promising directions,\n\nand open issues. Journal of visual communication and image representation, 10(1):39\u201362, 1999.\n\n[24] Hong Wu, Hanqing Lu, and Songde Ma. Willhunter: interactive image retrieval with multilevel relevance.\n\nIn ICPR, 2004.\n\n[25] Adriana Kovashka and Kristen Grauman. Attributes for image retrieval. In Visual Attributes. Springer,\n\n2017.\n\n[26] Adriana Kovashka and Kristen Grauman. Attribute pivots for guiding relevance feedback in image search.\n\nIn ICCV, 2013.\n\n[27] Devi Parikh and Kristen Grauman. Relative attributes. In ICCV, 2011.\n\n[28] Aron Yu and Kristen Grauman. Fine-grained comparisons with attributes. In Visual Attributes. Springer,\n\n2017.\n\n[29] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick,\n\nand Devi Parikh. Vqa: Visual question answering. In ICCV, 2015.\n\n[30] Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler.\n\nMovieqa: Understanding stories in movies through question-answering. In CVPR, 2016.\n\n[31] Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Tomas Mikolov, et al. Devise: A\n\ndeep visual-semantic embedding model. In NIPS, 2013.\n\n[32] Liwei Wang, Yin Li, and Svetlana Lazebnik. Learning deep structure-preserving image-text embeddings.\n\nIn CVPR, 2016.\n\n[33] Anna Rohrbach, Marcus Rohrbach, Ronghang Hu, Trevor Darrell, and Bernt Schiele. Grounding of textual\n\nphrases in images by reconstruction. In ECCV, 2016.\n\n10\n\n\f[34] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana\nLazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence\nmodels. In ICCV, 2015.\n\n[35] Jason D Williams and Steve Young. Partially observable markov decision processes for spoken dialog\n\nsystems. Computer Speech & Language, 21(2):393\u2013422, 2007.\n\n[36] Iulian Vlad Serban, Alessandro Sordoni, Yoshua Bengio, Aaron C Courville, and Joelle Pineau. Building\n\nend-to-end dialogue systems using generative hierarchical neural network models. In AAAI, 2016.\n\n[37] Antoine Bordes and Jason Weston. Learning end-to-end goal-oriented dialog. In ICLR, 2017.\n\n[38] Xiaoxiao Guo, Tim Klinger, Clemens Rosenbaum, Joseph P Bigus, Murray Campbell, Ban Kawas, Kartik\nTalamadupula, Gerry Tesauro, and Satinder Singh. Learning to query, reason, and answer questions on\nambiguous texts. In ICLR, 2017.\n\n[39] Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, Jos\u00e9 MF Moura, Devi Parikh, and\n\nDhruv Batra. Visual dialog. In CVPR, 2017.\n\n[40] Harm de Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron Courville.\n\nGuesswhat?! visual object discovery through multi-modal dialogue. In CVPR, 2017.\n\n[41] Paul Hongsuck Seo, Andreas Lehrmann, Bohyung Han, and Leonid Sigal. Visual reference resolution\n\nusing attention memory for visual dialog. In NIPS, 2017.\n\n[42] Abhishek Das, Satwik Kottur, Jos\u00e9 MF Moura, Stefan Lee, and Dhruv Batra. Learning cooperative visual\n\ndialog agents with deep reinforcement learning. In ICCV, 2017.\n\n[43] Florian Strub, Harm de Vries, Jeremie Mary, Bilal Piot, Aaron Courville, and Olivier Pietquin. End-to-end\n\noptimization of goal-driven and visually grounded dialogue systems. In IJCAI, 2017.\n\n[44] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.\n\nIn CVPR, 2016.\n\n[45] Yoon Kim. Convolutional neural networks for sentence classi\ufb01cation. In EMNLP, 2014.\n\n[46] Cheng Li, Paul Resnick, and Qiaozhu Mei. Multiple queries as bandit arms. In CIKM. ACM, 2016.\n\n[47] Xuanhui Wang, Cheng Li, Nadav Golbandi, Michael Bendersky, and Marc Najork. The lambdaloss\n\nframework for ranking metric optimization. In CIKM, 2018.\n\n[48] Xiujun Li, Zachary C Lipton, Bhuwan Dhingra, Lihong Li, Jianfeng Gao, and Yun-Nung Chen. A user\n\nsimulator for task-completion dialogues. arXiv preprint arXiv:1612.05688, 2016.\n\n[49] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in\n\nreferring expressions. In ECCV, 2016.\n\n[50] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel,\nand Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML,\n2015.\n\n[51] Xiaoxiao Guo, Satinder Singh, Honglak Lee, Richard L Lewis, and Xiaoshi Wang. Deep learning for\n\nreal-time atari game play using of\ufb02ine monte-carlo tree search planning. In NIPS, 2014.\n\n[52] Cheng Li, Yue Wang, Paul Resnick, and Qiaozhu Mei. Req-rec: High recall retrieval with query pooling\n\nand interactive classi\ufb01cation. In SIGIR. ACM, 2014.\n\n[53] Tamara L Berg, Alexander C Berg, and Jonathan Shih. Automatic attribute discovery and characterization\n\nfrom noisy web data. In ECCV, 2010.\n\n[54] X. Yang, T. Zhang, C. Xu, S. Yan, M. S. Hossain, and A. Ghoneim. Deep relative attributes. IEEE\n\nTransactions on Multimedia, 18(9):1832\u20131842, Sep. 2016.\n\n[55] Subhransu Maji. Discovering a lexicon of parts and attributes. In ECCV, 2012.\n\n11\n\n\f", "award": [], "sourceid": 390, "authors": [{"given_name": "Xiaoxiao", "family_name": "Guo", "institution": "IBM Research"}, {"given_name": "Hui", "family_name": "Wu", "institution": "IBM Research"}, {"given_name": "Yu", "family_name": "Cheng", "institution": "Microsoft AI & Research"}, {"given_name": "Steven", "family_name": "Rennie", "institution": "Fusemachines"}, {"given_name": "Gerald", "family_name": "Tesauro", "institution": "IBM TJ Watson Research Center"}, {"given_name": "Rogerio", "family_name": "Feris", "institution": "IBM Research AI"}]}