{"title": "Answerer in Questioner's Mind: Information Theoretic Approach to Goal-Oriented Visual Dialog", "book": "Advances in Neural Information Processing Systems", "page_first": 2579, "page_last": 2589, "abstract": "Goal-oriented dialog has been given attention due to its numerous applications in artificial intelligence.\nGoal-oriented dialogue tasks occur when a questioner asks an action-oriented question and an answerer responds with the intent of letting the questioner know a correct action to take. \nTo ask the adequate question, deep learning and reinforcement learning have been recently applied. \nHowever, these approaches struggle to find a competent recurrent neural questioner, owing to the complexity of learning a series of sentences.\nMotivated by theory of mind, we propose \"Answerer in Questioner's Mind\" (AQM), a novel information theoretic algorithm for goal-oriented dialog. \nWith AQM, a questioner asks and infers based on an approximated probabilistic model of the answerer.\nThe questioner figures out the answerer\u2019s intention via selecting a plausible question by explicitly calculating the information gain of the candidate intentions and possible answers to each question.\nWe test our framework on two goal-oriented visual dialog tasks: \"MNIST Counting Dialog\" and \"GuessWhat?!\".\nIn our experiments, AQM outperforms comparative algorithms by a large margin.", "full_text": "Answerer in Questioner\u2019s Mind:\nInformation Theoretic Approach to\n\nGoal-Oriented Visual Dialog\n\nSang-Woo Lee1\u2217, Yu-Jung Heo2, and Byoung-Tak Zhang2,3\n\nClova AI Research, Naver Corp1\n\nSeoul National University2\n\nSurromind Robotics3\n\nAbstract\n\nGoal-oriented dialog has been given attention due to its numerous applications in\narti\ufb01cial intelligence. Goal-oriented dialogue tasks occur when a questioner asks\nan action-oriented question and an answerer responds with the intent of letting\nthe questioner know a correct action to take. To ask the adequate question, deep\nlearning and reinforcement learning have been recently applied. However, these\napproaches struggle to \ufb01nd a competent recurrent neural questioner, owing to the\ncomplexity of learning a series of sentences. Motivated by theory of mind, we\npropose \u201cAnswerer in Questioner\u2019s Mind\u201d (AQM), a novel information theoretic\nalgorithm for goal-oriented dialog. With AQM, a questioner asks and infers based\non an approximated probabilistic model of the answerer. The questioner \ufb01gures out\nthe answerer\u2019s intention via selecting a plausible question by explicitly calculating\nthe information gain of the candidate intentions and possible answers to each\nquestion. We test our framework on two goal-oriented visual dialog tasks: \u201cMNIST\nCounting Dialog\u201d and \u201cGuessWhat?!\u201d. In our experiments, AQM outperforms\ncomparative algorithms by a large margin.\n\n1\n\nIntroduction\n\nGoal-oriented dialog is a classical arti\ufb01cial intelligence problem that needs to be addressed for digital\npersonal assistants, order-by-phone tools, and online customer service centers. Goal-oriented dialog\noccurs when a questioner asks an action-oriented question and an answerer responds with the intent\nof letting the questioner know a correct action to take. Signi\ufb01cant research on goal-oriented dialog\nhas tackled this problem using from the rule-based approach to the end-to-end neural approach [1\u20133].\nMotivated by the achievement of neural chit-chat dialog research [4], recent studies on goal-oriented\ndialogs have utilized deep learning, using massive data to train their neural networks in self-play\nenvironments. In this setting, two machine agents are trained to make a dialog to achieve the goal\nof the task in a cooperative way [5\u20137]. Many researchers attempted to solve goal-oriented dialog\ntasks by using the deep supervised learning (deep SL) approach [8] based on seq2seq models [9]\nor the deep reinforcement learning (deep RL) approach utilizing rewards obtained from the result\nof the dialog [10, 11]. However, these methods struggle to \ufb01nd a competent RNN model that uses\nback-propagation, owing to the complexity of learning a series of sentences. These algorithms tend to\ngenerate redundant sentences, making generated dialogs inef\ufb01cient [12, 13].\n\n\u2217Work carried out at Seoul National University\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\fFigure 1: (Left) Illustration of an AQM algorithm for goal-oriented visual dialog. AQM makes a\ndecision tree on the image for asking ef\ufb01cient questions. (Right) Examples of the GuessWhat?! game.\nThe goal of GuessWhat?! is to locate the correct object in the image. The green mask highlights the\ncorrect object.\n\nOur idea to deal with goal-oriented dialog is motivated by theory of mind [14], the ability to attribute\nmental states to others and to understand how our mental states are different. In this approach, an\nagent considers what the collaborator, the opposite agent cooperating in dialog, will respond by using\nan explicit approximated model of the collaborator. If one wishes to ef\ufb01ciently convey information\nto the other, it is best to converse in a way that maximizes the other\u2019s understanding [15]. For our\nmethod, we consider the mind to be beyond a part of mental states (e.g., belief, intent, knowledge).\nThe mind is the probabilistic distribution of the model of the collaborator itself.\nWe propose an \u201cAnswerer in Questioner\u2019s Mind\u201d (AQM) algorithm for goal-oriented dialog (Figure\n1 (Left)). AQM allows a questioner to ask appropriate consecutive questions with information\ntheoretic approach [16, 17]. AQM\u2019s questioner explicitly possesses an approximated model of the\nanswerer. The questioner utilizes the approximated model to calculate the information gain of the\ncandidate answerer\u2019s intentions and answers for each question. In the example of Figure 1 (Left),\nthe answerer\u2019s intention is the correct object highlighted by the green mask. Using the candidate\nobjects and the approximated answerer model, the questioner makes an ef\ufb01cient question which\nsplits out the candidate objects properly. \u201cIs this machinery?\u201d is selected in the \ufb01rst turn, because\nthe question separates the candidate objects evenly and thus has maximum information gain. In our\nmain experiment, AQM\u2019s question generator extracts proper questions from the training data, not\ngenerating new questions. However, in the discussion section, we extend AQM to generate novel\nquestions for the test image.\nWe test AQM mainly on goal-oriented visual dialog tasks in the self-play environment. Our main\nexperiment is conducted on \u201cGuessWhat?!\u201d, a cooperative two-player guessing game on the image\n(Figure 1 (Right)). AQM achieves an accuracy of 63.63% in 3 turns and 78.72% in 10 turns, outper-\nforming deep SL (46.8% in 5 turns) [6] and deep RL (52.3% in 4.1 turns) [18] algorithms. Though\nwe demonstrate the performance of our models in visual dialog tasks, our approach can be directly\napplied to general goal-oriented dialog where there is a non-visual context.\nOur main contributions are four folds.\n\noutperforms comparative methods.\n\n\u2022 We propose the AQM, a practical goal-oriented dialog system motivated by theory of mind.\nThe AQM framework is general and not rely on a speci\ufb01c model representation nor a\nlearning method. We compare various types of learning strategy on the model and selecting\nstrategy for the candidate questions.\n\u2022 We test our AQM on two goal-oriented visual dialog tasks, showing that our method\n\u2022 We use AQM as a tool to understand existing deep learning methods in goal-oriented dialog\nstudies. Section 5.1 and Appendix D include 1) the relationship between the hidden vector\nin comparative models and the posterior in AQM, 2) the relationship between the objective\nfunction of RL and AQM, and 3) a point to be considered on self-play with RL for making\nan agent to converse with a human.\n\u2022 We extend AQM to generate questions, in which case AQM can be understood as a way to\n\nboost the existing deep learning method in Section 5.2.\n\n2\n\n\f2 Previous Works\n\nOur study is related to various research \ufb01elds, including goal-oriented dialog [1\u20133, 10, 11], language\nemergence [5, 19], the theory of mind [20\u201323], referring game [22, 24], pragmatics [25\u201328], and\nvisual dialog [6, 7, 12, 13, 18]. In this section, we highlight three topics as below, obverter, opponent\nmodeling, and information gain.\nObverter Choi et al. recently applied the obverter technique [21], motivated by theory of mind, to\nstudy language emergence [22]. The task of the study is an image-description-match classi\ufb01cation.\nIn their experiments, one agent transmitted one sentence for describing an arti\ufb01cial image to the\ncollaborator agent. In their study, the obverter technique can be understood as that an agent plays\nboth questioner and answerer, maximizing the consistency between visual and language modules.\nTheir experimental results showed that their obverter technique generated a word corresponding to a\nspeci\ufb01c object (e.g. \u2018bbbbbbb{b,d}\u2019 for a blue box). They argued their method could be an alternative\nto RL-based language emergence systems. Compared to their model, however, AQM uses real images,\ncreates multi-turn dialog, and can be used for general goal-oriented dialog tasks.\nOpponent Modeling Studies on opponent modeling have treated simple games with a multi-agent\nenvironment where an agent competed with the other [23]. In the study of Foerster et al. [29], the\nagent has the model of the opponent and updates it assuming the opponent will be updated by gradient\ndescent with RL. They argued modeling opponent could be applied to track the non-stationary\nbehavior of an opponent agent. Their model outperformed classical RL methods in simple games,\nsuch as tic-tac-toe and rock-paper-scissors. On the other hand, AQM applied opponent modeling to a\ncooperative multi-agent setting. We believe that opponent modeling can also be applied to dialog\nsystems in which agents are partially cooperative and partially competitive.\nIn a broader sense, our study can also be understood as extending these studies to a multi-turn visual\ndialog, as the referring game is a special case of single-turn visual dialog.\nInformation Gain AQM\u2019s question-generator optimizes information gain using an approximated\ncollaborator model. However, the concept of utilizing information gain in a dialog task is not new for\na classical rule-based approach. Polifroni and Walker used information gain to build a dialog system\nfor restaurant recommendations [30]. They made a decision tree using information gain and asked a\nseries of informative questions about restaurant preferences. Rothe et al. applied a similar method\nto generate questions on a simple Battleship game experiment [31]. It is noticeable that they used\npre-de\ufb01ned logic to generate questions with information gain criteria to make novel (i.e., not shown\nin the training dataset) and human-like questions. Unlike these previous studies, AQM makes a new\ndecision tree for every new context; asking a question in AQM corresponds to constructing a node\nin decision tree classi\ufb01er. In the example of Figure 1 (Left), AQM makes a new decision tree for a\ntest image. AQM also considers uncertainty by deep learning, and does not require hand-made or\ndomain-speci\ufb01c rules.\n\n3 Answerer in Questioner\u2019s Mind (AQM)\n\nPreliminary In our experimental setting, two machine players, a questioner and an answerer, commu-\nnicate via natural dialog. Speci\ufb01cally, there exists a target class c, which is an answerer\u2019s intention or\na goal-action the questioner should perform. The answerer knows the class c, whereas the questioner\ndoes not. The goal of the dialog for the questioner is to \ufb01nd the correct class c by asking a series of\nquestions to the answerer. The answerer responds the answer to the given question.\nWe treat C, Qt, and At as random variables of class, t-th question, and t-th answer, respectively. c,\nqt, and at becomes their single instance. In a restaurant scenario example, qt can be \u201cWould you like\nto order?\u201d or \u201cWhat can I do for you?\u201d at can be \u201cTwo coffees, please.\u201d or \u201cWhat\u2019s the password for\nWi-Fi?\u201d c can then be \u201cReceive the order of two hot Americanos.\u201d or \u201cLet the customer know the\nWi-Fi password.\u201d\nModel Structure We illustrate the difference between the existing deep learning framework [6, 18,\n13] and the proposed AQM framework in Figure 2. The answerer systems of two frameworks are the\nsame. The answerer works as a kind of simple neural network models for visual question answering\n(VQA) and the network is trained on the training data. On the other hand, the questioner of two\nframeworks works differently.\n\n3\n\n\fFigure 2: Comparative illustration on modules of the existing deep learning framework and an AQM\nframework. The AQM\u2019s guesser computes Equation 2 and the AQM\u2019s question-generator computes\nEquation 3. ht\u22121 is the previous history (a1:t\u22121, q1:t\u22121).\n\nIn deep SL and RL methods [6, 18], the questioner has two RNN-based models, one is the question-\ngenerator to generate a question and the other is the guesser to classify a class. In AQM, these two\nRNN-based models are substituted by the mathematical calculation of the equation and argmax opera-\ntion, not the model. The question-generator selects a question from candidate questions sampled from\nQ-sampler, which makes a pre-de\ufb01ned question set before the game starts. The question-generator\ncalculates information gain I[C, At; qt, a1:t\u22121, q1:t\u22121] for each question in candidate questions. I\nis the information gain or mutual information of the class C and the current answer At, where the\nprevious history ht\u22121 = (a1:t\u22121, q1:t\u22121) and the current question qt are given. Note that maximizing\ninformation gain is the same as minimizing the conditional entropy of class C, given a current answer\nAt.\n\nI[C, At; qt, a1:t\u22121, q1:t\u22121]\n=H[C; a1:t\u22121, q1:t\u22121] \u2212 H[C|At; qt, a1:t\u22121, q1:t\u22121]\n\np(c|a1:t\u22121, q1:t\u22121)p(at|c, qt, a1:t\u22121, q1:t\u22121) ln\n\n(cid:88)\n\n(cid:88)\n\n=\n\np(at|c, qt, a1:t\u22121, q1:t\u22121)\np(at|qt, a1:t\u22121, q1:t\u22121)\n\n(1)\n\nat\n\nc\n\nThe guesser calculates the posterior of class p(c|a1:t, q1:t) \u221d p(c)(cid:81)t\n\nj=1 p(aj|c, qj, a1:j\u22121, q1:j\u22121).\nCalculation For the equation of both question-generator and guesser, the answerer\u2019s answer distri-\nbution p(at|c, qt, a1:t\u22121, q1:t\u22121) is required. The questioner has an approximated answer-generator\nnetwork to make the approximated answer distribution \u02dcp(at|c, qt, a1:t\u22121, q1:t\u22121), which we refer to\nas the likelihood \u02dcp. If at is a sentence, the probability of the answer-generator is extracted from the\nmultiplication of the word probability of RNN.\nThe AQM\u2019s guesser module calculates the posterior of class c, \u02c6p(c|a1:t, q1:t), based on the history\nht = (a1:t, q1:t), the likelihood \u02dcp, and the prior of class c, \u02c6p(cid:48)(c). Using the likelihood model, the\nguesser selects a maximum a posterior solution for classifying the class.\n\n\u02c6p(c|a1:t, q1:t) \u221d \u02c6p(cid:48)(c)\n\nt(cid:89)\n\nj=1\n\n\u02dcp(aj|c, qj, a1:j\u22121, q1:j\u22121)\n\n(2)\n\nWe use a term of likelihood as \u02dcp, prior as \u02c6p(cid:48), and posterior as \u02c6p from the perspective that the questioner\nclassi\ufb01es class c. During the conversation, \u02c6p can be calculated in a recursive way.\n\n4\n\n\fAlgorithm 1 AQM\u2019s Question-Generator\n\n\u02c6p(c) \u223c \u02c6p(cid:48)(c)-model\n\u02dcp(at|c, qt, a1:t\u22121, q1:t\u22121) \u223c \u02dcp(a|c, q)-model\nQ \u2190 Q-sampler\nfor t = 1:T do\n\nqt \u2190 argmaxqt\u2208Q \u02dcI[C, At; qt, a1:t\u22121, q1:t\u22121] in Eq. 3\nGet at from the answerer\nUpdate \u02c6p(c|a1:t, q1:t) \u221d \u02dcp(at|c, qt, a1:t\u22121, q1:t\u22121) \u00b7 \u02c6p(c|a1:t\u22121, q1:t\u22121)\n\nend for\n\nFigure 3: Procedural illustration on the AQM\u2019s question-generator.\n\nThe AQM\u2019s question-generator module selects q\u2217\nt , which has a maximum value of the approximated\ninformation gain \u02dcI[C, At; qt, a1:t\u22121, q1:t\u22121], or simply \u02dcI. To calculate the information gain \u02dcI, the\nquestion-generator module uses the likelihood \u02dcp and the posterior \u02c6p.\n\nq\u2217\nt = argmax\n\nqt\u2208Q\n\n= argmax\n\nqt\u2208Q\n\n\u02dcI[C, At; qt, a1:t\u22121, q1:t\u22121]\n\n(cid:88)\n\n(cid:88)\n\n\u02dcp(at|c, qt, a1:t\u22121, q1:t\u22121)\n\u02dcp(cid:48)(at|qt, a1:t\u22121, q1:t\u22121)\n\n\u02c6p(c|a1:t\u22121, q1:t\u22121)\u02dcp(at|c, qt, a1:t\u22121, q1:t\u22121) ln\n\nc\n\nat\n\nwhere \u02dcp(cid:48)(at|qt, a1:t\u22121, q1:t\u22121) = (cid:80)\n\n(3)\nc \u02c6p(c|a1:t\u22121, q1:t\u22121) \u00b7 \u02dcp(at|c, qt, a1:t\u22121, q1:t\u22121). Q-sampler is\nrequired to select the question from the candidate questions Q. In our main experiments in Section\n4.2, Q-sampler extracts candidate questions from the training data. In this case, AQM does not\ngenerate a new question for the test image. However, if Q-sampler uses a RNN-based model, AQM\ncan generate the question. We discuss this issue in Section 5.2.\nLearning In AQM, the answer-generator network in the questioner and the answerer does not\nshare the representation. Thus, we need to train the AQM\u2019s questioner. In the existing deep learning\nframework, SL and RL are used to train two RNN-based models of the questioner. In a typical deep\nSL method, questioner\u2019s RNNs are trained from the training data, which is the same or similar to the\ndata the answerer is trained from. In a typical deep RL method, the answerer and the questioner make\na conversation in the self-play environment. In this RL procedure, the questioner uses the answers\ngenerated from the answerer for end-to-end training, with reward from the result of the game. On the\nother hand, the AQM\u2019s questioner trains the approximated answer distribution of the answerer, the\nlikelihood \u02dcp. The likelihood \u02dcp can be obtained by learning training data as in deep SL methods, or\nusing the answers of the answerer obtained in the training phase of the self-play conversation as in\ndeep RL methods. As the objective function of RL or AQM does not guarantee human-like question\ngeneration [32], RL uses SL-based pre-training, whereas AQM uses an appropriate Q-sampler.\nAlgorithm 1 and Figure 3 explain the question-generator module procedure. The question-generator\nrequires the \u02c6p(cid:48)(c)-model for the prior, the \u02dcp(a|c, q)-model for the likelihood, and the Q-sampler for\nthe set of candidate questions. Additional explanations on AQM can be found in Appendix A.\n\n5\n\n\fFigure 4: (Left) Illustration of MNIST Counting Dialog, a simpli\ufb01ed version of MNIST Dialog\n[33]. (Right) Test accuracy of goal-oriented dialog from the MNIST Counting Dialog task. \u201cAcc\u201d is\nPropAcc, the average ratio of property recognition accuracy.\n\n4 Experiments\n\n4.1 MNIST Counting Dialog\n\nTo clearly explain the mechanism of AQM, we introduce the MNIST Counting Dialog task, which\nis a toy goal-oriented visual dialog problem, illustrated in Figure 4 (Left). Each image in MNIST\nCounting Dialog contains 16 small images of digit, each having four randomly assigned properties:\ncolor = {red, blue, green, purple, brown}, bgcolor = {cyan, yellow, white, silver, salmon}, number =\n{0, 1, \u00b7\u00b7\u00b7 , 9}, and style = {\ufb02at,stroke}. The goal of the MNIST Counting Dialog task is to inform the\nquestioner to pick the correct image among 10K candidate images via questioning and answering. In\nother words, class c is an index of the true target image (1 \u223c 10,000).\nFor the MNIST Counting Dialog task, we do not model the questioner and the answerer using neural\nnetworks. Instead, we de\ufb01ne the answer model in the questioner is count-based on 30K training\ndata. We set the average ratio of property recognition accuracy (PropAcc) \u03bb as 0.9, 0.95, and 1.0\nin Figure 4. For four properties such as color, bgcolor, number and style, each property recognition\naccuracy \u03bbcolor, \u03bbbgcolor, \u03bbnumber, \u03bbstyle is randomly sampled from an uniform distribution with a\nrange of [(2\u03bb \u2212 1), 1]. It contributes to add randomness or uncertainty to this task. For example, the\npercentage of correctly recognized color is 88% if \u03bbcolor is 0.88.\nAccording to the results in Figure 4, if the PropAcc \u03bb decreases, the accuracy of goal-oriented\ndialogue which has a goal to select the correct image is decreased by a large amount. Figure 4 (Right)\nshows that AQM nearly always chooses the true target image from 10K candidates in six turns if\nthe PropAcc \u03bb is 1.0. However, AQM also chooses correctly with a probability of 51% and 39%\n(accuracy of goal-oriented dialog) in six turns, when the PropAcc \u03bb is 0.95 and 0.90, respectively.\n\u201cRandom\u201d denotes a questioner with a random question-generator module and the AQM\u2019s guesser\nmodule. Detailed experimental settings can be found in Appendix B.\n\n4.2 GuessWhat?!\n\nGuessWhat?! Task GuessWhat?! is a cooperative two-player guessing game proposed by De Vries\net al. (Figure 1 (Right)) [6]. GuessWhat?! has received attention in the \ufb01eld of deep learning and\narti\ufb01cial intelligence as a testbed for research on the interplay of computer vision and dialog systems.\nThe goal of GuessWhat?! is to locate the correct object in a rich image scene by asking a sequence\nof questions. One participant, \u201cAnswerer\u201d, is randomly assigned an object in the image. The other\nparticipant, \u201cQuestioner,\u201d guesses the object assigned to the answerer. Both a questioner and an\nanswerer sees the image, but the correct object is known only to the answerer. To achieve the goal,\nthe questioner asks a series of questions, for which the answerer responds as \u201cyes,\u201d \u201cno,\u201d or \u201cn/a.\u201d\nThe questioner does not know a list of candidate objects while asking questions. When the questioner\ndecides to guess the correct object, a list of candidate objects is then revealed. A win occurs when\nthe questioner picks the correct object. The GuessWhat?! dataset contains 66,537 MSCOCO images\n[34], 155,280 games, and 831,889 question-answer pairs.\n\u02c6p(cid:48)(c)-model for the Prior The questioner does not know the list of candidate objects while asking\nquestions. This makes the GuessWhat?! task dif\ufb01cult, although the number of candidates is around 8.\n\n6\n\n\fFigure 5: Test accuracy from the GuessWhat?!.\nPrevious works do not report the performance\nchange with an increase in the number of turns.\n\nTable 1: Test accuracy from the GuessWhat?!.\n\nModel\nBaseline\nDeep SL (5-q) [6]\nDeep RL (4.1-q in Avg) [18]\nRandom-randQ-indA (5-q)\nRandom-randQ-depA (5-q)\nAQM-randQ-indA (5-q)\nAQM-countQ-indAhalf (5-q)\nAQM-countQ-indA (5-q)\nAQM-countQ-depAhalf (5-q)\nAQM-countQ-depA (5-q)\nAQM-countQ-depAhalf (10-q)\nAQM-countQ-depA (10-q)\nHuman [18]\n\nAccuracy\n\n16.04\n46.8\n52.3\n\n42.48 (\u00b1 0.84)\n46.36 (\u00b1 0.91)\n65.66 (\u00b1 0.55)\n61.64 (\u00b1 0.97)\n66.73 (\u00b1 0.76)\n70.90 (\u00b1 1.14)\n72.89 (\u00b1 0.70)\n77.35 (\u00b1 0.85)\n78.72 (\u00b1 0.54)\n\n84.4\n\nWe use YOLO9000, a real-time object detection algorithm, to estimate the set of candidate objects\n[35]. The prior \u02c6p(cid:48)(c) is set to 1/N, where N is the number of extracted objects.\n\u02dcp(a|q, c)-model for the Likelihood We use the answerer model from previous GuessWhat?! research\n[6]. The inputs of the answer-generator module consist of a VGG16 feature of a given context image,\na VGG16 feature of the cropped object in the context image, spatial and categorical information of\nthe cropped object, and the question qt at time step t. A simple multi-layer perceptron model uses\nthese features to classify the answer {yes, no, n/a}. Our answer-generator module assumes the answer\ndistribution is independent from the history ht\u22121 = (a1:t\u22121, q1:t\u22121). In other words, we approximate\nthe likelihood as \u02dcp(at|c, qt, a1:t\u22121, q1:t\u22121) \u221d \u02dcp(cid:48)(cid:48)(at|c, qt).\nWe use various strategy to train the questioner\u2019s approximated answer-generator network\n\u02dcp(at|c, qt, a1:t\u22121, q1:t\u22121) to approximate the answerer\u2019s answer distribution p(at|c, qt, a1:t\u22121, q1:t\u22121).\nIn \u201cindA,\u201d p and \u02dcp is trained separately for the same training data. In \u201cdepA,\u201d in which \u02dcp is trained\nfor the answer inferred from the answerer p, where the question and the image is also sampled from\nthe training data. The performance improvement of indA and depA setting would be achieved partly\nbecause the answerer and the questioner share the dataset. For ablation study, we also introduce\n\u201cindAhalf\u201d and \u201cdepAhalf\u201d setting. In indAhalf, p and \u02dcp is trained for the different dataset each, which\nthe training data is exclusively divided into halves. In depAhalf, p is trained for the \ufb01rst half training\ndata. After that, the questioner asks the questions in the second half training data to get the answer\nfrom p and use the answer as the training label.\nQ-sampler for the Candidate Question Set In the main experiments, we compare two Q-samplers\nwhich select the question from the training data. The \ufb01rst is \u201crandQ,\u201d which samples questions\nrandomly from the training data. The second is \u201ccountQ,\u201d which causes every other question from the\nset Q to be less dependent on the other. countQ checks the dependency of two questions with the\nfollowing rule: the probability of that two sampled questions\u2019 answers are the same cannot exceed\na \u02dcp\u2020(ai = a|qi, aj = a, qj) < 0.95, where \u02dcp\u2020(ai|qi, aj, qj) is derived from\nthe count of a pair of answers for two questions in the training data. \u02dcp made by indA is used for\ncountQ. We set the size of Q to 200.\nExperimental Results Figure 5 and Table 1 shows the experimental results. Figure 6 illustrates\nthe generated dialog. Our best algorithm, AQM-countQ-depA, achieved 63.63% in three turns,\noutperforming deep SL and deep RL algorithms. By allowing ten questions, the algorithms achieved\n78.72% and reached near-human performance. If the answerer\u2019s answer distribution p is directly\nused for the questioner\u2019s answer distribution \u02dcp (i.e., \u02dcp = p), AQM-countQ achieved 63.76% in three\nturns and 81.96% in ten turns (AQM-countQ-trueA in Figure 5). depA remarkably improved the\nscore in self-play but did not increased the quality of the generated dialog signi\ufb01cantly. On the other\nhand, countQ did not improve the score much but increased the quality of the generated dialog.\nIt is noticeable that the performance gap between indAhalf and indA is much larger than the gap\n\n95%. In other words,(cid:80)\n\n7\n\n\fFigure 6: Generated dialogs from our algorithm and the comparative algorithms. The tested games\nare sampled from the selected results of previous papers [6, 18].\n\nbetween depAhalf and depA. This result shows that the conversation between the questioner and\nthe answerer affects AQM\u2019s performance improvement more than sharing the training data between\nthe questioner and the answerer. The compared deep SL method used the question-generator with\nthe hierarchical recurrent encoder-decoder [36], achieving an accuracy of 46.8% in \ufb01ve turns [6].\nHowever, Random-randQ-depA achieved 46.36% in \ufb01ve turns, which is a competitive result to the\ndeep SL model. \u201cRandom\u201d denotes random question generation from the randQ set. The comparative\ndeep RL method applied reinforcement learning on long short-term memory, achieving 52.3% in\nabout 4.1 turns [18]. The deep RL has a module to decide whether the dialog is stopped or not. 4.1 is\nthe number of the averaged turns.\n\n5 Discussion\n\n5.1 Comparing AQM with SL and RL\n\nOur study replaced the task of training an RNN which generates questions with the task of training\na neural network which infers the probability of the answers. In the perspective of the hidden\nrepresentation to track the dialog, the contexts of history which AQM\u2019s questioner requires are the\nposterior \u02c6p and the history itself ht = (a1:t, q1:t) used as an input for the likelihood \u02dcp. In deep SL\nand RL methods, hidden neurons in RNN are expected to track the context of history. If the question\nto be asked is independent from the previous questions, the only context AQM should track is the\nposterior \u02c6p. In this case, the posterior \u02c6p in the yellow box of Figure 3 corresponds to the hidden vector\nof the RNN in the comparative dialog studies.\nMoreover, we argue that, in Appendix D, AQM and RL have a similar objective function, just as\ninformation gain in decision tree is used to classify. Many researchers have studied dialog systems\nfor cooperative games using deep RL, to increase the score in self-play environments [32, 37]. In\nAppendix D, we leverage AQM as a tool for analyzing the deep RL approach on goal-oriented dialog\ntasks from the perspective of theory of mind. According to our argument, training two agents to make\nplausible dialogs via rewards during self-play is not adaptable to a service scenario. To enable an\nagent to converse with a human, the opponent agent in self-play should model a human as much\nas possible. We prove that AQM and RL have a similar objective function, implying that RL-based\ntraining for the questioner can also be seen as implicit approximation on the answer distribution of\nthe answerer.\n\n8\n\n\f5.2 Generating Questions\n\nFor making a question more relevant to the new context of the dialog, a question needs to be generated.\nExtracting a question from the training dataset (randQ and countQ) is just one of the settings the\nAQM\u2019s Q-sampler can have. AQM can generate questions by using rule-based program [31] or a\nseq2seq models previously used in goal-oriented dialog studies. As a simple extension, we test the\n\u201cgen1Q\u201d setting, which uses a previous deep SL question-generator [6] to generate a \ufb01rst-turn question\nfor the test image. We use a beam search to make a set of questions sorted by its likelihood, and\nselect top-100 questions for the candidates. In the experiments on GuessWhat?!, AQM-gen1Q-depA\nachieves a slight performance improvement over AQM-countQ-depA at 2-q (49.79% \u2192 51.07%)\noutperforming the original deep SL method (46.8% in 5-q). However, at 5-q, AQM-gen1Q performs\nslightly worse than AQM-countQ-depA (72.89% \u2192 70.74%). If the Q-sampler generates questions\nthrough the seq2seq model using the history of dialog at every turn, the performance would be\nimproved further. Appendix E discusses the further direction of future works to make the question\nagent applicable for service.\nFigure 6 shows the generated dialogs of AQM-gen1Q. gen1Q tends to make more redundant sentences\nthan countQ because countQ carefully checks the similarity between questions. However, gen1Q\ntends to make the questions more related to the image. It is also noticeable that there are questions\nconcatenating two sentences with \u201cor\u201d in gen1Q. The score of the game is insuf\ufb01cient to evaluate\nthe quality of the generated dialog. Appendix C discusses the objective function of the goal-oriented\ndialog, mainly based on the case of goal-oriented visual dialog studies.\n\n6 Conclusion\n\nWe proposed \u201cAnswerer in Questioner\u2019s Mind\u201d (AQM), a practical goal-oriented dialog framework\nusing information-theoretic approach. In AQM, the questioner approximates the answerer\u2019s answer\ndistribution in dialog. In our experiments, AQM outperformed deep SL and RL methods. AQM can\nbe implemented in various manners, not relying on a speci\ufb01c model representation nor a learning\nmethod. We also extended AQM to generate question by applying a previously proposed deep SL\nmethod. In this case, AQM can be understood as a way to boost the existing deep learning method.\nThroughout the paper, we argued that considering the collaborator\u2019s mind in implementing an agent\nis useful and fundamental.\n\nAcknowledgements\n\nThe authors would like to thank Jin-Hwa Kim, Tong Gao, Cheolho Han, Wooyoung Kang, Jaehyun\nJun, Hwiyeol Jo, Byoung-Hee Kim, Kyoung Woon On, Sungjae Cho, Joonho Kim, Seungjae Jung,\nHanock Kwak, Donghyun Kwak, Christina Baek, Minjoon Seo, Marco Baroni, and Jung-Woo Ha\nfor helpful comments and editing. This work was supported by the Institute for Information &\nCommunications Technology Promotion (R0126-16-1072-SW.StarLab, 2017-0-01772-VTT , 2018-0-\n00622-RMI) and Korea Evaluation Institute of Industrial Technology (10060086-RISF) grant funded\nby the Korea government (MSIP, DAPA).\n\nReferences\n[1] Oliver Lemon, Kallirroi Georgila, James Henderson, and Matthew Stuttle. An isu dialogue system\nexhibiting reinforcement learning of dialogue policies: generic slot-\ufb01lling in the talk in-car system. In\nProceedings of the Eleventh Conference of the European Chapter of the Association for Computational\nLinguistics: Posters & Demonstrations, pages 119\u2013122. Association for Computational Linguistics, 2006.\n[2] Jason D Williams and Steve Young. Partially observable markov decision processes for spoken dialog\n\nsystems. Computer Speech & Language, 21(2):393\u2013422, 2007.\n\n[3] Antoine Bordes and Jason Weston. Learning end-to-end goal-oriented dialog. In ICLR, 2017.\n[4] Oriol Vinyals and Quoc Le. A neural conversational model. In ICML Deep Learning Workshop, 2015.\n[5] Angeliki Lazaridou, Karl Moritz Hermann Hermann, Karl Tuyls, and Stephen Clark. Emergence of\n\nlinguistic communication from referential games with symbolic and pixel input. In ICLR, 2018.\n\n9\n\n\f[6] Harm de Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron Courville.\nGuesswhat?! visual object discovery through multi-modal dialogue. In Proceedings of the IEEE Conference\non Computer Vision and Pattern Recognition, 2017.\n\n[7] Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, Jos\u00e9 MF Moura, Devi Parikh,\nand Dhruv Batra. Visual dialog. Proceedings of the IEEE Conference on Computer Vision and Pattern\nRecognition, 2017.\n\n[8] Tsung-Hsien Wen, David Vandyke, Nikola Mrksic, Milica Gasic, Lina M Rojas-Barahona, Pei-Hao Su,\nStefan Ultes, and Steve Young. A network-based end-to-end trainable task-oriented dialogue system. arXiv\npreprint arXiv:1604.04562, 2016.\n\n[9] Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger\nSchwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder\u2013decoder for statistical\nmachine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language\nProcessing (EMNLP), pages 1724\u20131734, 2014.\n\n[10] Tiancheng Zhao and Maxine Eskenazi. Towards end-to-end learning for dialog state tracking and manage-\n\nment using deep reinforcement learning. arXiv preprint arXiv:1606.02560, 2016.\n\n[11] Xuijun Li, Yun-Nung Chen, Lihong Li, and Jianfeng Gao. End-to-end task-completion neural dialogue\n\nsystems. arXiv preprint arXiv:1703.01008, 2017.\n\n[12] Jin-Hwa Kim, Devi Parikh, Dhruv Batra, Byoung-Tak Zhang, and Yuandong Tian. Codraw: Visual dialog\n\nfor collaborative drawing. arXiv preprint arXiv:1712.05558, 2017.\n\n[13] Abhishek Das, Satwik Kottur, Jos\u00e9 MF Moura, Stefan Lee, and Dhruv Batra. Learning cooperative visual\n\ndialog agents with deep reinforcement learning. arXiv preprint arXiv:1703.06585, 2017.\n\n[14] David Premack and Guy Woodruff. Does the chimpanzee have a theory of mind? Behavioral and Brain\n\nSciences, 1(4):515\u2013526, 1978.\n\n[15] Jerome S Bruner. Intention in the structure of action and interaction. Advances in infancy research, 1981.\n[16] David JC MacKay. Information theory, inference and learning algorithms. Cambridge university press,\n\n2003.\n\n[17] Byoung-Tak Zhang. Information-theoretic objective functions for lifelong learning. In AAAI Spring\n\nSymposium: Lifelong Machine Learning, pages 62\u201369, 2013.\n\n[18] Florian Strub, Harm de Vries, Jeremie Mary, Bilal Piot, Aaron Courville, and Olivier Pietquin. End-to-end\noptimization of goal-driven and visually grounded dialogue systems. arXiv preprint arXiv:1703.05423,\n2017.\n\n[19] Katrina Evtimova, Andrew Drozdov, Douwe Kiela, and Kyunghyun Cho. Emergent language in a multi-\n\nmodal, multi-step referential game. In ICLR, 2018.\n\n[20] Arjun Chandrasekaran, Deshraj Yadav, Prithvijit Chattopadhyay, Viraj Prabhu, and Devi Parikh. It takes\n\ntwo to tango: Towards theory of ai\u2019s mind. arXiv preprint arXiv:1704.00717, 2017.\n\n[21] John Batali. Computational simulations of the emergence of grammar. Approaches to the Evolution of\n\nLanguage: Social and Cognitive Bases, 405:426, 1998.\n\n[22] Edward Choi, Angeliki Lazaridou, and Nando de Freitas. Multi-agent compositional communication\n\nlearning from raw visual input. In ICLR, 2018.\n\n[23] Pablo Hernandez-Leal and Michael Kaisers. Learning against sequential opponents in repeated stochastic\ngames. In The 3rd Multi-disciplinary Conference on Reinforcement Learning and Decision Making, Ann\nArbor, 2017.\n\n[24] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy.\nGeneration and comprehension of unambiguous object descriptions. In Proceedings of the IEEE Conference\non Computer Vision and Pattern Recognition, pages 11\u201320, 2016.\n\n[25] Daniel Fried, Jacob Andreas, and Dan Klein. Uni\ufb01ed pragmatic models for generating and following\ninstructions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for\nComputational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages\n1951\u20131963, 2018.\n\n[26] Jacob Andreas and Dan Klein. Reasoning about pragmatics with neural listeners and speakers.\n\nIn\nProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages\n1173\u20131182, 2016.\n\n[27] Licheng Yu, Hao Tan, Mohit Bansal, and Tamara L Berg. A joint speakerlistener-reinforcer model for\n\nreferring expressions. In Computer Vision and Pattern Recognition (CVPR), volume 2, 2017.\n\n10\n\n\f[28] Will Monroe, Robert XD Hawkins, Noah D Goodman, and Christopher Potts. Colors in context: A\npragmatic neural model for grounded language understanding. Transactions of the Association for\nComputational Linguistics, 5:325\u2013338, 2017.\n\n[29] Jakob N Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor\n\nMordatch. Learning with opponent-learning awareness. arXiv preprint arXiv:1709.04326, 2017.\n\n[30] Joseph Polifroni and Marilyn Walker. Learning database content for spoken dialogue system design. In 5th\n\nInternational Conference on Language Resources and Evaluation (LREC), 2006.\n\n[31] Anselm Rothe, Brenden M Lake, and Todd Gureckis. Question asking as program generation. In Advances\n\nin Neural Information Processing Systems, pages 1046\u20131055, 2017.\n\n[32] Satwik Kottur, Jos\u00e9 Moura, Stefan Lee, and Dhruv Batra. Natural language does not emerge \u2018naturally\u2019in\nmulti-agent dialog. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language\nProcessing, pages 2962\u20132967, 2017.\n\n[33] Paul Hongsuck Seo, Andreas Lehrmann, Bohyung Han, and Leonid Sigal. Visual reference resolution\nusing attention memory for visual dialog. In Advances in Neural Information Processing Systems, 2017.\n[34] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll\u00e1r,\nand C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on\nComputer Vision, pages 740\u2013755. Springer, 2014.\n\n[35] Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. arXiv preprint arXiv:1612.08242,\n\n2016.\n\n[36] Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. Hierarchical\n\nneural network generative models for movie dialogues. arXiv preprint arXiv:1507.04808, 2015.\n\n[37] Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent popula-\n\ntions. arXiv preprint arXiv:1703.04908, 2017.\n\n[38] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption\ngenerator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages\n3156\u20133164, 2015.\n\n[39] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick,\nand Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE International Conference\non Computer Vision, pages 2425\u20132433, 2015.\n\n[40] Cheolho Han, Sang-Woo Lee, Yujung Heo, Wooyoung Kang, Jaehyun Jun, and Byoung-Tak Zhang. Criteria\nfor human-compatible ai in two-player vision-language tasks. In 2017 IJCAI Workshop on Linguistic and\nCognitive Approaches to Dialogue Agents, 2017.\n\n[41] Prithvijit Chattopadhyay, Deshraj Yadav, Viraj Prabhu, Arjun Chandrasekaran, Abhishek Das, Stefan Lee,\nDhruv Batra, and Devi Parikh. Evaluating visual conversational agents via cooperative human-ai games.\narXiv preprint arXiv:1708.05122, 2017.\n\n[42] R\u00e9mi Coulom. Ef\ufb01cient selectivity and backup operators in monte-carlo tree search. In International\n\nConference on Computers and Games, pages 72\u201383. Springer, 2006.\n\n[43] Andrew Y Ng and Michael I Jordan. On discriminative vs. generative classi\ufb01ers: A comparison of logistic\nregression and naive bayes. In Advances in Neural Information Processing Systems, pages 841\u2013848, 2002.\n[44] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez,\nThomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human\nknowledge. Nature, 550(7676):354, 2017.\n\n[45] Alex Kendall and Yarin Gal. What Uncertainties Do We Need in Bayesian Deep Learning for Computer\n\nVision? In Advances in Neural Information Processing Systems, 2017.\n\n[46] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep\n\nnetworks. In International Conference on Machine Learning, pages 1126\u20131135, 2017.\n\n11\n\n\f", "award": [], "sourceid": 1289, "authors": [{"given_name": "Sang-Woo", "family_name": "Lee", "institution": "Naver Corp."}, {"given_name": "Yu-Jung", "family_name": "Heo", "institution": "Seoul National University"}, {"given_name": "Byoung-Tak", "family_name": "Zhang", "institution": "Seoul National University & Surromind Robotics"}]}