{"title": "End-to-End Goal-Driven Web Navigation", "book": "Advances in Neural Information Processing Systems", "page_first": 1903, "page_last": 1911, "abstract": "We propose a goal-driven web navigation as a benchmark task for evaluating an agent with abilities to understand natural language and plan on partially observed environments. In this challenging task, an agent navigates through a website, which is represented as a graph consisting of web pages as nodes and hyperlinks as directed edges, to find a web page in which a query appears. The agent is required to have sophisticated high-level reasoning based on natural languages and efficient sequential decision-making capability to succeed. We release a software tool, called WebNav, that automatically transforms a website into this goal-driven web navigation task, and as an example, we make WikiNav, a dataset constructed from the English Wikipedia. We extensively evaluate different variants of neural net based artificial agents on WikiNav and observe that the proposed goal-driven web navigation well reflects the advances in models, making it a suitable benchmark for evaluating future progress. Furthermore, we extend the WikiNav with question-answer pairs from Jeopardy! and test the proposed agent based on recurrent neural networks against strong inverted index based search engines. The artificial agents trained on WikiNav outperforms the engined based approaches, demonstrating the capability of the proposed goal-driven navigation as a good proxy for measuring the progress in real-world tasks such as focused crawling and question-answering.", "full_text": "End-to-End Goal-Driven Web Navigation\n\nTandon School of Engineering\n\nCourant Institute of Mathematical Sciences\n\nRodrigo Nogueira\n\nNew York University\n\nrodrigonogueira@nyu.edu\n\nKyunghyun Cho\n\nNew York University\n\nkyunghyun.cho@nyu.edu\n\nAbstract\n\nWe propose a goal-driven web navigation as a benchmark task for evaluating an\nagent with abilities to understand natural language and plan on partially observed\nenvironments. In this challenging task, an agent navigates through a website,\nwhich is represented as a graph consisting of web pages as nodes and hyperlinks as\ndirected edges, to \ufb01nd a web page in which a query appears. The agent is required\nto have sophisticated high-level reasoning based on natural languages and ef\ufb01cient\nsequential decision-making capability to succeed. We release a software tool,\ncalled WebNav, that automatically transforms a website into this goal-driven web\nnavigation task, and as an example, we make WikiNav, a dataset constructed from\nthe English Wikipedia. We extensively evaluate different variants of neural net\nbased arti\ufb01cial agents on WikiNav and observe that the proposed goal-driven web\nnavigation well re\ufb02ects the advances in models, making it a suitable benchmark\nfor evaluating future progress. Furthermore, we extend the WikiNav with question-\nanswer pairs from Jeopardy! and test the proposed agent based on recurrent neural\nnetworks against strong inverted index based search engines. The arti\ufb01cial agents\ntrained on WikiNav outperforms the engined based approaches, demonstrating the\ncapability of the proposed goal-driven navigation as a good proxy for measuring\nthe progress in real-world tasks such as focused crawling and question-answering.\n\n1\n\nIntroduction\n\nIn recent years, there have been many exciting advances in building an arti\ufb01cial agent, which can be\ntrained with one learning algorithm, to solve many relatively large-scale, complicated tasks (see, e.g.,\n[8, 10, 6].) In much of these works, target tasks were computer games such as Atari games [8] and\nracing car game [6].\nThese successes have stimulated researchers to apply a similar learning mechanism to language-based\ntasks, such as multi-user dungeon (MUD) games [9, 4]. Instead of visual perception, an agent\nperceives the state of the world by its written description. A set of actions allowed to the agent is\neither \ufb01xed or dependent on the current state. This type of task can ef\ufb01ciently evaluate the agent\u2019s\nability of not only in planning but also language understanding.\nWe, however, notice that these MUD games do not exhibit the complex nature of natural languages\nto the full extent. For instance, the largest game world tested by Narasimhan et al. [9] uses a\nvocabulary of only 1340 unique words, and the largest game tested by He et al. [4] uses only 2258\nwords. Furthermore, the description of a state at each time step is almost always limited to the visual\ndescription of the current scene, lacking any use of higher-level concepts present in natural languages.\nIn this paper, we propose a goal-driven web navigation as a large-scale alternative to the text-based\ngames for evaluating arti\ufb01cial agents with natural language understanding and planning capability.\nThe proposed goal-driven web navigation consists of the whole website as a graph, in which the web\npages are nodes and hyperlinks are directed edges. An agent is given a query, which consists of one\n\n30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain.\n\n\for more sentences taken from a randomly selected web page in the graph, and navigates the network,\nstarting from a prede\ufb01ned starting node, to \ufb01nd a target node in which the query appears. Unlike\nthe text-based games, this task utilizes the existing text as it is, resulting in a large vocabulary with\na truly natural language description of the state. Furthermore, the task is more challenging as the\naction space greatly changes with respect to the state in which the agent is.\nWe release a software tool, called WebNav, that converts a given website into a goal-driven web\nnavigation task. As an example of its use, we provide WikiNav, which was built from English\nWikipedia. We design arti\ufb01cial agents based on neural networks (called NeuAgents) trained with\nsupervised learning, and report their respective performances on the benchmark task as well as the\nperformance of human volunteers. We observe that the dif\ufb01culty of a task generated by WebNav is\nwell controlled by two control parameters; (1) the maximum number of hops from a starting to a\ntarget node Nh and (2) the length of query Nq.\nFurthermore, we extend the WikiNav with an additional set of queries that are constructed from\nJeopardy! questions, to which we refer by WikiNav-Jeopardy. We evaluate the proposed NeuAgents\nagainst the three search-based strategies; (1) SimpleSearch, (2) Apache Lucene and (3) Google\nSearch API. The result in terms of document recall indicates that the NeuAgents outperform those\nsearch-based strategies, implying a potential for the proposed task as a good proxy for practical\napplications such as question-answering and focused crawling.\n\n2 Goal-driven Web Navigation\nA task T of goal-driven web navigation is characterized by\n\nT = (A, sS,G, q, R, \u2126).\n\n(1)\nThe world in which an agent A navigates is represented as a graph G = (N ,E). The graph consists\nof a set of nodes N = {si}NN\ni=1 and a set of directed edges E = {ei,j} connecting those nodes. Each\nnode represents a page of the website, which, in turn, is represented by the natural language text\nD(si) in it. There exists an edge going from a page si to sj if and only if there is a hyperlink in D(si)\nthat points to sj. One of the nodes is designated as a starting node sS from which any navigation\nbegins. A target node is the one whose natural language description contains a query q, and there\nmay be more than one target node.\nAt each time step, the agent A reads the natural language description D(st) of the current node in\nwhich the agent has landed. At no point, the whole world, consisting of the nodes and edges, nor its\nstructure or map (graph structure without any natural language description) is visible to the agent,\nthus making this task partially observed.\nOnce the agent A reads the description D(si) of the current node si, it can take one of the actions\navailable. A set of possible actions is de\ufb01ned as a union of all the outgoing edges ei,\u00b7 and the stop\naction, thus making the agent have state-dependent action space.\nEach edge ei,k corresponds to the agent jumping to a next node sk, while the stop action corresponds\nto the agent declaring that the current node si is one of the target nodes. Each edge ei,k is represented\nby the description of the next node D(sk). In other words, deciding which action to take is equivalent\nto taking a peek at each neighboring node and seeing whether that node is likely to lead ultimately to\na target node.\nThe agent A receives a reward R(si, q) when it chooses the stop action. This task uses a simple\nbinary reward, where\n\n(cid:26) 1,\n\nR(si, q) =\n\nif q \u2286 D(si)\n\n0, otherwise\n\nConstraints\nIt is clear that there exists an ultimate policy for the agent to succeed at every trial,\nwhich is to traverse the graph breadth-\ufb01rst until the agent \ufb01nds a node in which the query appears. To\navoid this kind of degenerate policies, the task includes a set of four rules/constraints \u2126:\n\n1. An agent can follow at most Nn edges at each node.\n2. An agent has a \ufb01nite memory of size smaller than T .\n\n2\n\n\fTable 1: Dataset Statistics of WikiNav-4-*, WikiNav-8-*, WikiNav-16-* and WikiNav-Jeopardy.\n\nWikiNav-4-* WikiNav-8-* WikiNav-16-* WikiNav-Jeopardy\n\nTrain\nValid\nTest\n\n6.0k\n1k\n1k\n\n1M\n20k\n20k\n\n12M\n20k\n20k\n\n113k\n10k\n10k\n\n3. An agent moves up to Nh hops away from sS.\n4. A query of size Nq comes from at least two hops away from the starting node.\n\nThe \ufb01rst constraint alone prevents degenerate policies, such as breadth-\ufb01rst search, forcing the agent\nto make good decisions as possible at each node. The second one further constraints ensure that the\nagent does not cheat by using earlier trials to reconstruct the whole graph structure (during test time)\nor to store the entire world in its memory (during training.) The third constraint, which is optional, is\nthere for computational consideration. The fourth constraint is included because the agent is allowed\nto read the content of a next node.\n\n3 WebNav: Software\n\nAs a part of this work, we build and release a software tool which turns a website into a goal-driven\nweb navigation task.1 We call this tool WebNav. Given a starting URL, the WebNav reads the whole\nwebsite, constructs a graph with the web pages in the website as nodes. Each node is assigned a\nunique identi\ufb01er si. The text content of each node D(si) is a cleaned version of the actual HTML\ncontent of the corresponding web page. The WebNav turns intra-site hyperlinks into a set of edges\nei,j.\nIn addition to transforming a website into a graph G from Eq. (1), the WebNav automatically selects\nqueries from the nodes\u2019 texts and divides them into training, validation, and test sets. We ensure that\nthere is no overlap among three sets by making each target node, from which a query is selected,\nbelongs to only one of them.\nEach generated example is de\ufb01ned as a tuple\n\nX = (q, s\u2217, p\u2217)\n\n(2)\nwhere q is a query from a web page s\u2217, which was found following a randomly selected path\np\u2217 = (sS, . . . , s\u2217). In other words, the WebNav starts from a starting page sS, random-walks the\ngraph for a prede\ufb01ned number of steps (Nh/2, in our case), reaches a target node s\u2217 and selects a\nquery q from D(s\u2217). A query consists of Nq sentences and is selected among the top-5 candidates\nin the target node with the highest average TF-IDF, thus discouraging the WebNav from choosing a\ntrivial query.\nFor the evaluation purpose alone, it is enough to use only a query q itself as an example. However,\nwe include both one target node (among potentially many other target nodes) and one path from the\nstarting node to this target node (again, among many possible connecting paths) so that they can be\nexploited when training an agent. They are not to be used when evaluating a trained agent.\n\n4 WikiNav: A Benchmark Task\n\nWith the WebNav, we built a benchmark goal-driven navigation task using Wikipedia as a target\nwebsite. We used the dump \ufb01le of the English Wikipedia from September 2015, which consists of\nmore than \ufb01ve million web pages. We built a set of separate tasks with different levels of dif\ufb01culty by\nvarying the maximum number of allowed hops Nh \u2208 {4, 8, 16} and the size of query Nq \u2208 {1, 2, 4}.\nWe refer to each task by WikiNav-Nh-Nq.\nFor each task, we generate training, validation and test examples from the pages half as many hops\naway from a starting page as the maximum number of hops allowed.2 We use \u201cCategory:Main topic\nclassi\ufb01cations\u201d as a starting node sS.\n\n1 The source code and datasets are publicly available at github.com/nyu-dl/WebNav.\n2 This limit is an arti\ufb01cial limit we chose for computational reasons.\n\n3\n\n\fTable 3: Sample query-answer pairs from WikiNav-Jeopardy.\nAnswer\n\nQuery\nFor the last 8 years of his life, Galileo was under\nhouse arrest for espousing this man\u2019s theory.\nIn the winter of 1971-72, a record 1,122 inches of snow fell Washington\nat Rainier Paradise Ranger Station in this state.\nThis company\u2019s Accutron watch, introduced in 1960,\nhad a guarantee of accuracy to within one minute a month.\n\nCopernicus\n\nBulova\n\nAs a minimal cleanup procedure, we excluded meta articles whose titles start with \u201cWikipedia\u201d.\nAny hyperlink that leads to a web page outside Wikipedia is removed in advance together with the\nfollowing sections: \u201cReferences\u201d, \u201cExternal Links\u201d, \u201cBibliography\u201d and \u201cPartial Bibliography\u201d.\nIn Table 2, we present basic per-article statistics of the\nEnglish Wikipedia. It is evident from these statistics that\nthe world of WikiNav-Nh-Nq is large and complicated,\neven after the cleanup procedure.\nWe ended up with a fairly small dataset for WikiNav-4-*,\nbut large for WikiNav-8-* and WikiNav-16-*. See Table 1\nfor details.\n\nHyperlinks Words\n462.5\n990.2\n132881\n\n\u221a\nAvg.\nVar\nMax\nMin\n\n4.29\n13.85\n300\n0\n\n1\n\nTable 2: Per-page statistics of English\nWikipedia.\n\n4.1 Related Work: Wikispeedia\n\nThis work is indeed not the \ufb01rst to notice the possibility of a website, or possibly the whole web, as a\nworld in which intelligent agents explore to achieve a certain goal. One most relevant recent work to\nours is perhaps Wikispeedia from [14, 12, 13].\nWest et al. [14, 12, 13] proposed the following game, called Wikispeedia. The game\u2019s world is nearly\nidentical to the goal-driven navigation task proposed in this work. More speci\ufb01cally, they converted\n\u201cWikipedia for Schools\u201d, which contains approximately 4,000 articles as of 2008, into a graph whose\nnodes are articles and directed edges are hyperlinks. From this graph, a pair of nodes is randomly\nselected and provided to an agent.\nThe agent\u2019s goal is to start from the \ufb01rst node, navigate the graph and reach the second node. Similarly\nto the WikiNav, the agent has access to the text content of the current nodes and all the immediate\nneighboring nodes. One major difference is that the target is given as a whole article, meaning that\nthere is a single target node in the Wikispeedia while there may be multiple target nodes in the\nproposed WikiNav.\nFrom this description, we see that the goal-driven web navigation is a generalization and re-framing\nof the Wikispeedia. First, we constrain a query to contain less information, making it much more\ndif\ufb01cult for an agent to navigate to a target node. Furthermore, a major research question by West and\nLeskovec [13] was to \u201cunderstand how humans navigate and \ufb01nd the information they are looking\nfor ,\u201d whereas in this work we are fully focused on proposing an automatic tool to build a challenging\ngoal-driven tasks for designing and evaluating arti\ufb01cial intelligent agents.\n\n5 WikiNav-Jeopardy: Jeopardy! on WikiNav\n\nOne of the potential practical applications utilizing the goal-drive navigation is question-answering\nbased on world knowledge. In this Q&A task, a query is a question, and an agent navigates a given\ninformation network, e.g., website, to retrieve an answer. In this section, we propose and describe\nan extension of the WikiNav, in which query-target pairs are constructed from actual Jeopardy!\nquestion-answer pairs. We refer to this extension of WikiNav by WikiNav-Jeopardy.\nWe \ufb01rst extract all the question-answer pairs from J! Archive3, which has more than 300k such\npairs. We keep only those pairs whose answers are titles of Wikipedia articles, leaving us with 133k\npairs. We divide those pairs into 113k training, 10k validation, and 10k test examples while carefully\n\n3 www.j-archive.com\n\n4\n\n\fensuring that no article appears in more than one partition. Additionally, we do not shuf\ufb02e the original\npairs to ensure that the train and test examples are from different episodes.\nFor each training pair, we \ufb01nd one path from the starting node \u201cMain Topic Classi\ufb01cation\u201d to the\ntarget node and include it for supervised learning. For reference, the average number of hops to\nthe target node is 5.8, the standard deviation is 1.2, and the maximum and minimum are 2 and 10,\nrespectively. See Table 3 for sample query-answer pairs.\n\n6 NeuAgent: Neural Network based Agent\n\n6.1 Model Description\n\nCore Function The core of the NeuAgent is a parametric function fcore that takes as input the\ncontent of the current node \u03c6c(si) and a query \u03c6q(q), and that returns the hidden state of the agent.\nThis parametric function fcore can be implemented either as a feedforward neural network fff:\n\nht = fff(\u03c6c(si), \u03c6q(q))\n\nwhich does not take into account the previous hidden state of the agent or as a recurrent neural\nnetwork frec:\n\nht = frec(ht\u22121, \u03c6c(si), \u03c6q(q)).\n\nWe refer to these two types of agents by NeuAgent-FF and NeuAgent-Rec, respectively. For the\nNeuAgent-FF, we use a single tanh layer, while we use long short-term memory (LSTM) units [5],\nwhich have recently become de facto standard, for the NeuAgent-Rec.\n\nBased on the new hidden state ht, the NeuAgent computes\nthe probability distribution over all the outgoing edges ei.\nThe probability of each outgoing edge is proportional to\nthe similarity between the hidden state ht such that\n\np(ei,j|\u02dcp) \u221d exp(cid:0)\u03c6c(sj)(cid:62)ht\n\n(cid:1) .\n\n(3)\n\nNote that the NeuAgent peeks at the content of the next\nnode sj by considering its vector representation \u03c6c(sj).\nIn addition to all the outgoing edges, we also allow the\nagent to stop with the probability\n\np(\u2205|\u02dcp) \u221d exp(cid:0)v(cid:62)\n\n(cid:1) ,\n\n\u2205 ht\n\nFigure 1: Graphical illustration of a\nsingle step performed by the baseline\nmodel, NeuAgent.\n\n(4)\nwhere the stop action vector v\u2205 is a trainable parameter.\nIn the case of NeuAgent-Rec, all these (unnormalized)\nprobabilities are conditioned on the history \u02dcp which is\na sequence of actions (nodes) selected by the agent so\nfar. We apply a softmax normalization on the unnormalized probabilities to obtain the probability\ndistribution over all the possible actions at the current node si.\nThe NeuAgent then selects its next action based on this action probability distribution (Eqs. (3) and\n(4)). If the stop action is chosen, the NeuAgent returns the current node as an answer and receives a\nreward R(si, q), which is one if correct and zero otherwise. If the agent selects one of the outgoing\nedges, it moves to the selected node and repeats this process of reading and acting.\nSee Fig. 1 for a single step of the described NeuAgent.\nContent Representation The NeuAgent represents the content of a node si as a vector \u03c6c(si) \u2208 Rd.\nIn this work, we use a continuous bag-of-words vector for each document:\n\n\u03c6c(si) =\n\n1\n\n|D(si)|\n\n|D(si)|(cid:88)\n\nk=1\n\nek.\n\nEach word vector ek is from a pretrained continuous bag-of-words model [7]. These word vectors\nare \ufb01xed throughout training.\n\n5\n\n\fQuery Representation In the case of a query, we consider two types of representation. The \ufb01rst\none is a continuous bag-of-words (BoW) vector, just as used for representing the content of a node.\nThe other one is a dynamic representation based on the attention mechanism [2].\nIn the attention-based query representation, the query is \ufb01rst projected into a set of context vectors.\nThe context vector of the k-th query word is\n\nwhere Wk(cid:48) \u2208 Rd\u00d7d and ek(cid:48) are respectively a trainable weight matrix and a pretrained word vector.\nu is the window size. Each context vector is scored at each time step t by \u03b2t\nk = fatt(ht\u22121, ck) w.r.t.\nthe previous hidden state of the NeuAgent, and all the scores are normalized to be positive and sum\nto one, i.e., \u03b1t\n. These normalized scores are used as the coef\ufb01cients in computing\nthe weighted-sum of query words to result in a query representation at time t:\n\nk = exp(\u03b2t\nk)\n\n(cid:80)|q|\n\nl=1 exp(\u03b2t\nl )\n\nk+u/2(cid:88)\n\nk(cid:48)=k\u2212u/2\n\nck =\n\nWk(cid:48)ek(cid:48),\n\n|q|(cid:88)\n\nk=1\n\n\u03c6q(q) =\n\n1\n|q|\n\n\u03b1t\n\nkck.\n\nLater, we empirically compare these two query representations.\n\n|p\u2217|(cid:88)\n\n6.2\n\nInference: Beam Search\n\nOnce the NeuAgent is trained, there are a number of approaches to using it for solving the proposed\ntask. The most naive approach is simply to let the agent make a greedy decision at each time step, i.e.,\nfollowing the outgoing edge with the highest probability arg maxk log p(ei,k| . . .). A better approach\nis to exploit the fact that the agent is allowed to explore up to Nn outgoing edges per node. We use a\nsimple, forward-only beam search with the beam width capped at Nn. The beam search simply keeps\nthe Nn most likely traces, in terms of log p(ei,k| . . .), at each time step.\n\n6.3 Training: Supervised Learning\n\nIn this paper, we investigate supervised learning, where we train the agent to follow an example trace\np\u2217 = (sS, . . . , s\u2217) included in the training set at each step (see Eq. (2)). In this case, the cost per\ntraining example is\n\nCsup = \u2212 log p(\u2205|p\u2217, q) \u2212\n\nlog p(p\u2217\n\nk|p\u2217\n\n