{"title": "On-the-Job Learning with Bayesian Decision Theory", "book": "Advances in Neural Information Processing Systems", "page_first": 3465, "page_last": 3473, "abstract": "Our goal is to deploy a high-accuracy system starting with zero training examples. We consider an \u201con-the-job\u201d setting, where as inputs arrive, we use real-time crowdsourcing to resolve uncertainty where needed and output our prediction when confident. As the model improves over time, the reliance on crowdsourcing queries decreases. We cast our setting as a stochastic game based on Bayesian decision theory, which allows us to balance latency, cost, and accuracy objectives in a principled way. Computing the optimal policy is intractable, so we develop an approximation based on Monte Carlo Tree Search. We tested our approach on three datasets-- named-entity recognition, sentiment classification, and image classification. On the NER task we obtained more than an order of magnitude reduction in cost compared to full human annotation, while boosting performance relative to the expert provided labels. We also achieve a 8% F1 improvement over having a single human label the whole set, and a 28% F1 improvement over online learning.", "full_text": "On-the-Job Learning with Bayesian Decision Theory\n\nKeenon Werling\n\nDepartment of Computer Science\n\nStanford University\n\nArun Chaganty\n\nDepartment of Computer Science\n\nStanford University\n\nkeenon@cs.stanford.edu\n\nchaganty@cs.stanford.edu\n\nPercy Liang\n\nDepartment of Computer Science\n\nStanford University\n\nChristopher D. Manning\n\nDepartment of Computer Science\n\nStanford University\n\npliang@cs.stanford.edu\n\nmanning@cs.stanford.edu\n\nAbstract\n\nOur goal is to deploy a high-accuracy system starting with zero training examples.\nWe consider an on-the-job setting, where as inputs arrive, we use real-time crowd-\nsourcing to resolve uncertainty where needed and output our prediction when con-\n\ufb01dent. As the model improves over time, the reliance on crowdsourcing queries\ndecreases. We cast our setting as a stochastic game based on Bayesian decision\ntheory, which allows us to balance latency, cost, and accuracy objectives in a prin-\ncipled way. Computing the optimal policy is intractable, so we develop an approx-\nimation based on Monte Carlo Tree Search. We tested our approach on three\ndatasets\u2014named-entity recognition, sentiment classi\ufb01cation, and image classi\ufb01-\ncation. On the NER task we obtained more than an order of magnitude reduction\nin cost compared to full human annotation, while boosting performance relative to\nthe expert provided labels. We also achieve a 8% F1 improvement over having a\nsingle human label the whole set, and a 28% F1 improvement over online learning.\n\n\u201cPoor is the pupil who does not surpass his master.\u201d\n\u2013 Leonardo da Vinci\n\n1\n\nIntroduction\n\nThere are two roads to an accurate AI system today: (i) gather a huge amount of labeled training\ndata [1] and do supervised learning [2]; or (ii) use crowdsourcing to directly perform the task [3, 4].\nHowever, both solutions require non-trivial amounts of time and money. In many situations, one\nwishes to build a new system \u2014 e.g., to do Twitter information extraction [5] to aid in disaster relief\nefforts or monitor public opinion \u2014 but one simply lacks the resources to follow either the pure ML\nor pure crowdsourcing road.\nIn this paper, we propose a framework called on-the-job learning (formalizing and extending ideas\n\ufb01rst implemented in [6]), in which we produce high quality results from the start without requiring\na trained model. When a new input arrives, the system can choose to asynchronously query the\ncrowd on parts of the input it is uncertain about (e.g. query about the label of a single token in a\nsentence). After collecting enough evidence the system makes a prediction. The goal is to maintain\nhigh accuracy by initially using the crowd as a crutch, but gradually becoming more self-suf\ufb01cient\nas the model improves. Online learning [7] and online active learning [8, 9, 10] are different in that\nthey do not actively seek new information prior to making a prediction, and cannot maintain high\naccuracy independent of the number of data instances seen so far. Active classi\ufb01cation [11], like us,\n\n1\n\n\fFigure 1: Named entity recognition on tweets in on-the-job learning.\n\nstrategically seeks information (by querying a subset of labels) prior to prediction, but it is based on\na static policy, whereas we improve the model during test time based on observed data.\nTo determine which queries to make, we model on-the-job learning as a stochastic game based on\na CRF prediction model. We use Bayesian decision theory to tradeoff latency, cost, and accuracy\nin a principled manner. Our framework naturally gives rise to intuitive strategies: To achieve high\naccuracy, we should ask for redundant labels to offset the noisy responses. To achieve low latency,\nwe should issue queries in parallel, whereas if latency is unimportant, we should issue queries se-\nquentially in order to be more adaptive. Computing the optimal policy is intractable, so we develop\nan approximation based on Monte Carlo tree search [12] and progressive widening to reason about\ncontinuous time [13].\nWe implemented and evaluated our system on three different tasks: named-entity recognition, sen-\ntiment classi\ufb01cation, and image classi\ufb01cation. On the NER task we obtained more than an order of\nmagnitude reduction in cost compared to full human annotation, while boosting performance rela-\ntive to the expert provided labels. We also achieve a 8% F1 improvement over having a single human\nlabel the whole set, and a 28% F1 improvement over online learning. An open-source implementa-\ntion of our system, dubbed LENSE for \u201cLearning from Expensive Noisy Slow Experts\u201d is available\nat http://www.github.com/keenon/lense.\n\n2 Problem formulation\n\nConsider a structured prediction problem from input x = (x1, . . . , xn) to output y = (y1, . . . , yn).\nFor example, for named-entity recognition (NER) on tweets, x is a sequence of words in the tweet\n(e.g., \u201con George str.\u201d) and y is the corresponding sequence of labels (e.g., NONE LOCATION\nLOCATION). The full set of labels of PERSON, LOCATION, RESOURCE, and NONE.\nIn the on-the-job learning setting, inputs arrive in a stream. On each input x, we make zero or more\nqueries q1, q2, . . . on the crowd to obtain labels (potentially more than once) for any positions in\nx. The responses r1, r2, . . . come back asynchronously, which are incorporated into our current\nprediction model p\u03b8. Figure 2 (left) shows one possible outcome: We query positions q1 = 2\n(\u201cGeorge\u201d) and q2 = 3 (\u201cstr.\u201d). The \ufb01rst query returns r1 = LOCATION, upon which we make\nanother query on the the same position q3 = 3 (\u201cGeorge\u201d), and so on. When we have suf\ufb01cient\ncon\ufb01dence about the entire output, we return the most likely prediction \u02c6y under the model. Each\nquery qi is issued at time si and the response comes back at time ti. Assume that each query costs\nm cents. Our goal is to choose queries to maximize accuracy, minimize latency and cost.\nWe make several remarks about this setting: First, we must make a prediction \u02c6y on each input x in\nthe stream, unlike in active learning, where we are only interested in the pool or stream of examples\nfor the purposes of building a good model. Second, the responses are used to update the prediction\n\n2\n\ny1x1y2x2y3x3r1y4x4y5x5SouponGeorgestr.#KatrinaRESOURCELOCATIONy1x1y2x2y3x3y4x4y5x5SouponGeorge\u201cGeorge\u201dstr.#Katrina32%: LOCATION2%: NONE12%: RESOURCE44%: PERSONDecide to ask a crowd worker in real timeGet beliefs under learned modelIncorporate feedback,return a predictionsoup on george str, #katrinalocationpersonnoneresourceWhat is `George` here?http://www.crowd-workers.com1.2.3.\f(a) Incorporating information from responses. The bar graphs\nrepresent the marginals over the labels for each token (indicated\nby the \ufb01rst character) at different points in time. The two time-\nlines show how the system updates its con\ufb01dence over labels\nbased on the crowd\u2019s responses. The system continues to issue\nqueries until it has suf\ufb01cient con\ufb01dence on its labels. See the\nparagraph on behavior in Section 3 for more information.\n\n(b) Game tree. An example of a partial\ngame tree constructed by the system when\ndeciding which action to take in the state\n\u03c3 = (1, (3), (0), (\u2205), (\u2205)), i.e. the query\nq1 = 3 has already been issued and the\nsystem must decide whether to issue an-\nother query or wait for a response to q1.\n\nFigure 2: Example behavior while running structure prediction on the tweet \u201cSoup on George str.\u201d\nWe omit the RESOURCE from the game tree for visual clarity.\n\nmodel, like in online learning. This allows the number of queries needed (and thus cost and latency)\nto decrease over time without compromising accuracy.\n3 Model\nWe model on-the-job learning as a stochastic game with two players: the system and the crowd.\nThe game starts with the system receiving input x and ends when the system turns in a set of labels\ny = (y1, . . . , yn). During the system\u2019s turn, the system may choose a query action q \u2208 {1, . . . , n}\nto ask the crowd to label yq. The system may also choose the wait action (q = \u2205W ) to wait for the\ncrowd to respond to a pending query or the return action (q = \u2205R) to terminate the game and return\nits prediction given responses received thus far. The system can make as many queries in a row (i.e.\nsimultaneously) as it wants, before deciding to wait or turn in.1 When the wait action is chosen,\nthe turn switches to the crowd, which provides a response r to one pending query, and advances\nthe game clock by the time taken for the crowd to respond. The turn then immediately reverts back\nto the system. When the game ends (the system chooses the return action), the system evaluates a\nutility that depends on the accuracy of its prediction, the number of queries issued and the total time\ntaken. The system should choose query and wait actions to maximize the utility of the prediction\neventually returned.\nIn the rest of this section, we describe the details of the game tree, our choice of utility and specify\nmodels for crowd responses, followed by a brief exploration of behavior admitted by our model.\n\nGame tree. Let us now formalize the game tree in terms of its states, actions, transitions and\nrewards; see Figure 2b for an example. The game state \u03c3 = (tnow, q, s, r, t) consists of the current\ntime tnow, the actions q = (q1, . . . , qk\u22121) that have been issued at times s = (s1, . . . , sk\u22121) and the\nresponses r = (r1, . . . , rk\u22121) that have been received at times t = (t1, . . . , tk\u22121). Let rj = \u2205 and\ntj = \u2205 iff qj is not a query action or its responses have not been received by time tnow.\nDuring the system\u2019s turn, when the system chooses an action qk, the state is updated to \u03c3(cid:48) =\n(tnow, q(cid:48), s(cid:48), r(cid:48), t(cid:48)), where q(cid:48) = (q1, . . . , qk), s(cid:48) = (s1, . . . , sk\u22121, tnow), r(cid:48) = (r1, . . . , rk\u22121,\u2205) and\nt(cid:48) = (t1, . . . , tk\u22121,\u2205). If qk \u2208 {1, . . . n}, then the system chooses another action from the new state\n\u03c3(cid:48). If qk = \u2205W , the crowd makes a stochastic move from \u03c3(cid:48). Finally, if qk = \u2205R, the game ends,\n\n1 This rules out the possibility of launching a query midway through waiting for the next response. However,\n\nwe feel like this is a reasonable limitation that signi\ufb01cantly simpli\ufb01es the search space.\n\n3\n\n1SoGsSoGsSoGsq1=1r1=resq2=3r2=loc2SoGsSoGsSoGsSoGsq1=1r1=resq2=3r2=perq4=3r4=locperlocresnoneLegend\u2205Wr1=loc0.47\u2205R4\u2205Wr1=loc0.27\u2205R=system=crowd\u03c3=(tnow,q,s,r,t)(1,(3),(0),(\u2205),(\u2205))(1.7,(3),(0),(1.7),(loc))\fand the system returns its best estimate of the labels using the responses it has received and obtains\na utility U (\u03c3) (de\ufb01ned later).\nLet F = {1 \u2264 j \u2264 k \u2212 1 | qj (cid:54)= \u2205W \u2227 rj = \u2205} be the set of in-\ufb02ight requests. During the crowd\u2019s\nturn (i.e. after the system chooses \u2205W ), the next response from the crowd, j\u2217 \u2208 F , is chosen: j\u2217 =\nj | sj, t(cid:48)\narg minj\u2208F t(cid:48)\nj is sampled from the response-time model, t(cid:48)\nj > tnow), for\nj\u2217 | x, r), and the state\neach j \u2208 F . Finally, a response is sampled using a response model, r(cid:48)\nis updated to \u03c3(cid:48) = (tj\u2217 , q, s, r(cid:48), t(cid:48)), where r(cid:48) = (r1, . . . , r(cid:48)\nj\u2217 , . . . , tk).\n\nj\u2217 , . . . , rk) and t(cid:48) = (t1, . . . , t(cid:48)\n\nj \u223c pT(t(cid:48)\nj\u2217 \u223c p(r(cid:48)\n\nj where t(cid:48)\n\nUtility. Under Bayesian decision theory,\nthe optimal choice for an action in state \u03c3 =\n(tnow, q, r, s, t) is the one that attains the maximum expected utility (i.e. value) for the game starting\nat \u03c3. Recall that the system can return at any time, at which point it receives a utility that trades\noff two things: The \ufb01rst is the accuracy of the MAP estimate according to the model\u2019s best guess\nof y incorporating all responses received by time \u03c4. The second is the cost of making queries: a\n(monetary) cost wM per query made and penalty of wT per unit of time taken. Formally, we de\ufb01ne\nthe utility to be:\n\nU (\u03c3) (cid:44) ExpAcc(p(y | x, q, s, r, t)) \u2212 (nQwM + tnowwT),\n\nExpAcc(p) = Ep(y)[Accuracy(arg max\n\ny(cid:48) p(y\n\n(cid:48)\n\n))],\n\n(1)\n(2)\n\nwhere nQ = |{j | qj \u2208 {1, . . . , n}| is the number of queries made, p(y | x, q, s, r, t) is a prediction\nmodel that incorporates the crowd\u2019s responses.\nThe utility of wait and return actions is computed by taking expectations over subsequent trajectories\nin the game tree. This is intractable to compute exactly, so we propose an approximate algorithm in\nSection 4.\n\nEnvironment model. The \ufb01nal component is a model of the environment (crowd). Given input\nx and queries q = (q1, . . . , qk) issued at times s = (s1, . . . , sk), we de\ufb01ne a distribution over the\noutput y, responses r = (r1, . . . , rk) and response times t = (t1, . . . , tk) as follows:\n\nk(cid:89)\n\np(y, r, t | x, q, s) (cid:44) p\u03b8(y | x)\n\npR(ri | yqi)pT(ti | si).\n\n(3)\n\ni=1\n\nThe three components are as follows: p\u03b8(y | x) is the prediction model (e.g. a standard linear-chain\nCRF); pR(r | yq) is the response model which describes the distribution of the crowd\u2019s response\nr for a given a query q when the true answer is yq; and pT(ti | si) speci\ufb01es the latency of query\nqi. The CRF model p\u03b8(y | x) is learned based on all actual responses (not simulated ones) using\nAdaGrad. To model annotation errors, we set pR(r | yq) = 0.7 iff r = yq,2 and distribute the\nremaining probability for r uniformly. Given this full model, we can compute p(r(cid:48) | x, r, q) simply\nby marginalizing out y and t from Equation 3. When conditioning on r, we ignore responses that\nhave not yet been received (i.e. when rj = \u2205 for some j).\n\nBehavior. Let\u2019s look at typical behavior that we expect the model and utility to capture. Figure 2a\nshows how the marginals over the labels change as the crowd provides responses for our running\nexample, i.e. named entity recognition for the sentence \u201cSoup on George str.\u201d. In the both timelines,\nthe system issues queries on \u201cSoup\u201d and \u201cGeorge\u201d because it is not con\ufb01dent about its predictions\nfor these tokens. In the \ufb01rst timeline, the crowd correctly responds that \u201cSoup\u201d is a resource and\nthat \u201cGeorge\u201d is a location. Integrating these responses, the system is also more con\ufb01dent about\nits prediction on \u201cstr.\u201d, and turns in the correct sequence of labels. In the second timeline, a crowd\nworker makes an error and labels \u201cGeorge\u201d to be a person. The system still has uncertainty on\n\u201cGeorge\u201d and issues an additional query which receives a correct response, following which the\nsystem turns in the correct sequence of labels. While the answer is still correct, the system could\nhave taken less time to respond by making an additional query on \u201cGeorge\u201d at the very beginning.\n\n2We found the humans we hired were roughly 70% accurate in our experiments\n\n4\n\n\f4 Game playing\n\nIn Section 3 we modeled on-the-job learning as a stochastic game played between the system and\nthe crowd. We now turn to the problem of actually \ufb01nding a policy that maximizes the expected\nutility, which is, of course, intractable because of the large state space.\nOur algorithm (Algorithm 1) combines ideas from Monte Carlo tree search [12] to systematically\nexplore the state space and progressive widening [13] to deal with the challenge of continuous vari-\nables (time). Some intuition about the algorithm is provided below. When simulating the system\u2019s\nturn, the next state (and hence action) is chosen using the upper con\ufb01dence tree (UCT) decision\nrule that trades off maximizing the value of the next state (exploitation) with the number of visits\n(exploration). The crowd\u2019s turn is simulated based on transitions de\ufb01ned in Section 3. To handle the\nunbounded fanout during the crowd\u2019s turn, we use progressive widening that maintains a current set\nof \u201cactive\u201d or \u201cexplored\u201d states, which is gradually grown with time. Let N (\u03c3) be the number of\ntimes a state has been visited, and C(\u03c3) be all successor states that the algorithm has sampled.\n\n(cid:46) Initialize visits, utility sum, and children\n\n(cid:46) Choose next state \u03c3(cid:48) using UCT\n\n(cid:46) Record observed utility\n\n(cid:46) Restrict continuous samples using PW\n\n(cid:110) V (\u03c3\n\n(cid:48)\n\n)\n\n(cid:113) log N (\u03c3)\n\n(cid:111)\n\nN (\u03c3(cid:48))\n\nincrement N (\u03c3)\nif system\u2019s turn then\n\u03c3(cid:48) \u2190 arg max\u03c3(cid:48)\nN (\u03c3(cid:48)) + c\nv \u2190MONTECARLOVALUE(\u03c3(cid:48))\nV (\u03c3) \u2190 V (\u03c3) + v\n(cid:112)\nreturn v\nelse if crowd\u2019s turn then\n\nAlgorithm 1 Approximating expected utility with MCTS and progressive widening\n1: For all \u03c3, N (\u03c3) \u2190 0, V (\u03c3) \u2190 0, C(\u03c3) \u2190 [ ]\n2: function MONTECARLOVALUE(state \u03c3)\n3:\n4:\n5:\n6:\n7:\n8:\n9:\n10:\n11:\n12:\n13:\n14:\n15:\n16:\n17:\n18:\n19:\n20: end function\n\n\u03c3(cid:48) is sampled from set of already visited C(\u03c3) based on (3)\n\u03c3(cid:48) is drawn based on (3)\nC(\u03c3) \u2190 C(\u03c3) \u222a {[\u03c3(cid:48)]}\n\nend if\nreturn MONTECARLOVALUE(\u03c3(cid:48))\n\nreturn utility U of \u03c3 according to (1)\n\nN (\u03c3)) \u2264 |C(\u03c3)| then\n\nif max(1,\n\nelse\n\nelse if game terminated then\n\nend if\n\n5 Experiments\n\nIn this section, we empirically evaluate our approach on three tasks. While the on-the-job setting we\npropose is targeted at scenarios where there is no data to begin with, we use existing labeled datasets\n(Table 1) to have a gold standard.\n\nBaselines. We evaluated the following four methods on each dataset:\n\n1. Human n-query: The majority vote of n human crowd workers was used as a prediction.\n2. Online learning: Uses a classi\ufb01er that trains on the gold output for all examples seen so\nfar and then returns the MLE as a prediction. This is the best possible of\ufb02ine system: it\nsees perfect information about all the data seen so far, but can not query the crowd while\nmaking a prediction.\n3. Threshold baseline: Uses the following heuristic: For each label, yi, we ask for m queries\nsuch that (1\u2212p\u03b8(yi | x))\u00d70.3m \u2265 0.98. Instead of computing the expected marginals over\nthe responses to queries in \ufb02ight, we simply count the in-\ufb02ight requests for a given variable,\nand reduces the uncertainty on that variable by a factor of 0.3. The system continues\nlaunching requests until the threshold (adjusted by number of queries in \ufb02ight) is crossed.\n\n5\n\n\fDataset (Examples) Task and notes\nNER (657)\n\nWe evaluate on the CoNLL-2003\nNER task3, a sequence labeling\nproblem over English sentences.\nWe only consider the four tags cor-\nresponding to persons, locations,\norganizations or none4.\n\nSentiment (1800)\n\nFace (1784)\n\nWe evaluate on a subset of the\nIMDB sentiment dataset [15] that\nconsists of 2000 polar movie re-\nviews; the goal is binary classi\ufb01ca-\ntion of documents into classes POS\nand NEG.\nWe evaluate on a celebrity face\nclassi\ufb01cation task [17]. Each im-\nage must be labeled as one of the\nfollowing four choices: Andersen\nCooper, Daniel Craig, Scarlet Jo-\nhansson or Miley Cyrus.\n\nFeatures\nWe used standard features [14]: the\ncurrent word, current lemma, pre-\nvious and next lemmas, lemmas in\na window of size three to the left\nand right, word shape and word\npre\ufb01x and suf\ufb01xes, as well as word\nembeddings.\nWe used two feature sets,\nthe\n\ufb01rst (UNIGRAMS) containing only\nword unigrams, and the second\n(RNN) that also contains sentence\nvector embeddings from [16].\n\nWe used the last layer of a 11-\nlayer AlexNet [2] trained on Ima-\ngeNet as input feature embeddings,\nthough we leave back-propagating\ninto the net to future work.\n\nTable 1: Datasets used in this paper and number of examples we evaluate on.\n\nNamed Entity Recognition\n\nSystem\n1-vote\n3-vote\n5-vote\nOnline\nThreshold\nLENSE\n\nDelay/tok Qs/tok\n1.0\n3.0\n5.0\nn/a\n0.61\n0.45\n\n467 ms\n750 ms\n1350 ms\nn/a\n414 ms\n267 ms\n\nPER F1\n90.2\n93.6\n95.5\n56.9\n95.2\n95.2\n\nLOC F1 ORG F1\n71.5\n74.5\n78.7\n51.4\n79.8\n81.7\n\n78.8\n85.1\n87.7\n74.6\n89.8\n89.7\n\nF1\n80.2\n85.4\n87.3\n60.9\n88.3\n88.8\n\nFace Identi\ufb01cation\n\nLatency Qs/ex Acc.\n93.6\n1216 ms\n99.1\n1782 ms\n2103 ms\n99.8\n79.9\nn/a\n93.5\n1680 ms\n1590 ms\n99.2\n\n1.0\n3.0\n5.0\nn/a\n2.66\n2.37\n\nTable 2: Results on NER and Face tasks comparing latencies, queries per token (Qs/tok) and perfor-\nmance metrics (F1 for NER and accuracy for Face).\n\nPredictions are made using MLE on the model given responses. The baseline does not\nreason about time and makes all its queries at the very beginning.\n\n4. LENSE: Our full system as described in Section 3.\n\nImplementation and crowdsourcing setup. We implemented the retainer model of [18] on Ama-\nzon Mechanical Turk to create a \u201cpool\u201d of crowd workers that could respond to queries in real-time.\nThe workers were given a short tutorial on each task before joining the pool to minimize systematic\nerrors caused by misunderstanding the task. We paid workers $1.00 to join the retainer pool and\nan additional $0.01 per query (for NER, since response times were much faster, we paid $0.005\nper query). Worker response times were generally in the range of 0.5\u20132 seconds for NER, 10\u201315\nseconds for Sentiment, and 1\u20134 seconds for Faces.\nWhen running experiments, we found that the results varied based on the current worker quality. To\ncontrol for variance in worker quality across our evaluations of the different methods, we collected\n5 worker responses and their delays on each label ahead of time5. During simulation we sample the\nworker responses and delays without replacement from this frozen pool of worker responses.\n\nSummary of results. Table 2 and Table 3 summarize the performance of the methods on the three\ntasks. On all three datasets, we found that on-the-job learning outperforms machine and human-only\n\n3http://www.cnts.ua.ac.be/conll2003/ner/\n4 The original also includes a \ufb01fth tag for miscellaneous, however the de\ufb01nition for miscellaneos is complex,\n\nmaking it very dif\ufb01cult for non-expert crowd workers to provide accurate labels.\n\n5These datasets are available in the code repository for this paper\n\n6\n\n\fSystem\n1-vote\n3-vote\n5-vote\nUNIGRAMS\nOnline\nThreshold\nLENSE\nRNN\nOnline\nThreshold\nLENSE\n\nLatency Qs/ex Acc.\n89.2\n95.8\n98.7\n\n6.6 s\n10.9 s\n13.5 s\n\n1.00\n3.00\n5.00\n\nn/a\n10.9 s\n11.7 s\n\nn/a\n11.0 s\n11.0 s\n\nn/a\n2.99\n3.48\n\nn/a\n2.85\n3.19\n\n78.1\n95.9\n98.6\n\n85.0\n96.0\n98.6\n\nFigure 3: Queries per example for LENSE on\nSentiment. With simple UNIGRAM features, the\nmodel quickly learns it does not have the ca-\npacity to answer con\ufb01dently and must query the\ncrowd. With more complex RNN features, the\nmodel learns to be more con\ufb01dent and queries\nthe crowd less over time.\n\nTable 3: Results on the Sentiment task compar-\ning latency, queries per example and accuracy.\n\nFigure 4: Comparing F1 and queries per token on the NER task over time. The left graph compares\nLENSE to online learning (which cannot query humans at test time). This highlights that LENSE\nmaintains high F1 scores even with very small training set sizes, by falling back the crowd when it\nis unsure. The right graph compares query rate over time to 1-vote. This clearly shows that as the\nmodel learns, it needs to query the crowd less.\ncomparisons on both quality and cost. On NER, we achieve an F1 of 88.4% at more than an order of\nmagnitude reduction on the cost of achieving comporable quality result using the 5-vote approach.\nOn Sentiment and Faces, we reduce costs for a comparable accuracy by a factor of around 2. For the\nlatter two tasks, both on-the-job learning methods perform less well than in NER. We suspect this\nis due to the presence of a dominant class (\u201cnone\u201d) in NER that the model can very quickly learn to\nexpend almost no effort on. LENSE outperforms the threshold baseline, supporting the importance\nof Bayesian decision theory.\nFigure 4 tracks the performance and cost of LENSE over time on the NER task. LENSE is not only\nable to consistently outperform other baselines, but the cost of the system steadily reduces over time.\nOn the NER task, we \ufb01nd that LENSE is able to trade off time to produce more accurate results than\nthe 1-vote baseline with fewer queries by waiting for responses before making another query.\nWhile on-the-job learning allows us to deploy quickly and ensure good results, we would like to\neventually operate without crowd supervision. Figure 3, we show the number of queries per example\non Sentiment with two different features sets, UNIGRAMS and RNN (as described in Table 1). With\nsimpler features (UNIGRAMS), the model saturates early and we will continue to need to query to\nthe crowd to achieve our accuracy target (as speci\ufb01ed by the loss function). On the other hand,\nusing richer features (RNN) the model is able to learn from the crowd and the amount of supervision\nneeded reduces over time. Note that even when the model capacity is limited, LENSE is able to\nguarantee a consistent, high level of performance.\n\n7\n\n 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 0 20 40 60 80 100 120 140 160 180 200Queries per exampleTimeUnigramsUnigrams + RNN embeddings 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 100 200 300 400 500 600 700F1TimeLENSEonline learning 0 0.2 0.4 0.6 0.8 1 1.2 1.4 0 100 200 300 400 500 600 700Queries per tokenTimeLENSE1 vote baseline\fReproducibility. All code, data, and experiments for this paper are available on CodaLab at\nhttps://www.codalab.org/worksheets/0x2ae89944846444539c2d08a0b7ff3f6f/.\n\n6 Related Work\n\nOn-the-job learning draws ideas from many areas: online learning, active learning, active classi\ufb01ca-\ntion, crowdsourcing, and structured prediction.\nOnline learning. The fundamental premise of online learning is that algorithms should improve\nwith time, and there is a rich body of work in this area [7]. In our setting, algorithms not only\nimprove over time, but maintain high accuracy from the beginning, whereas regret bounds only\nachieve this asymptotically.\nActive learning. Active learning (see [19] for a survey) algorithms strategically select most in-\nformative examples to build a classi\ufb01er. Online active learning [8, 9, 10] performs active learning\nin the online setting. Several authors have also considered using crowd workers as a noisy oracle\ne.g. [20, 21]. It differs from our setup in that it assumes that labels can only be observed after\nclassi\ufb01cation, which makes it nearly impossible to maintain high accuracy in the beginning.\nActive classi\ufb01cation. Active classi\ufb01cation [22, 23, 24] asks what are the most informative features\nto measure at test time. Existing active classi\ufb01cation algorithms rely on having a fully labeled\ndataset which is used to learn a static policy for when certain features should be queried, which does\nnot change at test time. On-the-job learning differs from active classi\ufb01cation in two respects: true\nlabels are never observed, and our system improves itself at test time by learning a stronger model.\nA notable exception is Legion:AR [6], which like us operates in on-the-job learning setting to for\nreal-time activity classi\ufb01cation. However, they do not explore the machine learning foundations\nassociated with operating in this setting, which is the aim of this paper.\nCrowdsourcing. A burgenoning subset of the crowdsourcing community overlaps with machine\nlearning. One example is Flock [25], which \ufb01rst crowdsources the identi\ufb01cation of features for an\nimage classi\ufb01cation task, and then asks the crowd to annotate these features so it can learn a decision\ntree. In another line of work, TurKontrol [26] models individual crowd worker reliability to optimize\nthe number of human votes needed to achieve con\ufb01dent consensus using a POMDP.\nStructured prediction. An important aspect our prediction tasks is that the output is structured,\nwhich leads to a much richer setting for one-the-job learning. Since tags are correlated, the impor-\ntance of a coherent framework for optimizing querying resources is increased. Making active partial\nobservations on structures and has been explored in the measurements framework of [27] and in the\ndistant supervision setting [28].\n7 Conclusion\n\nWe have introduced a new framework that learns from (noisy) crowds on-the-job to maintain high\naccuracy, and reducing cost signi\ufb01cantly over time. The technical core of our approach is modeling\nthe on-the-job setting as a stochastic game and using ideas from game playing to approximate the\noptimal policy. We have built a system, LENSE, which obtains signi\ufb01cant cost reductions over a\npure crowd approach and signi\ufb01cant accuracy improvements over a pure ML approach.\n\nAcknowledgments\n\nWe are grateful to Kelvin Guu and Volodymyr Kuleshov for useful feedback regarding the calibra-\ntion of our models and Amy Bearman for providing the image embeddings for the face classi\ufb01cation\nexperiments. We would also like to thank our anonymous reviewers for their helpful feedback. Fi-\nnally, our work was sponsored by a Sloan Fellowship to the third author.\n\nReferences\n[1] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei. ImageNet: A large-scale hierarchical image\n\ndatabase. In Computer Vision and Pattern Recognition (CVPR), pages 248\u2013255, 2009.\n\n[2] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classi\ufb01cation with deep convolutional neural\n\nnetworks. In Advances in Neural Information Processing Systems (NIPS), pages 1097\u20131105, 2012.\n\n8\n\n\f[3] M. S. Bernstein, G. Little, R. C. Miller, B. Hartmann, M. S. Ackerman, D. R. Karger, D. Crowell, and\nK. Panovich. Soylent: a word processor with a crowd inside. In Symposium on User Interface Software\nand Technology, pages 313\u2013322, 2010.\n\n[4] N. Kokkalis, T. K\u00a8ohn, C. Pfeiffer, D. Chornyi, M. S. Bernstein, and S. R. Klemmer. Emailvalet: Manag-\ning email overload through private, accountable crowdsourcing. In Conference on Computer Supported\nCooperative Work, pages 1291\u20131300, 2013.\n\n[5] C. Li, J. Weng, Q. He, Y. Yao, A. Datta, A. Sun, and B. Lee. Twiner: named entity recognition in targeted\ntwitter stream. In ACM Special Interest Group on Information Retreival (SIGIR), pages 721\u2013730, 2012.\n[6] Walter S Lasecki, Young Chol Song, Henry Kautz, and Jeffrey P Bigham. Real-time crowd labeling for\ndeployable activity recognition. In Proceedings of the 2013 conference on Computer supported coopera-\ntive work, 2013.\n\n[7] N. Cesa-Bianchi and G. Lugosi. Prediction, learning, and games. Cambridge University Press, 2006.\n[8] D. Helmbold and S. Panizza. Some label ef\ufb01cient learning results. In Conference on Learning Theory\n\n(COLT), pages 218\u2013230, 1997.\n\n[9] D. Sculley. Online active learning methods for fast label-ef\ufb01cient spam \ufb01ltering. In Conference on Email\n\nand Anti-spam (CEAS), 2007.\n\n[10] W. Chu, M. Zinkevich, L. Li, A. Thomas, and B. Tseng. Unbiased online active learning in data streams.\n\nIn International Conference on Knowledge Discovery and Data Mining (KDD), pages 195\u2013203, 2011.\n\n[11] T. Gao and D. Koller. Active classi\ufb01cation based on value of classi\ufb01er. In Advances in Neural Information\n\nProcessing Systems (NIPS), pages 1062\u20131070, 2011.\n\n[12] L. Kocsis and C. Szepesv\u00b4ari. Bandit based Monte-Carlo planning. In European Conference on Machine\n\nLearning (ECML), pages 282\u2013293, 2006.\n\n[13] R. Coulom. Computing elo ratings of move patterns in the game of go. Computer Games Workshop,\n\n2007.\n\n[14] J. R. Finkel, T. Grenager, and C. Manning. Incorporating non-local information into information extrac-\ntion systems by Gibbs sampling. In Association for Computational Linguistics (ACL), pages 363\u2013370,\n2005.\n\n[15] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts.\nLearning word vectors for sentiment analysis. In ACL: HLT, pages 142\u2013150, Portland, Oregon, USA,\nJune 2011. Association for Computational Linguistics.\n\n[16] R. Socher, A. Perelygin, J. Y. Wu, J. Chuang, C. D. Manning, A. Y. Ng, and C. Potts. Recursive deep mod-\nels for semantic compositionality over a sentiment treebank. In Empirical Methods in Natural Language\nProcessing (EMNLP), 2013.\n\n[17] N. Kumar, A. C. Berg, P. N. Belhumeur, and S. K. Nayar. Attribute and Simile Classi\ufb01ers for Face\n\nVeri\ufb01cation. In ICCV, Oct 2009.\n\n[18] M. S. Bernstein, J. Brandt, R. C. Miller, and D. R. Karger. Crowds in two seconds: Enabling realtime\n\ncrowd-powered interfaces. In User Interface Software and Technology, pages 33\u201342, 2011.\n\n[19] B. Settles. Active learning literature survey. Technical report, University of Wisconsin, Madison, 2010.\n[20] P. Donmez and J. G. Carbonell. Proactive learning: cost-sensitive active learning with multiple imperfect\n\noracles. In Conference on Information and Knowledge Management (CIKM), pages 619\u2013628, 2008.\n\n[21] D. Golovin, A. Krause, and D. Ray. Near-optimal Bayesian active learning with noisy observations. In\n\nAdvances in Neural Information Processing Systems (NIPS), pages 766\u2013774, 2010.\n\n[22] R. Greiner, A. J. Grove, and D. Roth. Learning cost-sensitive active classi\ufb01ers. Arti\ufb01cial Intelligence,\n\n139(2):137\u2013174, 2002.\n\n[23] X. Chai, L. Deng, Q. Yang, and C. X. Ling. Test-cost sensitive naive Bayes classi\ufb01cation. In International\n\nConference on Data Mining, pages 51\u201358, 2004.\n\n[24] S. Esmeir and S. Markovitch. Anytime induction of cost-sensitive trees. In Advances in Neural Informa-\n\ntion Processing Systems (NIPS), pages 425\u2013432, 2007.\n\n[25] J. Cheng and M. S. Bernstein. Flock: Hybrid Crowd-Machine learning classi\ufb01ers. In Proceedings of the\n18th ACM Conference on Computer Supported Cooperative Work & Social Computing, pages 600\u2013611,\n2015.\n\n[26] P. Dai, Mausam, and D. S. Weld. Decision-theoretic control of crowd-sourced work\ufb02ows. In Association\n\nfor the Advancement of Arti\ufb01cial Intelligence (AAAI), 2010.\n\n[27] P. Liang, M. I. Jordan, and D. Klein. Learning from measurements in exponential families. In Interna-\n\ntional Conference on Machine Learning (ICML), 2009.\n\n[28] G. Angeli, J. Tibshirani, J. Y. Wu, and C. D. Manning. Combining distant and partial supervision for\n\nrelation extraction. In Empirical Methods in Natural Language Processing (EMNLP), 2014.\n\n9\n\n\f", "award": [], "sourceid": 1922, "authors": [{"given_name": "Keenon", "family_name": "Werling", "institution": "Stanford University"}, {"given_name": "Arun Tejasvi", "family_name": "Chaganty", "institution": "Stanford"}, {"given_name": "Percy", "family_name": "Liang", "institution": "Stanford University"}, {"given_name": "Christopher", "family_name": "Manning", "institution": "Stanford University"}]}