{"title": "A KL-LUCB algorithm for Large-Scale Crowdsourcing", "book": "Advances in Neural Information Processing Systems", "page_first": 5894, "page_last": 5903, "abstract": "This paper focuses on best-arm identification in multi-armed bandits with bounded rewards. We develop an algorithm that is a fusion of lil-UCB and KL-LUCB, offering the best qualities of the two algorithms in one method. This is achieved by proving a novel anytime confidence bound for the mean of bounded distributions, which is the analogue of the LIL-type bounds recently developed for sub-Gaussian distributions. We corroborate our theoretical results with numerical experiments based on the New Yorker Cartoon Caption Contest.", "full_text": "A KL-LUCB Bandit Algorithm for\n\nLarge-Scale Crowdsourcing\n\nErvin T\u00e1nczos\u2217 and Robert Nowak\u2020\n\nUniversity of Wisconsin-Madison\n\ntanczos@wisc.edu,\n\nrdnowak@wisc.edu\n\nBob Mankoff\n\nFormer Cartoon Editor of the New Yorker\n\nbmankoff@hearst.com\n\nAbstract\n\nThis paper focuses on best-arm identi\ufb01cation in multi-armed bandits with bounded\nrewards. We develop an algorithm that is a fusion of lil-UCB and KL-LUCB,\noffering the best qualities of the two algorithms in one method. This is achieved by\nproving a novel anytime con\ufb01dence bound for the mean of bounded distributions,\nwhich is the analogue of the LIL-type bounds recently developed for sub-Gaussian\ndistributions. We corroborate our theoretical results with numerical experiments\nbased on the New Yorker Cartoon Caption Contest.\n\n1 Multi-Armed Bandits for Large-Scale Crowdsourcing\n\nThis paper develops a new multi-armed bandit (MAB) for large-scale crowdsourcing, in the style\nof the KL-UCB [4, 9, 3]. Our work is strongly motivated by crowdsourcing contests, like the New\nYorker Cartoon Caption contest [10]3. The new approach targets the \u201cbest-arm identi\ufb01cation problem\u201d\n[1] in the \ufb01xed con\ufb01dence setting and addresses two key limitations of existing theory and algorithms:\n\n(i) State of the art algorithms for best arm identi\ufb01cation are based on sub-Gaussian con\ufb01dence bounds\n\n[5] and fail to exploit the fact that rewards are usually bounded in crowdsourcing applications.\n\n(ii) Existing KL-UCB algorithms for best-arm identi\ufb01cation do exploit bounded rewards [8] , but have\nsuboptimal performance guarantees in the \ufb01xed con\ufb01dence setting, both in terms of dependence\non problem-dependent hardness parameters (Chernoff information) and on the number of arms,\nwhich can be large in crowdsourcing applications.\n\nThe new algorithm we propose and analyze is called lil-KLUCB, since it is inspired by the lil-UCB\nalgorithm [5] and the KL-LUCB algorithm [8]. The lil-UCB algorithm is based on sub-Gaussian\nbounds and has a sample complexity for best-arm identi\ufb01cation that scales as\n\n(cid:88)\n\ni\u22652\n\n\u2206\u22122\n\ni\n\nlog(\u03b4\u22121 log \u2206\u22122\n\ni\n\n) ,\n\nwhere \u03b4 \u2208 (0, 1) is the desired con\ufb01dence and \u2206i = \u00b51 \u2212 \u00b5i is the gap between the means of the\nbest arm (denoted as arm 1) and arm i. If the rewards are in [0, 1], then the KL-LUCB algorithm has\n\u2020This work was partially supported by the NSF grant IIS-1447449 and the AFSOR grant FA9550-13-1-0138.\n3For more details on the New Yorker Cartoon Caption Contest, see the Supplementary Materials.\n\n31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.\n\n\fa sample complexity scaling essentially like4(cid:88)\n\n(D\u2217\n\ni )\u22121 log(n\u03b4\u22121(D\u2217\n\ni )\u22121) ,\n\ni\u22652\ni := D\u2217(\u00b51, \u00b5i) is the Chernoff-information between a Ber(\u00b51)\nwhere n is the number of arms and D\u2217\nand a Ber(\u00b5i) random variable5. Ignoring the logarithmic factor, this bound is optimal for the case\nof Bernoulli rewards [7, 11]. Comparing these two bounds, we observe that KL-LUCB may offer\nbene\ufb01ts since D\u2217\ni /2, but lil-UCB has better logarithmic\ni and no explicit dependence on the number of arms n. Our new algorithm\ndependence on the \u22062\nlil-KLUCB offers the best of both worlds, providing a sample complexity that scales essentially like\n\ni = D\u2217(\u00b51, \u00b5i) \u2265 (\u00b51 \u2212 \u00b5i)2/2 = \u22062\n\n(cid:88)\n\ni\u22652\n\n(D\u2217\n\ni )\u22121 log(\u03b4\u22121 log(D\u2217\n\ni )\u22121) .\n\nThe key to this result is a novel anytime con\ufb01dence bound for sums of bounded random variables,\nwhich requires a signi\ufb01cant departure from previous analyses of KL-based con\ufb01dence bounds.\nThe practical bene\ufb01t of lil-KLUCB is illustrated in terms of the New Yorker Caption Contest problem\n[10]. The goal of that crowdsourcing task is to identify the funniest cartoon caption from a batch\nof n \u2248 5000 captions submitted to the contest each week. The crowd provides \u201c3-star\u201d ratings for\nthe captions, which can be mapped to {0, 1/2, 1}, for example. Unfortunately, many of the captions\nare not funny, getting average ratings close to 0 (and consequently very small variances). This\nfact, however, is ideal for KL-based con\ufb01dence intervals, which are signi\ufb01cantly tighter than those\nbased on sub-Gaussianity and the worst-case variance of 1/4. Compared to existing methods, the\nlil-KLUCB algorithm better addresses the two key features in this sort of application: (1) a very large\nnumber of arms, and (2) bounded reward distributions which, in many cases, have very low variance.\nIn certain instances, this can have a profound effect on sample complexity (e.g., O(n2) complexity\nfor algorithms using sub-Gaussian bounds vs. O(n log n) for lil-KLUCB, as shown in Table 1).\nThe paper is organized as follows. Section 2 de\ufb01nes the best-arm identi\ufb01cation problem, gives the\nlil-KLUCB algorithm and states the main results. We also brie\ufb02y review related literature, and\ncompare the performance of lil-KLUCB to that of previous algorithms. Section 3 provides the main\ntechnical contribution of the paper, a novel anytime con\ufb01dence bound for sums of bounded random\nvariables. Section 4 analyzes the performance of the lil-KLUCB algorithm. Section 5 provides\nexperimental support for the lil-KLUCB algorithm using data from the New Yorker Caption Contest.\n\n2 Problem Statement and Main Results\nConsider a MAB problem with n arms. We use the shorthand notation [n] := {1, . . . , n}. For every\ni \u2208 [n] let {Xi,j}j\u2208N denote the reward sequence of arm i, and suppose that P(Xi,j \u2208 [0, 1]) = 1 for\nall i \u2208 [n], j \u2208 N. Furthermore, assume that all rewards are independent, and that Xi,j \u223c Pi for all\nj \u2208 N. Let the mean reward of arm i be denoted by \u00b5i and assume w.l.o.g. that \u00b51 > \u00b52 \u2265 \u00b7\u00b7\u00b7 \u2265 \u00b5n.\nWe focus on the best-arm identi\ufb01cation problem in the \ufb01xed-con\ufb01dence setting. At every time t \u2208 N\nwe are allowed to select an arm to sample (based on past rewards) and observe the next element in\nits reward sequence. Based on the observed rewards, we wish to \ufb01nd the arm with the highest mean\nreward. In the \ufb01xed con\ufb01dence setting, we prescribe a probability of error \u03b4 \u2208 (0, 1) and our goal is\nto construct an algorithm that \ufb01nds the best arm with probability at least 1\u2212 \u03b4. Among 1\u2212 \u03b4 accurate\nalgorithms, one naturally favors those that require fewer samples. Hence proving upper bounds on\nthe sample complexity of a candidate algorithm is of prime importance.\nThe lil-KLUCB algorithm that we propose is a fusion of lil-UCB [5] and KL-LUCB [8], and its\noperation is essentially a special instance of LUCB++ [11]. At each time step t, let Ti(t) denote the\n\ntotal number of samples drawn from arm i so far, and let(cid:98)\u00b5i,Ti(t) denote corresponding empirical\n\nmean. The algorithm is based on lower and upper con\ufb01dence bounds of the following general form:\n\n4A more precise characterization of the sample complexity is given in Section 2.\n5The Chernoff-information between random variables Ber(x) and Ber(y) (0 < x < y < 1) is D\u2217(x, y) =\n1\u2212x and z\u2217 is the unique z \u2208 (x, y) such that\n\nx + (1 \u2212 z) log 1\u2212z\n\nD(z\u2217, x) = D(z\u2217, y), where D(z, x) = z log z\nD(z, x) = D(z, y).\n\n2\n\n\ffor each i \u2208 [n] and any \u03b4 \u2208 (0, 1)\n\n(cid:26)\n(cid:27)\nm <(cid:98)\u00b5i,Ti(t) : D(cid:0)(cid:98)\u00b5i,Ti(t), m(cid:1) \u2264 c log (\u03ba log2(2Ti(t))/\u03b4)\n(cid:26)\n(cid:27)\nm >(cid:98)\u00b5i,Ti(t) : D(cid:0)(cid:98)\u00b5i,Ti(t), m(cid:1) \u2264 c log (\u03ba log2(2Ti(t))/\u03b4)\n\nTi(t)\n\nTi(t)\n\nLi(t, \u03b4) = inf\n\nUi(t, \u03b4) = sup\n\nwhere c and \u03ba are small constants (de\ufb01ned in the next section). These bounds are designed so that\nwith probability at least 1 \u2212 \u03b4, Li(Ti(t), \u03b4) \u2264 \u00b5i \u2264 Ui(Ti(t), \u03b4) holds for all t \u2208 N. For any t \u2208 N\nlet TOP(t) be the index of the arm with the highest empirical mean, breaking ties at random. With\nthis notation, we state the lil-KLUCB algorithm and our main theoretical result.\n\nlil-KLUCB\n\n1. Initialize by sampling every arm once.\n2. While LTOP(t)(TTOP(t)(t), \u03b4/(n \u2212 1)) \u2264 max\ni(cid:54)=TOP(t)\n\nUi(Ti(t), \u03b4) do:\n\n\u2022 Sample the following two arms:\n\n\u2013 TOP(t), and\n\u2013 arg max\ni(cid:54)=TOP(t)\n\nUi(Ti(t), \u03b4)\n\nand update means and con\ufb01dence bounds.\n\n3. Output TOP(t)\n\nTheorem 1. For every i \u2265 2 let(cid:101)\u00b5i \u2208 (\u00b5i, \u00b51), and(cid:101)\u00b5 = maxi\u22652(cid:101)\u00b5i. With probability at least 1 \u2212 2\u03b4,\n\nlil-KLUCB returns the arm with the largest mean and the total number of samples it collects is upper\nbounded by\n\nc0 log(cid:0)(n \u2212 1)\u03b4\u22121 log D\u2217(\u00b51,(cid:101)\u00b5)\u22121(cid:1)\n\nD\u2217(\u00b51,(cid:101)\u00b5)\n\n(cid:88)\n\ni\u22652\n\n+\n\nc0 log(cid:0)\u03b4\u22121 log D\u2217(\u00b5i,(cid:101)\u00b5i)\u22121(cid:1)\n\nD\u2217(\u00b5i,(cid:101)\u00b5i)\n\n,\n\n(cid:101)\u00b52,...,(cid:101)\u00b5n\n\ninf\n\nwhere c0 is some universal constant, D\u2217(x, y) is the Chernoff-information.\nRemark 1. Note that the LUCB++ algorithm of [11] is general enough to handle identi\ufb01cation of\nthe top k arms (not just the best-arm). All arguments presented in this paper also go through when\nconsidering the top-k problem for k > 1. However, to keep the arguments clear and concise, we\nchose to focus on the best-arm problem only.\n\n2.1 Comparison with previous work\n\nWe now compare the sample complexity of lil-KLUCB to that of the two most closely related\nalgorithms, KL-LUCB [8] and lil-UCB [5]. For a detailed review of the history of MAB problems\nand the use of KL-con\ufb01dence intervals for bounded rewards, we refer the reader to [3, 9, 4].\nFor the KL-LUCB algorithm, Theorem 3 of [8] guarantees a high-probability sample complexity\nupper bound scaling as\n\n(D\u2217(\u00b5i, c))\u22121 log(cid:0)n\u03b4\u22121(D\u2217(\u00b5i, c))\u22121(cid:1) .\n\n(cid:88)\n\ni\u22651\n\ninf\n\nc\u2208(\u00b51,\u00b52)\n\nOur result improves this in two ways. On one hand, we eliminate the unnecessary logarithmic\ndependence on the number of arms n in every term. Note that the log n factor still appears in\nTheorem 1 in the term corresponding to the number of samples on the best arm. It is shown in [11]\nthat this factor is indeed unavoidable. The other improvement lil-KLUCB offers over KL-LUCB\nis improved logarithmic dependence on the Chernoff-information terms. This is due to the tighter\ncon\ufb01dence intervals derived in Section 3.\nComparing Theorem 1 to the sample complexity of lil-UCB, we see that the two are of the same form,\nthe exception being that the Chernoff-information terms take the place of the squared mean-gaps\n\n3\n\n\f(which arise due to the use of sub-Gaussian (SG) bounds). To give a sense of the improvement this\ncan provide, we compare the sums6\n\n(cid:88)\n\ni\u22652\n\nSKL =\n\n1\n\nD\u2217(\u00b5i, \u00b51)\n\nand SSG =\n\n1\n\u22062\ni\n\n.\n\n(cid:88)\n\ni\u22652\n\nLet \u00b5, \u00b5(cid:48) \u2208 (0, 1), \u00b5 < \u00b5(cid:48) and \u2206 = |\u00b5 \u2212 \u00b5(cid:48)|. Note that the Chernoff-information between Ber(\u00b5)\nand Ber(\u00b5(cid:48)) can be expressed as\n\nD\u2217(\u00b5, \u00b5(cid:48)) = max\nx\u2208[\u00b5,\u00b5(cid:48)]\n\nmin{D(x, \u00b5), D(x, \u00b5(cid:48))} = D(x\u2217, \u00b5) = D(x\u2217, \u00b5(cid:48)) = D(x\u2217,\u00b5)+D(x\u2217,\u00b5(cid:48))\n\n2\n\n,\n\nfor some unique x\u2217 \u2208 [\u00b5, \u00b5(cid:48)]. It follows that\n\nD\u2217(\u00b5, \u00b5(cid:48)) \u2265 min\nx\u2208[\u00b5,\u00b5(cid:48)]\n\nD(x, \u00b5) + D(x, \u00b5(cid:48))\n\n2\n\n(cid:112)\u00b5(\u00b5 + \u2206) +(cid:112)(1 \u2212 \u00b5)(1 \u2212 \u00b5 \u2212 \u2206)\n\n1\n\n.\n\n= log\n\nUsing this with every term in SKL gives us an upper bound on that sum. If the means are all bounded\nwell away from 0 and 1, then SKL may not differ that much from SSG. There are some situations\nhowever, when the two expressions behave radically differently. As an example, consider a situation\nwhen \u00b51 = 1. In this case we get\n\nSKL \u2264 (cid:88)\n\n2\n1\u2212\u2206i\n\n1\n\nlog\n\n\u2264 2\n\ni\u22652\n\n(cid:88)\n\ni\u22652\n\n1\n\u2206i\n\n(cid:28)(cid:88)\n\ni\u22652\n\n1\n\u22062\ni\n\n= SSG .\n\nTable 1 illustrates the difference between the scaling of the sums SKL and SSG when the gaps have\nthe parametric form \u2206i = (i/n)\u03b1.\n\nTable 1: SKL versus SSG for mean gaps \u2206i = ( i\n\n\u03b1\n\nSKL\nSSG\n\n\u2208 (0, 1/2)\n\nn\nn\n\n1/2\n\nn\n\nn log n\n\n\u2208 (1/2, 1)\n\nn )\u03b1, i = 1, . . . , n\n\u2208 (1,\u221e)\n\n1\n\nn\nn2\u03b1\n\nn log n\n\nn2\n\nn\u03b1\nn2\u03b1\n\nWe see that KL-type con\ufb01dence bounds can sometimes provide a signi\ufb01cant advantage in terms of\nthe sample complexity. Intuitively, the gains will be greatest when many of the means are close to 0\nor 1 (and hence have low variance). We will illustrate in Section 5 that such gains often also manifest\nin practical applications like the New Yorker Caption Contest problem.\n\n3 Anytime Con\ufb01dence Intervals for Sums of Bounded Random Variables\n\ni )\u22121 log log(D\u2217\n\ni )\u22121) samples from a suboptimal arm i, where D\u2217\n\nThe main step in our analysis is proving a sharp anytime con\ufb01dence bound for the mean of\nbounded random variables. These will be used to show, in Section 4, that lil-KLUCB draws at\nmost O((D\u2217\n:= D\u2217(\u00b51, \u00b5i) is\nthe Chernoff-information between a Ber(\u00b51) and a Ber(\u00b5i) random variable and arm 1 is the arm\nwith the largest mean. The iterated log factor is a necessary consequence of the law-of-the-iterated\nlogarithm [5], and in it is in this sense that we call the bound sharp. Prior work on MAB algorithms\nbased on KL-type con\ufb01dence bounds [4, 9, 3] did not focus on deriving tight anytime con\ufb01dence\nbounds.\nConsider a sequence of iid random variables Y1, Y2, . . . that are bounded in [0, 1] and have mean \u00b5.\n\nj\u2208[t] Yj be the empirical mean of the observations up to time t \u2208 N.\n\nLet(cid:98)\u00b5t = 1\n\nt\n\nTheorem 2. Let \u00b5 \u2208 [0, 1] and \u03b4 \u2208 (0, 1) be arbitrary. Fix any l \u2265 0 and set N = 2l, and de\ufb01ne\n\n(cid:80)\n\ni\n\n\uf8eb\uf8ed(cid:88)\n\nt\u2208[N ]\n\n(cid:88)\n\nk\u2265l\n\n\uf8f6\uf8f8 N\n\nN +1\n\n.\n\n\u03ba(N ) = \u03b41/(N +1)\n\n1{l(cid:54)=0} log2(2t)\u2212 N +1\n\nN + N\n\n(k + 1)\u2212 N +1\n\nN\n\n6Consulting the proof of Theorem 1 it is clear that the number of samples on the sub-optimal arms of\nlil-KLUCB scales essentially as SKL w.h.p. (ignoring doubly logarithmic terms), and a similar argument can be\nmade about lil-UCB. This justi\ufb01es considering these sums in order to compare lil-KLUCB and lil-UCB.\n\n4\n\n\f(cid:16)\n\n(cid:16)\n\n(cid:17)\n\n(cid:17)\n\n(i) De\ufb01ne the sequence zt \u2208 (0, 1 \u2212 \u00b5], t \u2208 N such that\n\nif a solution exists, and zt = 1 \u2212 \u00b5 otherwise. Then P (\u2203t \u2208 N : (cid:98)\u00b5t \u2212 \u00b5 > zt) \u2264 \u03b4.\n\nN +1 zt, \u00b5\n\n\u00b5 + N\n\nD\n\n=\n\nt\n\n,\n\nlog (\u03ba(N ) log2(2t)/\u03b4)\n\n(1)\n\n(ii) De\ufb01ne the sequence zt > 0, t \u2208 N such that\n\nif a solution exists, and zt = \u00b5 otherwise. Then P (\u2203t \u2208 N : (cid:98)\u00b5t \u2212 \u00b5 < \u2212zt) \u2264 \u03b4.\n\nN +1 zt, \u00b5\n\nD\n\n=\n\nt\n\n,\n\nlog (\u03ba(N ) log2(2t)/\u03b4)\n\n\u00b5 \u2212 N\n\nThe result above can be used to construct anytime con\ufb01dence bounds for the mean as follows. Consider\npart (i) of Theorem 2 and \ufb01x \u00b5. The result gives a sequence zt that upper bounds the deviations of\nthe empirical mean. It is de\ufb01ned through an equation of the form D(\u00b5 + N zt/(N + 1), \u00b5) = ft.\nNote that the arguments of the function on the left must be in the interval [0, 1], in particular\nN zt/(N + 1) < 1 \u2212 \u00b5, and the maximum of D(\u00b5 + x, \u00b5) for x > 0 is D(1, \u00b5) = log \u00b5\u22121. Hence,\nequation 1 does not have a solution if ft is too large (that is, if t is small). In these cases we set\nzt = 1 \u2212 \u00b5. However, since ft is decreasing, equation 1 does have a solution when t \u2265 T (for some\nT depending on \u00b5), and this solution is unique (since D(\u00b5 + x, \u00b5) is strictly increasing).\n\nWith high probability(cid:98)\u00b5t \u2212 \u00b5 \u2264 zt for all t \u2208 N by Theorem 2. Furthermore, the function D(\u00b5 + x, \u00b5)\n\nis increasing in x \u2265 0. By combining these facts we get that with probability at least 1 \u2212 \u03b4\n\nk\u2265l\n\n5\n\nOn the other hand\n\nD\n\nby de\ufb01nition. Chaining these two inequalities leads to the lower con\ufb01dence bound\n\n(cid:16)\n\nD\n\n\u00b5 + N\n\nN +1 zt, \u00b5\n\n.\n\nN +1 , \u00b5\n\n(cid:16) N(cid:98)\u00b5t+\u00b5\n\n(cid:17) \u2265 D\n(cid:17)\n(cid:17) \u2264 log (\u03ba(N ) log2(2t)/\u03b4)\n(cid:16) N(cid:98)\u00b5t+m\n(cid:16) N(cid:98)\u00b5t+m\n\n(cid:17) \u2264 log (\u03ba(N ) log2(2t)/\u03b4)\n(cid:27)\n(cid:17) \u2264 log (\u03ba(N ) log2(2t)/\u03b4)\n\nN +1 , m\n\nt\n\n,\n\n(cid:27)\n\n(cid:16)\n(cid:26)\n\n(cid:26)\n\n\u00b5 + N\n\nN +1 zt, \u00b5\n\nm <(cid:98)\u00b5t : D\n\nm >(cid:98)\u00b5t : D\n\n(2)\n\n(3)\n\nL(t, \u03b4) = inf\n\nwhich holds for all times t with probability at least 1 \u2212 \u03b4. Considering the left deviations of(cid:98)\u00b5t \u2212 \u00b5\n\nt\n\nwe can get an upper con\ufb01dence bound in a similar manner:\n\n.\n\nt\n\nN +1 , m\n\nU (t, \u03b4) = sup\n\nN +1 , m\n\nL(cid:48)(t, \u03b4) = inf\n\nThat is, for all times t, with probability at least 1 \u2212 2\u03b4 we have L(t, \u03b4) \u2264 (cid:98)\u00b5t \u2264 U (t, \u03b4).\nN +1 \u2248 (cid:98)\u00b5t, and thus\nNote that the constant log \u03ba(N ) \u2248 2 log2(N ), so the choice of N plays a relatively mild role in\nthe bounds. However, we note here that if N is suf\ufb01ciently large, then N(cid:98)\u00b5t+m\n(cid:17) \u2248 D ((cid:98)\u00b5t, m), in which case the bounds above are easily compared to those in prior\n(cid:16) N(cid:98)\u00b5t+m\n(cid:27)\n(cid:27)\n\nD\nworks [4, 9, 3]. We make this connection more precise and show that the con\ufb01dence intervals de\ufb01ned\nas\n\nm <(cid:98)\u00b5t : D ((cid:98)\u00b5t, m) \u2264 c(N ) log (\u03ba(N ) log2(2t)/\u03b4)\nm >(cid:98)\u00b5t : D ((cid:98)\u00b5t, m) \u2264 c(N ) log (\u03ba(N ) log2(2t)/\u03b4)\n\nTheorem 1 in the Supplementary Material, where the correctness of L(cid:48)(t, \u03b4) and U(cid:48)(t, \u03b4) is shown.\n\nsatisfy L(cid:48)(t, \u03b4) \u2264 (cid:98)\u00b5t \u2264 U(cid:48)(t, \u03b4) for all t, with probability 1 \u2212 2\u03b4. The constant c(N ) is de\ufb01ned in\nwe only prove part (i). Note that {(cid:98)\u00b5t \u2212 \u00b5 > zt} \u21d0\u21d2 {St > tzt}, where St =(cid:80)\nP(cid:0)\u2203t \u2208 [2k, 2k+1] : St > tzt\n\nProof of Theorem 2. The proofs of parts (i) and (ii) are completely analogous, hence in what follows\nj\u2208[t](Yj \u2212 \u00b5)\n\nP (\u2203t \u2208 N : St > tzt) \u2264 P (\u2203t \u2208 [N ] : St > tzt) +\n\ndenotes the centered sum up to time t. We start with a simple union bound\n\nU(cid:48)(t, \u03b4) = inf\n\n(cid:88)\n\n(cid:26)\n(cid:26)\n\n(cid:1) .\n\n, and\n\n(4)\n\nt\n\nt\n\n,\n\n\fFirst, we bound each summand in the second term individually. In an effort to save space, we de\ufb01ne\nthe event Ak = {\u2203t \u2208 [2k, 2k+1] : St > tzt}. Let tj,k = (1 + j\nN )2k. In what follows we use the\nnotation tj \u2261 tj,k. We have\n\nP (Ak) \u2264 (cid:88)\n\nP (\u2203t \u2208 [tj\u22121, tj] : St > tzt) \u2264 (cid:88)\n\nP(cid:0)\u2203t \u2208 [tj\u22121, tj] : St > tj\u22121ztj\u22121\n\n(cid:1) ,\n\nj\u2208[N ]\n\nj\u2208[N ]\n\nwhere the last step is true if tzt is non-decreasing in t. This technical claim is formally shown in\nLemma 1 in the Supplementary Material. However, to give a short heuristic, it is easy to see that\ntzt has an increasing lower bound. Noting that D(\u00b5 + x, \u00b5) is convex in x (the second derivative is\npositive), and that D(\u00b5, \u00b5) = 0, we have D(1, \u00b5)x \u2265 D(\u00b5 + x, \u00b5). Hence zt (cid:38) t\u22121 log log t.\nUsing a Chernoff-type bound together with Doob\u2019s inequality, we can continue as\n\nP (Ak) \u2264 inf\n\n(cid:88)\n\nj\u2208[N ]\n\nexp\n\nexp\n\n=\n\n\u03bb>0\n\nj\u2208[N ]\n\n\u2264 (cid:88)\n(cid:88)\nP (Ak) \u2264 (cid:88)\n(cid:88)\n\nj\u2208[N ]\n\n=\n\nj\u2208[N ]\n\nj\u2208[N ]\n\n(cid:1)(cid:1)\n\n\u03bb>0\n\ne\u03bbStj\n\n\u2212 sup\n\nP(cid:0)\u2203t \u2208 [tj\u22121, tj] : exp (\u03bbSt) > exp(cid:0)\u03bbtj\u22121ztj\u22121\n(cid:17)(cid:17)(cid:19)\n(cid:18)\ne\u03bb(Y1\u2212\u00b5)(cid:17)(cid:17)(cid:19)\n(cid:18)\ne\u03bb(\u03be\u2212\u00b5)(cid:17)(cid:17)(cid:19)\n\n\u03bbtj\u22121ztj\u22121 \u2212 log E(cid:16)\n(cid:16)\nN +j ztj\u22121 \u2212 log E(cid:16)\n(cid:16)\n(cid:18)\n\u03bb\u03b1jztj\u22121 \u2212 log E(cid:16)\n(cid:16)\nexp(cid:0)\u2212tjD(cid:0)\u00b5 + \u03b1jztj\u22121 , \u00b5(cid:1)(cid:1) ,\n\n\u2212tj sup\n\u03bb\u22650\n\n\u2212tj sup\n\u03bb\u22650\n\n\u03bb N +j\u22121\n\nexp\n\n.\n\n(5)\n\n(6)\n\nUsing E(e\u03bbY1) \u2264 E(e\u03bb\u03be) where \u03be \u223c Ber(\u00b5) (see Lemma 9 of [4]), and the notation \u03b1j = N +j\u22121\n\n,\n\nN +j\n\nsince the rate function of a Bernoulli random variable can be explicitly computed, namely we have\nsup\u03bb>0(\u03bbx \u2212 log E(e\u03bb\u03be)) = D(\u00b5 + x, \u00b5) (see [2]).\nAgain, we use the convexity of D(\u00b5 + x, \u00b5). For any \u03b1 \u2208 (0, 1) we have \u03b1D(\u00b5 + x, \u00b5) \u2265\nD(\u00b5 + \u03b1x, \u00b5), since D(\u00b5, \u00b5) = 0. Using this with \u03b1 =\n\nN\n\n\u03b1j (N +1) and x = \u03b1jztj\u22121, we get that\n\u00b5 + N\n\nN +1 ztj\u22121 , \u00b5\n\n.\n\n(cid:17)\n(cid:17)(cid:17)\n\nPlugging in the de\ufb01nition of tj and the sequence zt, and noting that \u03b4 < 1, we arrive at the bound\n\nRegarding the \ufb01rst term in (4), again using the Bernoulli rate function bound we have\n\n(cid:16)\n(cid:16)\n\nN\n\n\u03b1j (N +1) D(cid:0)\u00b5 + \u03b1jztj\u22121 , \u00b5(cid:1) \u2265 D\nP (Ak) \u2264 (cid:88)\n(cid:18)\n\n(cid:16)\u2212tj\n\nN +1\nN \u03b1jD\n\nj\u2208[N ]\n\nexp\n\nThis implies\n\n(cid:18)\nP (Ak) \u2264 (cid:88)\nP (\u2203t \u2208 [N ] : (cid:98)\u00b5t \u2212 \u00b5 > zt) \u2264 (cid:88)\n\n\u2212 N + 1\n\nj\u2208[N ]\n\nexp\n\nlog\n\nN\n\nUsing the convexity of D(\u00b5 + x, \u00b5) as before, we can continue as\n\nP (\u2203t \u2208 [N ] : (cid:98)\u00b5t \u2212 \u00b5 > zt) \u2264 (cid:88)\n\u2264 (cid:88)\n\n\u00b5 + N\n\nN +1 ztj\u22121, \u00b5\n\n.\n\n(7)\n\n(cid:19)(cid:19)\n\n(cid:16)\n\n(cid:17) N +1\n\nN\n\n.\n\n\u03ba(N )(k+1)\n\nexp (\u2212tD(\u00b5 + zt, \u00b5)) .\n\n\u03b4\n\nN\n\n)/\u03b4\n\nt\u2208[N ]\n\n\u2264 N\n\n\u03ba(N ) log2(2k+1 N + j \u2212 1\nP ((cid:98)\u00b5t \u2212 \u00b5 > zt) \u2264 (cid:88)\n(cid:16)\u2212t N +1\n(cid:16)\n(cid:17)(cid:17)\nN log (\u03ba(N ) log2(2t)/\u03b4)(cid:1)\nexp(cid:0)\u2212 N +1\n(cid:88)\n\nN +1 zt, \u00b5\n\n\u00b5 + N\n\nN D\n\nt\u2208[N ]\n\nt\u2208[N ]\n\nt\u2208[N ]\n\nexp\n\nN +1\n\nN \u03ba(N )\u2212 N +1\n\nN\n\nlog2(2t)\u2212 N +1\nN .\n\n\u2264 \u03b4\n\nt\u2208[N ]\n\n6\n\n\fPlugging the two bounds back into (4) we conclude that\n\nP (\u2203t : (cid:98)\u00b5t \u2212 \u00b5 > zt) \u2264 \u03b4\n\nN +1\n\nN \u03ba(N )\u2212 N +1\n\nN\n\n\uf8eb\uf8ed1{l(cid:54)=0} log2(2j)\u2212 N +1\n\nN +\n\n(cid:88)\n\nk\u2265l\n\n(k + 1)\u2212 N +1\n\nN\n\n\uf8f6\uf8f8 \u2264 \u03b4 ,\n\n(cid:88)\n\nj\u2208[N ]\n\nby the de\ufb01nition of \u03ba(N ).\n\n4 Analysis of lil-KLUCB\n\nRecall that the lil-KLUCB algorithm uses con\ufb01dence bounds of the form Ui(t, \u03b4) = sup{m >(cid:98)\u00b5t :\nD((cid:98)\u00b5t, m) \u2264 ft(\u03b4)} with some decreasing sequence ft(\u03b4). In this section we make this dependence\n\nexplicit, and use the notations Ui(ft(\u03b4)) and Li(ft(\u03b4)) for upper and lower con\ufb01dence bounds. For\nany \u0001 > 0 and i \u2208 [n], de\ufb01ne the events \u2126i(\u0001) = {\u2200t \u2208 N : \u00b5i \u2208 [Li(ft(\u0001)), Ui(ft(\u0001))]}.\nThe correctness of the algorithm follows from the correctness of the individual con\ufb01dence intervals,\nas is usually the case with LUCB algorithms. This is shown formally in Proposition 1 provided in the\nSupplementary Materials. The main focus in this section is to show a high probability upper bound\non the sample complexity. This can be done by combining arguments frequently used for analyzing\nLUCB algorithms and those used in the analysis of the lil-UCB [5]. The proof is very similar in spirit\nto that of the LUCB++ algorithm [11]. Due to spatial restrictions, we only provide a proof sketch\nhere, while the detailed proof is provided in the Supplementary Materials.\n\nProof sketch of Theorem 1. Observe that at each time step two things can happen (apart from stop-\nping): (1) Arm 1 is not sampled (two sub-optimal arms are sampled); (2) Arm 1 is sampled together\nwith some other (suboptimal) arm. Our aim is to upper bound the number of times any given arm is\nsampled for either of the reasons above. We do so by conditioning on the event\n\n\uf8f6\uf8f8 , for a certain choice of {\u03b4i} de\ufb01ned below.\n\n\uf8eb\uf8ed(cid:92)\n\ni\u22652\n\n\u2126(cid:48) = \u21261(\u03b4) \u2229\n\n\u2126i(\u03b4i)\n\nFor instance, if arm 1 is not sampled at a given time t, we know that TOP(t) (cid:54)= 1, which means\nthere must be an arm i \u2265 2 such that Ui(Ti(t), \u03b4) \u2265 U1(T1(t), \u03b4). However, on the event \u21261(\u03b4), the\nUCB of arm 1 is accurate, implying that Ui(Ti(t), \u03b4) \u2265 \u00b51. This implies that Ti(t) can not be too\n\nbig, since on \u2126i(\u03b4i),(cid:98)\u00b5i,t is \u201cclose\" to \u00b5i, and also Ui(Ti(t), \u03b4) is not much larger then(cid:98)\u00b5i. All this is\n\nmade formal in Lemma 2, yielding the following upper bound on number of times arm i is sampled\nfor reason (1):\n\n\u03c4i(\u03b4 \u00b7 \u03b4i) = min{t \u2208 N : ft(\u03b4 \u00b7 \u03b4i) < D\u2217(\u00b5i, \u00b51)} .\n\nSimilar arguments can be made about the number of samples of any suboptimal arm i for reason (2),\nand also the number of samples on arm 1. This results in the sample complexity upper bound\n\nK1 log(cid:0)(n \u2212 1)\u03b4\u22121 log D\u2217(\u00b51,(cid:101)\u00b5)\u22121(cid:1)\naccording to Theorem 1 in the Supplementary Material. Substituting \u03b3 = exp(\u2212D\u2217(\u00b5i,(cid:101)\u00b5i)z) we get\n\non the event \u2126(cid:48), where K1 is a universal constant. Finally, we de\ufb01ne the quantities \u03b4i = sup{\u0001 >\n0 : Ui(ft(\u0001)) \u2265 \u00b5i \u2200t \u2208 N}. Note that we have P(\u03b4i < \u03b3) = P(\u2203t \u2208 N : Ui(ft(\u03b3)) \u2265 \u00b5i) \u2264 \u03b3\n\nK1 log(cid:0)\u03b4\u22121 log D\u2217(\u00b5i,(cid:101)\u00b5i)\u22121(cid:1) + log \u03b4\u22121\n\nD\u2217(\u00b5i,(cid:101)\u00b5i)\n\nD\u2217(\u00b51,(cid:101)\u00b5)\n\n(cid:88)\n\ni\u22652\n\n+\n\n,\n\ni\n\nP(cid:16) log \u03b4\nD\u2217(\u00b5i,(cid:101)\u00b5i) \u2265 z\n\n\u22121\ni\n\n(cid:17) \u2264 exp(\u2212D\u2217(\u00b5i,(cid:101)\u00b5i)z) .\n\nHence {\u03b4i}i\u22652 are independent sub-exponential variables, which allows us to control their contribu-\ntion to the sum above using standard techniques.\n\n5 Real-World Crowdsourcing\n\nWe now compare the performance of lil-KLUCB to that of other algorithms in the literature. We do\nthis using both synthetic data and real data from the New Yorker Cartoon Caption contest [10]7. To\n\n7These data can be found at https://github.com/nextml/caption-contest-data\n\n7\n\n\fkeep comparisons fair, we run the same UCB algorithm for all the competing con\ufb01dence bounds.\nWe set N = 8 and \u03b4 = 0.01 in our experiments. The con\ufb01dence bounds are [KL]: the KL-bound\nderived based on Theorem 2, [SG1]: a matching sub-Gaussian bound derived using the proof of\nTheorem 2, using sub-Gaussian tails instead of the KL rate-function (the exact derivations are in the\nSupplementary Material), and [SG2]: the sharper sub-Gaussian bound provided by Theorem 8 of [7].\nWe compare these methods by computing the empirical probability that the best-arm is among the top\n5 empirically best arms, as a function of the total number of samples. We do so using using synthetic\ndata in Figure 5 , where the Bernoulli rewards simulate cases from Table 1, and using real human\nresponse data from two representative New Yorker caption contests in Figure 5.\n\nFigure 1: Probability of the best-arm in the top 5 empirically best arms, as a function of the number of samples,\nbased on 250 repetitions. \u00b5i = 1 \u2212 ((i \u2212 1)/n)\u03b1, with \u03b1 = 1 in the left panel, and \u03b1 = 1/2 in the right panel.\nThe mean-pro\ufb01le is shown above each plot. [KL] Blue; [SG1] Red; [SG2] Black.\n\nAs seen in Table 1, the KL con\ufb01dence bounds have the potential to greatly outperform the sub-\nGaussian ones. To illustrate this indeed translates into superior performance, we simulate two cases,\nwith means \u00b5i = 1 \u2212 ((i \u2212 1)/n)\u03b1, with \u03b1 = 1/2 and \u03b1 = 1, and n = 1000. As expected, the\nKL-based method requires signi\ufb01cantly fewer samples (about 20 % for \u03b1 = 1 and 30 % for \u03b1 = 1/2)\nto \ufb01nd the best arm. Furthermore, the arms with means below the median are sampled about 15 and\n25 % of the time respectively \u2013 key in crowdsourcing applications, since having participants answer\nfewer irrelevant (and potentially annoying) questions improves both ef\ufb01ciency and user experience.\n\nFigure 2: Probability of the best-arm in the top 5 empirically best arms vs. number of samples, based on 250\nbootstrapped repetitions. Data from New Yorker contest 558 (\u00b51 = 0.536) on left, and contest 512 (\u00b51 = 0.8)\non right. Mean-pro\ufb01le above each plot. [KL] Blue; [SG1] Red; [SG2] Black.\n\n8\n\n0.00.20.40.60.81.01.2P(best arm in top 5), alpha=1Number of samples (10 thousands)(Empirical) probability \u2212 250 trialsKaufmann lil\u2212UCBKL\u2212UCBSG lil\u2212UCB0.000.751.502.253.003.754.50llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll0.00.20.40.60.81.01.2P(best arm in top 5), alpha=1/2Number of samples (10 thousands)(Empirical) probability \u2212 250 trialsKaufmann lil\u2212UCBKL\u2212UCBSG lil\u2212UCB0.000.150.300.450.600.750.90llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll0.00.20.40.60.81.01.2P(best arm in top 5), Contest 558Number of samples (10 thousands)(Empirical) probability \u2212 250 trialsKaufmann lil\u2212UCBKL\u2212UCBSG lil\u2212UCB02468101214161820llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll0.00.20.40.60.81.01.2P(best arm in top 5), Contest 512Number of samples (10 thousands)(Empirical) probability \u2212 250 trialsKaufmann lil\u2212UCBKL\u2212UCBSG lil\u2212UCB0.000.751.502.253.003.754.50llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll\fTo see how these methods fair on real data, we also run these algorithms on bootstrapped human\nresponse data from the real New Yorker Caption Contest. The mean reward of the best arm in these\ncontests is usually between 0.5 and 0.85, hence we choose one contest from each end of this spectrum.\nAt the lower end of the spectrum, the three methods fair comparably. This is expected because the\nsub-Gaussian bounds are relatively good for means about 0.5. However, in cases where the top mean\nis signi\ufb01cantly larger than 0.5 we see a marked improvement in the KL-based algorithm.\n\nExtension to numerical experiments\n\nSince a large number of algorithms have been proposed in the literature for best arm identi\ufb01cation,\nwe include another algorithm in the numerical experiments for comparison.\nPreviously we compared lil-KLUCB to lil-UCB as a comparison for two reasons. First, this compari-\nson illustrates best the gains of using the novel anytime con\ufb01dence bounds as opposed to those using\nsub-Gaussian tails. Second, since lil-UCB is the state of the art algorithm, any other algorithm will\nlikely perform worse.\nThe authors of [6] compare a number of different best arm identi\ufb01cation methods, and conclude\nthat two of them seem to stand out: lil-UCB and Thompson sampling. Therefore, we now include\nThomspon sampling [Th] in our numerical experiments for the New Yorker data.\nWe implemented the method as prescribed in [6]. As can bee seen in Figure 5, Thompson sampling\nseems to perform somewhat worse than the previous methods in these two instances.\n\nFigure 3: Probability of the best-arm in the top 5 empirically best arms vs. number of samples, based on 250\nbootstrapped repetitions. Data from New Yorker contest 558 (\u00b51 = 0.536) on left, and contest 512 (\u00b51 = 0.8)\non right. Mean-pro\ufb01le above each plot. [KL] Blue; [SG1] Red; [SG2] Black; [Th] Purple.\n\n9\n\n0.00.20.40.60.81.01.2P(best arm in top 5), Contest 512Number of samples (10 thousands)(Empirical) probability \u2212 250 trialsKaufmann lil\u2212UCBKL\u2212UCBSG lil\u2212UCBThompson02468101214161820llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll0.00.20.40.60.81.01.2P(best arm in top 5), Contest 512Number of samples (10 thousands)(Empirical) probability \u2212 250 trialsKaufmann lil\u2212UCBKL\u2212UCBSG lil\u2212UCBThompson0.000.751.502.253.003.754.50llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll\fReferences\n[1] Jean-Yves Audibert and S\u00e9bastien Bubeck. Best arm identi\ufb01cation in multi-armed bandits. In COLT-23th\n\nConference on Learning Theory-2010, pages 13\u2013p, 2010.\n\n[2] St\u00e9phane Boucheron, G\u00e1bor Lugosi, and Pascal Massart. Concentration inequalities: A nonasymptotic\n\ntheory of independence. Oxford university press, 2013.\n\n[3] Olivier Capp\u00e9, Aur\u00e9lien Garivier, Odalric-Ambrym Maillard, R\u00e9mi Munos, Gilles Stoltz, et al. Kullback\u2013\nleibler upper con\ufb01dence bounds for optimal sequential allocation. The Annals of Statistics, 41(3):1516\u2013\n1541, 2013.\n\n[4] Aur\u00e9lien Garivier and Olivier Capp\u00e9. The kl-ucb algorithm for bounded stochastic bandits and beyond. In\n\nCOLT, pages 359\u2013376, 2011.\n\n[5] Kevin Jamieson, Matthew Malloy, Robert Nowak, and S\u00e9bastien Bubeck. lil-ucb: An optimal exploration\n\nalgorithm for multi-armed bandits. In Conference on Learning Theory, pages 423\u2013439, 2014.\n\n[6] Kevin G Jamieson, Lalit Jain, Chris Fernandez, Nicholas J Glattard, and Rob Nowak. Next: A system for\nreal-world development, evaluation, and application of active learning. In Advances in Neural Information\nProcessing Systems, pages 2656\u20132664, 2015.\n\n[7] Emilie Kaufmann, Olivier Capp\u00e9, and Aur\u00e9lien Garivier. On the complexity of best arm identi\ufb01cation in\n\nmulti-armed bandit models. The Journal of Machine Learning Research, 2016.\n\n[8] Emilie Kaufmann and Shivaram Kalyanakrishnan. Information complexity in bandit subset selection. In\n\nCOLT, pages 228\u2013251, 2013.\n\n[9] Odalric-Ambrym Maillard, R\u00e9mi Munos, Gilles Stoltz, et al. A \ufb01nite-time analysis of multi-armed bandits\n\nproblems with kullback-leibler divergences. In COLT, pages 497\u2013514, 2011.\n\n[10] B. Fox Rubin. How new yorker cartoons could teach computers to be funny. CNET News, 2016.\n\nhttps://www.cnet.com/news/how-new-yorker-cartoons-could-teach-computers-to-be-funny/.\n\n[11] Max Simchowitz, Kevin Jamieson, and Benjamin Recht. The simulator: Understanding adaptive sampling\n\nin the moderate-con\ufb01dence regime. arXiv preprint arXiv:1702.05186, 2017.\n\n10\n\n\f", "award": [], "sourceid": 3008, "authors": [{"given_name": "Ervin", "family_name": "Tanczos", "institution": "University of Wisconsin - Madison"}, {"given_name": "Robert", "family_name": "Nowak", "institution": "University of Wisconsion-Madison"}, {"given_name": "Bob", "family_name": "Mankoff", "institution": "Former Cartoon Editor of The New Yorker"}]}