{"title": "Secretary Ranking with Minimal Inversions", "book": "Advances in Neural Information Processing Systems", "page_first": 1051, "page_last": 1063, "abstract": "We study a secretary problem which captures the task of ranking in online settings. We term this problem the secretary ranking problem: elements from an ordered set arrive in random order and instead of picking the maximum element, the algorithm is asked to assign a rank, or position, to each of the elements. The rank assigned is irrevocable and is given knowing only the pairwise comparisons with elements previously arrived. The goal is to minimize the distance of the rank produced to the true rank of the elements measured by the Kendall-Tau distance, which corresponds to the number of pairs that are inverted with respect to the true order.\n\nOur main result is a matching upper and lower bound for the secretary ranking problem. We present an algorithm that ranks n elements with only O(n^{3/2}) inversions in expectation, and show that any algorithm necessarily suffers \\Omega(n^{3/2}) inversions when there are n available positions.  In terms of techniques, the analysis of our algorithm draws connections to linear probing in the hashing literature, while our lower bound result relies on a general anti-concentration bound for a generic balls and bins sampling process. We also consider the case where the number of positions m can be larger than the number of secretaries n and provide an improved bound by showing a  connection of this problem with random binary trees.", "full_text": "Secretary Ranking with Minimal Inversions\n\nSepehr Assadi\n\nRutgers University\n\nsepehr.assadi@rutgers.edu\n\nEric Balkanski\n\nHarvard University\n\nebalkans@gmail.com\n\nRenato Paes Leme\nGoogle Research\n\nrenatoppl@google.com\n\nAbstract\n\nWe study a secretary problem which captures the task of ranking in online settings.\nWe term this problem the secretary ranking problem: elements from an ordered set\narrive in random order and instead of picking the maximum element, the algorithm\nis asked to assign a rank, or position, to each of the elements. The rank assigned\nis irrevocable and is given knowing only the pairwise comparisons with elements\npreviously arrived. The goal is to minimize the distance of the rank produced to the\ntrue rank of the elements measured by the Kendall-Tau distance, which corresponds\nto the number of pairs that are inverted with respect to the true order.\nOur main result is a matching upper and lower bound for the secretary ranking\nproblem. We present an algorithm that ranks n elements with only O(n3/2) in-\nversions in expectation, and show that any algorithm necessarily suffers \u2126(n3/2)\ninversions when there are n available positions. In terms of techniques, the analysis\nof our algorithm draws connections to linear probing in the hashing literature, while\nour lower bound result relies on a general anti-concentration bound for a generic\nballs and bins sampling process. We also consider the case where the number of\npositions m can be larger than the number of secretaries n and provide an improved\nbound by showing a connection of this problem with random binary trees.\n\n1\n\nIntroduction\n\nThe secretary problem is one of the \ufb01rst problems studied in online algorithms\u2014in fact, it was\nextensively studied much before the \ufb01eld of online algorithms even existed. It \ufb01rst appeared in print\nin 1960 as a recreational problem in Martin Gardner\u2019s Mathematical Games column in Scienti\ufb01c\nAmerican. In the subsequent decade it caught the attention of many of the eminent probabilist\nresearchers like Lindley [Lin61], Dynkin [Dyn63], Chow et al. [CMRS64] and Gilbert and Mosteller\n[GM06] among others. In a very entertaining historical survey, Ferguson [Fer89] traces the origin of\nthe secretary problem to much earlier: Cayley in 1875 and Kepler in 1613 pose questions in the same\nspirit as the secretary problem.\nSecretary problem has been extended in numerous directions, see for example the surveys by\nSakaguchi [Sak95] and Freeman [Fre83]. The problem has had an enormous in\ufb02uence in computer\nscience and has provided some of the basic techniques in the \ufb01eld of online and approximation\nalgorithms. Babaioff et al extended this problem to matroid set systems [BIK07] and Knapsack\n[BIKK07] and perhaps more importantly, show that the secretary problem is a natural tool for\ndesigning online auctions. In the last decade, the secretary problem has also been extended to\nposets [KLVV11], submodular systems [BHZ10], general set systems [Rub16], stable matchings\n[BEF+17], non-uniform arrivals [KKN15] and applied to optimal data sampling [GD09], design of\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fprophet inequalities [AKW14, EHLM17], crowdsourcing systems [SM13], pricing in online settings\n[CEFJ14], online linear programming [AWY14] and online ad allocation [FHK+10].\nThe (admittedly incomplete) list of extensions and applications in the last paragraph serves to\nshowcase that the secretary problem has traditionally been a vehicle for deriving connections between\ndifferent sub\ufb01elds of computer science and a testbed of new techniques.\n\nRanking Secretaries. We consider a natural variant of the secretary problem that captures ranking\nfrom pairwise comparisons in online settings. In the secretary ranking problem, instead of selecting\nthe maximum element we are asked to rank each arriving element. In the process of deriving\nan optimal algorithm for this problem, we uncover novel connections between ranking and the\ntechnique of linear probing, which is one of the earliest techniques in the hashing literature studied\nby Knuth [Knu63], and also the expected height of random binary trees.\nIn the traditional secretary problem a decision maker is trying to hire a secretary. There is a total\norder over n secretaries and the goal of the algorithm is to hire the best secretary. The secretaries\nare assumed to arrive in a random order and the algorithm can only observe the relative rank of\neach secretary with respect to the previously interviewed ones. Once a secretary is interviewed,\nthe algorithm needs to decide whether to hire the current one or to irrevocably abandon the current\ncandidate and continue interviewing.\nIn our setting, there are m job positions and n secretaries. There is a known total order on positions.\nSecretaries arrive in random order and, as before, we can only compare a secretary with previously\ninterviewed ones. In our version, all secretaries will be hired and the decision of the algorithm is in\nwhich position to hire each secretary. Each position can be occupied by at most one secretary and\nhiring decisions are irrevocable. Ideally, the algorithm will hire the best secretary in the best position,\nthe second best secretary in the second best position and so on. The loss incurred by the algorithm\ncorresponds to the pairs that are incorrectly ordered, i.e., pairs where a better secretary is hired in a\nworse position.\nWe give two examples that illustrate scenarios where irrevocable ranking decisions occur online. The\n\ufb01rst is in the context of task assignments. For concreteness, consider a consulting \ufb01rm with teams of\ndifferent skill levels. Projects of different dif\ufb01culty arrive in an online fashion and when a project\narrives, the \ufb01rm needs to decide which team will execute. Of course, the most dif\ufb01cult projects should\ngo to the most skillful team. The second example is in the context of reward allocation. Consider a\nuniversity department that would like to assign the best scholarships available to the best students.\nHowever, scholarships arrive one at a time and the school needs to decide which student is assigned\nthat scholarship knowing only the relative quality of the scholarships arrived so far.\n\n1.1 Our Results and Techniques\n\nThe perhaps most natural case of the secretary ranking problem is when the numbers of positions\nand secretaries are the same, i.e. m = n, which we call the dense case. The trivial algorithm that\nassigns a random empty position for each arriving secretary incurs \u0398(n2) cost, since each pair of\nelements has probability 1/2 of being an inversion. On the other hand, \u2126(n) is a trivial lower bound\non the cost of any algorithm because nothing is known when the \ufb01rst element arrives. As such, there\nis a linear gap between the costs of the trivial upper and lower bounds for this secretary ranking\nproblem. Our main result is an asymptotically tight upper and lower bound on the loss incurred by\nthe algorithms for the secretary ranking problem.\nTheorem. There is an algorithm for the secretary ranking problem that computes a ranking with\nO(n3/2) inversions in expectation. Moreover, any algorithm for this problem makes \u2126(n3/2) inver-\nsions in expectation.\n\nThere are two challenges in designing an algorithm for secretary ranking. In earlier time steps, there\nare only a small number of comparisons observed and these do not contain suf\ufb01cient information\nto estimate the true rank of the arriving elements. In later time steps, we observe a large number\nof comparisons and using the randomness of elements arrival, the true rank of the elements can\nbe estimated well. However, the main dif\ufb01culty is that at these time steps many of the positions\nhave already been assigned to some element arrived earlier and are hence not available. The \ufb01rst\ninformation-theoretic challenge exacerbates this second issue. Previous bad placements might imply\n\n2\n\n\fthat all the desired positions are unavailable for the current element, causing a large cost even for an\nelement whose true rank is estimated accurately.\nThe algorithm needs to handle these two opposing challenges simultaneously. The main idea behind\nour algorithm is to estimate the rank of the current element using the observed comparisons and\nthen add some noise to these estimations to obtain additional randomness in the positions and avoid\npositively correlated mistakes. We then assign the current element to the closest empty position to\nthis noisy estimated rank. The main technical interest is in the analysis of this algorithm. We draw a\nconnection to the analysis of linear probing in the hashing literature [Knu63] to argue that under this\nextra noise, there often exists an empty position that is close to the estimated rank.\nFor the lower bound, we analyze the number of random pairwise comparisons needed to estimate\nthe rank of an element accurately. Such results are typically proven in the literature by using anti-\nconcentration inequalities. A main technical dif\ufb01culty is that most of the existing anti-concentration\ninequalities are for independent random variables while there is a correlation between the variables\nwe are considering. We prove, to the best of our knowledge, a new anti-concentration inequality for a\ngeneric balls in bins problem that involves correlated sampling.\n\n1.2 Related Work\n\nOur work is inserted in the vast line of literature on the secretary problem, which we brie\ufb02y discussed\nearlier. There has been a considerable amount of work on multiple-choice secretary problems\nwhere, instead of the single best element, multiple elements can be chosen as they arrive online\n[Kle05, BIKK07, BIK07, BHZ10, Rub16, KP09]. We note that in multiple-choice secretary problems,\nthe decision at arrival of an element is still binary, whereas in secretary ranking one of n positions\nmust be chosen. More closely related to our work is a paper of Babichenko et al. [BEF+17] where\nelements that arrive must also be assigned to a position. However, the objective is different and the\ngoal, which uses a game-theoretic notion of stable matching, is to maximize the number of elements\nthat are not in a blocking pair. Gobel et al. [GKT15] also studied an online appointment scheduling\nproblem in which the goal is to assign starting dates to a set of jobs arriving online. The objective\nhere is again different from the secretary ranking problem and is to minimize the total weight time of\nthe jobs.\nAnother related line of work in machine learning is the well-known problem of learning to rank that\nhas been extensively studied in recent years (e.g. [BSR+05, CQL+07, BRL07, XLW+08]). Two im-\nportant applications of this problem are search engines for document retrieval [L+09, RJ05, LXQ+07,\nCXL+06, XL07] and collaborative \ufb01ltering approaches to recommender systems [SLH10, SKB+12,\nLY08, WRdVR08]. There has been signi\ufb01cant interest recently in ranking from pairwise compar-\nisons [FRPU94, BFSC+13, CS15, SW17, JKSO16, HSRW16, DKMR14, BMW16, AAAK17]. To\nthe best of our knowledge, there has not been previous work on ranking from pairwise comparisons\nin an online setting.\nFinally, we also brie\ufb02y discuss hashing, since our main technique is related to linear probing. Linear\nprobing is a classic implementation of hash tables and was \ufb01rst analyzed theoretically by Knuth\nin 1963 [Knu63], in a report which is now regarded as the birth of algorithm analysis. Since then,\ndifferent variants of this problem mainly for hash functions with limited independence have been\nconsidered in the literature [SS90, PPR07, PT10]. Reviewing the vast literature on this subject is\nbeyond the scope of our paper and we refer the interested reader to these papers for more details.\nOrganization. The remainder of the paper is organized as follows. In Section 2 we formalize the\nsecretary ranking problem. In Section 3, we present and analyze our algorithm. Section 4 is devoted\nto showing the lower bound. We discuss the case where the number of positions m is different from\nthe number of elements n, as well as two natural extensions of the secretary ranking problem, in\nSection 5. Missing proofs and standard concentration bounds are postponed to the full version of the\npaper as well.\n\n2 Problem Setup\n\nIn the secretary ranking problem, there are n elements a1, . . . , an that arrive one at a time in an\nonline manner and in a uniformly random order. There is a total ordering among the elements, but\n\n3\n\n\fthe algorithm has only access to pairwise comparisons among the elements that have already arrived.\nIn other words, at time t, the algorithm only observes whether ai < aj for all i, j \u2264 t.\nThe secretary ranking problem generalizes the secretary problem in the following sense: in the\nsecretary problem, we are only interested in \ufb01nding the element with the highest rank. However, in\nthe secretary ranking problem, the goal is to assign a rank to every arrived element and construct a\ncomplete ranking of all elements. Similar to the secretary problem, we make the enabling assumption\nthat the order of elements arrival is uniformly random.1\nWe de\ufb01ne the rank function rk : {a1, . . . , an} \u2192 [n] as the true rank of the elements in the total\norder, i.e., ai < aj iff rk(ai) < rk(aj). Since the elements arrive uniformly at random, rk(\u00b7) is a\nrandom permutation. Upon arrival of an element at at time step t, the algorithm must, irrevocably,\nplace at in a position \u03c0(at) \u2208 [n] that is not yet occupied, in the sense that for at (cid:54)= as we must\nhave \u03c0(as) (cid:54)= \u03c0(at). Since the main goal of the algorithm is to place the elements as to re\ufb02ect the\ntrue rank as close as possible2, we refer to \u03c0(at) as the learned rank of at. The goal is to minimize\nthe number of pairwise mistakes induced by the learned ranking compared to the true ranking. A\npairwise mistake, or an inversion, is de\ufb01ned as a pair of elements ai, aj such that rk(ai) < rk(aj)\naccording to the true underlying ranking but \u03c0(ai) > \u03c0(aj) according to the learned ranking. We\nmeasure the cost of the algorithm in expectation over the randomness of both the arrival order of\nelements and the algorithm.\n\nMeasures of sortedness. We point out that the primary goal in the secretary ranking problem is\nto learn an ordering \u03c0 of the input elements which is as close as possible to their sorted order. As\nsuch, the cost suffered by an algorithm is given by a measure of sortedness of \u03c0 compared to the\ntrue ranking. There are various measures of sortedness studied in the literature depending on the\napplication. Our choice of using the number of inversions, also known as Kendall\u2019s tau measure, as\nthe cost of an algorithm is motivated by the importance of this measure and its close connection to\nother measures such as Spearman\u2019s footrule (see, e.g., Chapter 6B in [Dia88]).\nFor a mapping \u03c0 : [n] \u2192 [n], Kendall\u2019s tau K(\u03c0) measures the number of inversions in \u03c0, i.e.:\n\nK(\u03c0) := |{(i, j); (\u03c0(ai) \u2212 \u03c0(aj))(rk(ai) \u2212 rk(aj)) < 0}|.\n\n(cid:80)n\nAnother important measure of sortedness is Spearman\u2019s footrule F (\u03c0) given by: F (\u03c0) :=\ni=1 |rk(ai) \u2212 \u03c0(ai)| , which corresponds to the summation of distances between the true rank\nof each element and its current position. A celebrated result of Diaconis and Graham [DG77] shows\nthat these two measures are within a factor of two of each other, namely, K(\u03c0) \u2264 F (\u03c0) \u2264 2 \u00b7 K(\u03c0).\nWe refer to this inequality as the DG inequality throughout the paper. Thus, up to a factor of two, the\ngoals of minimizing the Kendall tau or Spearman\u2019s footrule distances are equivalent and, while the\nKendall tau distance is used in the formulation of the problem, we also use the Spearman\u2019s footrule\ndistance in the analysis.\n\n3 The Algorithm\n\nIn this section, we describe and analyze an algorithm for the secretary ranking problem. Our main\nalgorithmic result is the following theorem.\nTheorem 1. There exists an algorithm for the secretary ranking problem that incurs a cost of\nO(n\n\nn) in expectation.\n\n\u221a\n\nIn Section 4, we show that this cost incurred by the algorithm is asymptotically optimal.\n\n3.1 Description of the Algorithm\n\nThe general approach behind the algorithm in Theorem 1 is as follows.\n\n1It is straightforward to verify that when the ordering is adversarial, any algorithm incurs the trivial cost of\n\n\u2126(n2). For completeness, a proof is provided in the full version of the paper.\n\n2In other words, hire the better secretaries in better positions.\n\n4\n\n\fUpon the arrival of element at at time step t:\n\n1. Estimation step: Estimate the true rank of the arrived element at using the partial\n\ncomparisons seen so far.\n\n2. Assignment step: Find the nearest currently unassigned rank to this estimate and let\n\n\u03c0(at) be this position.\n\nWe now describe the algorithm in more details. A natural way to estimate the rank of the t-th element\nin the estimation step is to compute the rank of this element with respect to the previous t\u2212 1 elements\nseen so far and then scale this number to obtain an estimate of the rank of this element between 1 and\nn. However, for our analysis of the assignment step, we need to tweak this approach slightly: instead\nof simply rescaling and rounding, we add perturbation to the estimated rank and then round its value.\nThis gives a nice distribution of estimated ranks which is crucial for the analysis of the assignment\nstep. The assignment step then simply assigns a learned rank to the element as close as possible to its\nestimated rank. We formalize the algorithm in Algorithm 1.\n\nALGORITHM 1: Dense Ranking\n\n1 Input: a set R of n positions, denoted here by [n], and at most n online arrivals.\n2 for any time step t \u2208 [n] and element at do\n3\n4\n\nDe\ufb01ne rt := |{at(cid:48) | at(cid:48) < at and t(cid:48) < t}|.\nSample xt uniformly in the real interval [rt \u00b7 n\nSet the learned rank of at as \u03c0(at) = arg mini\u2208R\n\nt , (rt + 1) \u00b7 n\n\n(cid:12)(cid:12)(cid:12)i \u2212(cid:101)rk(at)\n\nt ] and choose (cid:101)rk(at) = (cid:100)xt(cid:101).\n(cid:12)(cid:12)(cid:12) and remove i from R.\n\n5\n6 end\n\nWe brie\ufb02y comment on the runtime of the algorithm. By using any self-balancing binary search\ntree\u2014such as a red-black tree or an AVL tree\u2014to store the ranking of the arrived elements as well as\nthe set R of available ranks separately, Algorithm 1 is implementable in O(log n) time for each step,\nso total O(n log n) worst-case time.\nWe also note some similarity between this algorithm and linear probing in hashing. Linear probing is\nan approach to resolving collisions in hashing where, when a key is hashed to a non-empty cell, the\nclosest neighboring cells are visited until an empty location is found for the key. The similarity is\napparent to our assignment step which \ufb01nds the nearest currently unassigned rank to the estimated\nrank of an element. The analysis of the assignment step follows similar ideas as the analysis for the\nlinear probing hashing scheme.\n\n3.2 The Analysis\n\nThe total number of inversions can be approximated within a factor of 2 by the Spearman\u2019s footrule.\nTherefore, we can write the cost of Algorithm 1 (up to a factor 2) as follows:\n\nn(cid:88)\n\n|rk(at) \u2212 \u03c0(at)| \u2264 n(cid:88)\n\nt=1\n\nt=1\n\n(cid:12)(cid:12)(cid:12)rk(at) \u2212(cid:101)rk(at)\n\n(cid:12)(cid:12)(cid:12) +\n\n(cid:12)(cid:12)(cid:12)(cid:101)rk(at) \u2212 \u03c0(at)\n(cid:12)(cid:12)(cid:12) .\n\nn(cid:88)\n\nt=1\n\nThis basically breaks the cost of the algorithm in two parts: one is the cost incurred by the estimation\nstep and the other one is the cost of the assignment step. Our analysis then consists of two main parts\nwhere each part bounds one of the terms in the RHS above. In particular, we \ufb01rst prove that given the\npartial comparisons seen so far, we can obtain a relatively good estimation to the rank of the arrived\nelement, and then in the second part, we show that we can typically \ufb01nd an unassigned position in the\nclose proximity of this estimated rank to assign to it. The following two lemmas capture each part\nseparately. In both lemmas, the randomness in the expectation is taken over the random arrivals and\nthe internal randomness of the algorithm:\n\nLemma 3.1 (Estimation Cost). In Algorithm 1, E(cid:104)(cid:80)n\nLemma 3.2 (Assignment Cost). In Algorithm 1, E(cid:104)(cid:80)n\n\nt=1\n\n(cid:12)(cid:12)(cid:12)(cid:105)\n(cid:12)(cid:12)(cid:12)rk(at) \u2212(cid:101)rk(at)\n(cid:12)(cid:12)(cid:12)(cid:105)\n(cid:12)(cid:12)(cid:12)(cid:101)rk(at) \u2212 \u03c0(at)\n\nt=1\n\n= O(n\n\n\u221a\nn).\n\u221a\n\n= O(n\n\nn).\n\n5\n\n\fTheorem 1 then follows immediately from these two lemmas and Eq (3.2). The main part of the\nargument is the analysis of the assignment cost, i.e., Lemma 3.2, and in particular its connection to\nlinear probing. The analysis for the estimation cost, i.e., Lemma 3.1, follows from standard Chernoff\nbound arguments and is deferred to the full version of the paper.\n\nAssignment cost: proof of Lemma 3.2.\nIt is useful to think of sampling a random permutation\nin the following recursive way: given a random permutation over t \u2212 1 elements, it is possible to\nobtain a random permutation over t elements by inserting the t-th element in a uniformly random\nposition between these t \u2212 1 elements. Formally, given \u03c3 : [t \u2212 1] \u2192 [t \u2212 1], if we sample a position\ni uniformly from [t] and generate permutation \u03c3(cid:48) : [t] \u2192 [t] such that:\n\n\uf8f1\uf8f4\uf8f2\uf8f4\uf8f3 i\n\n\u03c3(cid:48)(t(cid:48)) =\n\n\u03c3(t(cid:48))\n\u03c3(t(cid:48)) + 1\n\nif t(cid:48) = t\nif t(cid:48) < t and \u03c3(cid:48)(t(cid:48)) < i\nif t(cid:48) < t and \u03c3(cid:48)(t(cid:48)) > i\n\nthen \u03c3(cid:48) will be a random permutation over t elements. It is simple to see that just by \ufb01xing any\npermutation and computing the probability of it being generated by this process.\nThinking about sampling the permutation in this way is very convenient for this analysis since at the\nt-th step of the process, the relative order of the \ufb01rst t elements is \ufb01xed (even though the true ranks\ncan only be determined in the end). In that spirit, let us also de\ufb01ne for a permutation \u03c3 : [t] \u2192 [t] the\nevent O\u03c3 that \u03c3 is the relative ordering of the \ufb01rst t elements:\n\nO\u03c3 = {a\u03c3(1) < a\u03c3(2) < . . . < a\u03c3(t)}.\n\nThe following proposition asserts that the randomness of the arrival and the inner randomness of the\nalgorithm, ensures that the estimated ranks at each time step are chosen uniformly at random from all\npossible ranks in [n].\n\nProposition 3.3. The values of (cid:101)rk(a1), . . . ,(cid:101)rk(an) are i.i.d and uniformly chosen from [n].\n\nProof. First let us show that for any \ufb01xed permutation \u03c3 over t \u2212 1 elements, the relative rank rt\nde\ufb01ned in the algorithm is conditionally uniformly distributed in {0, . . . , t \u2212 1}. In other words:\n\nPr[rt = i | O\u03c3] =\n\n1\nt\n\n,\n\n\u2200i \u2208 {0, . . . , t \u2212 1}.\n\nSimply observe that there are exactly t permutations over t elements such that the permutation\ninduced in the \ufb01rst t \u2212 1 elements is \u03c3. Since we are sampling a random permutation in this process,\neach of these permutation are equally likely to happen. Moreover, since each permutation corresponds\nto inserting the t-the element in one of the t positions, we obtain the above equality.\nFurthermore, since the probability of each value of rt does not depend on the induced permutation\n\u03c3 over the \ufb01rst t \u2212 1 elements, then rt is independent of \u03c3. Since all the previous values rt(cid:48) are\ncompletely determined by \u03c3, rt is independent of all previous rt(cid:48) for t(cid:48) < t.\nFinally observe that if rt is random from {0, ..., t \u2212 1}, then xt is sampled at random from [0, n], so\n\nfar from the estimated rank we have to go in the assignment step, before we are able to assign a\nlearned rank to this element. This part of our analysis will be similar to the analysis of the linear\n\n(cid:101)rk(at) is sampled at random from [n]. Since for different values of t \u2208 [n], all rt are independent, all\nthe values of (cid:101)rk(at) are also independent.\nNow that we established that (cid:101)rk(at) are independent and uniform, our next task is to bound how\nprobing hashing scheme. If we are forced to let the learned rank of at be far away from (cid:101)rk(at),\n(cid:12)(cid:12)(cid:12)(cid:101)rk(at) \u2212 \u03c0(at)\n(cid:12)(cid:12)(cid:12) > k, then this necessarily means that all positions in the integer interval\n[(cid:101)rk(at) \u2212 k : (cid:101)rk(at) + k] must have already been assigned as a learned rank of some element. In the\nat(cid:48) among the t \u2212 1 elements that appear before the t-th element have estimated rank (cid:101)rk(at(cid:48)) \u2208 I.\n\nfollowing, we bound the probability of such an event happening for large values of k compared to the\ncurrent time step t.\nWe say that the integer interval I = [i : i + s \u2212 1] of size s is popular at time t, iff at least s elements\n\nsay\n\n6\n\n\f1,\n\nn\u2212t\n\n(cid:16)\n\n(cid:16)\n\n1,\n\nn\u2212t\n\n1,\n\nn\u2212t\n\n).\n\n(cid:17)2(cid:17)\n\n(cid:17)2(cid:17)\n\n(cid:16) t\n\n(cid:16) t\n\nis the key part of the argument.\n\nis popular at time t w.p. e\u2212\u2126(\u03b1).\n\n(cid:16) t\n(cid:17)2(cid:17)\n(cid:12)(cid:12)(cid:12) = O(max\n\nSince by Proposition 3.3 every element has probability s/n of having estimated rank in I and the\nestimated ranks are independent, we can bound the probability that I is popular using a standard\napplication of Chernoff bound (proof deferred to the full version of the paper).\nClaim 3.4. Let \u03b1 \u2265 1, an interval of size s \u2265 2\u03b1 max\n\n(cid:16)\nWe now use the above claim to bound the deviation between (cid:101)rk(at) and \u03c0(at). The following lemma\nLemma 3.5. For any t \u2264 n, we have E(cid:12)(cid:12)(cid:12)(cid:101)rk(at) \u2212 \u03c0(at)\nof length at least 2k\u03b1 which contains (cid:101)rk(at) and is popular.\nLet us prove the above claim then. Let I be the shortest integer interval [a : b] which contains (cid:101)rk(at)\n(cid:12)(cid:12)(cid:12) to be at least k\u03b1, the length of interval I needs\n\nProof. Fix any \u03b1 \u2265 1. We claim that, if the learned rank of at is a position which has distance at\nleast k\u03b1 = 4\u03b1 \u00b7 max\nfrom its estimated rank, then necessarily there exists an interval I\n\n(cid:12)(cid:12)(cid:12)(cid:101)rk(at) \u2212 \u03c0(at)\n\n(cid:90) \u221e\n(cid:90) \u221e\n\nand moreover both positions a and b are not assigned to a learned rank by time t (by this de\ufb01nition,\n\u03c0(at) would be either a or b). For\nto be at least 2k\u03b1. But for I to have length at least 2k\u03b1, we should have at least 2k elements from\na1, . . . , at\u22121 to have an estimated rank in I: this is simply because a and b are not yet assigned a\nrank by time t and hence any element at(cid:48) which has estimated rank outside the interval I is never\nassigned a learned rank inside I (otherwise the assignment step should pick a or b, a contradiction).\nWe are now ready to \ufb01nalize the proof. It is straightforward that in the above argument, it suf\ufb01ces to\nBy the above argument and Claim 3.4, for any \u03b1 \u2265 1, we have,\n\nonly consider the integer intervals [(cid:101)rk(at) \u2212 k\u03b1 : (cid:101)rk(at) + k\u03b1] parametrized by the choice of \u03b1 \u2265 1.\nE(cid:104)(cid:12)(cid:12)(cid:12)(cid:101)rk(at) \u2212 \u03c0(at)\n(cid:12)(cid:12)(cid:12)(cid:105) \u2264\n(cid:17) \u00b7 k\u03b1 \u00b7 d\u03b1\n\n(cid:16)(cid:12)(cid:12)(cid:12)(cid:101)rk(at) \u2212 \u03c0(at)\n(cid:12)(cid:12)(cid:12) > k\u03b1\n(cid:17) \u00b7 k\u03b1 \u00b7 d\u03b1\n(cid:16)\nInteger interval [(cid:101)rk(at) \u2212 k\u03b1 : (cid:101)rk(at) + k\u03b1] is popular\n(cid:90) \u221e\n(cid:17)2(cid:17)\n(cid:16) t\n(cid:16)\n(cid:16)\n(cid:16) t\n(cid:17)2(cid:17)\n(cid:12)(cid:12)(cid:12)(cid:101)rk(at) \u2212 \u03c0(at)\nE(cid:104)(cid:12)(cid:12)(cid:12)(cid:101)rk(at) \u2212 \u03c0(at)\n(cid:12)(cid:12)(cid:12)(cid:105)\nProof of Lemma 3.2. We have, E(cid:104)(cid:80)n\nin this case, we have E(cid:104)(cid:12)(cid:12)(cid:12)(cid:101)rk(at) \u2212 \u03c0(at)\n(cid:12)(cid:12)(cid:12)(cid:105)\nE(cid:104)(cid:12)(cid:12)(cid:12)(cid:101)rk(at) \u2212 \u03c0(at)\n(cid:12)(cid:12)(cid:12)(cid:105) \u2264 n which is trivially true and hence the total contribution of these\nterms to the above summation is only O(n). Also, when t > n \u2212 \u221a\n\u221a\nE(cid:104)(cid:12)(cid:12)(cid:12)(cid:101)rk(at) \u2212 \u03c0(at)\n(cid:12)(cid:12)(cid:12)(cid:105) \u2264 O(1) \u00b7(cid:80)n\u2212\u221a\nn). It remains to bound the total contribution of t \u2208 [n/2, n \u2212 \u221a\nLemma 3.5,(cid:80)n\u2212\u221a\n\nby linear-\nity of expectation. For any t < n/2, the maximum term in RHS of Lemma 3.5 is 1 and hence\n= O(1). Thus, the contribution of the \ufb01rst n/2 \u2212 1\nn, we can simply write\nn sum-\nn]. By\nn), where the equal-\n\n\u221a\nmands is also O(n\nn\nt=n/2\n\nn\nt=n/2\nity is by a simple calculation (see full version of the paper).\n\nWe are now ready to \ufb01nalize the proof of Lemma 3.2.\n\ne\u2212O(\u03b1) \u00b7 \u03b1 \u00b7 d\u03b1\n\n=(cid:80)n\n\n(cid:16) t\n\n= O(max\n\n1,\n\nn\u2212t\n\n= O(n\n\n\u03b1=0\n\n\u03b1=0\n\nn \u2212 t\n\n).\n\n\u2264\n\n\u2264\n\nClaim 3.4\n\nO(max\n\n1,\n\nn \u2212 t\n\n) \u00b7\n\n\u03b1=0\n\nt=1\n\nt=1\n\n(cid:17)2\n\n\u221a\n\nPr\n\nPr\n\n(cid:12)(cid:12)(cid:12)(cid:105)\n\n4 A Tight Lower Bound\n\nWe complement the algorithmic result from the previous section by showing that the cost incurred by\nthe algorithm is asymptotically optimal.\nTheorem 2. Any algorithm for the secretary ranking problem incurs \u2126(n\n\nn) cost in expectation.\n\n\u221a\n\n7\n\n\f\u221a\n\nt\n\nk\n\nt\n\nTo prove Theorem 2, we \ufb01rst show that no deterministic algorithm can achieve better than \u2126(n\nn)\ninversions and then use Yao\u2019s minimax principle to extend the lower bound to randomized algorithms\n(by simply \ufb01xing the randomness of the algorithm to obtain a deterministic one with the same\nperformance over the particular distribution of the input).\nThe main ingredient of our proof of Theorem 2 is an anti-concentration bound for sampling without\nreplacement which we cast as a balls in bins problem. We start by describing this balls in bin problem\nand prove the anti-concentration bound in Lemma 4.1. Lemma 4.2 then connects the problem of\nonline ranking to the balls in bins problem.\nTo continue, we introduce some asymptotic notation that is helpful for readability. We write v =\n\u03981(n) if variable v is linear in n, but also smaller and bounded away from n, i.e., v = cn for some\nconstant c such that 0 < c < 1.\nLemma 4.1. Assume there are n balls in a bin, r of which are red and the remaining n \u2212 r are blue.\nSuppose t < min(r, n \u2212 r) balls are drawn from the bin uniformly at random without replacement,\nand let Ek,t,r,n be the event that k out of those t balls are red. Then, if r = \u03981(n) and t = \u03981(n),\n\u221a\nfor every k \u2208 {0, . . . , t}: Pr (Ek,t,r,n) = O (1/\nOur high level approach toward proving Lemma 4.1 is as follows:\n\n(cid:1)/(cid:0)n\n(cid:1).\n1. We \ufb01rst use a counting argument to show that Pr (Ek,t,r,n) =(cid:0)r\n(cid:1)(cid:0)n\u2212r\n(cid:1)(cid:0)n\u2212r\n2. We then use Stirling\u2019s approximation to show(cid:0)r\n(cid:1)/(cid:0)n\n(cid:1) = O(n\u22121/2) for k = (cid:98) tr\n(cid:0)r\n(cid:1)(cid:0)n\u2212r\n(cid:1)/(cid:0)n\n(cid:1) \u2248 tr\n(cid:1) for k\u2217 \u2248\n(cid:1)/(cid:0)n\n(cid:1) \u2264(cid:0) r\n(cid:1)/(cid:0)n\n(cid:1)(cid:0) n\u2212r\n(cid:0)r\n(cid:1)(cid:0)n\u2212r\n\n3. Finally, with a max. likelihood argument, we show that arg maxk\u2208[n]\n\nn (cid:99).\nn .\n\nt\u2212k\n\nt\u2212k\n\nt\u2212k\n\nn) .\n\nk\n\nt\n\nt\n\nk\n\nk\u2217\n\nt\u2212k\n\nt\u2212k\u2217\n\nBy combining these, we have, Pr (Ek,t,r,n) \u2264 maxk\u2208[n]\nn (by the third step), which we bounded by O(n\u22121/2) (in the second step). The actual proof is\ntr\nhowever rather technical and is postponed to the full version of the paper.\n\u221a\nn) of\nThe next lemma shows that upon arrival of at, any position has probability at least O (1/\nbeing the correct rank for at, under some mild conditions. The proof of this lemma uses the previous\nanti-concentration bound for sampling without replacement by considering the elements smaller than\nat to be the red balls and the elements larger than at to be the blue balls. For at to have rank r and be\nthe kth element in the ranking so far, the \ufb01rst t \u2212 1 elements previously observed must contain k \u2212 1\nred balls out of the r \u2212 1 red balls and t \u2212 k blue balls out of the n \u2212 r blue balls.\nLemma 4.2. Fix any permutation \u03c3 of [t] and let O\u03c3 denote the event that a\u03c3(1) < a\u03c3(2) < . . . <\n\u221a\na\u03c3(t). If \u03c3(k) = t, k = \u03981(t) and t = \u03981(n) then for any r: Pr (rk(at) = r | O\u03c3) = O (1/\nProof. De\ufb01ne Ek as the event that \u201cat is the k-th smallest element in a1, . . . , at\u201d. We \ufb01rst have,\nPr (rk(at) = r | O\u03c3) = Pr (rk(at) = r | Ek) . This is simply because rk(at) is only a function of the\npairwise comparisons of at with other elements and does not depend on the ordering of the remaining\nelements between themselves. Moreover,\n\nn) .\n\nk\n\nt\n\nPr (rk(at) = r | Ek) = Pr (Ek | rk(at) = r) \u00b7 Pr (rk(at) = r)\n\nPr (Ek)\n\n= Pr (Ek | rk(at) = r) \u00b7 t\nn\n\nsince at is randomly partitioned across the [n] elements. Notice now that conditioned on rk(at) = r,\nthe event Ek is exactly the event Ek\u22121,t\u22121,r\u22121,n\u22121 in the sampling without replacement process\nde\ufb01ned in Lemma 4.1. The n \u2212 1 balls are all the elements but at, the r \u2212 1 red balls correspond\nto elements smaller than at, the n \u2212 r blue balls to elements larger than at, and t \u2212 1 balls drawn\nare the elements arrived before at. Finally, observe that Pr (r < k|Ek) = 0, so for r < k, the\nbound holds trivially. In the remaining cases, r = \u03981(n) and we use the bound in Lemma 4.1 with\nt/n = \u0398(1).\n\nUsing the previous lemma, we can lower bound the cost due to the t-th element. Fix any deterministic\nalgorithm A for the online ranking problem. Recall that \u03c0(at) denotes the learned rank of the item\nat arriving in the t-th time step. For any time step t \u2208 [n], we use costA(t) to denote the cost\nincurred by the algorithm A in positioning the item at. More formally, if rk(at) = i, we have\n\ncostA(t) :=(cid:12)(cid:12)i \u2212 \u03c0(a(t))(cid:12)(cid:12). Theorem 2 then follows by Yao\u2019s minimax principle principle and the\n\nfollowing lemma, whose proof appears in the appendix\nLemma 4.3. Fix any deterministic algorithm A. For any t = \u03981(n), E [costA(t)] = \u2126 (\n\n\u221a\n\nn).\n\n8\n\n\f5 Additional Results and Extensions\n\nIn the full version of the paper, we study two additional cases of the secretary ranking problem.\n\n\u2022 In the sparse case, we wish to compute how large the number m of positions needs to be such\nthat we incur no inversions. Clearly for m = 2n+1 \u2212 1 it is possible to obtain zero inversions\nwith probability 1 and for any number less than that it is also clear that any algorithm needs\nto cause inversions with non-zero probability. If we only want to achieve zero inversions\nwith high probability, how large does m need to be? By showing a connection between\nthe secretary problem and random binary trees, we show that for m \u2265 n\u03b1 for \u03b1 \u2248 2.998\nit is possible to design an algorithm that achieves zero inversion with probability 1 \u2212 o(1).\nThe constant \u03b1 here is obtained using the high probability bound on the height of a random\nbinary tree of n elements.\n\u2022 We combine the algorithms for the dense and sparse cases to obtain a general algorithm\nwith a bound on the expected number of inversions which smoothly interpolates between\nthe bounds obtained for the dense and sparse cases. This algorithm starts by running the\nalgorithm for the sparse case and when two elements are placed very closed to each other by\nthe sparse algorithm, we switch to use the algorithm for the dense case to assign a position\nto remaining elements with rank between these two close elements.\n\nWe discuss two extensions of the secretary ranking problem that we will include in the full version of\nthe paper.\n\n\u2022 Number of comparisons. A \ufb01rst extension is to minimize the number of pairwise compar-\nisons between elements needed to obtain a small number of inversions. It is possible to\nobtain O(n3/2) inversions with only O(n log n) comparisons, which is optimal up to lower\norder terms. Our algorithm maintains an ordering of the elements seen so far. Thus, when a\nnew element arrives, the algorithm can \ufb01nd the position of this new element in the current\nordering by binary search.\n\u2022 Noisy comparisons. Our main result extends to a noisy model where each comparison is,\nindependently, correct with probability 1 \u2212 \u0001 and wrong with probability \u0001. The impact of\nthe noise in the analysis is an additional \u0001n additive term per element in the estimation cost\nand no additional term for the assignment cost. The total cost is then O(n3/2 + \u0001n2).\nThe reason this can be done is that the analysis is modular and decomposes the error in an\nestimation and assignment terms. If we estimate the new rank using line 3 of Algorithm 1\nbut with noisy comparisons, we still have an unbiased estimator of the rank (but with larger\nvariance) and Proposition 3.3 still holds. Those are the only requirements for the analysis of\nthe assignment cost, so it remains unchanged.\n\nIf we have noisy comparisons and also want to improve the number of comparisons, the technique in\n(a) for reducing the number of comparisons no longer holds, but we can still reduce the number of\ncomparisons by estimating \u02dcr in line 3 of Algorithm 1 by sampling only a few of the previous elements\nand paying the corresponding cost in increased variance. The same analysis as in (b) still holds.\n\n9\n\n\fReferences\n[AAAK17] Arpit Agarwal, Shivani Agarwal, Sepehr Assadi, and Sanjeev Khanna. Learning with\nlimited rounds of adaptivity: Coin tossing, multi-armed bandits, and ranking from\npairwise comparisons. In Proceedings of the 30th Conference on Learning Theory,\nCOLT 2017, Amsterdam, The Netherlands, 7-10 July 2017, pages 39\u201375, 2017.\n\n[AKW14] Pablo D Azar, Robert Kleinberg, and S Matthew Weinberg. Prophet inequalities with\nlimited information. In Proceedings of the twenty-\ufb01fth annual ACM-SIAM sympo-\nsium on Discrete algorithms, pages 1358\u20131377. Society for Industrial and Applied\nMathematics, 2014.\n\n[AWY14] Shipra Agrawal, Zizhuo Wang, and Yinyu Ye. A dynamic near-optimal algorithm for\n\nonline linear programming. Operations Research, 62(4):876\u2013890, 2014.\n\n[BEF+17] Yakov Babichenko, Yuval Emek, Michal Feldman, Boaz Patt-Shamir, Ron Peretz, and\nRann Smorodinsky. Stable secretaries. In Proceedings of the 2017 ACM Conference\non Economics and Computation, EC \u201917, Cambridge, MA, USA, June 26-30, 2017,\npages 243\u2013244, 2017.\n\n[BFSC+13] R\u00f3bert Busa-Fekete, Balazs Szorenyi, Weiwei Cheng, Paul Weng, and Eyke H\u00fcller-\nmeier. Top-k selection based on adaptive sampling of noisy preferences. In Interna-\ntional Conference on Machine Learning, pages 1094\u20131102, 2013.\n\n[BHZ10] MohammadHossein Bateni, MohammadTaghi Hajiaghayi, and Morteza Zadimoghad-\ndam. Submodular secretary problem and extensions. In Approximation, Randomization,\nand Combinatorial Optimization. Algorithms and Techniques, pages 39\u201352. Springer,\n2010.\n\n[BIK07] Moshe Babaioff, Nicole Immorlica, and Robert Kleinberg. Matroids, secretary prob-\nlems, and online mechanisms. In Proceedings of the eighteenth annual ACM-SIAM\nsymposium on Discrete algorithms, pages 434\u2013443. Society for Industrial and Applied\nMathematics, 2007.\n\n[BIKK07] Moshe Babaioff, Nicole Immorlica, David Kempe, and Robert Kleinberg. A knap-\nIn Approximation, randomization, and\nsack secretary problem with applications.\ncombinatorial optimization. Algorithms and techniques, pages 16\u201328. Springer, 2007.\n\n[BMW16] Mark Braverman, Jieming Mao, and S Matthew Weinberg. Parallel algorithms for\nselect and partition with noisy comparisons. In Proceedings of the forty-eighth annual\nACM symposium on Theory of Computing, pages 851\u2013862. ACM, 2016.\n\n[BRL07] Christopher J Burges, Robert Ragno, and Quoc V Le. Learning to rank with nonsmooth\ncost functions. In Advances in neural information processing systems, pages 193\u2013200,\n2007.\n\n[BSR+05] Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton,\nand Greg Hullender. Learning to rank using gradient descent. In Proceedings of the\n22nd international conference on Machine learning, pages 89\u201396. ACM, 2005.\n\n[CEFJ14] Ilan Reuven Cohen, Alon Eden, Amos Fiat, and \u0141ukasz Je\u02d9z. Pricing online decisions:\nBeyond auctions. In Proceedings of the twenty-sixth annual ACM-SIAM symposium on\ndiscrete algorithms, pages 73\u201391. SIAM, 2014.\n\n[CMRS64] YS Chow, Sigaiti Moriguti, Herbert Robbins, and SM Samuels. Optimal selection\nIsrael Journal of mathematics,\n\nbased on relative rank (the \u201csecretary problem\u201d).\n2(2):81\u201390, 1964.\n\n[CQL+07] Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. Learning to rank:\nfrom pairwise approach to listwise approach. In Proceedings of the 24th international\nconference on Machine learning, pages 129\u2013136. ACM, 2007.\n\n[CS15] Yuxin Chen and Changho Suh. Spectral mle: Top-k rank aggregation from pairwise\ncomparisons. In International Conference on Machine Learning, pages 371\u2013380, 2015.\n\n10\n\n\f[CXL+06] Yunbo Cao, Jun Xu, Tie-Yan Liu, Hang Li, Yalou Huang, and Hsiao-Wuen Hon.\nIn Proceedings of the 29th annual\nAdapting ranking svm to document retrieval.\ninternational ACM SIGIR conference on Research and development in information\nretrieval, pages 186\u2013193. ACM, 2006.\n\n[DG77] Persi Diaconis and Ronald L Graham. Spearman\u2019s footrule as a measure of disarray.\nJournal of the Royal Statistical Society. Series B (Methodological), pages 262\u2013268,\n1977.\n\n[Dia88] Persi Diaconis. Group representations in probability and statistics. Lecture Notes-\n\nMonograph Series, 11:i\u2013192, 1988.\n\n[DKMR14] Susan Davidson, Sanjeev Khanna, Tova Milo, and Sudeepa Roy. Top-k and clustering\nwith noisy comparisons. ACM Transactions on Database Systems (TODS), 39(4):35,\n2014.\n\n[Dyn63] Eugene B Dynkin. The optimum choice of the instant for stopping a markov process.\n\nSoviet Mathematics, 4:627\u2013629, 1963.\n\n[EHLM17] Hossein Esfandiari, MohammadTaghi Hajiaghayi, Vahid Liaghat, and Morteza Mone-\nmizadeh. Prophet secretary. SIAM Journal on Discrete Mathematics, 31(3):1685\u20131701,\n2017.\n\n[Fer89] Thomas S Ferguson. Who solved the secretary problem? Statistical science, pages\n\n282\u2013289, 1989.\n\n[FHK+10] Jon Feldman, Monika Henzinger, Nitish Korula, Vahab S Mirrokni, and Cliff Stein.\nOnline stochastic packing applied to display ad allocation. In European Symposium on\nAlgorithms, pages 182\u2013194. Springer, 2010.\n\n[Fre83] PR Freeman. The secretary problem and its extensions: A review. International\n\nStatistical Review/Revue Internationale de Statistique, pages 189\u2013206, 1983.\n\n[FRPU94] Uriel Feige, Prabhakar Raghavan, David Peleg, and Eli Upfal. Computing with noisy\n\ninformation. SIAM Journal on Computing, 23(5):1001\u20131018, 1994.\n\n[GD09] Yogesh Girdhar and Gregory Dudek. Optimal online data sampling or how to hire the\nbest secretaries. In 2009 Canadian Conference on Computer and Robot Vision, pages\n292\u2013298. IEEE, 2009.\n\n[GKT15] Oliver G\u00f6bel, Thomas Kesselheim, and Andreas T\u00f6nnis. Online appointment schedul-\ning in the random order model. In Algorithms - ESA 2015 - 23rd Annual European\nSymposium, Patras, Greece, September 14-16, 2015, Proceedings, pages 680\u2013692,\n2015.\n\n[GM06] John P Gilbert and Frederick Mosteller. Recognizing the maximum of a sequence. In\n\nSelected Papers of Frederick Mosteller, pages 355\u2013398. Springer, 2006.\n\n[HSRW16] Reinhard Heckel, Nihar B Shah, Kannan Ramchandran, and Martin J Wainwright.\nActive ranking from pairwise comparisons and when parametric assumptions don\u2019t\nhelp. arXiv preprint arXiv:1606.08842, 2016.\n\n[JKSO16] Minje Jang, Sunghyun Kim, Changho Suh, and Sewoong Oh. Top-k ranking from pair-\nwise comparisons: When spectral ranking is optimal. arXiv preprint arXiv:1603.04153,\n2016.\n\n[KKN15] Thomas Kesselheim, Robert Kleinberg, and Rad Niazadeh. Secretary problems with\nnon-uniform arrival order. In Proceedings of the forty-seventh annual ACM symposium\non Theory of computing, pages 879\u2013888. ACM, 2015.\n\n[Kle05] Robert Kleinberg. A multiple-choice secretary algorithm with applications to online\nauctions. In Proceedings of the sixteenth annual ACM-SIAM symposium on Discrete\nalgorithms, pages 630\u2013631. Society for Industrial and Applied Mathematics, 2005.\n\n11\n\n\f[KLVV11] Ravi Kumar, Silvio Lattanzi, Sergei Vassilvitskii, and Andrea Vattani. Hiring a secretary\nfrom a poset. In Proceedings of the 12th ACM conference on Electronic commerce,\npages 39\u201348. ACM, 2011.\n\n[Knu63] Don Knuth. Notes on \u201copen\u201d addressing. Citeseer, 1963.\n\n[KP09] Nitish Korula and Martin P\u00e1l. Algorithms for secretary problems on graphs and hyper-\ngraphs. In Automata, Languages and Programming, 36th Internatilonal Colloquium,\nICALP 2009, Rhodes, Greece, July 5-12, 2009, Proceedings, Part II, pages 508\u2013520,\n2009.\n\n[L+09] Tie-Yan Liu et al. Learning to rank for information retrieval. Foundations and Trends R(cid:13)\n\nin Information Retrieval, 3(3):225\u2013331, 2009.\n\n[Lin61] Denis V Lindley. Dynamic programming and decision theory. Applied Statistics,\n\n10(1):39\u201351, 1961.\n\n[LXQ+07] Tie-Yan Liu, Jun Xu, Tao Qin, Wenying Xiong, and Hang Li. Letor: Benchmark\ndataset for research on learning to rank for information retrieval. In Proceedings of\nSIGIR 2007 workshop on learning to rank for information retrieval, volume 310. ACM\nAmsterdam, The Netherlands, 2007.\n\n[LY08] Nathan N Liu and Qiang Yang. Eigenrank: a ranking-oriented approach to collaborative\n\ufb01ltering. In Proceedings of the 31st annual international ACM SIGIR conference on\nResearch and development in information retrieval, pages 83\u201390. ACM, 2008.\n\n[PPR07] Anna Pagh, Rasmus Pagh, and Milan Ruzic. Linear probing with constant inde-\npendence. In Proceedings of the thirty-ninth annual ACM symposium on Theory of\ncomputing, pages 318\u2013327. ACM, 2007.\n\n[PT10] Mihai Patra\u00b8scu and Mikkel Thorup. On the k-independence required by linear probing\nand minwise independence. In International Colloquium on Automata, Languages,\nand Programming, pages 715\u2013726. Springer, 2010.\n\n[RJ05] Filip Radlinski and Thorsten Joachims. Query chains: learning to rank from implicit\nfeedback. In Proceedings of the eleventh ACM SIGKDD international conference on\nKnowledge discovery in data mining, pages 239\u2013248. ACM, 2005.\n\n[Rub16] Aviad Rubinstein. Beyond matroids: Secretary problem and prophet inequality with\ngeneral constraints. In Proceedings of the forty-eighth annual ACM symposium on\nTheory of Computing, pages 324\u2013332. ACM, 2016.\n\n[Sak95] Minoru Sakaguchi. Optimal stopping games: A review. Mathematica japonicae,\n\n42(2):343\u2013351, 1995.\n\n[SKB+12] Yue Shi, Alexandros Karatzoglou, Linas Baltrunas, Martha Larson, Nuria Oliver, and\nAlan Hanjalic. Climf: learning to maximize reciprocal rank with collaborative less-is-\nmore \ufb01ltering. In Proceedings of the sixth ACM conference on Recommender systems,\npages 139\u2013146. ACM, 2012.\n\n[SLH10] Yue Shi, Martha Larson, and Alan Hanjalic. List-wise learning to rank with matrix\nfactorization for collaborative \ufb01ltering. In Proceedings of the fourth ACM conference\non Recommender systems, pages 269\u2013272. ACM, 2010.\n\n[SM13] Yaron Singer and Manas Mittal. Pricing mechanisms for crowdsourcing markets. In\nProceedings of the 22nd international conference on World Wide Web, pages 1157\u2013\n1166. ACM, 2013.\n\n[SS90] Jeanette P Schmidt and Alan Siegel. The analysis of closed hashing under limited\nrandomness. In Proceedings of the twenty-second annual ACM symposium on Theory\nof computing, pages 224\u2013234. ACM, 1990.\n\n[SW17] Nihar B. Shah and Martin J. Wainwright. Simple, robust and optimal ranking from\npairwise comparisons. Journal of Machine Learning Research, 18:199:1\u2013199:38, 2017.\n\n12\n\n\f[WRdVR08] Jun Wang, Stephen Robertson, Arjen P de Vries, and Marcel JT Reinders. Probabilistic\nrelevance ranking for collaborative \ufb01ltering. Information Retrieval, 11(6):477\u2013497,\n2008.\n\n[XL07] Jun Xu and Hang Li. Adarank: a boosting algorithm for information retrieval. In\nProceedings of the 30th annual international ACM SIGIR conference on Research and\ndevelopment in information retrieval, pages 391\u2013398. ACM, 2007.\n\n[XLW+08] Fen Xia, Tie-Yan Liu, Jue Wang, Wensheng Zhang, and Hang Li. Listwise approach\nto learning to rank: theory and algorithm. In Proceedings of the 25th international\nconference on Machine learning, pages 1192\u20131199. ACM, 2008.\n\n13\n\n\f", "award": [], "sourceid": 622, "authors": [{"given_name": "Sepehr", "family_name": "Assadi", "institution": "Princeton University"}, {"given_name": "Eric", "family_name": "Balkanski", "institution": "Harvard University"}, {"given_name": "Renato", "family_name": "Leme", "institution": "Google Research"}]}