{"title": "Stochastic and Adversarial Online Learning without Hyperparameters", "book": "Advances in Neural Information Processing Systems", "page_first": 5059, "page_last": 5067, "abstract": "Most online optimization algorithms focus on one of two things: performing well in adversarial settings by adapting to unknown data parameters (such as Lipschitz constants), typically achieving $O(\\sqrt{T})$ regret, or performing well in stochastic settings where they can leverage some structure in the losses (such as strong convexity), typically achieving $O(\\log(T))$ regret. Algorithms that focus on the former problem hitherto achieved $O(\\sqrt{T})$ in the stochastic setting rather than $O(\\log(T))$. Here we introduce an online optimization algorithm that achieves $O(\\log^4(T))$ regret in a wide class of stochastic settings while gracefully degrading to the optimal $O(\\sqrt{T})$ regret in adversarial settings (up to logarithmic factors). Our algorithm does not require any prior knowledge about the data or tuning of parameters to achieve superior performance.", "full_text": "Stochastic and Adversarial Online Learning without\n\nHyperparameters\n\nAshok Cutkosky\n\nDepartment of Computer Science\n\nStanford University\n\nashokc@cs.stanford.edu\n\nKwabena Boahen\n\nDepartment of Bioengineering\n\nStanford University\n\nboahen@stanford.edu\n\nAbstract\n\nMost online optimization algorithms focus on one of two things: performing well\n\u221a\nin adversarial settings by adapting to unknown data parameters (such as Lipschitz\nT ) regret, or performing well in stochastic\nconstants), typically achieving O(\nsettings where they can leverage some structure in the losses (such as strong\n\u221a\nconvexity), typically achieving O(log(T )) regret. Algorithms that focus on the\nformer problem hitherto achieved O(\nT ) in the stochastic setting rather than\nO(log(T )). Here we introduce an online optimization algorithm that achieves\n\u221a\nO(log4(T )) regret in a wide class of stochastic settings while gracefully degrading\nto the optimal O(\nT ) regret in adversarial settings (up to logarithmic factors).\nOur algorithm does not require any prior knowledge about the data or tuning of\nparameters to achieve superior performance.\n\n1 Extending Adversarial Algorithms to Stochastic Settings\n\nThe online convex optimization (OCO) paradigm [1, 2] can be used to model a large number of\nscenarios of interest, such as streaming problems, adversarial environments, or stochastic optimization.\nIn brief, an OCO algorithm plays T rounds of a game in which on each round the algorithm outputs\na vector wt in some convex space W , and then receives a loss function (cid:96)t : W \u2192 R that is convex.\nThe algorithm\u2019s objective is to minimize regret, which is the total loss of all rounds relative to w(cid:63),\n\nthe minimizer of(cid:80)T\n\nt=1 (cid:96)t in W :\n\nT(cid:88)\n\nRT (w(cid:63)) =\n\n(cid:96)t(wt) \u2212 (cid:96)t(w(cid:63))\n\nt=1\n\n\u221a\n\nOCO algorithms typically either make as few as possible assumptions about the (cid:96)t while attempting\nto perform well (adversarial settings), or assume that the (cid:96)t have some particular structure that can\nbe leveraged to perform much better (stochastic settings). For the adversarial setting, the minimax\noptimal regret is O(BLmax\nT ), where B is the diameter of W and Lmax is the maximum Lipschitz\nconstant of the losses [3]. A wide variety of algorithms achieve this bound without prior knowledge of\none or both of B and Lmax [4, 5, 6, 7], resulting in hyperparameter-free algorithms. In the stochastic\nsetting, it was recently shown that for a class of problems (those satisfying the so-called Bernstein\ncondition), one can achieve regret O(dBLmax log(T )) where W \u2282 Rd using the METAGRAD\nalgorithm [8, 9]. This approach requires knowledge of the parameter Lmax.\nIn this paper, we extend an algorithm for the parameter-free adversarial setting [7] to the stochastic\nsetting, achieving both optimal regret in adversarial settings as well as logarithmic regret in a wide\nclass of stochastic settings, without needing to tune parameters. Our class of stochastic settings is\nthose for which E[\u2207(cid:96)t(wt)] is aligned with wt \u2212 w(cid:63), quanti\ufb01ed by a value \u03b1 that increases with\n\n31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.\n\n\fincreasing alignment. We call losses in this class \u03b1-acutely convex, and show that a single quadratic\nlower bound on the average loss is suf\ufb01cient to ensure high \u03b1.\nThis paper is organized as follows. In Section 2, we provide an overview of our approach. In Section\n3, we give explicit pseudo-code and prove our regret bounds for the adversarial setting. In Section\n4, we formally de\ufb01ne \u03b1-acute convexity and prove regret bounds for the acutely convex stochastic\nsetting. Finally, in Section 5, we give some motivating examples of acutely convex stochastic losses.\nSection 6 concludes the paper.\n\n2 Overview of Approach\n\n\u221a\n\nBefore giving the overview, we \ufb01x some notation. We assume our domain W is a closed convex\nsubset of a Hilbert space with 0 \u2208 W . We write gt to be an arbitrary subgradient of (cid:96)t at wt for all\nt, which we denote by gt \u2208 \u2202(cid:96)t(wt). Lmax is the maximum Lipschitz constant of all the (cid:96)t, and B\nw \u00b7 w. We observe\nis the diameter of the space W . The norm (cid:107) \u00b7 (cid:107) we use is the 2-norm: (cid:107)w(cid:107) =\nt=1 gt(wt \u2212 w(cid:63)). We will make heavy use of this\nt=1 gt(wt \u2212 w(cid:63)). Finally,\nt(cid:48)=1 gt(cid:48), and we use \u02dcO to suppress logarithmic terms in\n\nthat since each (cid:96)t is convex, we have RT (w(cid:63)) \u2264(cid:80)T\ninequality; every regret bound we state will in fact be an upper bound on(cid:80)T\nwe use a compressed sum notation g1:t =(cid:80)t\n(cid:19)\n(cid:118)(cid:117)(cid:117)(cid:116)Lmax\n\uf8eb\uf8ed\n\nbig-Oh notation. All proofs omitted from the main text appear in the appendix.\nOur algorithm works by trading off some performance in order to avoid knowledge of problem\nparameters. Prior analysis of the METAGRAD algorithm [9] showed that any algorithm guaranteeing\nRT (w(cid:63)) = \u02dcO\nwill obtain logarithmic regret for stochastic settings\nsatisfying the Bernstein condition. We will instead guarantee the weaker regret bound:\n\nt=1(gt \u00b7 (wt \u2212 w(cid:63)))2\n\n(cid:18)(cid:113)(cid:80)T\n\n(cid:107)gt(cid:107)(cid:107)wt \u2212 w(cid:63)(cid:107)2\n\nRT (w(cid:63)) \u2264 \u02dcO\n\n\uf8f6\uf8f8\n\n(1)\n\nT(cid:88)\n\nt=1\n\n\u221a\n\nwhich we will show in turn implies\nT regret in adversarial settings and logarithmic regret for\nacutely convex stochastic settings. Although (1) is weaker than the METAGRAD regret bound, we\ncan obtain it without prior knoweldge.\nIn order to come up with an algorithm that achieves the bound (1), we interpret it as the square root\nof E[(cid:107)w \u2212 w(cid:63)(cid:107)2], where w takes on value wt with probability proportional to (cid:107)gt(cid:107). This allows us to\nuse the bias-variance decomposition to write (1) as:\n\n\uf8eb\uf8ed(cid:107)w(cid:63) \u2212 w(cid:107)(cid:112)Lmax(cid:107)g(cid:107)1:T +\n\n(cid:118)(cid:117)(cid:117)(cid:116) T(cid:88)\n\n\uf8f6\uf8f8\n\nt=1\n\n(2)\n\nLmax(cid:107)gt(cid:107)(cid:107)wt \u2212 w(cid:107)2\n\nRT (w(cid:63)) \u2264 \u02dcO\n(cid:80)T\n(cid:112)(cid:107)g(cid:107)1:T ) simultaneously for all u \u2208 W [10, 6, 11, 7]. Thus if we knew w ahead\nt=1 (cid:107)gt(cid:107)wt\n(cid:107)g(cid:107)1:T\n(cid:112)(cid:107)g(cid:107)1:T ), the bias term of (2). We do not know w, but we can estimate it\n\nwhere w =\n. Certain algorithms for unconstrained OCO can achieve RT (u) =\n\u02dcO((cid:107)u(cid:107)Lmax\nof time, we could translate the predictions of one such algorithm by w to abtain RT (w(cid:63)) \u2264\n\u02dcO((cid:107)w(cid:63) \u2212 w(cid:107)Lmax\nover time. Errors in the estimation procedure will cause us to incur the variance term of (2). We\nimplement this strategy by modifying FREEREX [7], an unconstrained OCO algorithm that does not\nrequire prior knowledge of any parameters.\nOur modi\ufb01cation to FREEREX is very simple: we set wt = \u02c6wt + wt\u22121 where \u02c6wt is the tth output of\nFREEREX, and wt\u22121 is (approximately) a weighted average of the previous vectors w1, . . . , wt\u22121\nwith the weight of wt equal to (cid:107)gt(cid:107). This wt offset can be viewed as a kind of momentum term that\naccelerates us towards optimal points when the losses are stochastic (which tends to cause correlated\nwt and therefore large offsets), but has very little effect when the losses are adversarial (which tends\nto cause uncorrelated wt and therefore small offsets).\n\n2\n\n\f3 FREEREXMOMENTUM\n\n\u221a\n\nIn this section, we explicitly describe and analyze our algorithm, FREEREXMOMENTUM, a modi\ufb01ca-\ntion of FREEREX. FREEREX is a Follow-the-Regularized-Leader (FTRL) algorithm, which means\nthat for all t, there is some regularizer function \u03c8t such that wt+1 = argminW \u03c8t(w) + g1:t \u00b7 w.\n\u03c6(atw), where \u03c6(w) = ((cid:107)w(cid:107) + 1) log((cid:107)w(cid:107) + 1) \u2212 (cid:107)w(cid:107)\nSpeci\ufb01cally, FREEREX uses \u03c8t =\nand \u03b7t and at are speci\ufb01c numbers that grow over time as speci\ufb01ed in Algorithm 1. FREEREXMO-\nMENTUM\u2019s predictions are given by offsetting FREEREX\u2019s predictions wt+1 by a momentum term\n. We accomplish this by shifting the regularizers \u03c8t by wt, so that FREEREXMO-\nwt =\nMENTUM is FTRL with regularizers \u03c8t(w \u2212 wt).\nAlgorithm 1 FREEREXMOMENTUM\n\nt(cid:48)=1\n1+(cid:107)g(cid:107)1:t\n\n(cid:80)t\u22121\n\n(cid:107)gt(cid:48)(cid:107)wt\n\n5\nat\u03b7t\n\n\u2190 0, a0 \u2190 0, w1 \u2190 0, L0 \u2190 0, \u03c8(w) = ((cid:107)w(cid:107) + 1) log((cid:107)w(cid:107) + 1) \u2212 (cid:107)w(cid:107)\n\nInitialize: 1\n\u03b72\n0\nfor t = 1 to T do\n\nPlay wt\nReceive subgradient gt \u2208 \u2202(cid:96)t(wt)\nLt \u2190 max(Lt\u22121,(cid:107)gt(cid:107)). // Lt = maxt(cid:48)\u2264t (cid:107)gt(cid:107)\n\n(cid:16) 1\n\u2190 max\n(cid:80)t\u22121\n\n+ 2(cid:107)gt(cid:107)2, Lt(cid:107)g1:t(cid:107)(cid:17)\n(cid:104)\u221a\n\n5\u03c6(at(w\u2212wt)\n\n.\n\n1\n\u03b72\nt\n\n\u03b72\nt\u22121\n\nat \u2190 max(at\u22121, 1/(Lt\u03b7t)2)\n(cid:107)gt(cid:48)(cid:107)wt\nwt \u2190\nt(cid:48)=1\n1+(cid:107)g(cid:107)1:t\nwt+1 \u2190 argminW\n\nat\u03b7t\n\n+ g1:t \u00b7 w\n\n(cid:105)\n\nend for\n\n3.1 Regret Analysis\n\nWe leverage the description of FREEREXMOMENTUM in terms of shifted regularizers to prove a\nregret bound of the same form as (1) in four steps:\n\n\u03c8t\u22121(w+\n\nt (w+\n\nt+1) + gt \u00b7 (wt \u2212 w+\n\nt+1)\n\nt+1) \u2212 \u03c8+\nT\u22121(cid:88)\n\nt=1\n\n1. From [7] Theorem 13, we bound the regret by\n\nRT (w(cid:63)) \u2264 T(cid:88)\n\ngt \u00b7 (wt \u2212 w(cid:63))\n\nt=1\n\n\u2264 \u03c8T (w(cid:63)) +\n\nT(cid:88)\n\nt=1\n\n+ \u03c8+\n\nT (w(cid:63)) \u2212 \u03c8T (w(cid:63)) +\n\n\u03c8+\nt (w+\n\nt+2) \u2212 \u03c8t(w+\n\nt+2)\n\nt (w) \u2248\n\n\u221a\n\n5\u03c6(at(w\u2212wt\u22121)\n\nat\u03b7t\n\nt+1 = argminW \u03c8+\n\nis a version of \u03c8t shifted by wt\u22121 instead of wt, and\nt (w) + g1:tw. This breaks the regret out into two sums, one in which\nt+1) for which the two different functions are shifted\nt+2), for which the functions\n\nwhere \u03c8+\nw+\nwe have the term \u03c8t\u22121(w+\nby the same amount, and one with the term \u03c8+\nare shifted differently, but the arguments are the same.\nt are shifted by the same amount, the regret analysis for FREEREX\nin [7] applies to the second line of the regret bound, yielding a quantity similar to (cid:107)w(cid:63) \u2212\n\nt+2) \u2212 \u03c8t(w+\n\nt+1) \u2212 \u03c8+\n\nt (w+\n\nt (w+\n\n2. Because \u03c8t\u22121 and \u03c8+\n\nwT(cid:107)(cid:112)Lmax(cid:107)g(cid:107)1:T .\n\n3. Next, we analyze the third line. We show that wt \u2212 wt\u22121 cannot be too big, and use this\nt=1 Lmax(cid:107)gt(cid:107)(wt \u2212 wT )2.\n\nobservation to bound the third line with a quantity similar to\nAt this point we have enough results to prove a bound of the form (2) (see Theorem 1).\n\n4. Finally, we perform some algebraic manipulation on the bound from the \ufb01rst three steps to\n\nobtain a bound of the form (1) (see Corollary 2).\n\n3\n\n(cid:113)(cid:80)T\n\n\fThe details of Steps 1-3 procedure are in the appendix, resulting in Theorem 1, stated below. Step 4\nis carried out in Corollary 2, which follows.\nTheorem 1. Let \u03c8(w) = ((cid:107)w(cid:107)+1) log((cid:107)w(cid:107)+1)\u2212(cid:107)w(cid:107). Set Lt = maxt(cid:48)\u2264t (cid:107)gt(cid:48)(cid:107), and QT = 2\nDe\ufb01ne 1\n\u03b7t\nFREEREXMOMENTUM is bounded by:\n\n(cid:107)g(cid:107)1:T\n.\nLmax\nand at as in the pseudo-code for FREEREXMOMENTUM (Algorithm 1). Then the regret of\n\n\u03c8(QT (w(cid:63)\u2212wT ))+405Lmax+2LmaxB+3\n\n\u221a\nLmax\n\n2Lmax\n\nB log(BaT +1)\n\nT(cid:88)\n\ngt\u00b7(wt\u2212w(cid:63)) \u2264\n\n(cid:118)(cid:117)(cid:117)(cid:116)2Lmax\n\n(cid:32)\n\nt=1\n\n+\n\n\u221a\n\n5\nQT \u03b7T\n\n(cid:107)wT(cid:107)2 +\n\n(cid:33)(cid:18)\n\nT(cid:88)\n\nt=1\n\n(cid:107)gt(cid:107)(cid:107)wt \u2212 wT(cid:107)2\n\n\u221a\n\n1 + L1\n\n(cid:19)(cid:19)\n\n(cid:18) 1 + (cid:107)g(cid:107)1:T\n\n1 + (cid:107)g1(cid:107)\n\n2 + log\n\nlog(BaT + 1)\n\nCorollary 2. Under the assumptions and notation of Theorem 1, the regret of FREEREXMOMENTUM\nis bounded by:\n\nT(cid:88)\n\n\u221a\ngt \u00b7 (wt \u2212 w(cid:63)) \u2264 2\n\n5\n\n(cid:118)(cid:117)(cid:117)(cid:116)Lmax\n\n(cid:32)\n\nT(cid:88)\n\n(cid:107)w(cid:63)(cid:107)2 +\n\n(cid:107)gt(cid:107)(cid:107)w(cid:63) \u2212 wt(cid:107)2\n\nlog(2BT + 1)(2 + log(T ))\n\n(cid:33)\n\nt=1\n\nt=1\n\n+ 405Lmax + 2LmaxB + 3\n\n\u221a\nLmax\n\n\u221a\n\n2Lmax\n\n1 + L1\n\nB log(2BT + 1)\n\nObserve that since wt and w(cid:63) are both in W , (cid:107)w(cid:63)(cid:107) and (cid:107)wt \u2212 w(cid:63)(cid:107) both are at most B, so that\nCorollary 2 implies that FREEREXMOMENTUM achieves \u02dcO(BLmax\nT ) regret in the worst-case,\nwhich is optimal up to logarithmic factors.\n\n\u221a\n\n3.2 Ef\ufb01cient Implementation for L\u221e Balls\n\n(cid:104)\u221a\n\n(cid:105)\n\nA careful reader may notice that the procedure for FREEREXMOMENTUM involves computing\nargminW\n, which may not be easy if the solution wt+1 is on the boundary\nof W . When the wt+1 is not on the boundary of W , then we have a closed-form update:\n\n+ g1:t \u00b7 w\n\n5\u03c8(at(w\u2212wt)\n\nat\u03b7t\n\n(cid:20)\n\n(cid:18) \u03b7t(cid:107)g1:t(cid:107)\u221a\n\n(cid:19)\n\n(cid:21)\n\n\u2212 1\n\nwt+1 = wt \u2212 g1:t\n\nat(cid:107)g1:t(cid:107)\n\nexp\n\nthis section we offer a simple strategy for the case that W is an L\u221e ball, W =(cid:81)d\n\nHowever, when wt+1 lies on the boundary of W , it is not clear how to compute it for general W . In\n\ni=1[\u2212b, b].\n\n5\n\nIn this setting, we can use the standard trick (e.g. see [12]) of running a separate copy of FREEREX-\nMOMENTUM for each coordinate. That is, we observe that\n\n(3)\n\nRT (w(cid:63)) \u2264 T(cid:88)\n\nd(cid:88)\n\nT(cid:88)\n\ngt \u00b7 (wt \u2212 u) =\n\ngt,i(wt,i \u2212 ui)\n\n(4)\n\nt=1\n\ni=1\n\nt=1\n\nso that if we run an independent online learning algorithm on each coordinate, using the coordinates\nof the gradients gt,i as losses, then the total regret is at most the sum of the individual regrets. More\ndetailed pseudocode is given in Algorithm 2.\nCoordinate-wise FREEREXMOMENTUM is easily implementable in time O(d) per update because\nthe FREEREXMOMENTUM update is easy to perform in one dimension: if the update (3) is outside\nthe domain [\u2212b, b], simply set wt+1 to b or \u2212b, whichever is closer to the unconstrained update.\nTherefore, coordinate-wise FREEREXMOMENTUM can be computed in O(d) time per update.\nWe bound the regret of coordinate-wise FREEREXMOMENTUM using Corollary 2 and Equation (4),\nresulting the following Corollary.\n\n4\n\n\fAlgorithm 2 Coordinate-Wise FREEREXMOMENTUM\n\nInitialize: w1 = 0, d copies of FREEREXMOMENTUM, F1,. . . ,Fd, where each Fi uses domain\nW = [\u2212b, b].\nfor t = 1 to T do\n\nPlay wt, receive subgradient gt.\nfor i = 1 to d do\nGive gt,i to Fi.\nGet wt+1,i \u2208 [\u2212b, b] from Fi.\n\nend for\n\nend for\n\nCorollary 3. The regret of coordinate-wise FREEREXMOMENTUM is bounded by:\n\nT(cid:88)\n\n\u221a\ngt \u00b7 (wt \u2212 w(cid:63)) \u2264 2\n\n5\n\n(cid:118)(cid:117)(cid:117)(cid:116)dLmax\n\n(cid:32)\n\nT(cid:88)\n\nd(cid:107)w(cid:63)(cid:107)2 +\n\n(cid:107)gt(cid:107)(cid:107)w(cid:63) \u2212 wt(cid:107)2\n\nlog(2T b + 1)(2 + log(T ))\n\n(cid:33)\n\nt=1\n\nt=1\n\n+ 405dLmax + 2Lmaxdb + 3d\n\n\u221a\nLmax\n\n\u221a\n\n2Lmax\n\n1 + L1\n\nb log(2bT + 1)\n\n4 Logarithmic Regret in Stochastic Problems\n\nIn this section we formally de\ufb01ne \u03b1-acute convexity and show that FREEREXMOMENTUM achieves\nlogarithmic regret for \u03b1-acutely convex losses. As a warm-up, we \ufb01rst consider the simplest case in\nwhich the loss functions (cid:96)t are \ufb01xed, (cid:96)t = (cid:96) for all t. After showing logarithmic regret for this case,\nwe will then generalize to more complicated stochastic settings.\nIntuitively, an acutely convex loss function (cid:96) is one for which the gradient gt is aligned with the\nvector wt \u2212 w(cid:63) where w(cid:63) = argmin (cid:96), as de\ufb01ned below.\nDe\ufb01nition 4. A convex function (cid:96) is \u03b1-acutely convex on a set W if (cid:96) has a global minimum at some\nw(cid:63) \u2208 W and for all w \u2208 W , for all subgradients g \u2208 \u2202(cid:96)(w), we have\n\ng \u00b7 (w \u2212 w(cid:63)) \u2265 \u03b1(cid:107)g(cid:107)(cid:107)w \u2212 w(cid:63)(cid:107)2\n\nWith this de\ufb01nition in hand, we can show logarithmic regret in the case where (cid:96)t = (cid:96) for all t for\nsome \u03b1-acutely convex function (cid:96). From Corollary 2, with w(cid:63) = argmin (cid:96), we have\n\n(cid:107)w(cid:63)(cid:107) +\n\ngt \u00b7 (w(cid:63) \u2212 wt)\n\n(5)\n\nWhere the \u02dcO notation suppresses terms whose dependence on T is at most O(log2(T )). Now we\nneed a small Proposition:\nProposition 5. If a, b, c and d are non-negative constants such that\n\n\u221a\nx \u2264 a\n\nbx + c + d\n\nThen\n\nApplying Proposition 5 to Equation (5) with x =(cid:80)T\n\n\u221a\nx \u2264 4a2b + 2a\n\nc + 2d\n\nt=1 gt \u00b7 (wt \u2212 w(cid:63)) yields\n\n(cid:18) Lmax(cid:107)w(cid:63)(cid:107)\n\n(cid:19)\n\nRT (u) \u2264 \u02dcO\n\n\u03b1\n\n5\n\nT(cid:88)\n\nt=1\n\ngt \u00b7 (wt \u2212 w(cid:63)) \u2264 \u02dcO\n\n\u2264 \u02dcO\n\n\uf8eb\uf8ed\n\uf8eb\uf8ed\n\n(cid:118)(cid:117)(cid:117)(cid:116)Lmax\n(cid:118)(cid:117)(cid:117)(cid:116)Lmax\n\n(cid:32)\n(cid:32)\n\nT(cid:88)\nT(cid:88)\n\nt=1\n\nt=1\n\n1\n\u03b1\n\n(cid:107)w(cid:63)(cid:107)2 +\n\n(cid:107)gt(cid:107)(cid:107)w(cid:63) \u2212 wt(cid:107)2\n\n(cid:33)\uf8f6\uf8f8\n(cid:33)\uf8f6\uf8f8\n\n\fwhere the \u02dcO again suppresses logarithmic terms, now with dependence on T at most O(log4(T )).\nHaving shown that FREEREXMOMENTUM achieves logarithmic regret on \ufb01xed \u03b1-acutely convex\nlosses, we now generalize to stochastic losses. In order to do this we will necessarily have to\nmake some assumptions about the process generating the stochastic losses. We encapsulate these\nassumptions in a stochastic version of \u03b1-acute convexity, given below.\nDe\ufb01nition 6. Suppose for all t, gt is such that E[gt|g1, . . . gt\u22121] \u2208 \u2202(cid:96)(wt) for some convex function\n(cid:96) with minimum at w(cid:63). Then we say gt is \u03b1-acutely convex in expectation if:\n\nE[gt] \u00b7 (wt \u2212 w(cid:63)) \u2265 \u03b1 E[(cid:107)gt(cid:107)(cid:107)wt \u2212 w(cid:63)(cid:107)2]\n\nwhere all expectations are conditioned on g1, . . . , gt\u22121.\nUsing this de\ufb01nition, a fairly straightforward calculation gives us the following result.\nTheorem 7. Suppose gt is \u03b1-acutely convex in expectation and gt is bounded (cid:107)gt(cid:107) \u2264 Lmax with\nprobability 1. Then FREEREXMOMENTUM achieves expected regret:\n\n(cid:18) Lmax(cid:107)w(cid:63)(cid:107)\n\n(cid:19)\n\n\u03b1\n\nE[RT (w(cid:63))] \u2264 \u02dcO\n\nProof. Throughout this proof, all expectations are conditioned on prior subgradients. By Corollary 2\nand Jensen\u2019s inequality we have\n\u2264 E\n\n405Lmax + 2LmaxB + 3\n\ngt \u00b7 (wt \u2212 w(cid:63))\n\nB log(2BT + 1)\n\n\u221a\nLmax\n\n2Lmax\n\n\u221a\n\nE\n\n(cid:34) T(cid:88)\n\nt=1\n\n(cid:107)w(cid:63)(cid:107)2 +\n\nlog(2T B + 1)(2 + log(T ))\n\n\u2264 405Lmax + 2LmaxB + 3\n\nB log(2BT + 1)\n\n\uf8f9\uf8fb\n\nT(cid:88)\n\nt=1\n\nT(cid:88)\n\nt=1\n\n1 + L1\n\n(cid:33)\n\n(cid:107)gt(cid:107)(cid:107)w(cid:63) \u2212 wt(cid:107)2\n\u221a\n\u221a\n2Lmax\n\u03b4\n\nLmax\n\n(cid:33)\n\n(cid:33)\n\n\u221a\n\u221a\n2Lmax\n\u03b4\n\n(cid:107)w(cid:63)(cid:107)2 +\n\nE[(cid:107)gt(cid:107)(cid:107)w(cid:63) \u2212 wt(cid:107)2]\n\nlog(2T B + 1)(2 + log(T ))\n\n\u2264 405Lmax + 2LmaxB + 3\n\nLmax\n\nB log(2BT + 1)\n\n(cid:107)w(cid:63)(cid:107)2 +\n\n1\n\u03b1\n\nE[gt \u00b7 (wt \u2212 w(cid:63))]\n\nlog(2T B + 1)(2 + log(T ))\n\n. Then we have shown\n\n(cid:107)w(cid:63)(cid:107)2 +\n\nR\n\u03b1\n\nlog(2T B + 1)(2 + log(T ))\n\nLmax\n\n\u221a\n\u221a\n2Lmax\n\u03b4\n\n+ 405Lmax + 2LmaxB + 3\n\nB log(BT + 1)\n\nT(cid:88)\n\nt=1\n\n(cid:19)\n\n(cid:19)(cid:35)\n\n5\n\n\u221a\n\n(cid:32)\n\n\u221a\n+2\n\n(cid:35)\n(cid:20)\n(cid:118)(cid:117)(cid:117)(cid:116)Lmax\n(cid:32)\n(cid:118)(cid:117)(cid:117)(cid:116)Lmax\n(cid:118)(cid:117)(cid:117)(cid:116)Lmax\n(cid:32)\n(cid:105)\n(cid:115)\nt=1 gt(wt \u2212 w(cid:63))\nR \u2264 2\n\n+ 2\n\n+ 2\n\n\u221a\n\n\u221a\n\n5\n\n5\n\n5\n\nLmax\n\n(cid:18)\n\n(cid:18)\n\n(cid:34)(cid:115)\n\nT(cid:88)\n\nt=1\n\nSet R = E(cid:104)(cid:80)T\n\n= \u02dcO\n\nLmax\n\n(cid:107)w(cid:63)(cid:107)2 +\n\nR\n\u03b1\n\nAnd now we use Proposition 5 to conclude:\n\nE[gt \u00b7 (wt \u2212 w(cid:63))] = \u02dcO\n\n(cid:18) Lmax(cid:107)w(cid:63)(cid:107)\n\n(cid:19)\n\n\u03b1\n\nas desired, where again \u02dcO hides at most a O(log4(T )) dependence on T .\n\nExactly the same argument with an extra factor of d applies to the regret of FREEREXMOMENTUM\nwith coordinate-wise updates.\n\n6\n\n\f5 Examples of \u03b1-acute convexity in expectation\n\nIn this section, we show that \u03b1-acute convexity in expectation is a condition that arises in practice,\njustifying the relevance of our logarithmic regret bounds. To do this, we show that a quadratic lower\nbound on the expected loss implies \u03b1-acute convexity, demonstrating acutely convexity is a weaker\ncondition than strong convexity.\nProposition 8. Suppose E[gt|g1, . . . , gt\u22121] \u2208 \u2202(cid:96)(wt) for some convex (cid:96) such that for some \u00b5 > 0\nand w(cid:63) = argmin (cid:96), (cid:96)(w) \u2212 (cid:96)(w(cid:63)) \u2265 \u00b5\n2(cid:107)w \u2212 w(cid:63)(cid:107)2 for all w \u2208 W . Suppose (cid:107)g(cid:107) \u2264 Lmax with\nprobability 1. Then gt is\n\n-acutely convex in expectation.\n\n\u00b5\n\n2Lmax\n\nProof. By convexity and the hypothesis of the proposition: E[gt] \u00b7 (wt \u2212 w(cid:63)) \u2265 (cid:96)(wt) \u2212 (cid:96)(w(cid:63)) \u2265\n2(cid:107)wt \u2212 w(cid:63)(cid:107)2 \u2265 \u00b5\n\nE[(cid:107)gt(cid:107)(cid:107)wt \u2212 w(cid:63)(cid:107)2\n\n\u00b5\n\n2Lmax\n\nWith Proposition 8, we see that FREEREXMOMENTUM obtains logarithmic regret for any loss that is\nlarger than a quadratic, without requiring knowledge of the parameter \u00b5 or the Lipschitz bound Lmax.\nFurther, this result requires only the expected loss (cid:96) = E[(cid:96)t] to have a quadratic lower bound - the\nindividual losses (cid:96)t themselves need not do so.\nThe boundedness of W makes it surprisingly easy to have a quadratic lower bound. Although a\nquadratic lower bound for a function (cid:96) is easily implied by strong convexity, the quadratic lower\nbound is a signi\ufb01cantly weaker condition. For example, since W has diameter B, (cid:107)w(cid:107) \u2265 1\nB(cid:107)w(cid:107)2\nand so the absolute value is 1\nB -acutely convex, but not strongly convex. The following Proposition\nshows that existence of a quadratic lower bound is actually a local condition; so long as the expected\nloss (cid:96) has a quadratic lower bound in a neighborhood of w(cid:63), it must do so over the entire space W :\nProposition 9. Supppose (cid:96) : W \u2192 R is a convex function such that (cid:96)(w) \u2212 (cid:96)(w(cid:63)) \u2265 \u00b5\n2(cid:107)w \u2212 w(cid:63)(cid:107)\n\nfor all w with (cid:107)w \u2212 w(cid:63)(cid:107) \u2264 r. Then (cid:96)(w) \u2212 (cid:96)(w(cid:63)) \u2265 min(cid:0) \u00b5r\n\n2B , \u00b5\n\n2\n\n(cid:1)(cid:107)w \u2212 w(cid:63)(cid:107)2 for all w \u2208 W .\n(cid:105) \u2265 \u00b5r\n(cid:16) rw(cid:107)w(cid:107)\n(cid:104)\n\n(cid:17) \u2212 (cid:96)(w(cid:63))\n\n(cid:96)\n\nProof. We translate by w(cid:63) to assume without loss of generality that w(cid:63) = 0. Then the statement\nis clear for (cid:107)w(cid:107) \u2264 r. By convexity, (cid:96)(w) \u2212 (cid:96)(w(cid:63)) \u2265 (cid:107)w(cid:107)\n2 (cid:107)w(cid:107) \u2265\n2B(cid:107)w(cid:107)2.\n\n\u00b5r\n\nr\n\nFinally, we provide a simple motivating example of an interesting problem we can solve with an\n\u03b1-acutely convex loss that is not strongly convex: computing the median.\nProposition 10. Let W = [a, b], and (cid:96)t(w) = |w \u2212 xt| where each xt is drawn i.i.d. from some \ufb01xed\ndistribution with a continuous cumulative distribution function D, and assume D(x(cid:63)) = 1\n2 . Further,\nsuppose |2D(w) \u2212 1| \u2265 F|w \u2212 x(cid:63)| for all |w \u2212 x(cid:63)| \u2264 G. Suppose gt = (cid:96)(cid:48)\nt(wt) for wt (cid:54)= xt and\ngt = \u00b11 with equal probability if wt = xt. Then gt is min\n-acutely convex in expectation.\n\n(cid:16) F G\n\n(cid:17)\n\nb\u2212a , F\n\nProof. By a little calculation, E[gt] = (cid:96)(cid:48)(wt) = 2D(wt) \u2212 1, and E[|gt|] = 1. Since (cid:96)(cid:48)(x(cid:63)) = 0,\nw(cid:63) = x(cid:63) (the median). For |wt \u2212 x(cid:63)| \u2265 G, we have |2D(w) \u2212 1| \u2265 F G, which gives E[gt] \u00b7 (wt \u2212\nw(cid:63)) \u2265 F G\nE[|gt|](wt\u2212 w(cid:63))2. For |wt\u2212 x(cid:63)| \u2264 G, we have E[gt]\u00b7 (wt\u2212 w(cid:63)) \u2265 F E[|gt|](wt\u2212 w(cid:63))2,\nb\u2212a\nso that gt is min\n\n-acutely convex in expectation.\n\n(cid:16) F G\n\n(cid:17)\n\nb\u2212a , F\n\nProposition 10 shows that we can obtain low regret for an interesting stochastic problem without\ncurvature. The condition on the cumulative distribution function D is asking only that there be\npositive density in a neighborhood of the median; it would be satis\ufb01ed if D(cid:48)(w) \u2265 F for |w| \u2264 G.\nIf the expected loss (cid:96) is \u00b5-strongly convex, we can apply Proposition 8 to see that (cid:96) is \u00b5/2-aligned,\nand then use Theorem 7 to obtain a regret of \u02dcO(Lmax(cid:107)w(cid:63)(cid:107)/\u00b5). This is different from the usual regret\nbound of \u02dcO(L2\nmax/\u00b5) obtained by Online Newton Step [13], which is due to an inef\ufb01ciency in using\nthe wearker \u03b1-alignment condition. Instead, arguing from the regret bound of Corollary 2 directly,\nwe can recover the optimal regret bound:\n\n7\n\n\fCorollary 11. Suppose each (cid:96)t is an independent random variable with E[(cid:96)t] = (cid:96) for some \u00b5-strongly\nconvex (cid:96) with minimum at w(cid:63). Then the expected regret of FREEREXMOMENTUM satis\ufb01es\n\n(cid:34) T(cid:88)\n\nE\n\n(cid:35)\n\n(cid:96)(wt) \u2212 (cid:96)(w\u2217)\n\n\u2264 \u02dcO(L2\n\nmax/\u00b5)\n\nWhere the \u02dcO hides terms that are logarithmic in T B.\n\nt=1\n\nProof. From strong-convexity, we have\n\n(cid:107)wt \u2212 w(cid:63)(cid:107)2 \u2264 2\n\u00b5\n\n((cid:96)(wt) \u2212 (cid:96)(w(cid:63)))\n\nTherefore applying Corollary 2 we have\n\nE[RT (w(cid:63))] = E\n\n(cid:96)(wt) \u2212 (cid:96)(w\u2217)\n\n(cid:34) T(cid:88)\n\nt=1\n\n(cid:35)\n\n(cid:118)(cid:117)(cid:117)(cid:116)L2\n\uf8eb\uf8ed\n\u2264 \u02dcO((cid:112)L2\n\n\u2264 \u02dcO\n\n\uf8f6\uf8f8\n\n(cid:107)wt \u2212 w(cid:63)(cid:107)2]\n\nT(cid:88)\n\nt=1\n\nmax E[\n\nmax E[RT (w(cid:63))])\n\nSo that applying Proposition 5 we obtain the desired result.\n\nAs a result of Corollary 11, we see that FREEREXMOMENTUM obtains logarithmic regret for \u03b1-\naligned problems and also obtains the optimal (up to log factors) regret bound for \u00b5-strongly-convex\nproblems, all without requiring any knowledge of the parameters \u03b1 or \u00b5. This stands in contrast to\nprior algorithms that adapt to user-supplied curvature information such as Adaptive Gradient Descent\n[14] or (A,B)-prod [15].\n\n6 Conclusions and Open Problems\n\n\u03b1\n\nadversarial settings and \u02dcO(cid:0) LmaxB\n\n(cid:1) regret in \u03b1-acutely convex stochastic settings without requiring\n\nT ) regret in\n\n\u221a\n\nWe have presented an algorithm, FREEREXMOMENTUM, that achieves both \u02dcO(BLmax\n\nany prior information about any parameters. We further showed that a quadratic lower bound on\nthe expected loss implies acute convexity, so that while strong-convexity is suf\ufb01cient for acute\nconvexity, other important loss families such as the absolute loss may also be acutely convex. Since\nFREEREXMOMENTUM does not require prior information about any problem parameters, it does not\nrequire any hyperparameter tuning to be assured of good convergence. Therefore, the user need not\nactually know whether a particular problem is adversarial or acutely convex and stochastic, or really\nmuch of anything at all about the problem, in order to use FREEREXMOMENTUM.\nThere are still many interesting open questions in this area. First, we would like to \ufb01nd an ef\ufb01cient\nway to implement the FREEREXMOMENTUM algorithm or some variant directly, without appealing\nto coordinate-wise updates. This would enable us to remove the factor of d we incur by using\ncoordinate-wise updates. Second, our modi\ufb01cation to FREEREX is extremely simple and intuitive,\nbut our analysis makes use of some of the internal logic of FREEREX. It is possible, however, that\nany algorithm with suf\ufb01ciently low regret can be modi\ufb01ed in a similar way to achieve our results.\nFinally, we observe that while log4(T ) is much better than\nT asymptotically, it turns out that\nlog4(T ) >\nT for T < 1011, which casts the practical relevance of our logarithmic bounds in doubt.\nTherefore we hope that this work serves as a starting point for either new analysis or algorithm design\nthat further simpli\ufb01es and improves regret bounds.\n\n\u221a\n\n\u221a\n\nReferences\n[1] Martin Zinkevich. Online convex programming and generalized in\ufb01nitesimal gradient ascent. In Proceed-\n\nings of the 20th International Conference on Machine Learning (ICML-03), pages 928\u2013936, 2003.\n\n[2] Shai Shalev-Shwartz. Online learning and online convex optimization. Foundations and Trends in Machine\n\nLearning, 4(2):107\u2013194, 2011.\n\n8\n\n\f[3] Jacob Abernethy, Peter L Bartlett, Alexander Rakhlin, and Ambuj Tewari. Optimal strategies and min-\nimax lower bounds for online convex games. In Proceedings of the nineteenth annual conference on\ncomputational learning theory, 2008.\n\n[4] J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic\n\noptimization. In Conference on Learning Theory (COLT), 2010.\n\n[5] H. Brendan McMahan and Matthew Streeter. Adaptive bound optimization for online convex optimization.\n\nIn Proceedings of the 23rd Annual Conference on Learning Theory (COLT), 2010.\n\n[6] Francesco Orabona and D\u00e1vid P\u00e1l. Coin betting and parameter-free online learning.\n\nIn D. D. Lee,\nM. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing\nSystems 29, pages 577\u2013585. Curran Associates, Inc., 2016.\n\n[7] Ashok Cutkosky and Kwabena Boahen. Online learning without prior information. arXiv preprint\n\narXiv:1703.02629, 2017.\n\n[8] Tim van Erven and Wouter M Koolen. Metagrad: Multiple learning rates in online learning. In D. D.\nLee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information\nProcessing Systems 29, pages 3666\u20133674. Curran Associates, Inc., 2016.\n\n[9] Wouter M Koolen, Peter Gr\u00fcnwald, and Tim van Erven. Combining adversarial guarantees and stochastic\nfast rates in online learning. In Advances in Neural Information Processing Systems, pages 4457\u20134465,\n2016.\n\n[10] Francesco Orabona. Dimension-free exponentiated gradient. In Advances in Neural Information Processing\n\nSystems, pages 1806\u20131814, 2013.\n\n[11] Ashok Cutkosky and Kwabena A Boahen. Online convex optimization with unconstrained domains and\nlosses. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural\nInformation Processing Systems 29, pages 748\u2013756. Curran Associates, Inc., 2016.\n\n[12] Brendan Mcmahan and Matthew Streeter. No-regret algorithms for unconstrained online convex optimiza-\n\ntion. In Advances in neural information processing systems, pages 2402\u20132410, 2012.\n\n[13] Elad Hazan, Amit Agarwal, and Satyen Kale. Logarithmic regret algorithms for online convex optimization.\n\nMachine Learning, 69(2):169\u2013192, 2007.\n\n[14] Peter L Bartlett, Elad Hazan, and Alexander Rakhlin. Adaptive online gradient descent. In NIPS, volume 20,\n\npages 65\u201372, 2007.\n\n[15] Amir Sani, Gergely Neu, and Alessandro Lazaric. Exploiting easy data in online optimization. In Advances\n\nin Neural Information Processing Systems, pages 810\u2013818, 2014.\n\n9\n\n\f", "award": [], "sourceid": 2613, "authors": [{"given_name": "Ashok", "family_name": "Cutkosky", "institution": "Stanford University"}, {"given_name": "Kwabena", "family_name": "Boahen", "institution": "Stanford University"}]}