{"title": "High-Dimensional Optimization in Adaptive Random Subspaces", "book": "Advances in Neural Information Processing Systems", "page_first": 10847, "page_last": 10857, "abstract": "We propose a new randomized optimization method for high-dimensional problems which can be seen as a generalization of coordinate descent to random subspaces. We show that an adaptive sampling strategy for the random subspace significantly outperforms the oblivious sampling method, which is the common choice in the recent literature. The adaptive subspace can be efficiently generated by a correlated random matrix ensemble whose statistics mimic the input data. We prove that the improvement in the relative error of the solution can be tightly characterized in terms of the spectrum of the data matrix, and provide probabilistic upper-bounds. We then illustrate the consequences of our theory with data matrices of different spectral decay. Extensive experimental results show that the proposed approach offers significant speed ups in machine learning problems including logistic regression, kernel classification with random convolution layers and shallow neural networks with rectified linear units. Our analysis is based on convex analysis and Fenchel duality, and establishes connections to sketching and randomized matrix decompositions.", "full_text": "High-Dimensional Optimization in Adaptive Random\n\nSubspaces\n\nDepartment of Electrical Engineering\n\nDepartment of Electrical Engineering\n\nMert Pilanci\n\nStanford University\n\nJonathan Lacotte\n\nStanford University\n\nlacotte@stanford.edu\n\nMarco Pavone\n\nDepartment of Aeronautics &Astronautics\n\nStanford University\n\nAbstract\n\nWe propose a new randomized optimization method for high-dimensional problems\nwhich can be seen as a generalization of coordinate descent to random subspaces.\nWe show that an adaptive sampling strategy for the random subspace signi\ufb01cantly\noutperforms the oblivious sampling method, which is the common choice in the\nrecent literature. The adaptive subspace can be ef\ufb01ciently generated by a correlated\nrandom matrix ensemble whose statistics mimic the input data. We prove that\nthe improvement in the relative error of the solution can be tightly characterized\nin terms of the spectrum of the data matrix, and provide probabilistic upper-\nbounds. We then illustrate the consequences of our theory with data matrices of\ndifferent spectral decay. Extensive experimental results show that the proposed\napproach offers signi\ufb01cant speed ups in machine learning problems including\nlogistic regression, kernel classi\ufb01cation with random convolution layers and shallow\nneural networks with recti\ufb01ed linear units. Our analysis is based on convex analysis\nand Fenchel duality, and establishes connections to sketching and randomized\nmatrix decomposition.\n\n1\n\nIntroduction\n\nRandom Fourier features, Nystrom method and sketching techniques have been successful in large\nscale machine learning problems. The common practice is to employ oblivious sampling or sketching\nmatrices, which are typically randomized and \ufb01xed ahead of the time. However, it is not clear\nwhether one can do better by adapting the sketching matrices to data. In this paper, we show that\nadaptive sketching matrices can signi\ufb01cantly improve the approximation quality. We characterize the\napproximation error on the optimal solution in terms of the smoothness of the function, and spectral\nproperties of the data matrix.\nMany machine learning problems end up being high dimensional optimization problems, which\ntypically follow from forming the kernel matrix of a large dataset or mapping the data trough a high\ndimensional feature map, such as random Fourier features [20] or convolutional neural networks [13].\nSuch high dimensional representations induce higher computational and memory complexities, and\nresult in slower training of the models. Random projections are a classical way of performing\ndimensionality reduction, and are widely used in many algorithmic contexts [25]. Nevertheless, only\nrecently these methods have captured great attention as an effective way of performing dimensionality\nreduction in convex optimization. In the context of solving a linear system Ax = b and least-squares\noptimization, the authors of [14] propose a randomized iterative method with linear convergence rate,\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\f2\n\nwhich, at each iteration, performs a proximal update x(k+1) = argminx\u2208T (cid:107)x \u2212 x(k)(cid:107)\n2, where the\nnext iterate x(k+1) is restricted to lie within an af\ufb01ne subspace T = x(k) + range(A(cid:62)S), and S is a\nn\u00d7m dimension-reduction matrix with m (cid:54) min{n, d}. In the context of kernel ridge regression, the\nauthors of [31] propose to approximate the n-dimensional kernel matrix by sketching its columns to a\n\u2208 Rm, they show how to reconstruct an approximation(cid:101)x \u2208 Rn of the high dimensional\nlower m-dimensional subspace, chosen uniformly at random. From the low dimensional kernel ridge\nsolution \u03b1\u2217\nspectral properties of the kernel matrix K \u2013, the estimate(cid:101)x retains some statistical properties of x\u2217,\nsolution x\u2217\n\u2208 Rn. Provided that the sketching dimension m is large enough \u2013 as measured by the\ne.g., minimaxity. Similarly, in the broader context of classi\ufb01cation through convex loss functions,\nthe authors of [32, 33] propose to project the d-dimensional features of a given data matrix A to\n\u2208 Rm, their algorithm returns an estimate(cid:101)x \u2208 Rd of the optimal\na lower m-dimensional subspace, chosen independently of the data. After computing the optimal\n\u2208 Rd. Even though they provide formal guarantees on the estimation error (cid:107)(cid:101)x \u2212 x\u2217\nlow-dimensional classi\ufb01er \u03b1\u2217\nclassi\ufb01er x\u2217\n(cid:107)2,\ntheir results rely on several restrictive assumptions, that is, the data matrix A must be low rank, or, the\nclassi\ufb01er x\u2217 must lie in the span of the top few left singular vectors of A. Further, random subspace\noptimization has also been explored for large-scale trust region problems [27], also using a subspace\nchosen uniformly at random. Our proposed approach draws connections with the Gaussian Kaczmarz\nmethod proposed in [14] and the kernel sketching method in [31]. Differently, we are interested in\nsmooth, convex optimization problems with ridge regularization. In contrast to [32, 33], we do not\nmake any assumption on the optimal solution x\u2217.\nOur work relates to the considerable amount of literature on randomized approximations of high\ndimensional kernel matrices K. The typical approach consists of building a low-rank factorization of\nthe matrix K, using a random subset of its columns [28, 23, 9, 17]. The so-called Nystrom method\nhas proven to be effective empirically [30], and many research efforts have been devoted to improving\nand analyzing the performance of its many variants (e.g., uniform column sub-sampling, leverage-\nscore based sampling), especially in the context of kernel ridge regression [1, 2]. In a related vein,\nsketching methods have been proposed to reduce the space complexity of storing high-dimensional\ndata matrices [19, 31], by projecting their rows to a randomly chosen lower dimensional subspace.\nOur theoretical \ufb01ndings build on known results for low-rank factorization of positive semi-de\ufb01nite\n(p.s.d.) matrices [15, 5, 12, 29], and show intimate connections with kernel matrices sketching [31].\nLastly, our problem setting also draws connections with compressed sensing [7] where the goal is\nto recover a high dimensional structured signal from a small number of randomized, and usually\noblivious, measurements.\n\n1.1 Contributions\n\nIn this work, we propose a novel randomized subspace optimization method with strong solution\napproximation guarantees which outperform oblivious sampling methods. We derive probabilistic\nbounds on the error of approximation for general convex functions. We show that our method provides\na signi\ufb01cant improvement over the oblivious version, and theoretically quantify this observation as\nfunction of the spectral properties of the data matrix. We also introduce an iterative version of our\nmethod, which converges to the optimal solution by iterative re\ufb01nement.\n\n1.2 An overview of our results\nLet f : Rn \u2192 R be a convex and \u00b5-strongly smooth function, i.e., \u22072f (w) (cid:22) \u00b5In for all w \u2208 Rn,\nand A \u2208 Rn\u00d7d a high-dimensional matrix. We are interested in solving the primal problem\n\nx\u2217 = argmin\nx\u2208Rd\n\n\u03bb\n\n2(cid:107)x(cid:107)2\n2 ,\n\n(1)\nGiven a random matrix S \u2208 Rd\u00d7m with m (cid:28) d, we consider instead the sketched primal problem\n(2)\n\n\u03b1(cid:62)S(cid:62)S\u03b1 ,\n\nf (AS\u03b1) +\n\nf (Ax) +\n\n\u03b1\u2217\n\n\u2208 argmin\n\u03b1\u2208Rm\n\n\u03bb\n2\n\nwhere we effectively restrict the optimization domain to a lower m-dimensional subspace. In this\nwork, we explore the following questions: How can we estimate the original solution x\u2217 given the\nsketched solution \u03b1\u2217? Is a uniformly random subspace the optimal choice, e.g., S \u223c Gaussian i.i.d.?\n\n2\n\n\f(cid:107)2\n\n(cid:107)2/(cid:107)x\u2217\n\nOr, can we come up with an adaptive sampling distribution that is related to the matrix A, which\nyields stronger guarantees?\nby(cid:101)x = \u2212\u03bb\u22121A(cid:62)\nBy Fenchel duality analysis, we exhibit a natural candidate for an approximate solution to x\u2217, given\nmatrix of the form S = A(cid:62)(cid:101)S where (cid:101)S is typically Gaussian i.i.d., the relative error satis\ufb01es a high-\n\u2207f (AS\u03b1\u2217). Our main result (Section 2) establishes that, for an adaptive sketching\nprobability guarantee of the form (cid:107)(cid:101)x \u2212 x\u2217\n(cid:54) \u03b5, with \u03b5 < 1. Our error bound \u03b5 depends on\nthe smoothness parameter \u00b5, the regularization parameter \u03bb, the shape of the domain of the Fenchel\nconjugate f\u2217 and the spectral decay of the matrix A. Further, we show that this error can be explicitly\nthe adaptive matrix S = A(cid:62)(cid:101)S provides much stronger guarantees than oblivious sketching, where\ncontrolled in terms of the singular values of A, and we derive concrete bounds for several standard\nspectral pro\ufb01les, which arise in data analysis and machine learning. In particular, we show that using\na higher precision estimate(cid:101)x(T ) that satis\ufb01es (cid:107)(cid:101)x(T ) \u2212 x\u2217\nwe specialize our formal guarantees and empirical evaluations (Section 5) to Gaussian matrices (cid:101)S,\nbroader class of matrices (cid:101)S, such as Rademacher matrices, sub-sampled randomized Fourier (SRFT)\n\nS is independent of A. Then, we take advantage of the error contraction (i.e., \u03b5 < 1), and extend\nour adaptive sketching scheme to an iterative version (Section 3), which, after T iterations, returns\n(cid:54) \u03b5T . Throughout this work,\n\nwhich is a standard choice and yields the tightest error bounds. However, our approach extends to a\n\nor Hadamard (SRHT) transforms, and column sub-sampling matrices. Thus, it provides a general\nframework for random subspace optimization with strong solution guarantees.\n\n(cid:107)2/(cid:107)x\u2217\n\n(cid:107)2\n\n2 Convex optimization in adaptive random subspaces\nWe introduce the Fenchel conjugate of f, de\ufb01ned as f\u2217(z) := supw\u2208Rn\nthe relative error (cid:107)(cid:101)x \u2212 x\u2217\nconvex and its domain domf\u2217 := {z \u2208 Rn | f\u2217(z) < +\u221e} is a closed, convex set. Our control of\n(cid:107)2 is closely tied to controlling a distance between the respective\nsolutions of the dual problems of (1) and (2). The proof of the next two Propositions follow from\nstandard convex analysis arguments [22], and are deferred to Appendix C.\nProposition 1 (Fenchel Duality). Under the previous assumptions on f, it holds that\n\n(cid:8)w(cid:62)z \u2212 f (w)(cid:9), which is\n\n(cid:107)2/(cid:107)x\u2217\n\nmin\n\nf (Ax) +\n\nx\n\n2 = max\n\nz \u2212f\u2217(z) \u2212\n\n1\n\n2\u03bb(cid:107)A(cid:62)z(cid:107)\n2\n2 .\n\nThere exist an unique primal solution x\u2217 and an unique dual solution z\u2217. Further, we have Ax\u2217\n\u2208\n\u2202f\u2217(z\u2217), z\u2217 = \u2207f (Ax\u2217) and x\u2217 = \u2212 1\nProposition 2 (Fenchel Duality on Sketched Program). Strong duality holds for the sketched program\n\n\u03bb\n\n2(cid:107)x(cid:107)2\n\u03bb A(cid:62)z\u2217.\n\nmin\n\n\u03b1\n\nf (AS\u03b1) +\n\n\u03bb\n\n2(cid:107)S\u03b1(cid:107)2\n\n2 = max\n\ny \u2212f\u2217(y) \u2212\n\n1\n\n2\u03bb(cid:107)PSA(cid:62)y(cid:107)\n2\n2 ,\n\n(cid:32)\n\n(cid:33) 1\n\n2\n\nwhere PS = S(S(cid:62)S)\u2020S(cid:62) is the orthogonal projector onto the range of S. There exist a sketched\nprimal solution \u03b1\u2217 and an unique sketched dual solution y\u2217. Further, for any solution \u03b1\u2217, it holds\nthat AS\u03b1\u2217\nWe de\ufb01ne the following deterministic functional Zf which depends on f\u2217, the data matrix A and the\nsketching matrix S, and plays an important role in controlling the approximation error,\n\n\u2208 \u2202f\u2217(y\u2217) and y\u2217 = \u2207f (AS\u03b1\u2217).\n\nsup\n\nS A(cid:62)\u2206\n\n\u2206\u2208(domf\u2217\u2212z\u2217)\n\n,\n\n(3)\n\nZf \u2261 Zf (A, S) =\n\n\u2206(cid:62)AP \u22a5\n(cid:107)\u2206(cid:107)2\nS = I \u2212 PS is the orthogonal projector onto range(S)\n\n\u2207f (Ax\u2217) suggests the point (cid:101)x = \u2212\u03bb\u22121A(cid:62)\n\u22a5. The relationship x\u2217 =\nwhere P \u22a5\n\u2207f (AS\u03b1\u2217) as a candidate for approximat-\n\u2212\u03bb\u22121A(cid:62)\ning x\u2217. The Fenchel dual programs of (1) and (2) only differ in their quadratic regularization term,\nthat (cid:107)(cid:101)x \u2212 x\u2217\n2 and (cid:107)PSA(cid:62)y(cid:107)\n(cid:107)A(cid:62)z(cid:107)\nS A(cid:62)(z \u2212 y)(cid:107)2. As it holds\n(cid:107)2 = \u03bb\u22121(cid:107)A(cid:62)(z\u2217\n(cid:107)2 can be controlled in\nterms of the spectral norm (cid:107)P \u22a5\n(cid:107)2, or more sharply, in terms of the quantity Zf , which satis-\n\ufb01es Zf (cid:54) (cid:107)P \u22a5\nS A(cid:62)\n(cid:107)2. We formalize this statement in our next result, which proof is deferred to\nAppendix B.1.\n\n\u2212 y\u2217)(cid:107)2, we show that the error (cid:107)(cid:101)x \u2212 x\u2217\n2, which difference is tied to the quantity (cid:107)P \u22a5\n\nS A(cid:62)\n\n2\n\n2\n\n2\n\n3\n\n\f2\u03bb\n\n(cid:54)\n\n(cid:107)2\n\n(cid:107)2 ,\n\nf , we have\n\nZf(cid:107)x\u2217\n\nwhich further implies\n\n(cid:107)(cid:101)x \u2212 x\u2217\n(cid:107)(cid:101)x \u2212 x\u2217\n\nTheorem 1 (Deterministic bound). Let \u03b1\u2217 be any minimizer of the sketched program (2). Then,\nunder the condition \u03bb (cid:62) 2\u00b5Z 2\n\n(cid:114) \u00b5\n(cid:114) \u00b5\n2\u03bb(cid:107)P \u22a5\n(cid:107)2(cid:107)x\u2217\nFor an adaptive sketching matrix S = A(cid:62)(cid:101)S, we rewrite (cid:107)P \u22a5\n2 = (cid:107)K \u2212 K(cid:101)S((cid:101)S(cid:62)K(cid:101)S)\u2020(cid:101)S(cid:62)K(cid:107)2,\nS A(cid:62)\nfor randomized low-rank matrix factorization in the form K(cid:101)S((cid:101)S(cid:62)K(cid:101)S)\u2020(cid:101)S(cid:62)K of p.s.d. matrices\nof matrices (cid:101)S. For conciseness, we specialize our next result to adaptive Gaussian sketching, i.e.,\n(cid:101)S Gaussian i.i.d. Given a target rank k (cid:62) 2, we introduce a measure of the spectral tail of A as\n\nwhere K = AA(cid:62) is p.s.d. Combining our deterministic bound (5) with known results [15, 12, 5]\n\nK, we can give guarantees with high probability (w.h.p.) on the relative error for various types\n\nS A(cid:62)\n\n(cid:107)2 .\n\n(cid:107)2\n\n(4)\n\n(5)\n\n2\n\n(cid:107)\n\n(cid:54)\n\n(cid:16)\n\n(cid:80)\u03c1\n\n(cid:17) 1\n\n0R2\n\nk + 1\n\u03c32\nk\n\nj=k+1 \u03c32\nj\n\n2 , where \u03c1 is the rank of the matrix A and \u03c31 (cid:62) \u03c32 (cid:62) . . . (cid:62) \u03c3\u03c1\nRk(A) =\nits singular values. The proof of the next result follows from a combination of Theorem 1 and\nCorollary 10.9 in [15], and is deferred to Appendix B.2.\nCorollary 1 (High-probability bound). Given k (cid:54) min(n, d)/2 and a sketching dimension m = 2k,\n\u03bb (cid:62) 2\u00b5c2\n\nlet S = A(cid:62)(cid:101)S, with (cid:101)S \u2208 Rn\u00d7m Gaussian i.i.d. Then, for some universal constant c0 (cid:54) 36, provided\n\n(cid:114) \u00b5\nk(A), it holds with probability at least 1 \u2212 12e\u2212k that\nRk(A)(cid:107)x\u2217\n(cid:16) \u2206(cid:62)AP \u22a5\n(cid:17) 1\n(cid:107)2 .\nS A(cid:62)\u2206\nRemark 1. The quantity Zf := sup\u2206\u2208(domf\u2217\u2212z\u2217)\n2 is the eigenvalue of the matrix\n(cid:107)\u2206(cid:107)2\nP \u22a5\n\u2212 z\u2217) \u2229 S n\u22121, where S n\u22121 is the unit sphere\nS A(cid:62), restricted to the spherical cap K := (domf\u2217\nin dimension n. Thus, depending on the geometry of K, the deterministic bound (4) might be much\ntighter than (5), and yield a probabilistic bound better than (6). The investigation of such a result is\nleft for future work.\n\n(cid:107)(cid:101)x \u2212 x\u2217\n\n(cid:54) c0\n\n(cid:107)2\n\n(6)\n\n2\u03bb\n\n2\n\n2.1 Theoretical predictions as a function of spectral decay\n\nWe study the theoretical predictions given by (5) on the relative error, for different spectral decays\nof A and sketching methods, in particular, adaptive Gaussian sketching versus oblivious Gaussian\nk the eigenvalues of AA(cid:62).\nsketching and leverage score column sub-sampling [12]. We denote \u03bdk = \u03c32\nFor conciseness, we absorb \u00b5 into the eigenvalues by setting \u03bdk \u2261 \u00b5\u03bdk and \u00b5 \u2261 1. This re-scaling\nleaves the right-hand side of the bound (5) unchanged, and does not affect the analysis below. Then,\nwe assume that \u03bd1 = O(1), \u03bb \u2208 (\u03bd\u03c1, \u03bd1), and \u03bb \u2192 0 as n \u2192 +\u221e. These assumptions are standard\nin empirical risk minimization and kernel regression methods [11], which we focus on in Sections 4\nand 5. We consider three decaying schemes of practical interest. The matrix A has either a \ufb01nite-rank\n\u03c1, a \u03ba-exponential decay where \u03bdj \u223c e\u2212\u03baj and \u03ba > 0, or, a \u03b2-polynomial decay where \u03bdk \u223c j\u22122\u03b2\nand \u03b2 > 1/2. Among other examples, these decays are characteristic of various standard kernel\nfunctions, such as the polynomial, Gaussian and \ufb01rst-order Sobolev kernels [3]. Given a precision\n\u03b5 > 0 and a con\ufb01dence level \u03b7 \u2208 (0, 1), we denote by mA (resp. mO, mS) a suf\ufb01cient dimension for\nthe relative error. That is, with probability at least 1 \u2212 \u03b7, it holds that (cid:107)(cid:101)x \u2212 x\u2217\nwhich adaptive (resp. oblivious, leverage score) sketching yields the following (\u03b5, \u03b7)-guarantee on\nWe determine mA from our probabilistic regret bound (6). For mS, using our deterministic regret\nA(cid:62)(cid:101)S\nwhen (cid:101)S is a leverage score column sub-sampling matrix. To the best of our knowledge, the tightest\nbound (5), it then suf\ufb01ces to bound the spectral norm (cid:107)P \u22a5\nin terms of the eigenvalues \u03bdk,\nprovide an upper bound on the relative error (cid:107)(cid:101)x\u2212x\u2217\nbound has been given by [12] (see Lemma 5). For mO, we leverage results from [32]. The authors\n(cid:107)2, when S is Gaussian i.i.d. with variance\nd. It should be noted that their sketched solution \u03b1\u2217 is slightly different from ours. They solve\n\u03b1\u2217 = argmin f (AS\u03b1) + (2\u03bb)\u22121(cid:107)\u03b1(cid:107)2\n2, whereas we do include the matrix S in the regularization\n\n(cid:107)2 (cid:54) \u03b5.\n\n(cid:107)2/(cid:107)x\u2217\n\n(cid:107)2/(cid:107)x\u2217\n\nA(cid:62)\n\n(cid:107)2\n\n1\n\n4\n\n\f(cid:107)2\n2.\n\n2 \u2248 (cid:107)\u03b1\u2217\n(cid:107)2\n\nterm. One might wonder which regularizer works best when S is Gaussian i.i.d. Through extensive\nnumerical simulations, we observed a strongly similar performance. Further, standard Gaussian\nconcentration results yields that (cid:107)S\u03b1\u2217\nOur theoretical \ufb01ndings are summarized in Table 1, and we give the mathematical details of our\nderivations in Appendix D. For the sake of clarity, we provide in Table 1 lower bounds on the\nadaptive Gaussian sketching provides stronger guarantees on the relative error (cid:107)(cid:101)x \u2212 x\u2217\npredicted values mO and mS, and, thus, lower bounds on the ratios mO/mA and mS/mA. Overall,\n(cid:107)2/(cid:107)x\u2217\n(cid:107)2.\nTable 1: Sketching dimensions for a (\u03b5, \u03b7)-guarantee on the relative error (cid:107)(cid:101)x \u2212 x\u2217\n(cid:107)2/(cid:107)x\u2217\n(cid:107)2.\n(cid:16) 12\n(cid:17)\n(cid:16) 2d\n(cid:17)\n(cid:19)2\u2227 \u03b2\n(cid:16) 1\n(cid:19)\u22121+2\u2227 \u03b2\n\n(cid:16) 12\n\u03ba\u22121 log(cid:0) 1\n(cid:1) + log\n(cid:16) 2d\n\u03ba\u22121\u03b5\u22122 log(cid:0) 1\n(cid:1) log\n(cid:17)\n(cid:16) 1\n(cid:1) log\n\u03ba\u22121 log(cid:0) 1\n(cid:16)\n(cid:16) 1\n(cid:17)\n, \u03ba\u22121 log(cid:0) 1\n\n2\u03b2 \u03b5\nlog\n\u03b51/\u03b2\u22122 log(2d/\u03b7)\n\u2212 1\n\nLeverage score (mS)\nLower bound on mO\nmA\n\n(\u03b2 > 1/2)\n\u03bb\u22121/2\u03b2 \u03b5\u22121/\u03b2 + log\n\n(cid:16) 12\n(cid:17)\n(cid:16) 2\u03c1\n(cid:17)\n(cid:16) 4\u03c1\n\n\u03c1 + 1 + log\n(\u03c1 + 1)\u03b5\u22122 log\n\n2\u03b2 \u03b5\u22122 log\n\u2212 1\n\u03b2\n\n\u03b5\u22122+h log 2d, \u2200h > 0\n\nOblivious Gaussian (mO)\n\n\u03c1-rank matrix\n(\u03c1 (cid:28) n \u2227 d)\n\nAdaptive Gaussian (mA)\n\n\u03ba-exponential decay\n\n\u03b2-polynomial decay\n\n\u03b5\u22122 log \u03c1\n\n\u03b7\n\n\u03b2\u22121\n\n(\u03c1 + 1) log\n\n\u03b7\n\n\u2212 1\n\u03b2\n\n\u03bb\n\n2\u03b2 \u03b5\n\n(cid:17)\n\n\u03b7\n\n\u03bb\u03b5\n\n\u03bb\n\n(cid:17)\n\n\u03b7\n\n\u03b7\n\n(cid:17)\n(cid:17)\n\n\u03b7\n\n\u03b7\n\n\u2212 1\n\n\u03bb\n\n\u2212 1\n\n\u03bb\n\n(cid:18)\n\n(\u03ba > 0)\n\n\u03bb\u03b5\n\n\u03b7\n\n\u03b7\n\n\u03b2\u22121\n\nLower bound on mS\nmA\n\nlog \u03c1\n\nmin\n\nlog\n\n\u03b7\n\n(cid:18)\n\n(cid:1)(cid:17)\n\n\u03bb\u03b5\n\nWe illustrate numerically our predictions for adaptive Gaussian sketching versus oblivious Gaussian\nregression, with f (Ax) = n\u22121(cid:80)n\nsketching. With n = 1000 and d = 2000, we generate matrices Aexp and Apoly, with spectral\ndecay satisfying respectively \u03bdj \u223c ne\u22120.1j and \u03bdj \u223c nj\u22122. First, we perform binary logistic\ni x) where (cid:96)yi(z) = yi log(1+e\u2212z)+(1\u2212yi) log(1+ez),\nrelative error (cid:107)(cid:101)x\u2212x\u2217\ny \u2208 {0, 1}n and ai is the i-th row of A. For the polynomial (resp. exponential) decay, we expect the\n(2n)\u22121(cid:80)n\n(cid:107)2 to follow w.h.p. a decay proportional to m\u22121 (resp. e\u22120.05m). Figure 1\ncon\ufb01rms those predictions. We repeat the same experiments with a second loss function, f (Ax) =\ni x)yi. The latter is a convex relaxation of the penalty 1\n2 for\n\ufb01tting a shallow neural network with a ReLU non-linearity. Again, Figure 1 con\ufb01rms our predictions,\nand we observe that the adaptive method performs much better than the oblivious sketch.\n\n(cid:107)2/(cid:107)x\u2217\n+\u22122(a(cid:62)\n\n2(cid:107)(Ax)+\u2212y(cid:107)2\n\ni=1 (cid:96)yi(a(cid:62)\n\ni=1(a(cid:62)\n\ni x)2\n\nFigure 1: Relative error versus sketching dimension m \u2208 {2k | 3 (cid:54) k (cid:54) 10} of adaptive Gaussian\nsketching (red) and oblivious Gaussian sketching (green), for the ReLU and logistic models, and the\nexponential and polynomial decays. We use \u03bb = 10\u22124 for all simulations. Results are averaged over\n10 trials. Bar plots show (twice) the empirical standard deviations.\n\n3 Algorithms for adaptive subspace sketching\n\n3.1 Numerical conditioning and generic algorithm\n\nA standard quantity to characterize the capacity of a convex program to be solved ef\ufb01ciently is its\ncondition number [6], which, for the primal (1) and (adaptive) sketched program (2), are given by\n\n(cid:0)A(cid:62)\n\n\u22072f (Ax)A(cid:1)\n\n\u03bb + supx \u03c31\n\u03bb + inf x \u03c3d (A(cid:62)\u22072f (Ax)A)\n\n\u03ba =\n\n,\n\n\u03baS =\n\nsup\u03b1 \u03c31\n\ninf \u03b1 \u03c3d\n\nThe latter can be signi\ufb01cantly larger than \u03ba, up to \u03baS \u2248 \u03ba\nvariable overcomes this issue. With AS,\u2020 = AS(S(cid:62)S)\u2212 1\n\n(cid:16)(cid:101)S(cid:62)A(\u03bbI + A(cid:62)\n(cid:17)\n\u22072f (AA(cid:62)(cid:101)S\u03b1)A)A(cid:62)(cid:101)S\n(cid:16)(cid:101)S(cid:62)A(\u03bbI + A(cid:62)\u22072f (AA(cid:62)(cid:101)S\u03b1)A)A(cid:62)(cid:101)S\n(cid:17) .\n\u03c31((cid:101)S(cid:62)AA(cid:62)(cid:101)S)\n\u03c3m((cid:101)S(cid:62)AA(cid:62)(cid:101)S) (cid:29) \u03ba. A simple change of\n\n2 , we solve instead the optimization problem\n\n5\n\n2528(a)ReLU\u2013polynomial10\u2212210005001000(b)ReLU\u2013exponential10\u2212410\u221221002528(c)Logistic\u2013polynomial10\u2212410\u2212210005001000(d)Logistic\u2013exponential10\u2212510\u2212310\u22121\f\u03bb\n\nf (AS,\u2020\u03b1\u2020) +\n\n2(cid:107)\u03b1\u2020(cid:107)2\n2.\n\n\u03b1\u2217\n\u2020 = argmin\n\u03b1\u2020\u2208Rm\n\u2207f (AS,\u2020\u03b1\u2217\n\nIt holds that (cid:101)x = \u2212\u03bb\u22121A(cid:62)\n\u2020). The additional complexity induced by this change of\nvariables comes from computing the (square-root) pseudo-inverse of S(cid:62)S, which requires O(m3)\n\ufb02ops via a singular value decomposition. When m is small, this additional computation is negligible\nand numerically stable, and the re-scaled sketched program (7) is actually better conditioned that the\noriginal primal program (1), as stated in the next result that we prove in Appendix C.3.\nProposition 3. Under adaptive sketching, the condition number \u03ba\u2020 of the re-scaled sketched pro-\ngram (7) satis\ufb01es \u03ba\u2020 (cid:54) \u03ba with probability 1.\n\n(7)\n\nAlgorithm 1: Generic algorithm for adaptive sketching.\n\nInput :Data matrix A \u2208 Rn\u00d7d, random matrix (cid:101)S \u2208 Rn\u00d7m and parameter \u03bb > 0.\n1 Compute the sketching matrix S = A(cid:62)(cid:101)S, and, the sketched matrix AS = AS.\n2 Compute the re-scaling matrix R =(cid:0)S(cid:62)S(cid:1)\u2212 1\n(cid:17)\n3 Solve the convex optimization problem (7), and return(cid:101)x = \u2212 1\n\nAS,\u2020\u03b1\u2217\n\u2020\n\n\u03bb A(cid:62)\n\n\u2207f\n\n(cid:16)\n\n.\n\n2 , and the re-scaled sketched matrix AS,\u2020 = ASR.\n\n2(cid:107)SQ\u03b1(cid:107)2\n\n\u2207f (ASQ\u03b1\u2217\nQ).\n\n3.2 Error contraction and almost exact recovery of the optimal solution\n\nWe observed a drastic practical performance improvement between solving the sketched program as\nformulated in (2) and its well-conditioned version (7).\nIf the chosen sketch dimension m is itself prohibitively large for computing the matrix Q = (S(cid:62)S)\u2212 1\n2 ,\none might consider a pre-conditioning matrix Q, which is faster to compute, and such that the\nmatrix SQ is well-conditioned. Typically, one might compute a matrix Q based on an approximate\nsingular value decomposition of the matrix S(cid:62)S. Then, one solves the optimization problem\n\u03b1\u2217\nQ = argmin\u03b1\u2208Rm f (ASQ\u03b1) + \u03bb\n\n2. Provided that Q is invertible, it holds that(cid:101)x satis\ufb01es\n\n(cid:107)2 w.h.p., and, with \u03b5 < 1 provided\nthat \u03bb is large enough. Here, we extend Algorithm 1 to an iterative version which takes advantage of\n\n(cid:101)x = \u2212\u03bb\u22121A(cid:62)\nThe estimate(cid:101)x satis\ufb01es a guarantee of the form (cid:107)(cid:101)x\u2212 x\u2217\nthis error contraction, and which is relevant when a high-precision estimate(cid:101)x is needed.\nInput :Data matrix A \u2208 Rn\u00d7d, random matrix (cid:101)S \u2208 Rn\u00d7m, iterations number T , parameter \u03bb > 0.\n1 Compute the sketched matrix AS,\u2020 as in Algorithm 1. Set(cid:101)x(0) = 0.\nCompute a(t) = A(cid:101)x(t\u22121), and, b(t) =(cid:0)S(cid:62)S(cid:1)\u2212 1\n2 S(cid:62)(cid:101)x(t\u22121).\n\nAlgorithm 2: Iterative adaptive sketching\n\n(cid:107)2 (cid:54) \u03b5(cid:107)x\u2217\n\n2 for t = 1, 2, . . . , T do\n3\n4\n\nSolve the following convex optimization problem\n\nf (AS,\u2020\u03b1\u2020 + a(t)) +\n\n\u03bb\n2(cid:107)\u03b1\u2020 + b(t)(cid:107)2\n2 .\n\n\u03b1(t)\u2020 = argmin\n\u03b1\u2020\u2208Rm\n\u03bb A(cid:62)\n\n5 end\n\nUpdate the solution by(cid:101)x(t) = \u2212 1\n6 Return the last iterate(cid:101)x(T ).\nmatrix AS,\u2020 has to be computed only once, at the beginning of the procedure. The output(cid:101)x(T )\n\nA key advantage is that, at each iteration, the same sketching matrix S is used. Thus, the sketched\n\n\u2207f (AS,\u2020\u03b1(t)\u2020 + a(t)).\n\nsatis\ufb01es the following recovery property, which empirical bene\ufb01ts are illustrated in Figure 2.\nTheorem 2. After T iterations of Algorithm 2, provided that \u03bb (cid:62) 2\u00b5Z 2\n\nf , it holds that\n\n(8)\n\n(cid:107)(cid:101)x(T ) \u2212 x\u2217\n\n(cid:107)2 (cid:54)\n\n(cid:32) \u00b5Z 2\n\n(cid:33) T\n\n2\n\nf\n2\u03bb\n\n6\n\n(cid:107)x\u2217\n\n(cid:107)2 .\n\n(9)\n\n\fFurther, if S = A(cid:62)(cid:101)S where (cid:101)S \u2208 Rn\u00d7m with i.i.d. Gaussian entries and m = 2k for some target\nk(A), the approximate solution(cid:101)x(T ) satis\ufb01es with probability at least 1 \u2212 12e\u2212k,\n\nrank k (cid:62) 2, then, for some universal constant c0 (cid:54) 36, after T iterations of Algorithm 2, provided\nthat \u03bb (cid:62) 2c2\n\n0\u00b5R2\n\n(cid:18) c2\n\n(cid:107)(cid:101)x(T ) \u2212 x\u2217\n\n(cid:107)2 (cid:54)\n\n(cid:19) T\n\n2\n\nk(A)\n\n0\u00b5R2\n2\u03bb\n\n(cid:107)x\u2217\n\n(cid:107)2 .\n\n(10)\n\nFigure 2: Relative error versus sketching dimension m \u2208 {2k | 3 (cid:54) k (cid:54) 10} of adaptive Gaussian\nsketching for (a) the iterative method (Algorithm 2) and (b) the power method (see Remark 2). We use\nthe MNIST dataset with images mapped through 10000-dimensional random Fourier features [20]\nfor even-vs-odd classi\ufb01cation using binary logistic loss, and, \u03bb = 10\u22125. Results are averaged over 20\ntrials. Bar plots show (twice) the empirical standard deviations.\n\nRemark 2. An immediate extension of Algorithms 1 and 2 consists in using the power method [15].\nGiven q \u2208 N, one uses the sketching matrix S = (A(cid:62)A)\napproximation error (cid:107)AA(cid:62)\n\nA(cid:62)(cid:101)S. The larger q, the smaller the\n(cid:107)2 (see Corollary 10.10 in [15]). Of pratical interest\nare data matrices A with a spectral pro\ufb01le starting with a fast decay, and then becoming \ufb02at. This\nhappens typically for A of the form A = A + W , where A has a fast decay and W is a noise matrix\nwith, for instance, independent subgaussian rows [26]. Our results easily extend to this setting and\nwe illustrate its empirical bene\ufb01ts in Figure 2.\n\n\u2212 AS(S(cid:62)S)\u2020S(cid:62)A(cid:62)\n\nq\n\n4 Application to empirical risk minimization and kernel methods\n\nBy the representer theorem, the primal program (1) can be re-formulated as\n\nwhere K = AA(cid:62). Clearly, it holds that x\u2217 = A(cid:62)w\u2217. Given a matrix (cid:101)S with i.i.d. Gaussian entries,\n\n\u2208 argmin\nw\u2208Rn\n\nf (Kw) +\n\nwe consider the sketched version of the kernelized primal program (11),\n\n(11)\n\nw(cid:62)Kw ,\n\n\u03bb\n2\n\nw\u2217\n\nf (K(cid:101)S\u03b1) +\n\n\u03b1(cid:62)(cid:101)S(cid:62)K(cid:101)S\u03b1 .\n\n\u03bb\n2\n\n\u03b1\u2217\n\n(12)\n\n\u2208 argmin\n\u03b1\u2208Rm\n\nThe sketched program (12) is exactly our adaptive Gaussian sketched program (2). Thus, setting (cid:101)w =\n\u2212\u03bb\u22121\u2207f (K(cid:101)S\u03b1\u2217), it holds that(cid:101)x = A(cid:62)(cid:101)w. Since the relative error (cid:107)(cid:101)x \u2212 x\u2217\nby the decay of the eigenvalues of K, so does the relative error (cid:107)A(cid:62)((cid:101)w \u2212 w\u2217)(cid:107)2/(cid:107)A(cid:62)w\u2217\n(cid:107)2/(cid:107)x\u2217\n(cid:107)2 is controlled\n(cid:16)\n(cid:17)\n(cid:107)2. More\n2(cid:101)S\ngenerally, the latter statements are still true if K is any positive semi-de\ufb01nite matrix, and, if we\nreplace A(cid:62) by any square-root matrix of K. Here, we denote Zf \u2261 Zf\n(see Eq. (3)).\nTheorem 3. Let K \u2208 Rn\u00d7n be any positive semi-de\ufb01nite matrix. Let w\u2217 be any minimizer of the\nsolution (cid:101)w = \u2212 1\nkernel program (11) and \u03b1\u2217 be any minimizer of its sketched version (12). De\ufb01ne the approximate\n\nf , then it holds that\n\n2 , K 1\n\nK 1\n\n\u03bb\u2207f (K(cid:101)S\u03b1\u2217). If \u03bb (cid:62) 2\u00b5Z 2\n2 ((cid:101)w \u2212 w\u2217)(cid:107)2\n\n(cid:107)K\n\n1\n\n(cid:54)\n\nZf(cid:107)K\n\n1\n\n2 w\u2217\n\n(cid:107)2 .\n\n(13)\n\n2\u03bb\n\n(cid:114) \u00b5\n\n7\n\n242628210(b)Powermethod10\u22121100q=0q=1q=2242628210(a)Iterativemethod10\u2212210\u22121100T=1T=2T=3\fFor a positive de\ufb01nite kernel k : Rd \u00d7 Rd \u2192 R and a data matrix A = [a1, . . . , an](cid:62)\n\u2208 Rn\u00d7d,\nlet K be the empirical kernel matrix, with Kij = k(ai, aj). Let \u03d5(\u00b7) \u2208 RD be a random feature\nmap [20, 8], such as random Fourier features or a random convolutional neural net. We are interested\nin the computational complexities of forming the sketched versions of the primal (1), the kernel\nprimal (11) and the primal (1) with \u03d5(A) instead of A. We compare the complexities of adaptive and\noblivious sketching and uniform column sub-sampling. Table 2 shows that all three methods have\nsimilar complexities for computing AS and \u03d5(A)S. Adaptive sketching exhibits an additional factor\n\npractice, the latter is negligible compared to the cost of forming \u03d5(A) which, for instance, corresponds\nto a forward pass over the whole dataset in the case of a convolutional neural network. On the other\nhand, uniform column sub-sampling is signi\ufb01cantly faster in order to form the sketched kernel matrix\n\n2 that comes from computing the correlated sketching matrices S = A(cid:62)(cid:101)S and S = \u03d5(A)(cid:62)(cid:101)S. In\nK(cid:101)S, which relates to the well-known computational advantages of kernel Nystrom methods [30].\nTable 2: Complexity of forming the sketched programs, given A \u2208 Rn\u00d7d. We denote dk the number\nof \ufb02ops to evaluate the kernel product k(a, a(cid:48)), and, d\u03d5 the number of \ufb02ops for a forward-pass \u03d5(a).\nNote that these complexities could be reduced through parallelization.\n\nK(cid:101)S\n\n-\n\nO (dknm)\n\nO (2mdn) O (d\u03d5n) + O (2mDn) O(cid:0)dkn2(cid:1) + O(cid:0)mn2(cid:1)\n\n\u03d5(A)S\n\nAS\n\nAdaptive sketching\nOblivious sketching\n\nO (mdn)\nUniform column sub-sampling O (mdn)\n\nO (d\u03d5n) + O (mDn)\nO (d\u03d5n) + O (mDn)\n\n5 Numerical evaluation of adaptive Gaussian sketching\n\n(cid:107)2\n\n2\n\n\u2212\u03b3(cid:107)a \u2212 a(cid:48)\n\n(cid:1) [20, 18]. For MNIST and\n\nWe evaluate Algorithm 1 on MNIST and CIFAR10. First, we aim to show that the sketching\ndimension can be considerably smaller than the original dimension while retaining (almost) the same\ntest classi\ufb01cation accuracy. Second, we aim to get signi\ufb01cant speed-ups in achieving a high-accuracy\nclassi\ufb01er. To solve the primal program (1), we use two standard algorithms, stochastic gradient\ndescent (SGD) with (best) \ufb01xed step size and stochastic variance reduction gradient (SVRG) [16]\nwith (best) \ufb01xed step size and frequency update of the gradient correction. To solve the adaptive\nsketched program (2), we use SGD, SVRG and the sub-sampled Newton method [4, 10] \u2013 which we\nrefer to as Sketch-SGD, Sketch-SVRG and Sketch-Newton. The latter is well-suited to the sketched\nprogram, as the low-dimensional Hessian matrix can be quickly inverted at each iteration. For both\ndatasets, we use 50000 training and 10000 testing images. We transform each image using a random\n\nFourier feature map \u03d5(\u00b7) \u2208 RD, i.e., (cid:104)\u03d5(a), \u03d5(a(cid:48))(cid:105) \u2248 exp(cid:0)\nCIFAR10, we choose respectively D = 10000 and \u03b3 = 0.02, and, D = 60000 and \u03b3 = 0.002, so\nthat the primal is respectively 10000-dimensional and 60000-dimensional. Then, we train a classi\ufb01er\nvia a sequence of binary logistic regressions \u2013 which allow for ef\ufb01cient computation of the Hessian\nFirst, we evaluate the test classi\ufb01cation error of (cid:101)x. We solve to optimality the primal and\nand implementation of the Sketch-Newton algorithm \u2013, using a one-vs-all procedure.\nsketched programs for values of \u03bb \u2208 {10\u22124, 5 \u00b7 10\u22125, 10\u22125, 5 \u00b7 10\u22126} and sketching dimensions\nm \u2208 {64, 128, 256, 512, 1024}. In Table 3 are reported the results, which are averaged over 20 trials\nfor MNIST and 10 trials for CIFAR10, and, empirical variances are reported in Appendix A. Overall,\nthe adaptive sketched program yields a high-accuracy classi\ufb01er for most couples (\u03bb, m). Further, we\nmatch the best primal classi\ufb01er with values of m as small as 256 for MNIST and 512 for CIFAR10,\nwhich respectively corresponds to a dimension reduction by a factor \u2248 40 and \u2248 120. These results\nadditionally suggest that adaptive Gaussian sketching introduces an implicit regularization effect,\nwhich might be related to the bene\ufb01ts of spectral cutoff estimators. For instance, on CIFAR10, using\n\u03bb = 10\u22125 and m = 512, we obtain an improvement in test accuracy by more than 2% compared to\nvalue of m increases, the test classi\ufb01cation error of(cid:101)x increases to that of x\u2217, until matching it.\nx\u2217. Further, over some sketching dimension threshold under which the performance is bad, as the\n(Nystrom method) for which S = A(cid:62)(cid:101)S with(cid:101)S a column sub-sampling matrix. As reported in Table 4,\n\nFurther, we evaluate the test classi\ufb01cation error of two sketching baselines, that is, oblivious Gaussian\nsketching for which the matrix S has i.i.d. Gaussian entries, and, adaptive column sub-sampling\n\nadaptive Gaussian sketching performs better for a wide range of values of sketching size m and\nregularization parameter \u03bb.\n\n8\n\n\fTable 3: Test classi\ufb01cation error of adaptive Gaussian sketching on MNIST and CIFAR10 datasets.\n\n\u03bb\n10\u22124\n5 \u00b7 10\u22125\n10\u22125\n5 \u00b7 10\u22126\n\nx\u2217\nMNIST\n5.4\n4.6\n2.8\n2.5\n\n(cid:101)x64\n\n(cid:101)x128\n\n(cid:101)x256\n\n(cid:101)x512\n\n(cid:101)x1024\n\n4.8\n4.1\n8.1\n11.8\n\n4.8\n3.8\n3.4\n4.9\n\n5.2\n4.0\n2.4\n2.8\n\n5.3\n4.3\n2.5\n2.6\n\n5.4\n4.5\n2.8\n2.4\n\n(cid:101)x64\n\n(cid:101)x128\n\n(cid:101)x256\n\n(cid:101)x512\n\n(cid:101)x1024\n\n-\n\n52.1\n60.1\n63.6\n\n-\n\n50.5\n54.5\n59.8\n\n-\n\n50.6\n47.7\n51.9\n\n-\n\n50.8\n45.9\n47.7\n\n-\n\n51.0\n46.2\n45.8\n\nx\u2217\n\nCIFAR\n-\n\n51.6\n48.2\n47.6\n\nTable 4: Test classi\ufb01cation error on MNIST and CIFAR10. \"AG\": Adaptive Gaussian sketch, \"Ob\":\nOblivious Gaussian sketch, \"N\": Nystrom method.\n\n(cid:101)xAG\n(cid:101)xAG\n\n256\n\n(cid:101)xAG\n(cid:101)xAG\n\n(cid:101)xOb\n(cid:101)xOb\n\n256\n\n(cid:101)xOb\n(cid:101)xOb\n\n256\n\nx\u2217\nMNIST\n256\n4.6 % 4.0 % 4.5 % 25.2 % 8.5% 5.0 %\n2.5%\n\n1024\n4.6\n2.8% 2.4% 30.1% 9.4% 3.0% 2.7%\n\n1024\n\n1024\n\n256\n\nCIFAR\n\nx\u2217\n1024\n51.6 % 50.6% 51.0% 88.2% 70.5% 55.8% 53.1%\n47.6% 51.9% 45.8% 88.9% 80.1% 57.2% 55.8%\n\n1024\n\n1024\n\n256\n\n(cid:101)xN\n(cid:101)xN\n\n(cid:101)xN\n(cid:101)xN\n\n\u03bb\n\n5 \u00b7 10\u22125\n5 \u00b7 10\u22126\n\n\u03bb\n\n5 \u00b7 10\u22125\n5 \u00b7 10\u22126\n\nThen, we compare the test classi\ufb01cation error versus wall-clock time of the optimization algorithms\nmentioned above. Figure 3 shows results for some values of m and \u03bb. We observe some speed-ups\non the 10000-dimensional MNIST problem, in particular for Sketch-SGD and for Sketch-SVRG, for\nwhich computing the gradient correction is relatively fast. Such speed-ups are even more signi\ufb01cant\non the 60000-dimensional CIFAR10 problem, especially for Sketch-Newton. A few iterations of\n\nSketch-Newton suf\ufb01ce to almost reach the minimum(cid:101)x, with a per-iteration time which is relatively\n\nsmall thanks to dimensionality reduction. Hence, it is more than 10 times faster to reach the best test\naccuracy using the sketched program. In addition to random Fourier features mapping, we carry out\nanother set of experiments with the CIFAR10 dataset, in which we pre-process the images. That is,\nsimilarly to [24, 21], we map each image through a random convolutional layer. Then, we kernelize\nthese processed images using a Gaussian kernel with \u03b3 = 2\u00b710\u22125. Using our implementation, the best\ntest accuracy of the kernel primal program (11) we obtained is 73.1%. Sketch-SGD, Sketch-SVRG\nand Sketch-Newton \u2013 applied to the sketched kernel program (12) \u2013 match this test accuracy, with\nsigni\ufb01cant speed-ups, as reported in Figure 3.\n\nFigure 3: Test classi\ufb01cation error (percentage) versus wall-clock time (seconds).\n\nAcknowledgements\n\nThis work was partially supported by the National Science Foundation under grant IIS-1838179 and\nOf\ufb01ce of Naval Research, ONR YIP Program, under contract N00014-17-1-2433.\n\n9\n\n050100(a)MNIST,m=512,\u03bb=10\u22125.2510SGDSVRGSketch-SVRGSketch-NewtonSketch-SGD102103(b)CIFAR,m=256,\u03bb=10\u22125.485060SGDSketch-SVRGSketch-NewtonSketch-SGD102103(c)CIFAR(randomlayer),m=1024,\u03bb=5.10\u22126.273040SGDSketch-SVRGSketch-NewtonSketch-SGD\fReferences\n[1] A. Alaoui and M. W. Mahoney. Fast randomized kernel ridge regression with statistical\n\nguarantees. In Advances in Neural Information Processing Systems, pages 775\u2013783, 2015.\n\n[2] F. Bach. Sharp analysis of low-rank kernel matrix approximations. In Conference on Learning\n\nTheory, pages 185\u2013209, 2013.\n\n[3] A. Berlinet and C. Thomas-Agnan. Reproducing kernel Hilbert spaces in probability and\n\nstatistics. Kluwer Adademic, 2004.\n\n[4] R. Bollapragada, R. H. Byrd, and J. Nocedal. Exact and inexact subsampled newton methods\n\nfor optimization. IMA Journal of Numerical Analysis, 39(2):545\u2013578, 2018.\n\n[5] C. Boutsidis and A. Gittens. Improved matrix algorithms via the subsampled randomized\nhadamard transform. SIAM Journal on Matrix Analysis and Applications, 34(3):1301\u20131340,\n2013.\n\n[6] S. Boyd and L. Vandenberghe. Convex optimization. Cambridge university press, 2004.\n\n[7] E. J. Candes, J. K. Romberg, and T. Tao. Stable signal recovery from incomplete and inaccurate\nmeasurements. Communications on Pure and Applied Mathematics: A Journal Issued by the\nCourant Institute of Mathematical Sciences, 59(8):1207\u20131223, 2006.\n\n[8] A. Coates and A. Y. Ng. Learning feature representations with k-means. In Neural networks:\n\nTricks of the trade, pages 561\u2013580. Springer, 2012.\n\n[9] P. Drineas and M. W. Mahoney. On the nystr\u00f6m method for approximating a gram matrix for\nimproved kernel-based learning. journal of machine learning research, 6(Dec):2153\u20132175,\n2005.\n\n[10] M. A. Erdogdu and A. Montanari. Convergence rates of sub-sampled newton methods. arXiv\n\npreprint arXiv:1508.02810, 2015.\n\n[11] J. Friedman, T. Hastie, and R. Tibshirani. The elements of statistical learning, volume 1.\n\n[12] A. Gittens and M. W. Mahoney. Revisiting the nystr\u00f6m method for improved large-scale\n\nmachine learning. The Journal of Machine Learning Research, 17(1):3977\u20134041, 2016.\n\n[13] I. Goodfellow, Y. Bengio, and A. Courville. Deep learning. MIT press, 2016.\n\n[14] R. M. Gower and P. Richt\u00e1rik. Randomized iterative methods for linear systems. SIAM Journal\n\non Matrix Analysis and Applications, 36(4):1660\u20131690, 2015.\n\n[15] N. Halko, P.-G. Martinsson, and J. A. Tropp. Finding structure with randomness: Probabilistic\nalgorithms for constructing approximate matrix decompositions. SIAM review, 53(2):217\u2013288,\n2011.\n\n[16] R. Johnson and T. Zhang. Accelerating stochastic gradient descent using predictive variance\n\nreduction. In Advances in neural information processing systems, pages 315\u2013323, 2013.\n\n[17] S. Kumar, M. Mohri, and A. Talwalkar. Sampling methods for the nystr\u00f6m method. Journal of\n\nMachine Learning Research, 13(Apr):981\u20131006, 2012.\n\n[18] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel,\nP. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher,\nM. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine\nLearning Research, 12:2825\u20132830, 2011.\n\n[19] M. Pilanci and M. J. Wainwright. Randomized sketches of convex programs with sharp\n\nguarantees. IEEE Transactions on Information Theory, 61(9):5096\u20135115, 2015.\n\n10\n\n\f[20] A. Rahimi and B. Recht. Random features for large-scale kernel machines. In Advances in\n\nneural information processing systems, pages 1177\u20131184, 2008.\n\n[21] B. Recht, R. Roelofs, L. Schmidt, and V. Shankar. Do cifar-10 classi\ufb01ers generalize to cifar-10?\n\narXiv preprint arXiv:1806.00451, 2018.\n\n[22] R. T. Rockafellar. Convex analysis. Princeton university press, 2015.\n\n[23] A. J. Smola and B. Sch\u00f6lkopf. A tutorial on support vector regression. Statistics and computing,\n\n14(3):199\u2013222, 2004.\n\n[24] S. Tu, R. Roelofs, S. Venkataraman, and B. Recht. Large scale kernel learning using block\n\ncoordinate descent. arXiv preprint arXiv:1602.05310, 2016.\n\n[25] S. S. Vempala. The random projection method, volume 65. American Mathematical Soc., 2005.\n\n[26] R. Vershynin. High-dimensional probability: An introduction with applications in data science,\n\nvolume 47. Cambridge University Press, 2018.\n\n[27] K. Vu, P.-L. Poirion, C. D\u2019Ambrosio, and L. Liberti. Random projections for trust region\n\nsubproblems. arXiv preprint arXiv:1706.02730, 2017.\n\n[28] C. K. Williams and M. Seeger. Using the nystr\u00f6m method to speed up kernel machines. In\n\nAdvances in neural information processing systems, pages 682\u2013688, 2001.\n\n[29] R. Witten and E. Candes. Randomized algorithms for low-rank matrix factorizations: sharp\n\nperformance bounds. Algorithmica, 72(1):264\u2013281, 2015.\n\n[30] T. Yang, Y.-F. Li, M. Mahdavi, R. Jin, and Z.-H. Zhou. Nystr\u00f6m method vs random fourier\nfeatures: A theoretical and empirical comparison. In Advances in neural information processing\nsystems, pages 476\u2013484, 2012.\n\n[31] Y. Yang, M. Pilanci, M. J. Wainwright, et al. Randomized sketches for kernels: Fast and optimal\n\nnonparametric regression. The Annals of Statistics, 45(3):991\u20131023, 2017.\n\n[32] L. Zhang, M. Mahdavi, R. Jin, T. Yang, and S. Zhu. Recovering the optimal solution by dual\n\nrandom projection. In Conference on Learning Theory, pages 135\u2013157, 2013.\n\n[33] L. Zhang, M. Mahdavi, R. Jin, T. Yang, and S. Zhu. Random projections for classi\ufb01cation: A\n\nrecovery approach. IEEE Transactions on Information Theory, 60(11):7300\u20137316, 2014.\n\n11\n\n\f", "award": [], "sourceid": 5785, "authors": [{"given_name": "Jonathan", "family_name": "Lacotte", "institution": "Stanford University"}, {"given_name": "Mert", "family_name": "Pilanci", "institution": "Stanford"}, {"given_name": "Marco", "family_name": "Pavone", "institution": "Stanford University"}]}