{"title": "The Entire Regularization Path for the Support Vector Machine", "book": "Advances in Neural Information Processing Systems", "page_first": 561, "page_last": 568, "abstract": null, "full_text": " The Entire Regularization Path for the Support\n Vector Machine\n\n\n\n Trevor Hastie Saharon Rosset\n Department of Statistics IBM Watson Research Center\n Stanford University P.O. Box 218\n Stanford, CA 94305, USA Yorktown Heights, N.Y. 10598\n hastie@stanford.edu srosset@us.ibm.com\n\n\n Robert Tibshirani Ji Zhu\n Department of Statistics Department of Statistics\n Stanford University University of Michigan\n Stanford, CA 94305, USA Ann Arbor, MI 48109-1092\n tibs@stanford.edu jizhu@umich.edu\n\n\n\n\n Abstract\n\n In this paper we argue that the choice of the SVM cost parameter can be\n critical. We then derive an algorithm that can fit the entire path of SVM\n solutions for every value of the cost parameter, with essentially the same\n computational cost as fitting one SVM model.\n\n\n\n1 Introduction\n\nWe have a set of n training pairs xi, yi, where xi Rp is a p-vector of real valued predictors\n(attributes) for the ith observation, yi {-1, +1} codes its binary response. The standard\ncriterion for fitting the linear SVM )[1, 2, 3] is\n\n n\n 1\n min ||||2 + C i, (1)\n 0, 2 i=1\n subject to, for each i: yi(0 + xT ) 1 - \n i i.\n\nHere the i are non-negative slack variables that allow points to be on the wrong side of their\n\"soft margin\" (f (x) = 1), as well as the decision boundary, and C is a cost parameter that\ncontrols the amount of overlap. If the data are separable, then for sufficiently large C the\nsolution achieves the maximal margin separator; if not, the solution achieves the minimum\noverlap solution with largest margin.\n\nAlternatively, we can formulate the problem using a (hinge) Loss + Penalty criterion [4, 5]:\n\n n \n min [1 - yi(0 + T xi)]+ + ||||2. (2)\n 0, 2\n i=1\n\nThe regularization parameter in (2) corresponds to 1/C, with C in (1).\n\n\f\nThis latter formulation emphasizes the role of regularization. In many situations we have\nsufficient variables (e.g. gene expression arrays) to guarantee separation. We may never-\ntheless avoid the maximum margin separator ( 0), which is governed by observations\non the boundary, in favor of a more regularized solution involving more observations.\n\nThe nonlinear kernel SVMs can be represented in this form as well. With kernel K and\nf (x) = 0 + n \n i=1 iK (x, xi), we solve [5]\n n n n n\n \n min [1 - yi(0 + iK(xi, xj))] + jj K(xj, x ).\n j (3)\n 0, 2\n i=1 j=1 j=1 j =1\n\nOften the regularization parameter C (or ) is regarded as a genuine \"nuisance\". Software\npackages, such as the widely used SVMlight [6], provide default settings for C.\n\nTo illustrate the effect of regularization, we generated data from a pair of mixture densities,\ndescribed in detail in [5, Chapter 2]. We used an SVM with a radial kernel K(x, x ) =\nexp(-||x - x ||2). Figure 1 shows the test error as a function of C for these data, using\nfour different values for . Here we see a dramatic range in the correct choice for C (or\n = 1/C). When = 5, the most regularized model is called for; when = 0.1, the least\nregularized.\n\n\n Test Error Curves - SVM with Radial Kernel\n\n = 5 = 1 = 0.5 = 0.1\n\n 0.35\n\n\n\n\n\n 0.30\n\n\n\n\n\n Test Error\n 0.25\n\n\n\n\n\n 0.20\n\n 1e-01 1e+01 1e+03 1e-01 1e+01 1e+03 1e-01 1e+01 1e+03 1e-01 1e+01 1e+03\n\n\n\n C = 1/\n\nFigure 1: Test error curves for the mixture example, using four different values for the radial kernel\nparameter .\n\n\nOne of the reasons that investigators avoid extensive exploration of C is the computational\ncost involved. In this paper we develop an algorithm which fits the entire path of SVM\nsolutions [0(C), (C)], for all possible values of C, with essentially the computational\ncost of fitting a single model for a particular value of C. Our algorithm exploits the fact\nthat the Lagrange multipliers implicit in (1) are piecewise-linear in C. This also means that\nthe coefficients ^\n (C) are also piecewise-linear in C. This is true for all SVM models, both\nlinear and nonlinear kernel-based SVMs.\n\n\n2 Problem Setup\n\nWe use a criterion equivalent to (1), implementing the formulation in (2):\n\n n \n min i + T subject to 1 - yif (xi) i; i 0; f (x) = 0 + T x. (4)\n ,0 2\n i=1\n\nInitially we consider only linear SVMs to get the intuitive flavor of our procedure; we then\ngeneralize to kernel SVMs.\n\n\f\nWe construct the Lagrange primal function\n\n n n n\n \n LP : i + T + i(1 - yif (xi) - i) - ii (5)\n 2\n i=1 i=1 i=1\n\nand set the derivatives to zero. This gives\n\n n\n 1\n : = iyixi (6)\n i=1\n n\n : yii = 0, (7)\n 0 i=1\n\nalong with the KKT conditions\n\n i(1 - yif (xi) - i) = 0 (8)\n ii = 0 (9)\n 1 - i - i = 0 (10)\n\nWe see that 0 i 1, with i = 1 when i > 0 (which is when yif (xi) < 1). Also\nwhen yif (xi) > 1, i = 0 since no cost is incurred, and i = 0. When yif (xi) = 1, i\ncan lie between 0 and 1.\n\nThe usual Lagrange multipliers associated with the solution to (1) are = \n i i/ = Ci.\nWe prefer our formulation here since our i [0, 1], and this simplifies the definition of\nthe paths we define.\n\nWe wish to find the entire solution path for all values of 0. Our basic idea is as follows.\nWe start with large and decrease it toward zero, keeping track of all the events that occur\nalong the way. As decreases, |||| increases, and hence the width of the margin decreases.\nAs this width decreases, points move from being inside to outside their margins. Their\ncorresponding i change from i = 1 when they are inside their margin (yif (xi) < 1) to\ni = 0 when they are outside their margin (yif (xi) > 1). By continuity, points must linger\non the margin (yif (xi) = 1) while their i decrease from 1 to 0. We will see that the i()\ntrajectories are piecewise-linear in , which affords a great computational savings: as long\nas we can establish the break points, all values in between can be found by simple linear\ninterpolation. Note that points can return to the margin, after having passed through it.\n\nIt is easy to show that if the i() are piecewise linear in , then both (C) = C\n i i(C )\nand (C) are piecewise linear in C. It turns out that 0(C) is also piecewise linear in C.\n\nOur algorithm keeps track of the following sets:\n\n M = {i : yif(xi) = 1, 0 i 1}, M for Margin\n I = {i : yif(xi) < 1, i = 1}, I for Inside the margin\n O = {i : yif(xi) > 1, i = 0}, O for Outside the margin\n\n3 The Algorithm\n\nDue to space restrictions, we show some details here; the rest can be found in [7].\n\n\nInitialization\n\nThe initial conditions depend on whether the classes are balanced or not (n+ = n-). The\nbalanced case is easier. For very large , |||| is small, and the the margin is very wide,\n\n\f\nall points are in O, and hence i = 1i. From (6) this means the orientation of is fixed\nuntil the i change. The margin narrows as decreases, but the orientation remains fixed.\nBecause of (7), the narrowing margin must connect with an outermost member of each\nclass simultaneously. These points are easily identified, and this establishes the first event,\nthe first tenants of M, and 0.\n\nWhen n- = n+, the setup is more complex. In order to satisfy the constraint (7), a\nquadratic programming algorithm is needed to obtain the initial configuration. See [7] for\ndetails.\n\n\nKernels\n\nThe development so far has been in the original feature space. It is easy to see that the\nentire development carries through with \"kernels\" as well. In this case f (x) = 0 + g(x),\nand the only change that occurs is that (6) is changed to\n\n n\n 1\n g(xi) = jyjK(xi, xj), i = 1, . . . , n, (11)\n j=1\n\nor j() = jyj/ using the notation in (3). Hereafter we will develop our algorithm for\nthis more general kernel case.\n\n\nThe Path\n\nThe algorithm hinges on the set of points M sitting on the margin. We consider M at the\npoint that an event has occurred:\n\n 1. The initial event, which means 2 or more points start in M, with their initial values\n of [0, 1].\n 2. A point from I has just entered M, with its value of i initially 1.\n 3. A point from O has reentered M, with its value of i initially 0.\n 4. One or more points in M has left the set, to join either O or I.\n\nWhichever the case, for continuity reasons this set will stay stable until the next event\noccurs, since to pass through M, a point's i must change from 0 to 1 or vice versa. Since\nall points in M have yif (xi) = 1, we can establish a path for their i.\n\nWe use the subscript to index the sets above immediately after the th event has occurred.\nSuppose |M | = m, and let , \n i 0 and be the values of these parameters at the point of\nentry. Likewise f is the function at this point. For convenience we define 0 = 0, and\nhence = \n 0 0.\n\nSince \n n\n 1\n f (x) = y \n j j K (x, xj ) + 0 , (12)\n j=1\n\nfor > > +1 we can write\n\n \n f (x) = f (x) - f (x) + f (x)\n \n \n 1\n = ( \n j - )yj K (x, xj ) + (0 - ) + f (x) . (13)\n j 0\n jM\n\n\f\nThe second line follows because all the observations in I have their i = 1, and those in\nO have their i = 0, for this range of . Since each of the m points xi M are to stay\non the margin, we have that\n \n 1 ( \n j - )yiyj K (xi, xj ) + yi(0 - ) + = 1, i M . (14)\n j 0\n jM\n\n\nWriting j = - \n j j , from (14) we have\n\n\n jyiyjK(xi, xj) + yi0 = - , i M . (15)\n jM\n\n\nFurthermore, since at all times n y\n i=1 ii = 0, we have that\n\n\n yjj = 0. (16)\n jM\n\nEquations (15) and (16) constitute m + 1 linear equations in m + 1 unknowns j, and can\nbe solved. The j and hence j will change linearly in , until the next event occurs:\n\n j = - ( - )b\n j j , j {0} M . (17)\n\nSee [7] for more precise details on solving these equations.\n\nFrom (13) we have\n \n f (x) = f (x) - h (x) + h (x), (18)\n \n\nwhere\n h (x) = yjbjK(x, xj) + b0 (19)\n jM\n\nThus the function itself changes in a piecewise-inverse manner in .\n\n\nFinding +1\n\nThe paths continue until one of the following events occur:\n\n 1. One of the i for i M reaches a boundary (0 or 1). For each i the value of \n for which this occurs is easily established.\n\n 2. One of the points in I or O attains yif (xi) = 1.\n\nBy examining these conditions, we can establish the largest < for which an event\noccurs, and hence establish +1 and update the sets.\n\n\nTermination\n\nIn the separable case, we terminate when I becomes empty. At this point, all the i in (4)\nare zero, and further movement increases the norm of unnecessarily.\n\nIn the non-separable case, runs all the way down to zero. For this to happen without f\n\"blowing up\" in (18), we must have f - h = 0, and hence the boundary and margins\nremain fixed at a point where \n i i is as small as possible, and the margin is as wide as\npossible subject to this constraint.\n\n\f\n3.1 Computational Complexity\n\nAt any update event along the path of our algorithm, the main computational burden is\nsolving the system of equations of size m = |M |. While this normally involves O(m3)\ncomputations, since M +1 differs from M by typically one observation, inverse updat-\ning can reduce the computations to O(m2). The computation of h (xi) in (19) requires\nO(nm ) computations. Beyond that, several checks of cost O(n) are needed to evaluate\nthe next move.\n\n\n\n 4\n 3\n 223423\n 1 3\n 1 2\n 4\n 112\n 3\n 1\n 4 1\n 22\n 311\n 3 1\n 2\n 3 1\n 2\n 4 3 1\n 2 1\n 2\n 3 1\n 2 1\n 2 1\n 2 1\n 3 2 111\n 2 111111\n 3 2 1111\n 4 33 2 1111\n 1e+01 2 11\n 4 3 2 1\n 3 22 11111111\n 4 3 2 1\n 2 1111111111\n 2 1\n 2 11111\n 3 2 1\n 2 111111\n 33 2 11\n 2 11\n 33 2 1111\n 2\n 4 2 111\n 3 2 111\n 3 2 11\n 2 1\n 2 11\n 44 3 2 1\n 2 111\n 3 2 1111\n 3 2 111\n 0 11\n 4 33 2 11\n 2 1\n 3 2 1 = 5\n 3 2 1111\n 22 111\n 4 2 1\n 0 1\n 2 111\n 3 1\n 33 2 1\n 2 111\n 3 2 1\n 2 111\n 4 2 1\n 1 1\n 3 2 1111\n 4 3\n 4 3 2 1\n 2\n 3 2 1\n 1 2 1\n 3 2 11\n 1 1 3 2 11\n 4 3 2 111\n 1 1 2 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1 1\n 1\n 1\n 4 3 2 1\n 2 11\n 3 2 1\n 1 1 11111\n 4 33 2 1\n 1 1\n 1\n 1\n 1\n 1 2\n 3 2 1\n 33 22 1\n 1 1 1\n 4 2 11\n 1 1 1\n 1\n 1\n 1\n 1 3 2 1\n 4 2 11\n 3 2 1\n 1 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1 1\n = 5 4 2 1\n 3 2 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n\n 1 1 1\n 3 11\n 1 1 1 1\n 1\n 1 1\n 1\n 1\n 1\n 1 4 2\n 3 2 1\n 2 1\n 3 2 1\n 1\n 1\n 1 1 1 1\n 1\n 1 1 1\n 1\n 1\n 1\n 1\n 1\n 1 1\n 2\n 4 33 2 1\n 1\n 1\n 1 1 1 1 1\n 1\n 1\n 1\n 1\n 1 3 2 1\n 2 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n\n 1 1 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1 3 22 11\n 4 3 2 1\n 0 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1 1e-03 4 3 22 11\n 1\n 1\n 1\n 1\n 1 3 22\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n\n 4 3\n 4 3 2 11\n 4 3 222 1\n alue 3 2 1\n 3 2 1\n 4 3 2 11\n 22 1\n = 1\n 4 333 2 1\n 3 2 1\n 2\n v 4 3 2 1\n 3 2 1\n 2 1\n 4 3\nSize 33 22 1\n 44 3 2 11\n 4 3 22\n 3 222 1\n 4 3 2 1\n 3 1\n 4 33 2\n 33 22 1\n 3 2\n 4 3 2\n 4 3 2 1\n 4 3 2\n 3 2\n 08 2 1\n 4 2\n 3 2\n = 0.5\n 4 2\n = 1 33 2\n 3 2\n 4 3\n 4 3 2\n gin 3 22\n 3 2\n 44 2\n 3 2\n Eigen 1e-07 3 2\n 4 3 22\n 3 2\n 4 33 22\n 2 3\n 2 2 4 3\n 4 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n\n 2 2\n 2 2\n 4 3 22\n 4 3 2\n 3\n = 0.1\n 4 33 222\n 4 333 22\n 2\n 2\n 2\n 2\n 2\n 2\n 2 4 2\n 3\n 4 3\n 06 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n\n 2 4 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2 2\n 2\n 2 3\n 4 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n Mar = 0.5 2\n 4 3 2\n 33 22\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3 4 3 2\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n\n 4 33 22\n 4 33 2\n 4 3 22\n 3\n 44 33\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n\n 2\n 3\n 4 2\n 4 3 2\n 3\n = 0.1 4 3 2\n 44 3 22\n 4 3\n 4 33 2\n 1e-11 2\n 3 2\n 3\n 04 3 2\n 3\n 3\n 3 444 33 2\n 3 22\n 2 3 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n\n\n\n 3 4\n 4\n 4\n 4\n 4\n 4 4\n 4\n 4\n 4\n 4 3 22\n 4 4 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3 2\n 4 3\n 4\n 4\n 4 4 3 2\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4 4 2\n 4 33\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 3\n 4 3\n 4 2\n 4 4 2\n 4 3\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4 4\n 4\n 4\n 4 4 3 2\n 3\n 4\n 4\n 4 4 3\n 44444444444444444444444444444444444444444444444444444444444 3 2\n 4 2\n 4444444444444444444444444444444444444444444444444444\n 4 4 3 2\n 4\n 4\n 4\n 4\n 4 4 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4 3 3 2 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n\n 4\n 4\n 4 4 4 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n\n\n\n\n 3 2\n 333333333333333333333333333333333333333333\n 4 4\n 4\n 4 4\n 4\n 4 4 3\n 3\n 3\n 3\n 3\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n\n 4\n 4\n 4 4 3 222222224\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 3\n 4\n 4\n 0 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n 1\n\n\n\n\n\n 1\n 1\n 1\n 1\n 1 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4 4\n 4\n 4\n 4\n 4 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n 2\n\n\n\n\n\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4\n 4 3\n 1 4 4 2\n\n 1e-15\n 1e-04 1e-02 1e+00 0 50 100 150 200\n Sequence Number\n\n\n\nFigure 2: [Left] The margin sizes |M | as a function of , for different values of the radial-kernel\nparameter . The vertical lines show the positions used to compare the times with libsvm. [Right]\nThe eigenvalues (on the log scale) for the kernel matrices K corresponding to the four values\nof .The larger eigenvalues correspond in this case to smoother eigenfunctions, the small ones to\nrougher. The rougher eigenfunctions get penalized exponentially more than the smoother ones. For\nsmaller values of , the effective dimension of the space is truncated.\n\n\n\nAlthough we have no hard results, our experience so far suggests that the total number \nof moves is O(k min(n+, n-)), for k around 4 - 6; hence typically some small multiple\nc of n. If the average size of M is m, this suggests the total computational burden is\nO(cn2m + nm2), which is similar to that of a single SVM fit.\n\nOur R function SvmPath computes all 632 steps in the mixture example (n+ = n- =\n100, radial kernel, = 1) in 1.44(0.02) secs on a Pentium 4, 2Ghz Linux machine; the svm\nfunction (using the optimized code libsvm, from the R library e1071) takes 9.28(0.06)\nseconds to compute the solution at 10 points along the path. Hence it takes our procedure\nabout 50% more time to compute the entire path, than it costs libsvm to compute a typical\nsingle solution.\n\n\n4 Mixture simulation continued\n\nThe in Figure 1 are the entire collection of change points as described in Section 3. We\nwere at first surprised to discover that not all these sequences achieved zero training errors\non the 200 training data points, at their least regularized fit. In fact the minimal training er-\nrors, and the corresponding values for are summarized in Table 1. It is sometimes argued\nthat the implicit feature space is \"infinite dimensional\" for this kernel, which suggests that\nperfect separation is always possible. The last row of the table shows the effective rank of\nthe 200200 kernel Gram matrix K (which we defined to be the number of singular values\ngreater than 10-12). In general a full rank K is required to achieve perfect separation. This\nrank-deficiency of the Gram matrix has been noted by a number of other authors.\n\nThis emphasizes the fact that not all features in the feature map implied by K are of equal\n\n\f\n 5 1 0.5 0.1\n Training Errors 0 12 21 33\n Effective Rank 200 177 143 76\n\nTable 1: The number of minimal training errors for different values of the radial kernel scale param-\neter , for the mixture simulation example. Also shown is the effective rank of the 200 200 Gram\nmatrix K .\n\n\n\nstature; many of them are shrunk way down to zero. Rephrasing, the regularization in (3)\npenalizes unit-norm features by the inverse of their eigenvalues, which effectively anni-\nhilates some, depending on . Small implies wide, flat kernels, and a suppression of\nwiggly, \"rough\" functions.\n\nWriting (3) in matrix form,\n\n \n min L[y, K] + T K, (20)\n 0, 2\n\nwe reparametrize using the eigen-decomposition of K = UDUT . Let K = U where\n = DUT . Then (20) becomes\n\n \n min L[y, U] + T D-1. (21)\n 0, 2\n\nNow the columns of U are unit-norm basis functions (in R2) spanning the column space\nof K; from (21) we see that those members corresponding to near-zero eigenvalues (the\nelements of the diagonal matrix D) get heavily penalized and hence ignored. Figure 2\nshows the elements of D for the four values of .\n\n\n5 Discussion\n\nOur work on the SVM path algorithm was inspired by early work on exact path algorithms\nin other settings. \"Least Angle Regression\" [8] show that the coefficient path for the se-\nquence of \"lasso\" coefficients is piecewise linear. The lasso uses a quadratic criterion,\nwith an L1 constraint. In fact, any model with an L1 constraint and a quadratic, piecewise\nquadratic, piecewise linear, or mixed quadratic and linear loss function, will have piecewise\nlinear coefficient paths, which can be calculated exactly and efficiently for all values of \n[9]. This includes the L1 SVM [10].\n\nThe SVM model has a quadratic constraint and a piecewise linear (\"hinge\") loss function.\nThis leads to a piecewise linear path in the dual space, hence the Lagrange coefficients i\nare piecewise linear.\n\nOf course, quadratic criterion + quadratic constraints also lead to exact path solutions, as\nin the classic ridge regression case, since a closed form solution is obtained via the SVD.\n\nThe general techniques employed in this paper are known as parametric programming in\nconvex optimization. After completing this work, it was brought to our attention that [11]\nreported on the picewise-linear nature of the lagrange multipliers, although they did not\ndevelop the path algorithm. [12, 13] employ techniques similar to ours in incremental\nlearning for SVMs. These authors do not construct exact paths as we do, but rather focus\non updating and downdating the solutions as more (or less) data arises. [14] allow for\nupdating the parameters as well, but again do not construct entire solution paths.\n\nThe SvmPath has been implemented in the R computing environment, and is available\nfrom the R website.\n\n\f\nAcknowledgements\n\nThe authors thank Jerome Friedman for helpful discussions, and Mee-Young Park for as-\nsisting with some of the computations. Trevor Hastie was partially supported by grant\nDMS-0204162 from the National Science Foundation, and grant RO1-EB0011988-08 from\nthe National Institutes of Health.\n\n\nReferences\n\n [1] B. Boser, I. Guyon, and V. Vapnik. A training algorithm for optimal margin classifiers.\n In Proceedings of COLT II, Philadelphia, PA, 1992.\n\n [2] C. Cortes and V. Vapnik. Support vector networks. Machine Learning, 20:125, 1995.\n\n [3] Bernard Scholkopf and Alex Smola. Learning with Kernels: Support Vector Ma-\n chines, Regularization, Optimization, and Beyond (Adaptive Computation and Ma-\n chine Learning). MIT Press, 2001.\n\n [4] G. Wahba, Y. Lin, and H. Zhang. Gacv for support vector machines. In A.J. Smola,\n P.L. Bartlett, B. Scholkopf, and D. Schuurmans, editors, Advances in Large Margin\n Classifiers, pages 297311, Cambridge, MA, 2000. MIT Press.\n\n [5] T. Hastie, R. Tibshirani, and J. Friedman. The Elements of Statistical Learning; Data\n mining, Inference and Prediction. Springer Verlag, New York, 2001.\n\n [6] Thorsten Joachims. Practical Advances in Kernel Methods -- Support Vector Learn-\n ing, chapter Making large scale SVM learning practical. MIT Press, 1999. see\n http://svmlight.joachims.org.\n\n [7] Trevor Hastie, Saharon Rosset, Robert Tibshirani, and Ji Zhu. The entire regulariza-\n tion path for the support vector machine. Journal of Machine Learning Research,\n (5):13911415, 2004.\n\n [8] B. Efron, T. Hastie, I. Johnstone, and R.. Tibshirani. Least angle regression. Technical\n report, Stanford University, 2002.\n\n [9] Saharon Rosset and Ji Zhu. Piecewise linear regularized so-\n lution paths. Technical report, Stanford University, 2003.\n http://www-stat.stanford.edu/saharon/papers/piecewise.ps.\n[10] Ji Zhu, Saharon Rosset, Trevor Hastie, and Robert Tibshirani. L1 norm support vector\n machines. Technical report, Stanford University, 2003.\n\n[11] Massimiliano Pontil and Alessandro Verri. Properties of support vector machines.\n Neural Comput., 10(4):955974, 1998.\n\n[12] Shai Fine and Katya Scheinberg. Incas: An incremental active set method for svm.\n Technical report, IBM Research Labs, Haifa, 2002.\n\n[13] G. Cauwenberghs and T. Poggio. Incremental and decremental support vector ma-\n chine learning. In Advances in Neural Information Processing Systems (NIPS*2000),\n volume 13. MIT Press, Cambridge, MA, 2001.\n\n[14] Christopher Diehl and Gert Cauwenberghs. Svm incremental learning, adaptation and\n optimization. In Proceedings of the 2003 International Joint Conference on Neural\n Networks, pages 26852690, 2003. Special series on Incremental Learning.\n\n\f\n", "award": [], "sourceid": 2713, "authors": [{"given_name": "Saharon", "family_name": "Rosset", "institution": null}, {"given_name": "Robert", "family_name": "Tibshirani", "institution": null}, {"given_name": "Ji", "family_name": "Zhu", "institution": null}, {"given_name": "Trevor", "family_name": "Hastie", "institution": null}]}