{"title": "Automatic Capacity Tuning of Very Large VC-Dimension Classifiers", "book": "Advances in Neural Information Processing Systems", "page_first": 147, "page_last": 155, "abstract": null, "full_text": "Automatic Capacity Tuning \n\nof Very Large VC-dimension Classifiers \n\nI. Guyon \nAT&T Bell Labs, \n50 Fremont st., 6th floor, \nSan Francisco, CA 94105 \nisabelle@neural.att.com \n\nB. Boser\u00b7 \n\nEECS Department, \n\nUniversity of California, \n\nBerkeley, CA 94720 \n\nboser@eecs.berkeley.edu \n\nV. Vapnik \nAT&T Bell Labs, \nRoom 4G-314, \nHolmdel, NJ 07733 \nv lad@neural.att.com \n\nAbstract \n\nLarge VC-dimension classifiers can learn difficult tasks, but are usually \nimpractical because they generalize well only if they are trained with huge \nquantities of data. In this paper we show that even high-order polynomial \nclassifiers in high dimensional spaces can be trained with a small amount \nof training data and yet generalize better than classifiers with a smaller \nVC-dimension. This is achieved with a maximum margin algorithm (the \nGeneralized Portrait). The technique is applicable to a wide variety of \nclassifiers, including Perceptrons, polynomial classifiers (sigma-pi unit net(cid:173)\nworks) and Radial Basis Functions. The effective number of parameters is \nadjusted automatically by the training algorithm to match the complexity \nof the problem. It is shown to equal the number of those training patterns \nwhich are closest patterns to the decision boundary (supporting patterns). \nBounds on the generalization error and the speed of convergence of the al(cid:173)\ngorithm are given. Experimental results on handwritten digit recognition \ndemonstrate good generalization compared to other algorithms. \n\n1 \n\nINTRODUCTION \n\nBoth experimental evidence and theoretical studies [1] link the generalization of a \nclassifier to the error on the training examples and the capacity of the classifier. \n\n\u00b7Part of this work was done while B. Boser was at AT&T Bell Laboratories. He is now \n\nat the University of California, Berkeley. \n\n147 \n\n\f148 \n\nGuyon, Boser, and Vapnik \n\nClassifiers with a large number of adjustable parameters, and therefore large ca(cid:173)\npacity, likely learn the training set without error, but exhibit poor generalization. \nConversely, a classifier with insufficient capacity might not be able to learn the task \nat all. The goal of capacity tuning methods is to find the optimal capacity which \nminimizes the expected generalization error for a given amount of training data. \n\nCapacity tuning techniques include: starting with a low capacity system and allocat(cid:173)\ning more parameters as needed or starting with an large capacity system and elim(cid:173)\ninating unnecessary adjustable parameters with regularization. The first method \nrequires searching in the space of classifier structures which possibly contains many \nlocal minima. The second method is computationally inefficient since it does not \navoid adjusting a large number of parameters although the effective number of pa(cid:173)\nrameters may be small. \n\nWith the method proposed in this paper, the capacity of some very large VC(cid:173)\ndimension classifiers is adjusted automatically in the process of training. The prob(cid:173)\nlem is formulated as a quadratic programming problem which has a single global \nminimum. Only the effective parameters get adjusted during training which ensures \ncompu tational efficiency. \n\n1.1 MAXIMUM MARGIN AND SUPPORTING PATTERNS \n\nHere is a familiar problem: Given is a limited number of training examples from two \nclasses A and B; find the linear decision boundary which yields best generalization \nperformance. When the training data is scarce, there exists usually many errorless \nseparations (figure 1.1). This is especially true when the dimension of input space \n(i.e. the number of tunable parameters) is large compared to the number of training \nexamples. The question arises which of these solutions to choose? The one solution \nthat achieves the largest possible margin between the decision boundary and the \ntraining patterns (figure 1.2) is optimal in the \"minimax\" sense [2] (see section 2.2). \nThis choice is intuitively justifiable: a new example from class A is likely to fall \nwithin or near the convex envelope of the examples of class A (and similarly for \nclass B). By providing the largest possible \"safety\" margin, we minimize the chances \nthat examples from class A and B cross the border to the wrong side. \n\nAn important property of the maximum margin solution is that it is only depen(cid:173)\ndent upon a restricted number of training examples, called supporting patterns (or \ninformative patterns). These are those examples which lie on the margin and there(cid:173)\nfore are closest to the decision boundary (figure 1.2). The number m of linearly \nindependent supporting patterns satisfies the inequality: \n\nm ~ min(N + 1,p). \n\n(1) \nIn this inequality, (N + 1) is the number of adjustable parameters and equals the \nVapnik-Chervonenkis dimension (VC-dimension) [2], and p is the number of training \nexamples. In reference [3], we show that the generalization error is bounded by m/p \nand therefore m is a measure of complexity of the learning problem. Because m is \nbounded by p and is generally a lot smaller than p, the maximum margin solution \nobtains good generalization even when the problem is grossly underdetermined, \ni.e. the number of training patterns p is much smaller than the number of adjustable \nparameters, N + 1. In section 2.3 we show that the existence of supporting patterns \nis advantageous for computational reasons as well. \n\n\fAutomatic Capacity Tuning of Very Large VC-dimension Classifiers \n\n149 \n\n-.-\n\nA \n\nx \n\n8 \n\u2022 \n\u2022\u2022 \u2022 \n\nXi \n\n--.-\n\nA \n\n-\n\nII \n\n8 \n\n(1) \n\n(2) \n\nFigure 1: Linear separations. \n(1) When many linear decision rules separate the training set, which one to choose? \n(2) The maximum margin solution. The distance to the decision boundary of the \nclosest training patterns is maximized. The grey shading indicates the margin area \nin which no pattern falls. The supporting patterns (in white) lie on the margin. \n\n1.2 NON-LINEAR CLASSIFIERS \n\nAlthough algorithms that maximize the margin between classes have been known \nfor many years [4, 2], they have for computational reasons so far been limited to the \nspecial case of finding linear separations and consequently to relatively simple clas(cid:173)\nsification problems. In this paper, we present an extension to one of these maximum \nmargin training algorithms called the \"Generalized Portrait Method\" (G P) [2] to \nvarious non-linear classifiers, including including Perceptrons, polynomial classifiers \n(sigma-pi unit networks) and kernel classifiers (Radial Basis Functions) (figure 2). \nThe new algorithm trains efficiently very high VC-dimension classifiers with a huge \nnumber of tunable parameters. Despite the large number of free parameters, the \nsolution exhibits good generalization due to the inherent regularization of the max(cid:173)\nimum margin cost function. \n\nAs an example, let us consider the case of a second order polynomial classifier. Its \ndecision surface is described by the following equation: \n\n2::::: WiXi + 2::::: WijXiXj + b = O. \n\ni \n\ni,j \n\n(2) \n\nhe Wi, Wij and b are adjustable parameters, and Xi are the coordinates of a pattern \nx. If n is the dimension of input pattern x, the number of adjustable parameters \nof the second order polynomial classifier is [n( n + 1 )/2] + 1. In general, the number \nof adjustable parameters of a qth order polynomial is of the order of N ~ n q \u2022 \nThe G P algorithm has been tested on the problem of handwritten digit recognition. \nThe input patterns consist of 16 X 16 pixel images (n = 256). The results achieved \n\n\f150 \n\nGuyon, Boser, and Vapnik \n\n256 \n3.104 \n8.10 7 \n4.109 \n1 . 1012 \n\n10.5 0 \n5.8% \n5.2% \n4.9% \n5.2% \n\nTable 1: Handwritten digit recognition experiments. The first database \n(DB1) consists of 1200 clean images recorded from ten subjects. Half of this data \nis used for training, and the other half is used to evaluate the generalization per(cid:173)\nformance. The other database (DB2) consists of 7300 images for training and 2000 \nfor testing and has been recorded from actual mail pieces. We use ten polynomial \nclassification functions of order q, separating one class against all others. We list the \nnumber N of adjustable parameters, the error rates on the test set and the average \nnumber <m>of supporting patterns per separating hypersurface. The results com(cid:173)\npare favorably to neural network classifiers which minimize the mean squared error \nwith backpropagation. For the one layer network (linear classifier),the error on the \ntest set is 12.7 % on DB1 and larger than 25 % on DB2. The lowest error rate for \nDB2, 4.9 %, obtained with a forth order polynomial, is comparable to the 5.1 % \nerror obtained with a multi-layer neural network with sophisticated architecture \nbeing trained and tested on the same data [6]. \n\nwith polynomial classifiers of order q are summarized in table 1. Also listed is \nthe number of adjustable parameters, N. This quantity increases rapidly with q \nand quickly reaches a level that is computationally intractable for algorithms thdt \nexplicitly compute each parameter [5]. Moreover, as N increases, the learning prob(cid:173)\nlem becomes grossly underdetermined: the number of training patterns (p = 600 \nfor DB1 and p = 7300 for DB2) becomes very small compared to N. Nevertheless, \ngood generalization is achieved as shown by the experimental results listed in the \ntable. This is a consequence of the inherent regularization of the algorithm. \n\nAn important concern is the sensitivity of the maximum margin solution to the \npresence of outliers in the training data. It is indeed important to remove undesired \noutliers (such as meaningless or mislabeled patterns) to get best generalization \nperformance. Conversely, \"good\" outliers (such as examples of rare styles) must be \nkept. Cleaning techniques have been developed based on the re-examination by a \nhuman supervisor of those supporting patterns which result in the largest increase of \nthe margin when removed, and thus, are the most likely candidates for outliers [3]. \nIn our experiments on DB2 with linear classifiers, the error rate on the test set \ndropped from 15.2% to 10.5% after cleaning the training data (not the test data). \n\n2 ALGORITHM DESIGN \n\nThe properties of the G P algorithm arise from merging two separate ideas: Training \nin dual space, and minimizing the maximum loss. For large VC-dimension classifiers \n(N ~ p), the first idea reduces the number of effective parameters to be actually \n\n\fAutomatic Capacity Tuning of Very Large VC-dimension Classifiers \n\n151 \n\ncomputed from N to p. The second idea reduces it from p to m. \n\n2.1 DUALITY \n\nWe seek a decision function for pattern vectors x of dimension n belonging to either \nof two classes A and B. The input to the training algorithm is a set of p examples \nXi with labels Yi: \n\n(3) \n\nwhere {Yk = 1 \n\nYk =-1 \n\nif Xk E class A \nif Xk E class B. \n\nFrom these training examples the algorithm finds the parameters of the decision \nfunction D(x) during a learning phase. After training, the classification of unknown \npatterns is predicted according to the following rule: \n\nx E A if D(x) > 0 \nx E B otherwise. \n\n(4) \n\nWe limit ourselves to classifiers linear in their parameters, but not restricted to \nlinear dependences in their input components, such as Perceptrons and kernel-based \nclassifiers. Perceptrons [5] have a decision function defined as: \n\nD(x) = w . <p(x) + b = L Wi<Pi(X) + b, \n\nN \n\ni=l \n\n(5) \n\nwhere the <Pi are predefined functions of x, and the Wi and b are the adjustable \nparameters of the decision function. This definition encompasses that of polynomial \nclassifiers. In that particular case, the <Pi are products of components of vector x(see \nequation 2). Kernel-based classifiers, have a decision function defined as: \n\np \n\nD(x) = L CtkI\u00abXk, x) + b, \n\nk=l \n\n(6) \n\nThe coefficients Ctk and the bias b are the parameters to be adjusted and the Xk \nare the training patterns. The function I< is a predefined kernel, for example a \npotential function [7] or any Radial Basis Function (see for instance [8]). \n\nPerceptrons and RBF's are often considered two very distinct approaches to classifi(cid:173)\ncation. However, for a number of training algorithms, the resulting decision function \ncan be cast either in the form of equation (5) or (6). This has been pointed out \nin the literature for the Perceptron and potential function algorithms [7], for the \npolynomial classifiers trained with pseudo-inverse [9] and more recently for regular(cid:173)\nization algorithms and RBF's [8]. In those cases, Perceptrons and RBF's constitute \ndual representations of the same decision function. \n\nThe duality principle can be understood simply in the case of Heb b 's learning rule. \nThe weight vector of a linear Perceptron (<pi(X) = Xi), trained with Hebb's rule, is \nsimply the average of all training patterns Xk, multiplied by their class membership \npolarity Yk: \n\n1 p \n\nw = - LYkXk . \n\nP k=l \n\n\f152 \n\nGuyon, Boser, and Vapnik \n\nSubstituting this solution into equation (5), we obtain the dual representation \n\nD(x) = w\u00b7 x + b = - 2: Yk Xk . X + b . \n\n1 p \n\nP k=l \n\nThe corresponding kernel classifier has kernel K(x, x') = X\u00b7 x' and the dual param(cid:173)\neters ctk are equal to (l/p)Yk. \nIn general, a training algorithm for Perceptron classifiers admits a dual kernel rep(cid:173)\nresentation if its solution is a linear combination of the training patterns in ip-space: \n\np \n\nw = L ctkip(Xk) . \n\nk=l \n\n(7) \n\nReciprocally, a kernel classifier admits a dual Perceptron representation if the kernel \nfunction possesses a finite (or infinite) expansion of the form: \n\nK(x, x') = L ipi(X) ipi(X/) . \n\n(8) \n\nSuch is the case for instance for some symmetric kernels [10]. \nthat we have been using include \n\nExamples of kernels \n\nK(x, x') \nK(x, x') \nK(x, x') \nK(x, x') \nK(x, x') \nK(x, x') \n\n(x. x, + l)q \ntanh (1' x . x') \nexp (1' x . x') - 1 \nexp (-lIx - x/1l2/-y) \nexp (-lIx - x/ll/-Y) \n(x. x' + l)q exp (-llx - x/ll/-Y) \n\n(polynomial of order q), \n(neural units), \n(exponential) , \n(gaussian RBF), \n(exponential RBF), \n(mixed polynomial & RBF). \n\n(9) \nThese kernels have positive parameters (the integer q or the real number -y) which \ncan be determined with a Structural Risk Minimization or Cross-Validation proce(cid:173)\ndure (see for instance [2]). More elaborate kernels incorporating known invariances \nof the data could be used also. \nThe G P algorithm computes the maximum margin solution in the kernel represen(cid:173)\ntation. This is crucial for making the computation tractable when training very \nlarge VC-dimension classifiers. Training a classifier in the kernel representation is \ncomputationally advantageous when the dimension N of vectors w (or the VC(cid:173)\ndimension N + 1) is large compared to the number of parameters ctk, which equals \nthe number of training patterns p. This is always true if the kernel function pos(cid:173)\nsesses an infinite expansions (8). The experimental results listed in table 1 indicate \nthat this argument holds in practice even for low order polynomial expansions when \nthe dimension n of input space is sufficiently large. \n\n2.2 MINIMIZING THE MAXIMUM LOSS \n\nThe margin, defined as the Euclidean distance between the decision boundary and \nthe closest training patterns in ip-space can be computed as \n\n(10) \n\n\fAutomatic Capacity Tuning of Very Large VC-dimension Classifiers \n\n153 \n\nThe goal of the maximum margin training algorithm is to find the decision function \nD(x) which maximizes M, that is the solution of the optimization problem \n\n. YkD(Xk) \n\nm~x~n IIwll \n\n. \n\n(11) \n\nThe solution w of this problem depends only on those patterns which are on the \nmargin, i.e. the ones that are closest to the decision boundary, called supporting \npatterns. It can be shown that w can indeed be represented as a linear combination \nof the supporting patterns in ip-space [4, 2, 3] (see section 2.3). \n\nIn the classical framework of loss minimization, problem 11 is equivalent to mini(cid:173)\nmizing (over w) the maximum loss. The loss function is defined as \n\nThis \"minimax\" approach contrasts with training algorithms which minimize the \naverage loss. For example, backpropagation minimizes the mean squared error \n(MSE), which is the average of \n\nThe benefit of minimax algorithms is that the solution is a function only of a \nrestricted number of training patterns, namely the supporting patterns. This results \nin high computational efficiency in those cases when the number m of supporting \npatterns is small compared to both the total number of training patterns p and the \ndimension N of ip-space. \n\n2.3 THE GENERALIZED PORTRAIT \n\nThe G P algorithm consists in formulating the problem 11 in the dual a-space as \nthe quadratic programming problem of maximizing the cost function \n\nJ(a, b) = L ak (1- bYk) - -a . H . a, \n\n1 \n\n2 \n\np \n\nk=l \n\nunder the constrains ak > 0 [4, 2]. The p x p square matrix H has elements: \n\nHkl = YkYIK(Xk,Xl). \n\nwhere K(x, x') is a kernel, such as the ones proposed in (9), which can be expanded \nas in (8). Examples are shown in figure 2. K(x, x') is not restricted to the dot \nproduct K(x, x') = x . x' as in the original formulation of the GP algorithm [2]. \nIn order for a unique solution to exist, H must be positive definite. The bias b can \nbe either fixed or optimized together with the parameters ak. This case introduces \nanother set of constraints: Ek Ykak = 0 [4]. \nThe quadratic programming problem thus defined can be solved efficiently by stan(cid:173)\ndard numerical methods [11]. Numerical computation can be further reduced by \nprocessing iteratively small chunks of data [2]. The computational time is linear the \ndimension n of x-space (not the dimension N of ip-space) and in the number p of \ntraining examples and polynomial in the number m < min(N + 1,p) of supporting \n\n\f154 \n\nGuyon, Boser, and Vapnik \n\nXi \n\n~!;~;t;.':<l \n\n'.r-\n\nX \n\n\u2022 \u2022 A \n\u2022\u2022 \n\u2022 \n\n\u2022 \n\n\u2022 \n\n\u2022 \n8 \n\u2022 \n\u2022\u2022\u2022 \n\n\u2022 \n\u2022 \n\n(1 ) \n\n\u2022\u2022 A \n\u2022\u2022 \n\u2022 \n\n\u2022 \n\n\u2022 \n\u2022 \n\n(2) \n\n\u2022 \n\n\u2022 \n8 \n\u2022 \n\u2022 \u2022\u2022 \n\nFigure 2: Non-linear separations. \nDecision boundaries obtained by maximizing the margin in ip-space (see text). The \ngrey shading indicates the margin area projected back to x-space. The supporting \npatterns (white) lie on the margin. (1) Polynomial classifier of order two (sigma-pi \nunit network), with kernel K(x, x') = (x. x' + 1)2. (2) Kernel classifier (RBF) with \nkernel K(x,x) = (exp -llx - x'lI/lO). \n\npatterns. It can be theoretically proven that it is a polynomial in m of order lower \nthan 10, but experimentally an order 2 was observed. \nOnly the supporting patterns appear in the solution with non-zero weight a'k: \n\nD(x) = LYka'kK(Xk, x) + h, \n\nk \n\nSubstituting (8) in D(x), we obtain: \n\nW = LYka'kip(Xk) . \n\nk \n\n(12) \n\n(13) \n\nUsing the kernel representation, with a factorized kernel (such as 9), the classifica(cid:173)\ntion time is linear in n (not N) and in m (not p). \n\n3 CONCLUSIONS \n\nWe presented an algorithm to train in high dimensional spaces polynomial classifiers \nand Radial Basis functions which has remarquable computational and generaliza(cid:173)\ntion performances. The algorithms seeks the solution with the largest possible \nmargin on both side of the decision boundary. The properties of the algorithm arise \nfrom the fact that the solution is a function only of a small number of supporting \npatterns, namely those training examples that are closest to the decision boundary. \nThe generalization error of the maximum margin classifier is bounded by the ratio \n\n\fAutomatic Capacity Tuning of Very Large VC-dimension Classifiers \n\n155 \n\nof the number of linearly independent supporting patterns and the number of train(cid:173)\ning examples. This bound is tighter than a bound based on the VC-dimension of \nthe classifier family. For further improvement of the generalization error, outliers \ncorresponding to supporting patterns with large elk can be eliminated automati(cid:173)\ncally or with the assistance of a supervisor. This feature suggests other interesting \napplications of the maximum margin algorithm for database cleaning. \n\nAcknowledgements \n\nWe wish to thank our colleagues at UC Berkeley and AT&T Bell Laboratories for \nmany suggestions and stimulating discussions. Comments by L. Bottou, C. Cortes, \nS. Sanders, S. Solla, A. Zakhor, are gratefully acknowledged. We are especially in(cid:173)\ndebted to R. Baldick and D. Hochbaum for investigating the polynomial convergence \nproperty, S. Hein for providing the code for constrained nonlinear optimization, and \nD. Haussler and M. Warmuth for help and advice regarding performance bounds. \n\nReferences \n\n[1] 1. Guyon, V. Vapnik, B. Boser, L. Bottou, and S.A. Solla. Structural risk \nminimization for character recognition. In J. Moody and et aI., editors, NIPS \n4, San Mateo CA, 1992. IEEE, Morgan Kaufmann. \n\n[2] V.N. Vapnik. Estimation of dependences based on empirical data. Springer, \n\nNew York, 1982. \n\n[3] B. Boser, 1. Guyon, and V. Vapnik. An training algorithm for optimal mar(cid:173)\n\ngin classifiers. In Fifth Annual Workshop on Computational Learning Theory, \npages 144-152, Pittsburgh, July 1992. ACM. \n\n[4] P. F. Lambert. Designing patterns recognizers with extremal paradigm infor(cid:173)\n\nmation. In Watanabe S., editor, Methodologies of Pattern Recognition, pages \n359-391. Academic Press, 1969. \n\n[5] R.O. Duda and P.E. Hart. Pattern Classification And Scene Analysis. Wiley \n\nand Son, 1973. \n\n[6] Y. Le Cun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, \n\nand 1. D. Jackel. Back-propagation applied to handwritten zipcode recogni(cid:173)\ntion. Neural Computation, 1(4):541-551, 1989. \n\n[7] M.A. Aizerman, E.M. Braverman, and L.I. Rozonoer. Theoretical foundations \nof the potential function method in pattern recognition learning. Automation \nand Remote Control, 25:821-837, 1964. \n\n[8] T. Poggio and F. Girosi. Regularization algorithms for learning that are equiv(cid:173)\n\nalent to multilayer networks. Science, 247:978 - 982, February 1990. \n\n[9] T. Poggio. On optimal nonlinear associative recall. Bioi. Cybern., 19:201,1975. \n[10] G.F Roach. Green's Functions. Cambridge University Press, Cambridge, 1982 \n\n(second ed.). \n\n[11] D. Luenberger. Linear and Non-linear Programming. Addidon Wesley, 1984. \n\n\f", "award": [], "sourceid": 653, "authors": [{"given_name": "I.", "family_name": "Guyon", "institution": null}, {"given_name": "B.", "family_name": "Boser", "institution": null}, {"given_name": "V.", "family_name": "Vapnik", "institution": null}]}