{"title": "Overfitting in Neural Nets: Backpropagation, Conjugate Gradient, and Early Stopping", "book": "Advances in Neural Information Processing Systems", "page_first": 402, "page_last": 408, "abstract": null, "full_text": "Overfitting in Neural Nets: Backpropagation, \n\nConjugate Gradient, and Early Stopping \n\nRich Caruana \nCALD,CMU \n\n5000 Forbes Ave. \n\nPittsburgh, PA 15213 \ncaruana@cs.cmu.edu \n\nSteve Lawrence \n\nNEC Research Institute \n4 Independence Way \nPrinceton, NJ 08540 \n\nLee Giles \n\nInformation Sciences \nPenn State University \n\nUniversity Park, PA 16801 \n\nlawrence@ research. nj. nec. com \n\ngiles@ist.psu.edu \n\nAbstract \n\nThe conventional wisdom is that backprop nets with excess hidden units \ngeneralize poorly. We show that nets with excess capacity generalize \nwell when trained with backprop and early stopping. Experiments sug(cid:173)\ngest two reasons for this: 1) Overfitting can vary significantly in different \nregions of the model. Excess capacity allows better fit to regions of high \nnon-linearity, and backprop often avoids overfitting the regions of low \nnon-linearity. 2) Regardless of size, nets learn task subcomponents in \nsimilar sequence. Big nets pass through stages similar to those learned \nby smaller nets. Early stopping can stop training the large net when it \ngeneralizes comparably to a smaller net. We also show that conjugate \ngradient can yield worse generalization because it overfits regions of low \nnon-linearity when learning to fit regions of high non-linearity. \n\nIntroduction \n\n1 \nIt is commonly believed that large multi-layer perceptrons (MLPs) generalize poorly: nets \nwith too much capacity overfit the training data. Restricting net capacity prevents overfit(cid:173)\nting because the net has insufficient capacity to learn models that are too complex. This \nbelief is consistent with a VC-dimension analysis of net capacity vs. generalization: the \nmore free parameters in the net the larger the VC-dimension of the hypothesis space, and \nthe less likely the training sample is large enough to select a (nearly) correct hypothesis [2]. \n\nOnce it became feasible to train large nets on real problems, a number of MLP users noted \nthat the overfitting they expected from nets with excess capacity did not occur. Large nets \nappeared to generalize as well as smaller nets -\nsometimes better. The earliest report \nof this that we are aware of is Martin and Pittman in 1991: \"We find only marginal and \ninconsistent indications that constraining net capacity improves generalization\" [7]. \n\nWe present empirical results showing that MLPs with excess capacity often do not over(cid:173)\nfit. On the contrary, we observe that large nets often generalize better than small nets of \nsufficient capacity. Backprop appears to use excess capacity to better fit regions of high \nnon-linearity, while still fitting regions of low non-linearity smoothly. (This desirable be(cid:173)\nhavior can disappear if a fast training algorithm such as conjugate gradient is used instead \nof backprop.) Nets with excess capacity trained with backprop appear first to learn models \nsimilar to models learned by smaller nets. If early stopping is used, training of the large net \ncan be halted when the large net's model is similar to models learned by smaller nets. \n\n\fApprOXlUlat l o D \nT rain i ng Data \nT urgel Functi on '-Vilhau! Noise \n\n-\n\n>< \n\nx \n\nApproxunatlo n \nT rain i ng Data \nT urg .. t Functi on 'Without Noise \n\n- 1 \n\n- 1 \n\nOrder 10 \n\nOrder 20 \n\n15~----------~----------~ \n\nTraining Data \n>< TurS .. 1 Functi on '\\Vi l h out N oise \n\n>< \n\nApprOXLnlutlon -\n\n- 1 5 O~-----------'~O----------~20 \n\n- 1 5 O~-----------'~O----------~20 \n\n10 Hidden Nodes \n\n50 Hidden Nodes \n\nFigure 1: Top: Polynomial fit to data from y = sin( x /3) + v . Order 20 overfits. Bottom: Small and \nlarge MLPs fit to same data. The large MLP does not overfit significantly more than the small MLP. \n\n2 Overfitting \nMuch has been written about overfitting and the bias/variance tradeoff in neural nets and \nother machine learning models [2, 12, 4, 8, 5, 13, 6] . The top of Figure 1 illustrates \npolynomial overfitting. We created a training dataset by evaluating y = sin( x /3) + lJ \nat 0 1 I I 2, ... ,20 where lJ is a uniformly distributed random variable between -0.25 and \n0.25. We fit polynomial models with orders 2-20 to the data. Underfitting occurs with \norder 2. The fit is good with order 10. As the order (and number of parameters) increases, \nhowever, significant overfitting (poor generalization) occurs. At order 20, the polynomial \nfits the training data well, but interpolates poorly. \n\nThe bottom of Figure 1 shows MLPs fit to the data. We used a single hidden layer MLP, \nbackpropagation (BP), and 100,000 stochastic updates. The learning rate was reduced \nlinearly to zero from an initial rate of 0.5 (reducing the learning rate improves convergence, \nand linear reduction performs sintilarly to other schedules [3]). This schedule and number \nof updates trains the MLPs to completion. (We examine early stopping in Section 4.) As \nwith polynomials, the smallest net with one hidden unit (HU) (4 weights weights) underfits \nthe data. The fit is good with two HU (7 weights). Unlike polynomials, however, networks \nwith 10 HU (31 weights) and 50 HU (151 weights) also yield good models. MLPs with \nseven times as many parameters as data points trained with BP do not significantly overfit \nthis data. The experiments in Section 4 confirm that this bias of BP-trained MLPs towards \nsmooth models is not limited to the simple 2-D problem used here. \n\n3 Local Overfitting \nRegularization methods such as weight decay typically assume that overfitting is a global \nphenomenon. But overfitting can vary significantly in different regions of a model. Figure \n2 shows polynomial fits for data generated from the following equation: \n\n- cos( x) + v \ncos(3(x - iT)) + V \n\n0 :::; x < iT \niT:::; X :::; 2iT \n\nY = \n{ \nFive equally spaced points were generated in the first region, and 15 in the second region, \nso that the two regions have different data densities and different underlying functions. \nOverfitting is different in the two regions. In Figure 2 the order 6 model fits the left region \n\n(Equation 1) \n\n\fO roe r 2 Approxim .. tio n \nTrain ing Dat a \nT .. r get Fu n c tio n 'Withou t No,,.., \n\n-\n\n+ \n\nOroe r I'i Approxi m .. t ion \nT raining D a . a \nT .. r ge. Fu n ction 'W ithou . Noio.c \n\n-\n\n+ \n\nOrder 2 \n\nOrder 6 \n\nOroer 10 A.t~..i~\\,',';~''::;~ - . -\n\nT arget Fu n c tio n 'Withou. No !o.c \n\nOroer ll'i ~t~..i~i:,';~':'~ - . -\n\nT arget Fu n c tio n \"\"ithoU! Noise \n\n'oL---~--~--~----~--~--~U \n\nOrder 10 \n\nOrder 16 \n\nFigure 2: Polynomial approximation of data from Equation 1 as the order of the model is increased \nfrom 2 to 16. The overfitting behavior differs in the left and right hand regions. \n\nwell, but larger models overfit it. The order 6 model underfits the region on the right, and \nthe order 10 model fits it better. No model performs well on both regions. Figure 3 shows \nMLPs trained on the same data (20,000 batch updates, learning rate linearly reduced to \nzero starting at 0.5). Small nets underfit. Larger nets, however, fit the entire function well \nwithout significant overfitting in the left region. \n\nThe ability of MLPs to fit both regions of low and high non-linearity well (without over(cid:173)\nfitting) depends on the training algorithm. Conjugate gradient (CG) is the most popular \nsecond order method. CG results in lower training error for this problem, but overfits sig(cid:173)\nnificantly. Figure 4 shows results for 10 trials for BP and CG. Large BP nets generalize \nbetter on this problem -- even the optimal size CG net is prone to overfitting. The degree \nof overfitting varies in different regions. When the net is large enough to fit the region of \nhigh non-linearity, overfitting is often seen in the region of low non-linearity. \n\n4 Generalization, Network Capacity, and Early Stopping \nThe results in Sections 2 and 3 suggest that BP nets are less prone to overfitting than \nexpected. But MLPs can and do overfit. This section examines overfitting vs. net size \non seven problems: NETtalk [10], 7 and 12 bit parity, an inverse kinematic model for a \nrobot arm (thanks to Sebastian Thrun for the simulator), Base 1 and Base 2: two sonar \nmodeling problems using data collected from a robot wondering hallways at CMU, and \nvision data used to learn to steer an autonomous car [9]. These problems exhibit a variety \nof characteristics. Some are Boolean. Others are continuous. Some have noise. Others are \nnoise-free. Some have many inputs or outputs. Others have few inputs or outputs. \n4.1 Results \nFor each problem we used small training sets (100-1000 points, depending on the problem) \nso that overfitting was possible. We trained fully connected feedforward MLPs with one \nhidden layer whose size varied from 2 to 800 HU (about 500-100,000 parameters). All the \nnets were trained with BP using stochastic updates, learning rate 0.1, and momentum 0.9. \n\nWe used early stopping for regularization because it doesn't interfere with backprop's abil(cid:173)\nity to control capacity locally. Early stopping combined with backprop is so effective that \nvery large nets can be trained without significant overfitting. Section 4.2 explains why. \n\n\f/ \n\n-\n\n\" -\n\n\"'/ \n\n1 Hidden Unit \n\n4 Hidden Units \n\n10 Hidden Units \n\n100 Hidden Units \n\nFigure 3: MLP approximation using backpropagation (BP) training of data from Equation 1 as the \nnumber of hidden units is increased. No significant overfitting can be seen. \n\n07 \n\n06 \n\n05 \n\n0 4 \n\nOJ \n\n02 \n\n01 \n\n0 7 \n\n06 \n\n05 \n\n0 4 \n\nOJ \n\n0 2 \n\n0 1 \n\nOJ \n\n'\" ::l \nZ \n~ \n\n25 \n\n50 \n\n5 \n\n10 \n\nNumbe! of Hidden Ncdes \n\n5 \n\n10 \n\n25 \n\n50 \n\nNumbei cI Hidden Nodes \n\nFigure 4: Test Normalized Mean Squared Error for MLPs trained with BP (left) and CG (right). \nResults are shown with both box-whiskers plots and the mean plus and minus one standard deviation. \n\nFigure 5 shows generalization curves for four of the problems. Examining the results for \nall seven problems, we observe that on only three (Base 1, Base 2, and ALVINN), do \nnets that are too large yield worse generalization than smaller networks, but the loss is \nsurprisingly small. Many trials were required before statistical tests confirmed that the \ndifferences between the optimal size net and the largest net were significant. Moreover, the \nresults suggest that generalization is hurt more by using a net that is a little too small than \nby using one that is far too large, i.e., it is better to make nets too large than too small. \n\nFor most tasks and net sizes, we trained well beyond the point where generalization per(cid:173)\nformance peaked. Because we had complete generalization curves, we noticed something \nunexpected. On some tasks, small nets overtrained considerably. The NETtalk graph in \nFigure 5 is a good example. Regularization (e.g., early stopping) is critical for nets of all \nsizes -\n\nnot just ones that are too big. Nets with restricted capacity can overtrain. \n\n4.2 Why Excess Capacity Does Not Hurt \nBP nets initialized with small weights can develop large weights only after the number of \nupdates is large. Thus BP nets consider hypotheses with small weights before hypotheses \nwith large weights. Nets with large weights have more representational power, so simple \nhypotheses are explored before complex hypotheses. \n\n\f0.17 _ -..... - - , - -..... - - , - - - - , \n\nNETtalk \n\n0.2 .----..... - - , - -..... - - , - - - - , \n\nInverse Kinematics \n\n0.16 \n\n0.15 \n\n0.14 \n\n0.13 \n\n0 . 12 ' - - - - - ' - - - ' - - - - - ' - - - ' - - - - ' \n\no \n\n100000 200000 300000 400000 500000 \n\nPattern Presentations \n\nBase 1: Average of 10 Runs \n\no . 15 r