{"title": "On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks", "book": "Advances in Neural Information Processing Systems", "page_first": 13888, "page_last": 13899, "abstract": "Mixup~\\cite{zhang2017mixup} is  a recently proposed  method for training deep neural networks  where additional samples are generated during training  by convexly combining random pairs of images and their associated labels. While simple to implement, it has shown to be a surprisingly effective method of data augmentation for image classification;  DNNs trained with mixup show noticeable gains in classification performance on a number of  image classification benchmarks. In this work, we discuss a hitherto untouched aspect of mixup training -- the calibration and predictive uncertainty   of models trained with mixup. We find that DNNs trained with mixup  are significantly better calibrated --  i.e the predicted softmax scores  are  much better indicators of the actual likelihood of a correct prediction --  than DNNs trained in the regular fashion. We conduct experiments on a number of image classification architectures and datasets --  including large-scale datasets like ImageNet -- and find this to be the case.  \n\tAdditionally, we find that merely mixing features does not result in the same calibration benefit and that the label smoothing in mixup training plays a significant role in improving calibration.  Finally,  we also observe that mixup-trained DNNs are less prone to over-confident predictions on out-of-distribution and random-noise data.  We conclude that the  typical overconfidence seen in neural networks, even on in-distribution data is likely a consequence of training with hard labels, suggesting that mixup training be employed for classification tasks where predictive uncertainty is a significant concern.", "full_text": "On Mixup Training: Improved Calibration and\nPredictive Uncertainty for Deep Neural Networks\n\nSunil Thulasidasan\u2217\u2217,1,2, Gopinath Chennupati1, Jeff Bilmes2,\n\nTanmoy Bhattacharya1, Sarah Michalak1\n\n2Department of Electrical and Computer Engineering, University of Washington\n\n1Los Alamos National Laboratory\n\nAbstract\n\nMixup [37] is a recently proposed method for training deep neural networks where\nadditional samples are generated during training by convexly combining random\npairs of images and their associated labels. While simple to implement, it has shown\nto be a surprisingly effective method of data augmentation for image classi\ufb01cation:\nDNNs trained with mixup show noticeable gains in classi\ufb01cation performance on\na number of image classi\ufb01cation benchmarks. In this work, we discuss a hitherto\nuntouched aspect of mixup training \u2013 the calibration and predictive uncertainty of\nmodels trained with mixup. We \ufb01nd that DNNs trained with mixup are signi\ufb01cantly\nbetter calibrated \u2013 i.e the predicted softmax scores are much better indicators of\nthe actual likelihood of a correct prediction \u2013 than DNNs trained in the regular\nfashion. We conduct experiments on a number of image classi\ufb01cation architectures\nand datasets \u2013 including large-scale datasets like ImageNet \u2013 and \ufb01nd this to be\nthe case. Additionally, we \ufb01nd that merely mixing features does not result in the\nsame calibration bene\ufb01t and that the label smoothing in mixup training plays a\nsigni\ufb01cant role in improving calibration. Finally, we also observe that mixup-\ntrained DNNs are less prone to over-con\ufb01dent predictions on out-of-distribution\nand random-noise data. We conclude that the typical overcon\ufb01dence seen in neural\nnetworks, even on in-distribution data is likely a consequence of training with hard\nlabels, suggesting that mixup training be employed for classi\ufb01cation tasks where\npredictive uncertainty is a signi\ufb01cant concern.\n\n1\n\nIntroduction: Overcon\ufb01dence and Uncertainty in Deep Learning\n\nMachine learning algorithms are replacing or expected to increasingly replace humans in decision-\nmaking pipelines. With the deployment of AI-based systems in high risk \ufb01elds such as medical\ndiagnosis [24], autonomous vehicle control [20] and the legal sector [1], the major challenges of\nthe upcoming era are thus going to be in issues of uncertainty and trust-worthiness of a classi\ufb01er.\nWith deep neural networks having established supremacy in many pattern recognition tasks, it is the\npredictive uncertainty of these types of classi\ufb01ers that will be of increasing importance. The DNN\nmust not only be accurate, but also indicate when it is likely to get the wrong answer. This allows\nthe decision-making to be routed to a human or another more accurate, but possibly more expensive,\nclassi\ufb01er, with the assumption being that the additional cost incurred is greatly surpassed by the\nconsequences of a wrong prediction.\nFor this reason, quantifying the predictive uncertainty and con\ufb01dence calibration for deep neural\nnetworks has seen increased attention in recent years ([6, 18, 7, 9, 19, 15, 28]). One of the \ufb01rst works\nto examine the issue of calibration for modern neural networks was [9]; in a well-calibrated classi\ufb01er,\n\n\u2217Correspondence to sunil@lanl.gov\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\f(a)\n\n(b)\n\n(c)\n\n(d)\n\n(e)\n\n(f)\n\n(g)\n\n(h)\n\n(i)\n\n(j)\n\nFigure 1: Joint density plot of accuracy vs con\ufb01dence (captured by the winning softmax score) on\nthe CIFAR-100 validation set at different training epochs for the VGG-16 deep neural network. Top\nRow: In regular training, the DNN moves from under-con\ufb01dence, at the beginning of training, to\novercon\ufb01dence at the end. A well-calibrated classi\ufb01er would have most of the density lying on\nthe x = y gray line. Bottom Row: Training with mixup on the same architecture and dataset. At\ncorresponding epochs, the network is much better calibrated.\n\npredictive scores should be indicative of the actual likelihood of correctness. The authors in [9]\nshow signi\ufb01cant empirical evidence that modern deep neural networks are poorly calibrated, with\ndepth, weight decay and batch normalization all in\ufb02uencing calibration. Modern architectures, it\nturns out, are prone to overcon\ufb01dence, meaning accuracy is likely to be lower than what is indicated\nby the predictive score. The top row in Figure 1 illustrates this phenomena: shown are a series\nof joint density plots of the average winning score and accuracy of a VGG-16 [29] network over\nthe CIFAR-100 [17] validation set, plotted at different epochs. Both the con\ufb01dence (captured by\nthe winning score) as well as accuracy start out low and gradually increase as the network learns.\nHowever, what is interesting \u2013 and concerning \u2013 is that the con\ufb01dence always leads accuracy in the\nlater stages of training. Towards the end of training, accuracy saturates while con\ufb01dence continues\nto increase resulting in a very sharply peaked distribution of winning scores and an overcon\ufb01dent\nmodel.\nMost modern DNNs, when trained for classi\ufb01cation in a supervised learning setting, are trained using\none-hot encoded labels that have all the probability mass in one class; the training labels are thus\nzero-entropy signals that admit no uncertainty about the input. The DNN is thus, in some sense,\ntrained to become overcon\ufb01dent. Hence a worthwhile line of exploration is whether principled\napproaches to label smoothing can somehow temper overcon\ufb01dence. Label smoothing and related\nwork has been explored before [30, 27]. In this work, we carry out an exploration along these lines by\ninvestigating the effect of the recently proposed mixup [37] method of training deep neural networks.\nIn mixup, additional synthetic samples are generated during training by convexly combining random\npairs of images and, importantly, their labels as well. While simple to implement, it has shown to be a\nsurprisingly effective method of data augmentation: DNNs trained with mixup show noticeable gains\nin classi\ufb01cation performance on a number of image classi\ufb01cation benchmarks. However neither the\noriginal work nor any subsequent extensions to mixup [33, 10, 22] have explored the effect of mixup\non predictive uncertainty and DNN calibration; this is precisely what we aim to do in this paper.\nOur \ufb01ndings are as follows: mixup trained DNNs are signi\ufb01cantly better calibrated \u2013 i.e the predicted\nsoftmax scores are much better indicators of the actual likelihood of a correct prediction \u2013 than DNNs\ntrained without mixup (see Figure 1 bottom row for an example). We also observe that merely mixing\nfeatures does not result in the same calibration bene\ufb01t and that the label smoothing in mixup training\nplays a signi\ufb01cant role in improving calibration. Further, we also observe that mixup-trained DNNs\nare less prone to over-con\ufb01dent predictions on out-of-distribution and random-noise data. We note\nhere that in this work we do not consider the calibration and uncertainty over adversarially perturbed\ninputs; we leave that for future exploration.\n\n2\n\n0.00.20.40.60.81.0Mean of Winning Score0.00.20.40.60.81.0accuracyEpoch00.00.20.40.60.81.0Mean of Winning Score0.00.20.40.60.81.0accuracyEpoch200.00.20.40.60.81.0Mean of Winning Score0.00.20.40.60.81.0accuracyEpoch610.00.20.40.60.81.0Mean of Winning Score0.00.20.40.60.81.0accuracyEpoch1200.00.20.40.60.81.0Mean of Winning Score0.00.20.40.60.81.0accuracyEpoch1610.00.51.00.000.250.500.751.00AccuracyAccuracyEpoch00.00.51.00.000.250.500.751.00Epoch200.00.51.0Confidence0.000.250.500.751.00Epoch610.00.51.00.000.250.500.751.00Epoch1200.00.51.00.000.250.500.751.00Epoch161\fThe rest of the paper is organized as follows: Section 2 provides a brief overview of the mixup\ntraining process. Section 3 discusses calibration metrics, experimental setup and mixup\u2019s calibration\nbene\ufb01ts results on image data. In Section 5 we explore in more detail the effect of mixup-based label\nsmoothing on calibration. In Section 7 we show additional evidence of the bene\ufb01t of mixup training\non predictive uncertainty when dealing with unseen data. Further discussions and conclusions are in\nSection 8\n\n2 An Overview of Mixup Training\n\nMixup training [37] is based on the principle of Vicinal Risk Minimization [3](VRM): the classi\ufb01er\nis trained not only on the training data, but also in the vicinity of each training sample. The vicinal\npoints are generated according to the following simple rule introduced in [37]:\n\n\u02dcx = \u03bbxi + (1 \u2212 \u03bb)xj\n\u02dcy = \u03bbyi + (1 \u2212 \u03bb)yj\n\nwhere xi and xj are two randomly sampled input points, and yi and yj are their associated one-hot\nencoded labels. This has the effect of the empirical Dirac delta distribution\n\ncentered at (xi, yi) being replaced with the empirical vicinal distribution\n\nP\u03b4(x, y) =\n\n1\nn\n\n\u03b4(x = xi, y = yi)\n\nP\u03bd(\u02dcx, \u02dcy) =\n\n\u03bd(\u02dcx, \u02dcy|xi, yi)\n\nn(cid:88)\nn(cid:88)\n\ni\n\n1\nn\n\ni\n\nm(cid:88)\n\ni=1\n\nThe vicinal samples (\u02dcx, \u02dcy) are generated as above, and during training minimization is performed on\nthe empirical vicinal risk:\n\nR\u03bd(f ) =\n\n1\nm\n\nL(f ( \u02dcxi), \u02dcyi)\n\nwhere L is the standard cross-entropy loss, but calculated on the soft-labels \u02dcyi instead of hard\nlabels. Training this way not only augments the feature set \u02dcX, but the induced set of soft-labels\nalso encourages the strength of the classi\ufb01cation regions to vary linearly betweens samples. The\nexperiments in [37] and related work in [14, 33, 10] show noticeable performance gains in various\nimage classi\ufb01cation tasks.\nThe linear interpolator \u03bb \u2208 [0, 1] that determines the mixing ratio is drawn from a symmetric Beta\ndistribution, Beta(\u03b1, \u03b1), where \u03b1 is the hyper-parameter that controls the strength of the interpolation\nbetween pairs of images and the associated smoothing of the training labels. \u03b1 = 0 recovers the base\ncase corresponding to zero-entropy training labels (one-hot encodings, in which case the resulting\nimage is either just xi or xj), while a high value of \u03b1 ends up in always averaging the inputs and\nlabels. The authors in [37] remark that relatively smaller values of \u03b1 \u2208 [0.1, 0.4] gave the best\nperforming results for classi\ufb01cation, while high values of \u03b1 resulted in signi\ufb01cant under-\ufb01tting. In\nthis work, we also look at the effect of \u03b1 on calibration performance.\n\n3 Experiments\n\nWe perform numerous experiments to analyze the effect of mixup training on the calibration of the\nresulting trained classi\ufb01ers on both image and NLP data. We experiment with various deep architec-\ntures and standard datasets, including large-scale training with ImageNet. In all the experiments in\nthis paper, we only apply mixup to pairs of images as done in [37]. The mixup functionality was\nimplemented using the mixup authors\u2019 code available at [36].\n\n3.1 Setup\n\nFor the small-scale image experiments, we use the following datasets in our experiments: STL-10 [4],\nCIFAR-10 and CIFAR-100 [17] and Fashion-MNIST [34]. For STL-10, we use the VGG-16 [29]\n\n3\n\n\fnetwork. CIFAR-10 and CIFAR-100 experiments were carried out on VGG-16 as well as ResNet-34\nmodels. For Fashion-MNIST, we used a ResNet-18 model. For all experiments, we use batch\nnormalization, weight decay of 5 \u00d7 10\u22124, trained the network using SGD with Nesterov momentum,\ntraining for 200 epochs with an initial learning rate of 0.1 halved at 2 at 60,120 and 160 epochs.\nUnless otherwise noted, calibration results are reported for the best performing epoch on the validation\nset.\n\n3.2 Calibration Metrics\n\nWe measure the calibration of the network as follows (and as described in [9]): predictions are\ngrouped into M interval bins of equal size. Let Bm be the set of samples whose prediction scores\n(the winning softmax score) fall into bin Bm. The accuracy and con\ufb01dence of Bm are de\ufb01ned as\n\nwhere \u02c6pi is the con\ufb01dence (winning score) of sample i. The Expected Calibration Error (ECE) is\nthen de\ufb01ned as:\n\nacc(Bm) =\n\nconf(Bm) =\n\nM(cid:88)\n\nm=1\n\n|Bm|\nn\n\n1\n|Bm|\n1\n|Bm|\n\n\u02c6pi\n\niinBm\n\niinBm\n\n1(\u02c6yi = yi)\n\n(cid:88)\n(cid:88)\n(cid:12)(cid:12)(cid:12)(cid:12)acc(Bm) \u2212 conf(Bm)\n(cid:12)(cid:12)(cid:12)(cid:12)\n(cid:16)\n\nECE =\n\n(cid:104)\n\nM(cid:88)\n\nm=1\n\n|Bm|\nn\n\nIn high-risk applications, con\ufb01dent but wrong predictions can be especially harmful; thus we also\nde\ufb01ne an additional calibration metric \u2013 the Overcon\ufb01dence Error (OE)\u2013 as follows\n\nOE =\n\nconf(Bm) \u00d7 max\n\nconf(Bm) \u2212 acc(Bm), 0\n\n(cid:17)(cid:105)\n\nThis penalizes predictions by the weight of the con\ufb01dence but only when con\ufb01dence exceeds accuracy;\nthus overcon\ufb01dent bins incur a high penalty.\n\n3.3 Comparison Methods\n\nSince mixup produces smoothed labels over mixtures of inputs, we compare the calibration perfor-\nmance of mixup to two other label smoothing techniques:\n\nby distributing an \u0001 mass over the other (i.e., non ground-truth) classes.\n\n\u2022 \u0001\u2212label smoothing described in [30] where the one-hot encoded training signal is smoothed\n\u2022 We also compare the performance of mixup against the entropy-regularized loss (ERL) de-\nscribed in [27] that discourages the neural network from being over-con\ufb01dent by penalizing\nlow-entropy distributions.\n\nOur baseline comparison is regular training where no label smoothing or mixing of features is applied\n(no-mixup). We also note that in this section we do not compare against the temperature scaling\nmethod described in [9], which is a post-training calibration method and will generally produce\nwell-calibrated scores. Here we would like to see the effect of label smoothing while training;\nexperiments with temperature scaling are reported in Section 7.\n\n3.4 Results\n\nResults on the various datasets and architectures are shown in Figure 2. While the performance gains\nin validation accuracy are generally consistent with the results reported in [37], it is the effects on\nnetwork calibration that we focus here. The top row shows a calibration scatter plot for STL-10\nand CIFAR-100, highlighting the effect of mixup training. In a well calibrated model, where the\ncon\ufb01dence matches the accuracy most of the points will be on x = y line. We see that in the base\ncase, both for STL-10 and CIFAR-100, most of the points tend to lie in the overcon\ufb01dent region. The\nmixup case is much better calibrated, noticeably in the high-con\ufb01dence regions.\n\n4\n\n\f(a)\n\n(b)\n\n(c)\n\n(d)\n\n(e)\n\n(h)\n\n(f)\n\n(i)\n\n(g)\n\n(j)\n\nFigure 2: Calibration results for mixup and base-case on various image datasets and architectures.\nTop Row: Scatterplots for accuracy and con\ufb01dence for STL-10(a,b) and CIFAR-100(c,d). The mixup\ncase is much better calibrated with the points lying closer to the x = y line, while in the base case,\npoints tend to lie in the overcon\ufb01dent region. Middle Row: Mixup versus comparison methods.\nlabel smoothing is the \u0001-label smoothing method, while ERL is the entropy regularized loss. Bottom\nRow: Calibration Error (e) and Overcon\ufb01dence error (f) on various architectures. Experiments\nsuggest best ECE is achieved in the [0.2,0.4] range for \u03b1 (h), while overcon\ufb01dence error decreases\nmonotonically with \u03b1 due to under\ufb01tting (i). Accuracy behavior for differently calibrated models is\nshowin in (j).\n\nThe bar plots in the middle row show the results on various combinations of datasets and architectures\non accuracy and calibration against comparison methods. We report the calibration error for the\nbest performing model (in terms of validation accuracy). For label smoothing, an \u0001 \u2208 [0.05, 0.1]\nperformed best while for ERL, the best-performing con\ufb01dence penalty hyper-parameter was 0.1. The\ntrends in the comparison are clear: label smoothing either via \u0001-smoothing, ERL or mixup generally\nprovides a calibration advantage and tempers overcon\ufb01dence, with the latter generally performing the\nbest in comparison to other methods. We also show the effect on ECE as we vary the hyperparameter\n\u03b1 of the mixing parameter distribution. For very low values of \u03b1, the behavior is similar to the base\ncase (as expected), but ECE also noticeably worsens for higher values of \u03b1 due to the model being\nunder-con\ufb01dent. Indeed, mixup models can be undercon\ufb01dent if \u03b1 is large whic is related to manifold\nintrusion [10]: a mixed-up sample is more likely to lie away from the original manifold and thus be\naffected by manifold intrusion if \u03b1 is large. Overcon\ufb01dence alone decreases monotonically as we\nincrease \u03b1 as shown in Figure 2i. We also show the accuracy of mixup models at various levels of\n\n5\n\n0.000.250.500.751.00Score of Winning Class0.00.20.40.60.81.0accuracySTL-10/VGG-16: Calibrationno mixup0.000.250.500.751.00Score of Winning Class0.00.20.40.60.81.0accuracySTL-10/VGG-16: Calibrationmixup0.000.250.500.751.00Score of Winning Class0.00.20.40.60.81.0accuracyCIFAR-100/ResNet34: Calibrationno mixup0.000.250.500.751.00Score of Winning Class0.00.20.40.60.81.0accuracyCIFAR-100/ResNet34: CalibrationmixupSTL-10VGG-16CIFAR-100ResNet-34CIFAR-10ResNet-34FashionResNet-180.000.010.020.030.040.050.060.070.08Calibration ErrorCalibration Errorno_mixupmixuplabel_smoothingERLSTL-10VGG-16CIFAR-100ResNet-34CIFAR-10ResNet-34FashionResNet-180.000.010.020.030.040.050.060.07Overconfidence ErrorOverconfidence Errorno_mixupmixuplabel_smoothingERLSTL-10VGG-16CIFAR-100ResNet-34CIFAR-10ResNet-34FashionResNet-18020406080100Validation AccuracyAccuracyno_mixupmixuplabel_smoothingERL0.10.20.30.40.5alpha0.020.040.060.08Calibration ErrorCalibration Error0.10.20.30.40.5alpha0.0000.0050.0100.0150.020OverconfidenceOverconfidence0.000.020.040.06Calibration Error7580859095100accuracyAccuracy vs CalibrationSTL-10/VGG-16CIFAR-100/ResNet-34CIFAR-10/ResNet-34Fashion-MNIST/ResNet-18\fcalibration determined by \u03b1. As can be seen, a well-tuned \u03b1 can result in a better-calibrated model\nwith very little loss in performance. Our classi\ufb01cation results here are consistent with those reported\nin [37] where the best performing \u03b1 was in the [0.1, .0.4] range.\n\n3.4.1 Large-scale Experiments on ImageNet\n\n(a)\n\n(b)\n\n(c)\n\nFigure 3: Calibration on ImageNet for ResNet architectures\n\nHere we report the results of calibration metrics resulting form mixup training on the 1000-class\nversion of the ImageNet [5] data comprising of over 1.2 million images. One of the advantages of\nmixup and its implementation is that it adds very little overhead to the training time, and thus can be\neasily applied to large scale datasets like ImageNet. We perform distributed parallel training using the\nsynchronous version of stochastic gradient descent. We use the learning-rate schedule described in\n[8] on a 32-GPU cluster and train till 93% accuracy is reached over the top-5 predictions. We test on\ntwo modern state-of-the-art archictures: ResNet-50 [12] and ResNext-101 (32x4d) [35]. The results\nare shown in Figure 3. The scatter-plot showing calibration for ResNext-101 architecture suggests\nthat mixup training provides noticeable bene\ufb01ts even in the large-data scenario, where the models\nshould be less prone to over-\ufb01tting the one-hot labels. On the deeper ResNext, mixup provides better\ncalibration than the label smoothing models, though this same effect was not visible for the ResNet-50\nmodel. However, both calibration error and overcon\ufb01dence show noticeable improvements using\nlabel smoothing over the baseline. The mixup model did however achieve a consistently higher\nclassi\ufb01cation performance of \u2248 0.4 percent over the other methods.\n\n4 Experiments on Natural Language Data\n\n(a)\n\n(b)\n\n(c)\n\nFigure 4: Accuracy, calibration and overcon\ufb01dence on various NLP datasets\n\nWhile mixup was originally suggested as a method to mostly improve performance on image\nclassi\ufb01cation tasks, here we explore the effect of mixup training in the natural language processing\n(NLP) domain. A straight-forward mixing of inputs (as in pixel-mixing in images) will generally\nproduce nonsense input since the semantics are unclear. To avoid this, we modify the mixup strategy\nto perform mixup on the embeddings layer rather than directly on the input documents. We note\nthat this approach is similar to the recent work described in [11] that utilizes mixup for improving\nsentence classi\ufb01cation which is among the few works, besides ours, studying the effects of mixup in\n\n6\n\n0.20.40.60.8Confidence0.20.40.60.81.0AccuracyImageNet/ResNext-101no mixupmixupResNext-1010.000.020.040.06Calibration ErrorImageNetResNet-500.000.010.020.030.040.05Calibration ErrorImageNetno_mixupmixuplabel_smoothingERLTRECIMDBMR020406080Validation Accuracyno_mixupmixup-feats-onlymixupERLlabel_smoothingTRECIMDBMR0.0000.0250.0500.0750.1000.1250.1500.1750.200Expected Calibration Errorno_mixupmixup-feats-onlymixupERLlabel_smoothingTRECIMDBMR0.0000.0250.0500.0750.1000.1250.1500.1750.200Overconfidence Penaltyno_mixupmixup-feats-onlymixupERLlabel_smoothing\fthe NLP domain. For our experiments, we employ mixup on NLP data for text classi\ufb01cation using the\nMR [25], TREC [21] and [23] datasets.We train a CNN for sentence classi\ufb01cation (Sentence-level\nCNN) [16], where we initialize all the words with pre-trained GloVe [26] embeddings, which are\nmodi\ufb01ed while training on each dataset. For the remaining parameters, we use the values suggested\nin [16]. We refrain from training the most recent NLP models [13, 2, 38], since our aim here is not to\nshow state-of-art classi\ufb01cation performance on these datasets, but to study the effect on calibration.\nAlso, the design of the more recent NLP models makes embedding mixup less straightforward.\nNevertheless, the performance bene\ufb01ts on calibration, shown in Figure 4 are evident where mixup\nprovides noticeable gains for all datasets, both in terms of calibration and overcon\ufb01dence. We leave\nfurther exploration of principled strategies for mixup for NLP as future work.\n\n5 Effect of Soft Labels on Calibration\n\nSo far we have seen that mixup consistently leads to better calibrated networks compared to the\nbase case, in addition to improving classi\ufb01cation performance as has been observed in a number of\nworks [33, 10, 22]. This behavior is not surprising given that mixup is a form of data augmentation:\nin mixup training, due to random sampling of both images as well as the mixing parameter \u03bb, the\nprobability that the learner sees the same image twice is small. This has a strong regularizing effect\nin terms of preventing memorization and over-\ufb01tting, even for high-capacity neural networks. Indeed,\nunlike regular training, the train loss in the mixup case is always signi\ufb01cantly higher than the base\ncase as observed by the mixup authors [37]. From the perspective of statistical learning theory, the\nimproved calibration of a mixup classi\ufb01er can be viewed as the classi\ufb01er learning the true posteriors\nin the in\ufb01nite data limit [32] due to the signi\ufb01cant amount of data augmentation resulting from\nthe random combination in mixup. However this leads to the following question: if the improved\ncalibration is essentially an effect of data augmentation, does simply combining the images but not\ncombining the labels provide the same calibration bene\ufb01t?\nWe perform a series of experiments on various image datasets and architectures to explore this\nquestion. Results from the earlier sections show that existing label smoothing techniques that\nincrease the entropy of the training signal do provide better calibration without exploiting any data\naugmentation effects and thus we expect to see this in play in the mixup case as well. In the latter case,\nthe entropies of the train labels are determined by the \u03b1 parameter of the Beta(\u03b1, \u03b1) distribution\nfrom which the mixing parameter is sampled. The distribution of training entropies for a few cases\nof \u03b1 are shown in Figure 5. The base-case is equivalent to \u03b1 = 0 (not shown) where the entropy\ndistribution is a point-mass at 0.\n\n(a)\n\n(b)\n\n(c)\n\n(d)\n\nFigure 5: Entropy distribution of training labels as a function of the \u03b1 parameter of the Beta(\u03b1, \u03b1)\ndistribution from which the mixing parameter is sampled.\n\nTo tease out the effect of full mixup versus only mixing features, we convexly combine images as\nbefore, but the resulting image assumes the hard label of the nearer class; this provides data augmen-\ntation without the label smoothing effect. Results on a number of benchmarks and architectures are\nshown in Figure 6. The results are clear: merely mixing features does not provide the calibration ben-\ne\ufb01t seen in the full-mixup case suggesting that the point-mass distributions in hard-coded labels are\ncontributing factors to overcon\ufb01dence. As in label smoothing and entropy regularization, having (or\nenforcing via a loss penalty) a non-zero mass in more than one class prevents the largest pre-softmax\nlogit from becoming much larger than the others tempering overcon\ufb01dence and leading to improved\ncalibration.\n\n7\n\n0.00.20.40.6alpha=0.10.00.20.40.6alpha=0.30.00.20.40.6alpha=0.50.00.20.40.6alpha=1.0\fIn addition to feature and label mixing, a recent extension to mixup [33] also proposes convexly\ncombining the representations in the hidden layer of the network; we report the calibration effects of\nthis approach in the supplementary material.\n\n(a)\n\n(b)\n\n(c)\n\n(d)\n\n(e)\n\nFigure 6: Calibration performance when only features are mixed vs. full mixup, on various datasets\nand architectures\n\n6 Effect of Extended Training on Mixup Calibration\n\nAs remarked in the previous section, one of the contributing factors to improved calibration in mixup\nis the signi\ufb01cant data augmentation aspect of mixup training, where the model is unlikely to see\nthe same mixed-up sample more than once. The natural question here is whether these models will\neventually become overcon\ufb01dent if trained for much longer periods. Below, we show the training\ncurves for a few extended training experiments where the models were trained for 1000 epochs:\nfor the baseline (i.e when \u03b1 = 0.), the train loss and accuracy approach 0 and 100% respectively\n(i.e., over-\ufb01tting), while in the mixup case (non-zero \u03b1\u2019s), the strong data augmentation prevents\nover-\ufb01tting. This behavior is sustained over the entire duration of the training as can be seen in\nthe corresponding values of ECE. Mixup models, even when trained for much longer, continue to\nhave a low calibration error, suggesting that the mixing of data has a sustained inhibitive effect on\nover-\ufb01tting the training data (the training loss for mixup continues to be signi\ufb01cantly higher than\nbaseline even after extended training) and preventing the model from becoming overcon\ufb01dent.\n\nFigure 7: Training loss and calibration error under extended training for CIFAR-10 and CIFAR-100\nwith mixup. Baseline train error (orange) goes to zero early on while mixup continues to have\nnon-zero training loss even after 1000 epochs. Meanwhile, calibration error for mixup does not\nexhibit an upward trend even after extended training.\n\n7 Testing on Out-of-Distribution and Random Data\n\nIn this section, we explore the effect\nof mixup training when predicting on\nsamples from unseen classes (out-of-\ndistribution) and random noise im-\nages. We \ufb01rst train a VGG-16 network\non in-distribution data (STL-10) and\nthen predict on classes sampled from\nthe ImageNet database that have not\nbeen encountered during training. For\nthe random noise images, we test on\ngaussian random noise with the same\nmean and variance as the training set.\n\nFigure 8: Distribution of winning scores of various mod-\nels when tested on out-of-distribution and gaussian noise\nsamples, after being trained on the STL-10 dataset.\n\n8\n\nSTL-10VGG-160.000.010.020.030.04Calibration ErrorCIFAR-100ResNet-340.000.020.040.06Calibration ErrorCIFAR-100VGG-160.0000.0250.0500.0750.100Calibration Errormixup-feats-onlymixupFashionResNet-180.0000.0050.0100.0150.020Calibration ErrorResNext-101ImageNet0.000.020.040.06Calibration Error02505007501000epoch01Train LossCIFAR-10/ResNet-18MixupBaseline02505007501000epoch0.050.10ECECIFAR-10/ResNet-18Mixup ECE02505007501000epoch024Train LossCIFAR-100/ResNet-18MixupBaseline02505007501000epoch0.0250.0500.075ECECIFAR-100/ResNet-18Mixup ECE0.00.20.40.60.81.0ConfidencePredictions on Out-of-Distribution Imagesno mixupmixuptemp_scaledMC dropout avg (p=0.3)0.00.20.40.60.81.0ConfidencePredictions on Random Noise Imagesno mixupmixuptemp_scaledMC dropout avg (p=0.3)\fWe compare the performance of a mixup-trained model with that of the baseline, as well as a\ntemperature calibrated pre-trained baseline as described in [9]. Since the latter is a post-training\ncalibration method, we expect it to be well calibrated on in-distribution data. We also compare the\nprediction uncertainty using the Monte Carlo dropout method described in [6] where multiple forward\npasses using dropout are made during test-time. We average predictions over 10 runs. The distribution\nover prediction scores for out-of-distribution and random data for mixup and comparison methods\nare shown in Figure 8. The differences versus the baseline are striking; in both cases, the mixup DNN\nis noticeably less con\ufb01dent than its non-mixup counterpart, with the score distribution being nearly\nperfectly separable in the random noise case. While temperature scaling is more conservative than\nmixup on real but out-of-sample data, it is noticeably more overcon\ufb01dent in the random-noise case.\nFurther, mixup performs signi\ufb01cantly better than MC-dropout in both cases.\nIn Table 1, we also show a comparison of the performance\nof the aforementioned models for reliably detecting of out-of-\ndistributon and random-noise data, using Area under the ROC\ncurve as the metric. Mixup is the best performing model in both\ncases, signi\ufb01cantly outperforming the others as a random-noise\ndetector. Temperature scaling, while producing well-calibrated\nmodels for in-distribution data is not a reliable detector. The\nscaling process reduces the con\ufb01dence on both in and out-of-\ndistribution data, signi\ufb01cantly reducing the ability to discrimi-\nnate between these two types of data. Mixup, on the other hand,\ndoes well in both cases. The results here suggest that the effect of training with interpolated samples\nand the resulting label smoothing tempers over-con\ufb01dence in regions away from the training data.\nWhile these experiments were limited to two datasets and one architecture, the results indicate that\ntraining by minimizing vicinal risk can be an effective way to enhance reliability of predictions in\nDNNs. Note that since mixup trains the model by convexly combining pairs of images, the synthe-\nsized images all lie within the convex hull of the training data. In the suppllementary material, we\nprovide results on the prediction con\ufb01dence when images lie outside the convex hull of the training\nset.\n\nTable 1: Out-of-category detection\nresults for the DAC on STL-10 and\nTiny ImageNet.\n\nBaseline\nMixup (\u03b1=0.4)\nTemp. Scaling\nDropout(p=0.3)\n\nMethod\n\nAUROC(In/Out)\nSTL-10/\nGaussian\n73.28\n95.93\n54.2\n70.57\n\nSTL-10/\nImageNet\n80.57\n83.28\n56.2\n78.93\n\n8 Conclusion and Future Work\n\nWe presented results on an unexplored area of mixup based training \u2013 its effect on DNN calibration\nand predictive uncertainty. Existing empirical work has conclusively shown the bene\ufb01ts of mixup\nfor boosting classi\ufb01cation performance; in this work, we show an additional important bene\ufb01t \u2013\nmixup trained networks turn out to be better calibrated and provide more reliable estimates both for\nin-sample and out-of-sample data (being under-con\ufb01dent in the latter case).\nThere are possibly multiple reasons for this: the data augmentation provided by mixup is a form of\nregularization that prevents over-\ufb01tting and memorization, tempering overcon\ufb01dence in the process.\nThe label smoothing resulting from mixup might be viewed as a form of entropic regularization on\nthe training signals, again preventing the DNN from driving the training error to zero. The results\nin the paper provide further evidence that training with hard labels is likely one of the contributing\nfactors leading to overcon\ufb01dence seen in modern neural networks. Recent work [33] has shown how\nthe classi\ufb01cation regions in mixup are smoother, without sudden jumps from one high con\ufb01dence\nregion to the other suggesting that the lack of sharp boundary transitions in classi\ufb01cation regions play\nan important role in producing well-calibrated classi\ufb01ers.\nSince mixup is implemented while training, it can also be employed with post-training calibration\nlike temperature scaling, model perturbations like the dropout method or even the ensemble models\ndescribed in [18]. Further mixup based models can also be combined with rejection classi\ufb01ers, both\nduring training [31] for dealing with label noise, as well as inference [7] to improve the training and\nclassi\ufb01cation pipeline in modern deep learning. Indeed, the classi\ufb01cation performance-boost coupled\nwith the well-calibrated nature of mixup trained DNNs as studied in this paper suggest that mixup\nbased training be employed in situations where predictive uncertainty is a signi\ufb01cant concern.\n\n9\n\n\fAcknowledgments\n\nWe would like to thank the anonymous referees for their valuable suggestions on improving the\npaper. The authors were supported in part by the Joint Design of Advanced Computing Solutions for\nCancer (JDACS4C) program established by the U.S. Department of Energy (DOE) and the National\nCancer Institute (NCI) of the National Institutes of Health. This work was performed under the\nauspices of the U.S. Department of Energy by Los Alamos National Laboratory under Contract\nDE-AC5206NA25396. This work was also supported in part by the CONIX Research Center, one of\nsix centers in JUMP, a Semiconductor Research Corporation (SRC) program sponsored by DARPA.\n\nReferences\n[1] Richard Berk. An impact assessment of machine learning risk forecasts on parole board\n\ndecisions and recidivism. Journal of Experimental Criminology, 13(2):193\u2013216, 2017.\n\n[2] Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah\nConstant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. Universal sentence encoder.\narXiv preprint arXiv:1803.11175, 2018.\n\n[3] Olivier Chapelle, Jason Weston, L\u00b4eon Bottou, and Vladimir Vapnik. Vicinal risk minimization.\n\nIn Advances in neural information processing systems, pages 416\u2013422, 2001.\n\n[4] Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsuper-\nvised feature learning. In Proceedings of the fourteenth international conference on arti\ufb01cial\nintelligence and statistics, pages 215\u2013223, 2011.\n\n[5] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale\nhierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009.\nIEEE Conference on, pages 248\u2013255. Ieee, 2009.\n\n[6] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model\nuncertainty in deep learning. In international conference on machine learning, pages 1050\u20131059,\n2016.\n\n[7] Yonatan Geifman and Ran El-Yaniv. Selective classi\ufb01cation for deep neural networks. In\n\nAdvances in neural information processing systems, pages 4885\u20134894, 2017.\n\n[8] Priya Goyal, Piotr Doll\u00b4ar, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola,\nAndrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training\nimagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.\n\n[9] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural\n\nnetworks. arXiv preprint arXiv:1706.04599, 2017.\n\n[10] Hongyu Guo, Yongyi Mao, and Richong Zhang. Mixup as locally linear out-of-manifold\n\nregularization. arXiv preprint arXiv:1809.02499, 2018.\n\n[11] Hongyu Guo, Yongyi Mao, and Richong Zhang. Augmenting data with mixup for sentence\n\nclassi\ufb01cation: An empirical study. arXiv preprint arXiv:1905.08941, 2019.\n\n[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image\nrecognition. In Proceedings of the IEEE conference on computer vision and pattern recognition,\npages 770\u2013778, 2016.\n\n[13] Jeremy Howard and Sebastian Ruder. Universal language model \ufb01ne-tuning for text classi\ufb01ca-\n\ntion. arXiv preprint arXiv:1801.06146, 2018.\n\n[14] Hiroshi Inoue. Data augmentation by pairing samples for images classi\ufb01cation. arXiv preprint\n\narXiv:1801.02929, 2018.\n\n[15] Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for\ncomputer vision? In Advances in neural information processing systems, pages 5574\u20135584,\n2017.\n\n[16] Yoon Kim. Convolutional neural networks for sentence classi\ufb01cation.\n\narXiv:1408.5882, 2014.\n\narXiv preprint\n\n[17] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images.\n\nTechnical report, Citeseer, 2009.\n\n10\n\n\f[18] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable\npredictive uncertainty estimation using deep ensembles. In Advances in Neural Information\nProcessing Systems, pages 6402\u20136413, 2017.\n\n[19] Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training con\ufb01dence-calibrated classi\ufb01ers\n\nfor detecting out-of-distribution samples. arXiv preprint arXiv:1711.09325, 2017.\n\n[20] Jesse Levinson, Jake Askeland, Jan Becker, Jennifer Dolson, David Held, Soeren Kammel,\nJ Zico Kolter, Dirk Langer, Oliver Pink, Vaughan Pratt, et al. Towards fully autonomous driving:\nSystems and algorithms. In Intelligent Vehicles Symposium (IV), 2011 IEEE, pages 163\u2013168.\nIEEE, 2011.\n\n[21] Xin Li and Dan Roth. Learning question classi\ufb01ers. In Proceedings of the 19th international\nconference on Computational linguistics-Volume 1, pages 1\u20137. Association for Computational\nLinguistics, 2002.\n\n[22] Daojun Liang, Feng Yang, Tian Zhang, and Peter Yang. Understanding mixup training methods.\n\nIEEE Access, 6:58774\u201358783, 2018.\n\n[23] Andrew L Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher\nPotts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting\nof the association for computational linguistics: Human language technologies-volume 1, pages\n142\u2013150. Association for Computational Linguistics, 2011.\n\n[24] Riccardo Miotto, Li Li, Brian A Kidd, and Joel T Dudley. Deep patient: an unsupervised\nrepresentation to predict the future of patients from the electronic health records. Scienti\ufb01c\nreports, 6:26094, 2016.\n\n[25] Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization\nwith respect to rating scales. In Proceedings of the 43rd annual meeting on association for\ncomputational linguistics, pages 115\u2013124. Association for Computational Linguistics, 2005.\n\n[26] Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for\nword representation. In Proceedings of the 2014 conference on empirical methods in natural\nlanguage processing (EMNLP), pages 1532\u20131543, 2014.\n\n[27] Gabriel Pereyra, George Tucker, Jan Chorowski, \u0141ukasz Kaiser, and Geoffrey Hinton. Reg-\narXiv preprint\n\nularizing neural networks by penalizing con\ufb01dent output distributions.\narXiv:1701.06548, 2017.\n\n[28] Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify\nclassi\ufb01cation uncertainty. In Advances in Neural Information Processing Systems, pages 3183\u2013\n3193, 2018.\n\n[29] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale\n\nimage recognition. arXiv preprint arXiv:1409.1556, 2014.\n\n[30] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re-\nthinking the inception architecture for computer vision. In Proceedings of the IEEE conference\non computer vision and pattern recognition, pages 2818\u20132826, 2016.\n\n[31] Sunil Thulasidasan, Tanmoy Bhattacharya, Jeff Bilmes, Gopinath Chennupati, and Jamal\nMohd-Yusof. Combating label noise in deep learning using abstention. arXiv preprint\narXiv:1905.10964, 2019.\n\n[32] Vladimir N Vapnik and A Ya Chervonenkis. On the uniform convergence of relative frequencies\n\nof events to their probabilities. In Measures of complexity, pages 11\u201330. Springer, 2015.\n\n[33] Vikas Verma, Alex Lamb, Christopher Beckham, Aaron Courville, Ioannis Mitliagkis, and\nYoshua Bengio. Manifold mixup: Encouraging meaningful on-manifold interpolation as a\nregularizer. arXiv preprint arXiv:1806.05236, 2018.\n\n[34] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for\n\nbenchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.\n\n[35] Saining Xie, Ross Girshick, Piotr Doll\u00b4ar, Zhuowen Tu, and Kaiming He. Aggregated residual\ntransformations for deep neural networks. In Proceedings of the IEEE conference on computer\nvision and pattern recognition, pages 1492\u20131500, 2017.\n[36] Hongyi Zhang. https://github.com/hongyi-zhang/mixup.\n\n11\n\n\f[37] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond\n\nempirical risk minimization. arXiv preprint arXiv:1710.09412, 2017.\n\n[38] Peng Zhou, Zhenyu Qi, Suncong Zheng, Jiaming Xu, Hongyun Bao, and Bo Xu. Text classi\ufb01-\ncation improved by integrating bidirectional lstm with two-dimensional max pooling. arXiv\npreprint arXiv:1611.06639, 2016.\n\n12\n\n\f", "award": [], "sourceid": 7763, "authors": [{"given_name": "Sunil", "family_name": "Thulasidasan", "institution": "Los Alamos National Laboratory & University of Washington"}, {"given_name": "Gopinath", "family_name": "Chennupati", "institution": "Los Alamos National Laboratory"}, {"given_name": "Jeff", "family_name": "Bilmes", "institution": "University of Washington, Seattle"}, {"given_name": "Tanmoy", "family_name": "Bhattacharya", "institution": "Los Alamos National Laboratory"}, {"given_name": "Sarah", "family_name": "Michalak", "institution": "Los Alamos National Laboratory"}]}