{"title": "ImageNet Classification with Deep Convolutional Neural Networks", "book": "Advances in Neural Information Processing Systems", "page_first": 1097, "page_last": 1105, "abstract": "We trained a large, deep convolutional neural network to classify the 1.3 million high-resolution images in the LSVRC-2010 ImageNet training set into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of 39.7\\% and 18.9\\% which is considerably better than the previous state-of-the-art results. The neural network, which has 60 million parameters and 500,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and two globally connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implementation of convolutional nets. To reduce overfitting in the globally connected layers we employed a new regularization method that proved to be very effective.", "full_text": "ImageNet Classi\ufb01cation with Deep Convolutional\n\nNeural Networks\n\nAlex Krizhevsky\n\nUniversity of Toronto\n\nkriz@cs.utoronto.ca\n\nIlya Sutskever\n\nUniversity of Toronto\n\nilya@cs.utoronto.ca\n\nGeoffrey E. Hinton\nUniversity of Toronto\n\nhinton@cs.utoronto.ca\n\nAbstract\n\nWe trained a large, deep convolutional neural network to classify the 1.2 million\nhigh-resolution images in the ImageNet LSVRC-2010 contest into the 1000 dif-\nferent classes. On the test data, we achieved top-1 and top-5 error rates of 37.5%\nand 17.0% which is considerably better than the previous state-of-the-art. The\nneural network, which has 60 million parameters and 650,000 neurons, consists\nof \ufb01ve convolutional layers, some of which are followed by max-pooling layers,\nand three fully-connected layers with a \ufb01nal 1000-way softmax. To make train-\ning faster, we used non-saturating neurons and a very ef\ufb01cient GPU implemen-\ntation of the convolution operation. To reduce over\ufb01tting in the fully-connected\nlayers we employed a recently-developed regularization method called \u201cdropout\u201d\nthat proved to be very effective. We also entered a variant of this model in the\nILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%,\ncompared to 26.2% achieved by the second-best entry.\n\n1\n\nIntroduction\n\nCurrent approaches to object recognition make essential use of machine learning methods. To im-\nprove their performance, we can collect larger datasets, learn more powerful models, and use bet-\nter techniques for preventing over\ufb01tting. Until recently, datasets of labeled images were relatively\nsmall \u2014 on the order of tens of thousands of images (e.g., NORB [16], Caltech-101/256 [8, 9], and\nCIFAR-10/100 [12]). Simple recognition tasks can be solved quite well with datasets of this size,\nespecially if they are augmented with label-preserving transformations. For example, the current-\nbest error rate on the MNIST digit-recognition task (<0.3%) approaches human performance [4].\nBut objects in realistic settings exhibit considerable variability, so to learn to recognize them it is\nnecessary to use much larger training sets. And indeed, the shortcomings of small image datasets\nhave been widely recognized (e.g., Pinto et al. [21]), but it has only recently become possible to col-\nlect labeled datasets with millions of images. The new larger datasets include LabelMe [23], which\nconsists of hundreds of thousands of fully-segmented images, and ImageNet [6], which consists of\nover 15 million labeled high-resolution images in over 22,000 categories.\nTo learn about thousands of objects from millions of images, we need a model with a large learning\ncapacity. However, the immense complexity of the object recognition task means that this prob-\nlem cannot be speci\ufb01ed even by a dataset as large as ImageNet, so our model should also have lots\nof prior knowledge to compensate for all the data we don\u2019t have. Convolutional neural networks\n(CNNs) constitute one such class of models [16, 11, 13, 18, 15, 22, 26]. Their capacity can be con-\ntrolled by varying their depth and breadth, and they also make strong and mostly correct assumptions\nabout the nature of images (namely, stationarity of statistics and locality of pixel dependencies).\nThus, compared to standard feedforward neural networks with similarly-sized layers, CNNs have\nmuch fewer connections and parameters and so they are easier to train, while their theoretically-best\nperformance is likely to be only slightly worse.\n\n1\n\n\fDespite the attractive qualities of CNNs, and despite the relative ef\ufb01ciency of their local architecture,\nthey have still been prohibitively expensive to apply in large scale to high-resolution images. Luck-\nily, current GPUs, paired with a highly-optimized implementation of 2D convolution, are powerful\nenough to facilitate the training of interestingly-large CNNs, and recent datasets such as ImageNet\ncontain enough labeled examples to train such models without severe over\ufb01tting.\nThe speci\ufb01c contributions of this paper are as follows: we trained one of the largest convolutional\nneural networks to date on the subsets of ImageNet used in the ILSVRC-2010 and ILSVRC-2012\ncompetitions [2] and achieved by far the best results ever reported on these datasets. We wrote a\nhighly-optimized GPU implementation of 2D convolution and all the other operations inherent in\ntraining convolutional neural networks, which we make available publicly1. Our network contains\na number of new and unusual features which improve its performance and reduce its training time,\nwhich are detailed in Section 3. The size of our network made over\ufb01tting a signi\ufb01cant problem, even\nwith 1.2 million labeled training examples, so we used several effective techniques for preventing\nover\ufb01tting, which are described in Section 4. Our \ufb01nal network contains \ufb01ve convolutional and\nthree fully-connected layers, and this depth seems to be important: we found that removing any\nconvolutional layer (each of which contains no more than 1% of the model\u2019s parameters) resulted in\ninferior performance.\nIn the end, the network\u2019s size is limited mainly by the amount of memory available on current GPUs\nand by the amount of training time that we are willing to tolerate. Our network takes between \ufb01ve\nand six days to train on two GTX 580 3GB GPUs. All of our experiments suggest that our results\ncan be improved simply by waiting for faster GPUs and bigger datasets to become available.\n\n2 The Dataset\n\nImageNet is a dataset of over 15 million labeled high-resolution images belonging to roughly 22,000\ncategories. The images were collected from the web and labeled by human labelers using Ama-\nzon\u2019s Mechanical Turk crowd-sourcing tool. Starting in 2010, as part of the Pascal Visual Object\nChallenge, an annual competition called the ImageNet Large-Scale Visual Recognition Challenge\n(ILSVRC) has been held. ILSVRC uses a subset of ImageNet with roughly 1000 images in each of\n1000 categories. In all, there are roughly 1.2 million training images, 50,000 validation images, and\n150,000 testing images.\nILSVRC-2010 is the only version of ILSVRC for which the test set labels are available, so this is\nthe version on which we performed most of our experiments. Since we also entered our model in\nthe ILSVRC-2012 competition, in Section 6 we report our results on this version of the dataset as\nwell, for which test set labels are unavailable. On ImageNet, it is customary to report two error rates:\ntop-1 and top-5, where the top-5 error rate is the fraction of test images for which the correct label\nis not among the \ufb01ve labels considered most probable by the model.\nImageNet consists of variable-resolution images, while our system requires a constant input dimen-\nsionality. Therefore, we down-sampled the images to a \ufb01xed resolution of 256 \u00d7 256. Given a\nrectangular image, we \ufb01rst rescaled the image such that the shorter side was of length 256, and then\ncropped out the central 256\u00d7256 patch from the resulting image. We did not pre-process the images\nin any other way, except for subtracting the mean activity over the training set from each pixel. So\nwe trained our network on the (centered) raw RGB values of the pixels.\n\n3 The Architecture\n\nThe architecture of our network is summarized in Figure 2.\nIt contains eight learned layers \u2014\n\ufb01ve convolutional and three fully-connected. Below, we describe some of the novel or unusual\nfeatures of our network\u2019s architecture. Sections 3.1-3.4 are sorted according to our estimation of\ntheir importance, with the most important \ufb01rst.\n\n1http://code.google.com/p/cuda-convnet/\n\n2\n\n\f3.1 ReLU Nonlinearity\n\nThe standard way to model a neuron\u2019s output f as\na function of its input x is with f (x) = tanh(x)\nor f (x) = (1 + e\u2212x)\u22121.\nIn terms of training time\nwith gradient descent, these saturating nonlinearities\nare much slower than the non-saturating nonlinearity\nf (x) = max(0, x). Following Nair and Hinton [20],\nwe refer to neurons with this nonlinearity as Recti\ufb01ed\nLinear Units (ReLUs). Deep convolutional neural net-\nworks with ReLUs train several times faster than their\nequivalents with tanh units. This is demonstrated in\nFigure 1, which shows the number of iterations re-\nquired to reach 25% training error on the CIFAR-10\ndataset for a particular four-layer convolutional net-\nwork. This plot shows that we would not have been\nable to experiment with such large neural networks for\nthis work if we had used traditional saturating neuron\nmodels.\nWe are not the \ufb01rst to consider alternatives to tradi-\ntional neuron models in CNNs. For example, Jarrett\net al. [11] claim that the nonlinearity f (x) = |tanh(x)|\nworks particularly well with their type of contrast nor-\nmalization followed by local average pooling on the\nCaltech-101 dataset. However, on this dataset the pri-\nmary concern is preventing over\ufb01tting, so the effect\nthey are observing is different from the accelerated\nability to \ufb01t the training set which we report when us-\ning ReLUs. Faster learning has a great in\ufb02uence on the\nperformance of large models trained on large datasets.\n\n3.2 Training on Multiple GPUs\n\nFigure 1: A four-layer convolutional neural\nnetwork with ReLUs (solid line) reaches a 25%\ntraining error rate on CIFAR-10 six times faster\nthan an equivalent network with tanh neurons\n(dashed line). The learning rates for each net-\nwork were chosen independently to make train-\ning as fast as possible. No regularization of\nany kind was employed. The magnitude of the\neffect demonstrated here varies with network\narchitecture, but networks with ReLUs consis-\ntently learn several times faster than equivalents\nwith saturating neurons.\n\nA single GTX 580 GPU has only 3GB of memory, which limits the maximum size of the networks\nthat can be trained on it. It turns out that 1.2 million training examples are enough to train networks\nwhich are too big to \ufb01t on one GPU. Therefore we spread the net across two GPUs. Current GPUs\nare particularly well-suited to cross-GPU parallelization, as they are able to read from and write to\none another\u2019s memory directly, without going through host machine memory. The parallelization\nscheme that we employ essentially puts half of the kernels (or neurons) on each GPU, with one\nadditional trick: the GPUs communicate only in certain layers. This means that, for example, the\nkernels of layer 3 take input from all kernel maps in layer 2. However, kernels in layer 4 take input\nonly from those kernel maps in layer 3 which reside on the same GPU. Choosing the pattern of\nconnectivity is a problem for cross-validation, but this allows us to precisely tune the amount of\ncommunication until it is an acceptable fraction of the amount of computation.\nThe resultant architecture is somewhat similar to that of the \u201ccolumnar\u201d CNN employed by Cire\u00b8san\net al. [5], except that our columns are not independent (see Figure 2). This scheme reduces our top-1\nand top-5 error rates by 1.7% and 1.2%, respectively, as compared with a net with half as many\nkernels in each convolutional layer trained on one GPU. The two-GPU net takes slightly less time\nto train than the one-GPU net2.\n\n2The one-GPU net actually has the same number of kernels as the two-GPU net in the \ufb01nal convolutional\nlayer. This is because most of the net\u2019s parameters are in the \ufb01rst fully-connected layer, which takes the last\nconvolutional layer as input. So to make the two nets have approximately the same number of parameters, we\ndid not halve the size of the \ufb01nal convolutional layer (nor the fully-conneced layers which follow). Therefore\nthis comparison is biased in favor of the one-GPU net, since it is bigger than \u201chalf the size\u201d of the two-GPU\nnet.\n\n3\n\n\f3.3 Local Response Normalization\n\nReLUs have the desirable property that they do not require input normalization to prevent them\nfrom saturating. If at least some training examples produce a positive input to a ReLU, learning will\nhappen in that neuron. However, we still \ufb01nd that the following local normalization scheme aids\nx,y the activity of a neuron computed by applying kernel i at position\ngeneralization. Denoting by ai\n(x, y) and then applying the ReLU nonlinearity, the response-normalized activity bi\nx,y is given by\nthe expression\n\n\uf8eb\uf8edk + \u03b1\n\nmin(N\u22121,i+n/2)(cid:88)\n\nj=max(0,i\u2212n/2)\n\n\uf8f6\uf8f8\u03b2\n\n(aj\n\nx,y)2\n\nbi\nx,y = ai\n\nx,y/\n\nwhere the sum runs over n \u201cadjacent\u201d kernel maps at the same spatial position, and N is the total\nnumber of kernels in the layer. The ordering of the kernel maps is of course arbitrary and determined\nbefore training begins. This sort of response normalization implements a form of lateral inhibition\ninspired by the type found in real neurons, creating competition for big activities amongst neuron\noutputs computed using different kernels. The constants k, n, \u03b1, and \u03b2 are hyper-parameters whose\nvalues are determined using a validation set; we used k = 2, n = 5, \u03b1 = 10\u22124, and \u03b2 = 0.75. We\napplied this normalization after applying the ReLU nonlinearity in certain layers (see Section 3.5).\nThis scheme bears some resemblance to the local contrast normalization scheme of Jarrett et al. [11],\nbut ours would be more correctly termed \u201cbrightness normalization\u201d, since we do not subtract the\nmean activity. Response normalization reduces our top-1 and top-5 error rates by 1.4% and 1.2%,\nrespectively. We also veri\ufb01ed the effectiveness of this scheme on the CIFAR-10 dataset: a four-layer\nCNN achieved a 13% test error rate without normalization and 11% with normalization3.\n\n3.4 Overlapping Pooling\n\nPooling layers in CNNs summarize the outputs of neighboring groups of neurons in the same kernel\nmap. Traditionally, the neighborhoods summarized by adjacent pooling units do not overlap (e.g.,\n[17, 11, 4]). To be more precise, a pooling layer can be thought of as consisting of a grid of pooling\nunits spaced s pixels apart, each summarizing a neighborhood of size z \u00d7 z centered at the location\nof the pooling unit. If we set s = z, we obtain traditional local pooling as commonly employed\nin CNNs. If we set s < z, we obtain overlapping pooling. This is what we use throughout our\nnetwork, with s = 2 and z = 3. This scheme reduces the top-1 and top-5 error rates by 0.4% and\n0.3%, respectively, as compared with the non-overlapping scheme s = 2, z = 2, which produces\noutput of equivalent dimensions. We generally observe during training that models with overlapping\npooling \ufb01nd it slightly more dif\ufb01cult to over\ufb01t.\n\n3.5 Overall Architecture\n\nNow we are ready to describe the overall architecture of our CNN. As depicted in Figure 2, the net\ncontains eight layers with weights; the \ufb01rst \ufb01ve are convolutional and the remaining three are fully-\nconnected. The output of the last fully-connected layer is fed to a 1000-way softmax which produces\na distribution over the 1000 class labels. Our network maximizes the multinomial logistic regression\nobjective, which is equivalent to maximizing the average across training cases of the log-probability\nof the correct label under the prediction distribution.\nThe kernels of the second, fourth, and \ufb01fth convolutional layers are connected only to those kernel\nmaps in the previous layer which reside on the same GPU (see Figure 2). The kernels of the third\nconvolutional layer are connected to all kernel maps in the second layer. The neurons in the fully-\nconnected layers are connected to all neurons in the previous layer. Response-normalization layers\nfollow the \ufb01rst and second convolutional layers. Max-pooling layers, of the kind described in Section\n3.4, follow both response-normalization layers as well as the \ufb01fth convolutional layer. The ReLU\nnon-linearity is applied to the output of every convolutional and fully-connected layer.\nThe \ufb01rst convolutional layer \ufb01lters the 224\u00d7 224\u00d7 3 input image with 96 kernels of size 11\u00d7 11\u00d7 3\nwith a stride of 4 pixels (this is the distance between the receptive \ufb01eld centers of neighboring\n\n3We cannot describe this network in detail due to space constraints, but it is speci\ufb01ed precisely by the code\n\nand parameter \ufb01les provided here: http://code.google.com/p/cuda-convnet/.\n\n4\n\n\fFigure 2: An illustration of the architecture of our CNN, explicitly showing the delineation of responsibilities\nbetween the two GPUs. One GPU runs the layer-parts at the top of the \ufb01gure while the other runs the layer-parts\nat the bottom. The GPUs communicate only at certain layers. The network\u2019s input is 150,528-dimensional, and\nthe number of neurons in the network\u2019s remaining layers is given by 253,440\u2013186,624\u201364,896\u201364,896\u201343,264\u2013\n4096\u20134096\u20131000.\n\nneurons in a kernel map). The second convolutional layer takes as input the (response-normalized\nand pooled) output of the \ufb01rst convolutional layer and \ufb01lters it with 256 kernels of size 5 \u00d7 5 \u00d7 48.\nThe third, fourth, and \ufb01fth convolutional layers are connected to one another without any intervening\npooling or normalization layers. The third convolutional layer has 384 kernels of size 3 \u00d7 3 \u00d7\n256 connected to the (normalized, pooled) outputs of the second convolutional layer. The fourth\nconvolutional layer has 384 kernels of size 3 \u00d7 3 \u00d7 192 , and the \ufb01fth convolutional layer has 256\nkernels of size 3 \u00d7 3 \u00d7 192. The fully-connected layers have 4096 neurons each.\n\n4 Reducing Over\ufb01tting\n\nOur neural network architecture has 60 million parameters. Although the 1000 classes of ILSVRC\nmake each training example impose 10 bits of constraint on the mapping from image to label, this\nturns out to be insuf\ufb01cient to learn so many parameters without considerable over\ufb01tting. Below, we\ndescribe the two primary ways in which we combat over\ufb01tting.\n\n4.1 Data Augmentation\n\nThe easiest and most common method to reduce over\ufb01tting on image data is to arti\ufb01cially enlarge\nthe dataset using label-preserving transformations (e.g., [25, 4, 5]). We employ two distinct forms\nof data augmentation, both of which allow transformed images to be produced from the original\nimages with very little computation, so the transformed images do not need to be stored on disk.\nIn our implementation, the transformed images are generated in Python code on the CPU while the\nGPU is training on the previous batch of images. So these data augmentation schemes are, in effect,\ncomputationally free.\nThe \ufb01rst form of data augmentation consists of generating image translations and horizontal re\ufb02ec-\ntions. We do this by extracting random 224\u00d7 224 patches (and their horizontal re\ufb02ections) from the\n256\u00d7256 images and training our network on these extracted patches4. This increases the size of our\ntraining set by a factor of 2048, though the resulting training examples are, of course, highly inter-\ndependent. Without this scheme, our network suffers from substantial over\ufb01tting, which would have\nforced us to use much smaller networks. At test time, the network makes a prediction by extracting\n\ufb01ve 224 \u00d7 224 patches (the four corner patches and the center patch) as well as their horizontal\nre\ufb02ections (hence ten patches in all), and averaging the predictions made by the network\u2019s softmax\nlayer on the ten patches.\nThe second form of data augmentation consists of altering the intensities of the RGB channels in\ntraining images. Speci\ufb01cally, we perform PCA on the set of RGB pixel values throughout the\nImageNet training set. To each training image, we add multiples of the found principal components,\n\n4This is the reason why the input images in Figure 2 are 224 \u00d7 224 \u00d7 3-dimensional.\n\n5\n\n\fwith magnitudes proportional to the corresponding eigenvalues times a random variable drawn from\na Gaussian with mean zero and standard deviation 0.1. Therefore to each RGB image pixel Ixy =\n[I R\n\nxy]T we add the following quantity:\n\nxy, I G\n\nxy, I B\n\n[p1, p2, p3][\u03b11\u03bb1, \u03b12\u03bb2, \u03b13\u03bb3]T\n\nwhere pi and \u03bbi are ith eigenvector and eigenvalue of the 3 \u00d7 3 covariance matrix of RGB pixel\nvalues, respectively, and \u03b1i is the aforementioned random variable. Each \u03b1i is drawn only once\nfor all the pixels of a particular training image until that image is used for training again, at which\npoint it is re-drawn. This scheme approximately captures an important property of natural images,\nnamely, that object identity is invariant to changes in the intensity and color of the illumination. This\nscheme reduces the top-1 error rate by over 1%.\n\n4.2 Dropout\n\nCombining the predictions of many different models is a very successful way to reduce test errors\n[1, 3], but it appears to be too expensive for big neural networks that already take several days\nto train. There is, however, a very ef\ufb01cient version of model combination that only costs about a\nfactor of two during training. The recently-introduced technique, called \u201cdropout\u201d [10], consists\nof setting to zero the output of each hidden neuron with probability 0.5. The neurons which are\n\u201cdropped out\u201d in this way do not contribute to the forward pass and do not participate in back-\npropagation. So every time an input is presented, the neural network samples a different architecture,\nbut all these architectures share weights. This technique reduces complex co-adaptations of neurons,\nsince a neuron cannot rely on the presence of particular other neurons. It is, therefore, forced to\nlearn more robust features that are useful in conjunction with many different random subsets of the\nother neurons. At test time, we use all the neurons but multiply their outputs by 0.5, which is a\nreasonable approximation to taking the geometric mean of the predictive distributions produced by\nthe exponentially-many dropout networks.\nWe use dropout in the \ufb01rst two fully-connected layers of Figure 2. Without dropout, our network ex-\nhibits substantial over\ufb01tting. Dropout roughly doubles the number of iterations required to converge.\n\n5 Details of learning\n\nWe trained our models using stochastic gradient descent\nwith a batch size of 128 examples, momentum of 0.9, and\nweight decay of 0.0005. We found that this small amount\nof weight decay was important for the model to learn. In\nother words, weight decay here is not merely a regularizer:\nit reduces the model\u2019s training error. The update rule for\nweight w was\n\nvi+1\n\n:= 0.9 \u00b7 vi \u2212 0.0005 \u00b7 \u0001 \u00b7 wi \u2212 \u0001 \u00b7\n\nwi+1\n\n:= wi + vi+1\n\n(cid:28) \u2202L\n\n\u2202w\n\n(cid:12)(cid:12)wi\n\n(cid:29)\n\nDi\n\nFigure 3: 96 convolutional kernels of size\n11\u00d711\u00d73 learned by the \ufb01rst convolutional\nlayer on the 224\u00d7224\u00d73 input images. The\ntop 48 kernels were learned on GPU 1 while\nthe bottom 48 kernels were learned on GPU\n2. See Section 6.1 for details.\n\n(cid:68) \u2202L\n\n(cid:69)\n\n(cid:12)(cid:12)wi\n\n\u2202w\n\nwhere i is the iteration index, v is the momentum variable, \u0001 is the learning rate, and\nis\nthe average over the ith batch Di of the derivative of the objective with respect to w, evaluated at\nwi.\nWe initialized the weights in each layer from a zero-mean Gaussian distribution with standard de-\nviation 0.01. We initialized the neuron biases in the second, fourth, and \ufb01fth convolutional layers,\nas well as in the fully-connected hidden layers, with the constant 1. This initialization accelerates\nthe early stages of learning by providing the ReLUs with positive inputs. We initialized the neuron\nbiases in the remaining layers with the constant 0.\nWe used an equal learning rate for all layers, which we adjusted manually throughout training.\nThe heuristic which we followed was to divide the learning rate by 10 when the validation error\nrate stopped improving with the current learning rate. The learning rate was initialized at 0.01 and\n\nDi\n\n6\n\n\freduced three times prior to termination. We trained the network for roughly 90 cycles through the\ntraining set of 1.2 million images, which took \ufb01ve to six days on two NVIDIA GTX 580 3GB GPUs.\n\n6 Results\n\nOur results on ILSVRC-2010 are summarized in Table 1. Our network achieves top-1 and top-5\ntest set error rates of 37.5% and 17.0%5. The best performance achieved during the ILSVRC-\n2010 competition was 47.1% and 28.2% with an approach that averages the predictions produced\nfrom six sparse-coding models trained on different features [2], and since then the best pub-\nlished results are 45.7% and 25.7% with an approach that averages the predictions of two classi-\n\ufb01ers trained on Fisher Vectors (FVs) computed from two types of densely-sampled features [24].\n\nTop-1\nTop-5\n47.1% 28.2%\n45.7% 25.7%\n37.5% 17.0%\n\nModel\nSparse coding [2]\nSIFT + FVs [24]\nCNN\n\nTable 1: Comparison of results on ILSVRC-\nIn italics are best results\n2010 test set.\nachieved by others.\n\nWe also entered our model in the ILSVRC-2012 com-\npetition and report our results in Table 2. Since the\nILSVRC-2012 test set labels are not publicly available,\nwe cannot report test error rates for all the models that\nwe tried.\nIn the remainder of this paragraph, we use\nvalidation and test error rates interchangeably because\nin our experience they do not differ by more than 0.1%\n(see Table 2). The CNN described in this paper achieves\na top-5 error rate of 18.2%. Averaging the predictions\nof \ufb01ve similar CNNs gives an error rate of 16.4%. Training one CNN, with an extra sixth con-\nvolutional layer over the last pooling layer, to classify the entire ImageNet Fall 2011 release\n(15M images, 22K categories), and then \u201c\ufb01ne-tuning\u201d it on ILSVRC-2012 gives an error rate of\n16.6%. Averaging the predictions of two CNNs that were pre-trained on the entire Fall 2011 re-\nlease with the aforementioned \ufb01ve CNNs gives an error rate of 15.3%. The second-best con-\ntest entry achieved an error rate of 26.2% with an approach that averages the predictions of sev-\neral classi\ufb01ers trained on FVs computed from different types of densely-sampled features [7].\nFinally, we also report our error\nrates on the Fall 2009 version of\nImageNet with 10,184 categories\nand 8.9 million images. On this\ndataset we follow the convention\nin the literature of using half of\nthe images for training and half\nfor testing. Since there is no es-\ntablished test set, our split neces-\nsarily differs from the splits used\nby previous authors, but this does\nnot affect the results appreciably.\nOur top-1 and top-5 error rates\non this dataset are 67.4% and\n40.9%, attained by the net described above but with an additional, sixth convolutional layer over the\nlast pooling layer. The best published results on this dataset are 78.1% and 60.9% [19].\n\nTable 2: Comparison of error rates on ILSVRC-2012 validation and\ntest sets. In italics are best results achieved by others. Models with an\nasterisk* were \u201cpre-trained\u201d to classify the entire ImageNet 2011 Fall\nrelease. See Section 6 for details.\n\nModel\nSIFT + FVs [7]\n1 CNN\n5 CNNs\n1 CNN*\n7 CNNs*\n\nTop-1 (val) Top-5 (val) Top-5 (test)\n\n\u2014\n\n40.7%\n38.1%\n39.0%\n36.7%\n\n\u2014\n\n18.2%\n16.4%\n16.6%\n15.4%\n\n26.2%\n\n16.4%\n\n\u2014\n\n\u2014\n\n15.3%\n\n6.1 Qualitative Evaluations\n\nFigure 3 shows the convolutional kernels learned by the network\u2019s two data-connected layers. The\nnetwork has learned a variety of frequency- and orientation-selective kernels, as well as various col-\nored blobs. Notice the specialization exhibited by the two GPUs, a result of the restricted connec-\ntivity described in Section 3.5. The kernels on GPU 1 are largely color-agnostic, while the kernels\non on GPU 2 are largely color-speci\ufb01c. This kind of specialization occurs during every run and is\nindependent of any particular random weight initialization (modulo a renumbering of the GPUs).\n\n5The error rates without averaging predictions over ten patches as described in Section 4.1 are 39.0% and\n\n18.3%.\n\n7\n\n\fFigure 4: (Left) Eight ILSVRC-2010 test images and the \ufb01ve labels considered most probable by our model.\nThe correct label is written under each image, and the probability assigned to the correct label is also shown\nwith a red bar (if it happens to be in the top 5). (Right) Five ILSVRC-2010 test images in the \ufb01rst column. The\nremaining columns show the six training images that produce feature vectors in the last hidden layer with the\nsmallest Euclidean distance from the feature vector for the test image.\n\nIn the left panel of Figure 4 we qualitatively assess what the network has learned by computing its\ntop-5 predictions on eight test images. Notice that even off-center objects, such as the mite in the\ntop-left, can be recognized by the net. Most of the top-5 labels appear reasonable. For example,\nonly other types of cat are considered plausible labels for the leopard. In some cases (grille, cherry)\nthere is genuine ambiguity about the intended focus of the photograph.\nAnother way to probe the network\u2019s visual knowledge is to consider the feature activations induced\nby an image at the last, 4096-dimensional hidden layer. If two images produce feature activation\nvectors with a small Euclidean separation, we can say that the higher levels of the neural network\nconsider them to be similar. Figure 4 shows \ufb01ve images from the test set and the six images from\nthe training set that are most similar to each of them according to this measure. Notice that at the\npixel level, the retrieved training images are generally not close in L2 to the query images in the \ufb01rst\ncolumn. For example, the retrieved dogs and elephants appear in a variety of poses. We present the\nresults for many more test images in the supplementary material.\nComputing similarity by using Euclidean distance between two 4096-dimensional, real-valued vec-\ntors is inef\ufb01cient, but it could be made ef\ufb01cient by training an auto-encoder to compress these vectors\nto short binary codes. This should produce a much better image retrieval method than applying auto-\nencoders to the raw pixels [14], which does not make use of image labels and hence has a tendency\nto retrieve images with similar patterns of edges, whether or not they are semantically similar.\n\n7 Discussion\n\nOur results show that a large, deep convolutional neural network is capable of achieving record-\nbreaking results on a highly challenging dataset using purely supervised learning.\nIt is notable\nthat our network\u2019s performance degrades if a single convolutional layer is removed. For example,\nremoving any of the middle layers results in a loss of about 2% for the top-1 performance of the\nnetwork. So the depth really is important for achieving our results.\nTo simplify our experiments, we did not use any unsupervised pre-training even though we expect\nthat it will help, especially if we obtain enough computational power to signi\ufb01cantly increase the\nsize of the network without obtaining a corresponding increase in the amount of labeled data. Thus\nfar, our results have improved as we have made our network larger and trained it longer but we still\nhave many orders of magnitude to go in order to match the infero-temporal pathway of the human\nvisual system. Ultimately we would like to use very large and deep convolutional nets on video\nsequences where the temporal structure provides very helpful information that is missing or far less\nobvious in static images.\n\n8\n\n\fReferences\n[1] R.M. Bell and Y. Koren. Lessons from the net\ufb02ix prize challenge. ACM SIGKDD Explorations Newsletter,\n\n[2] A. Berg, J. Deng, and L. Fei-Fei.\n\nLarge scale visual recognition challenge 2010. www.image-\n\n9(2):75\u201379, 2007.\n\nnet.org/challenges. 2010.\n\n[3] L. Breiman. Random forests. Machine learning, 45(1):5\u201332, 2001.\n[4] D. Cire\u00b8san, U. Meier, and J. Schmidhuber. Multi-column deep neural networks for image classi\ufb01cation.\n\nArxiv preprint arXiv:1202.2745, 2012.\n\n[5] D.C. Cire\u00b8san, U. Meier, J. Masci, L.M. Gambardella, and J. Schmidhuber. High-performance neural\n\nnetworks for visual object classi\ufb01cation. Arxiv preprint arXiv:1102.0183, 2011.\n\n[6] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.\n\nImageNet: A Large-Scale Hierarchical\n\nImage Database. In CVPR09, 2009.\n\n[7] J. Deng, A. Berg, S. Satheesh, H. Su, A. Khosla, and L. Fei-Fei.\nhttp://www.image-net.org/challenges/LSVRC/2012/.\n\nILSVRC-2012, 2012. URL\n\n[8] L. Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training examples: An\nincremental bayesian approach tested on 101 object categories. Computer Vision and Image Understand-\ning, 106(1):59\u201370, 2007.\n\n[9] G. Grif\ufb01n, A. Holub, and P. Perona. Caltech-256 object category dataset. Technical Report 7694, Cali-\n\nfornia Institute of Technology, 2007. URL http://authors.library.caltech.edu/7694.\n\n[10] G.E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R.R. Salakhutdinov. Improving neural net-\n\nworks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580, 2012.\n\n[11] K. Jarrett, K. Kavukcuoglu, M. A. Ranzato, and Y. LeCun. What is the best multi-stage architecture for\n\nobject recognition? In International Conference on Computer Vision, pages 2146\u20132153. IEEE, 2009.\n\n[12] A. Krizhevsky. Learning multiple layers of features from tiny images. Master\u2019s thesis, Department of\n\nComputer Science, University of Toronto, 2009.\n\n[13] A. Krizhevsky. Convolutional deep belief networks on cifar-10. Unpublished manuscript, 2010.\n[14] A. Krizhevsky and G.E. Hinton. Using very deep autoencoders for content-based image retrieval.\n\nIn\n\nESANN, 2011.\n\n[15] Y. Le Cun, B. Boser, J.S. Denker, D. Henderson, R.E. Howard, W. Hubbard, L.D. Jackel, et al. Hand-\nwritten digit recognition with a back-propagation network. In Advances in neural information processing\nsystems, 1990.\n\n[16] Y. LeCun, F.J. Huang, and L. Bottou. Learning methods for generic object recognition with invariance to\npose and lighting. In Computer Vision and Pattern Recognition, 2004. CVPR 2004. Proceedings of the\n2004 IEEE Computer Society Conference on, volume 2, pages II\u201397. IEEE, 2004.\n\n[17] Y. LeCun, K. Kavukcuoglu, and C. Farabet. Convolutional networks and applications in vision.\n\nIn\nCircuits and Systems (ISCAS), Proceedings of 2010 IEEE International Symposium on, pages 253\u2013256.\nIEEE, 2010.\n\n[18] H. Lee, R. Grosse, R. Ranganath, and A.Y. Ng. Convolutional deep belief networks for scalable unsuper-\nvised learning of hierarchical representations. In Proceedings of the 26th Annual International Conference\non Machine Learning, pages 609\u2013616. ACM, 2009.\n\n[19] T. Mensink, J. Verbeek, F. Perronnin, and G. Csurka. Metric Learning for Large Scale Image Classi\ufb01-\ncation: Generalizing to New Classes at Near-Zero Cost. In ECCV - European Conference on Computer\nVision, Florence, Italy, October 2012.\n\n[20] V. Nair and G. E. Hinton. Recti\ufb01ed linear units improve restricted boltzmann machines. In Proc. 27th\n\nInternational Conference on Machine Learning, 2010.\n\n[21] N. Pinto, D.D. Cox, and J.J. DiCarlo. Why is real-world visual object recognition hard? PLoS computa-\n\ntional biology, 4(1):e27, 2008.\n\n[22] N. Pinto, D. Doukhan, J.J. DiCarlo, and D.D. Cox. A high-throughput screening approach to discovering\ngood forms of biologically inspired visual representation. PLoS computational biology, 5(11):e1000579,\n2009.\n\n[23] B.C. Russell, A. Torralba, K.P. Murphy, and W.T. Freeman. Labelme: a database and web-based tool for\n\nimage annotation. International journal of computer vision, 77(1):157\u2013173, 2008.\n\n[24] J. S\u00e1nchez and F. Perronnin. High-dimensional signature compression for large-scale image classi\ufb01cation.\nIn Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pages 1665\u20131672. IEEE,\n2011.\n\n[25] P.Y. Simard, D. Steinkraus, and J.C. Platt. Best practices for convolutional neural networks applied to\nvisual document analysis. In Proceedings of the Seventh International Conference on Document Analysis\nand Recognition, volume 2, pages 958\u2013962, 2003.\n\n[26] S.C. Turaga, J.F. Murray, V. Jain, F. Roth, M. Helmstaedter, K. Briggman, W. Denk, and H.S. Seung. Con-\nvolutional networks can learn to generate af\ufb01nity graphs for image segmentation. Neural Computation,\n22(2):511\u2013538, 2010.\n\n9\n\n\f", "award": [], "sourceid": 534, "authors": [{"given_name": "Alex", "family_name": "Krizhevsky", "institution": null}, {"given_name": "Ilya", "family_name": "Sutskever", "institution": null}, {"given_name": "Geoffrey", "family_name": "Hinton", "institution": null}]}