{"title": "DeViSE: A Deep Visual-Semantic Embedding Model", "book": "Advances in Neural Information Processing Systems", "page_first": 2121, "page_last": 2129, "abstract": "Modern visual recognition systems are often limited in their ability to scale to large numbers of object categories. This limitation is in part due to the increasing difficulty of acquiring sufficient training data in the form of labeled images as the number of object categories grows. One remedy is to leverage data from other sources -- such as text data -- both to train visual models and to constrain their predictions. In this paper we present a new deep visual-semantic embedding model trained to identify visual objects using both labeled image data as well as semantic information gleaned from unannotated text. We demonstrate that this model matches state-of-the-art performance on the 1000-class ImageNet object recognition challenge while making more semantically reasonable errors, and also show that the semantic information can be exploited to make predictions about tens of thousands of image labels not observed during training. Semantic knowledge improves such zero-shot predictions by up to 65%, achieving hit rates of up to 10% across thousands of novel labels never seen by the visual model.", "full_text": "DeViSE: A Deep Visual-Semantic Embedding Model\n\nAndrea Frome*, Greg S. Corrado*, Jonathon Shlens*, Samy Bengio\n\nJeffrey Dean, Marc\u2019Aurelio Ranzato, Tomas Mikolov\n\n* These authors contributed equally.\n\n{afrome, gcorrado, shlens, bengio, jeff, ranzato\u2020, tmikolov}@google.com\n\nGoogle, Inc.\n\nMountain View, CA, USA\n\nAbstract\n\nModern visual recognition systems are often limited in their ability to scale to\nlarge numbers of object categories. This limitation is in part due to the increasing\ndif\ufb01culty of acquiring suf\ufb01cient training data in the form of labeled images as the\nnumber of object categories grows. One remedy is to leverage data from other\nsources \u2013 such as text data \u2013 both to train visual models and to constrain their pre-\ndictions. In this paper we present a new deep visual-semantic embedding model\ntrained to identify visual objects using both labeled image data as well as seman-\ntic information gleaned from unannotated text. We demonstrate that this model\nmatches state-of-the-art performance on the 1000-class ImageNet object recogni-\ntion challenge while making more semantically reasonable errors, and also show\nthat the semantic information can be exploited to make predictions about tens\nof thousands of image labels not observed during training. Semantic knowledge\nimproves such zero-shot predictions achieving hit rates of up to 18% across thou-\nsands of novel labels never seen by the visual model.\n\n1\n\nIntroduction\n\nThe visual world is populated with a vast number of objects, the most appropriate labeling of which\nis often ambiguous, task speci\ufb01c, or admits multiple equally correct answers. Yet state-of-the-\nart vision systems attempt to solve recognition tasks by arti\ufb01cially assigning images to a small\nnumber of rigidly de\ufb01ned classes. This has led to building labeled image data sets according to\nthese arti\ufb01cial categories and in turn to building visual recognition systems based on N-way discrete\nclassi\ufb01ers. While growing the number of labels and labeled images has improved the utility of\nvisual recognition systems [7], scaling such systems beyond a limited number of discrete categories\nremains an unsolved problem. This problem is exacerbated by the fact that N-way discrete classi\ufb01ers\ntreat all labels as disconnected and unrelated, resulting in visual recognition systems that cannot\ntransfer semantic information about learned labels to unseen words or phrases. One way of dealing\nwith this issue is to respect the natural continuity of visual space instead of arti\ufb01cially partitioning\nit into disjoint categories [20].\nWe propose an approach that addresses these shortcomings by training a visual recognition model\nwith both labeled images and a comparatively large and independent dataset \u2013 semantic information\nfrom unannotated text data. This deep visual-semantic embedding model (DeViSE) leverages textual\ndata to learn semantic relationships between labels, and explicitly maps images into a rich semantic\nembedding space. We show that this model performs comparably to state-of-the-art visual object\nclassi\ufb01ers when trained and evaluated on \ufb02at 1-of-N metrics, while simultaneously making fewer\nsemantically unreasonable mistakes along the way. Furthermore, we show that the model leverages\n\n\u2020Current af\ufb01liation: Facebook, Inc.\n\n1\n\n\fvisual and semantic similarity to correctly predict object category labels for unseen categories, i.e.\n\u201czero-shot\u201d classi\ufb01cation, even when the number of unseen visual categories is 20,000 for a model\ntrained on just 1,000 categories.\n\n2 Previous Work\n\nThe current state-of-the-art approach to image classi\ufb01cation is a deep convolutional neural network\ntrained with a softmax output layer (i.e. multinomial logistic regression) that has as many units\nas the number of classes (see, for instance [11]). However, as the number of classes grows, the\ndistinction between classes blurs, and it becomes increasingly dif\ufb01cult to obtain suf\ufb01cient numbers\nof training images for rare concepts.\nOne solution to this problem, termed WSABIE [20], is to train a joint embedding model of both im-\nages and labels, by employing an online learning-to-rank algorithm. The proposed model contained\ntwo sets of parameters: (1) a linear mapping from image features to the joint embedding space, and\n(2) an embedding vector for each possible label. Compared to the proposed approach, WSABIE\nonly explored linear mappings from image features to the embedding space, and the available labels\nwere only those provided in the image training set. It could thus not generalize to new classes.\nMore recently, Socher et al [18] presented a model for zero-shot learning where a deep neural\nnetwork was \ufb01rst trained in an unsupervised manner from many images in order to obtain a rich\nimage representation [3]; in parallel, a neural network language model [2] was trained in order to\nobtain embedding representations for thousands of common terms. The authors trained a linear\nmapping between the image representations and the word embeddings representing 8 classes for\nwhich they had labeled images, thus linking the image representation space to the embedding space.\nThis last step was performed using a mean-squared error criterion. They also trained a simple model\nto determine if a given image was from any of the 8 original classes or not (i.e., an outlier detector).\nWhen the model determined an image to be in the set of 8 classes, a separately trained softmax\nmodel was used to perform the 8-way classi\ufb01cation; otherwise the model predicted the nearest class\nin the embedding space (in their setting, only 2 outlier classes were considered). Their model differs\nfrom our proposed approach in several ways: \ufb01rst and foremost, the scale, as our model considers\n1,000 known classes for the image model and up to 20,000 unknown classes, instead of respectively\n8 and 2; second, in [18] there is an inherent trade-off between the quality of predictions for trained\nand outlier classes; third, by using a different visual model, different language model, and different\ntraining objective, we were able to train a single uni\ufb01ed model that uses only embeddings.\nThere has been other recent work showing impressive zero-shot performance on visual recognition\ntasks [12, 17, 16], however all of these rely on a curated source of semantic information for the\nlabels: the WordNet hierarchy is used in [12] and [17], and [16] uses a knowledge base containing\ndescriptive properties for each class. By contrast, our approach learns its semantic representation\ndirectly from unannotated data.\n\n3 Proposed Approach\n\nOur objective is to leverage semantic knowledge learned in the text domain, and transfer it to a model\ntrained for visual object recognition. We begin by pre-training a simple neural language model well-\nsuited for learning semantically-meaningful, dense vector representations of words [13]. In parallel,\nwe pre-train a state-of-the-art deep neural network for visual object recognition [11], complete with\na traditional softmax output layer. We then construct a deep visual-semantic model by taking the\nlower layers of the pre-trained visual object recognition network and re-training them to predict the\nvector representation of the image label text as learned by the language model. These three training\nphases are detailed below.\n\n3.1 Language Model Pre-training\n\nThe skip-gram text modeling architecture introduced by Mikolov et al [13, 14] has been shown to\nef\ufb01ciently learn semantically-meaningful \ufb02oating point representations of terms from unannotated\ntext. The model learns to represent each term as a \ufb01xed length embedding vector by predicting\nadjacent terms in the document (Figure 1a, right). We call these vector representations embedding\n\n2\n\n\fFigure 1: (a) Left: a visual object categorization network with a softmax output layer; Right: a skip-gram\nlanguage model; Center: our joint model, which is initialized with parameters pre-trained at the lower layers\nof the other two models. (b) t-SNE visualization [19] of a subset of the ILSVRC 2012 1K label embeddings\nlearned using skip-gram.\n\nvectors. Because synonyms tend to appear in similar contexts, this simple objective function drives\nthe model to learn similar embedding vectors for semantically related words.\nWe trained a skip-gram text model on a corpus of 5.7 million documents (5.4 billion words) extracted\nfrom wikipedia.org. The text of the web pages was tokenized into a lexicon of roughly 155,000\nsingle- and multi-word terms consisting of common English words and phrases as well as terms from\ncommonly used visual object recognition datasets [7]. Our skip-gram model used a hierarchical\nsoftmax layer for predicting adjacent terms and was trained using a 20-word window with a single\npass through the corpus. For more details and a pointer to open-source code, see [13].\nWe trained skip-gram models of varying hidden dimensions, ranging from 100-D to 2,000-D, and\nfound 500- and 1,000-D embeddings to be a good compromise between training speed, semantic\nquality, and the ultimate performance of the DeViSE model described below. The semantic quality\nof the embedding representations learned by these models is impressive.1 A visualization of the lan-\nguage embedding space over a subset of ImageNet labels indicates that the language model learned\na rich semantic structure that could be exploited in vision tasks (Figure 1b).\n\n3.2 Visual Model Pre-training\n\nThe visual model architecture we employ is based on the winning model for the 1,000-class Ima-\ngeNet Large Scale Visual Recognition Challenge (ILSVRC) 2012 [11, 6]. The deep neural network\nmodel consists of several convolutional \ufb01ltering, local contrast normalization, and max-pooling lay-\ners, followed by several fully connected neural network layers trained using the dropout regular-\nization technique [10]. We trained this model with a softmax output layer, as described in [11], to\npredict one of 1,000 object categories from the ILSVRC 2012 1K dataset [7], and were able to repro-\nduce their results. This trained model serves both as our benchmark for performance comparisons,\nas well as the initialization for our joint model.\n\n3.3 Deep Visual-Semantic Embedding Model\n\nOur deep visual-semantic embedding model (DeViSE) is initialized from these two pre-trained neu-\nral network models (Figure 1a). The embedding vectors learned by the language model are unit\nnormed and used to map label terms into target vector representations2.\nThe core visual model, with its softmax prediction layer now removed, is trained to predict these\nvectors for each image, by means of a projection layer and a similarity metric. The projection layer\nis a linear transformation that maps the 4,096-D representation at the top of our core visual model\ninto the 500- or 1,000-D representation native to our language model.\n\n1For example, the 9 nearest terms to tiger shark using cosine distance are bull shark, blacktip shark, shark,\noceanic whitetip shark, sandbar shark, dusky shark, blue shark, requiem shark, and great white shark. The\n9 nearest terms to car are cars, muscle car, sports car, compact car, automobile, racing car, pickup truck,\ndealership, and sedans.\n\n2In [13], which introduced the skip-gram model for text, cosine similarity between vectors is used for\nmeasuring semantic similarity. Unit-norming the vectors and using dot product similarity is an equivalent\nsimilarity measurement.\n\n3\n\ntransportationdogsbirdsmusical instrumentsaquatic lifeinsectsanimalsclothingfoodreptilesembeddingvectorlookup tableembeddingvectorlookup tablesimilarity metriclabelimagecorevisualmodeltransformationDeep Visual SemanticEmbedding Modelimagelabelsoftmax layercorevisualmodelTraditionalVisual Modelsource wordnearby wordsoftmax layerSkip-gramLanguage ModelparameterinitializationparameterinitializationAB\fThe choice of loss function proved to be important. We used a combination of dot-product similarity\nand hinge rank loss (similar to [20]) such that the model was trained to produce a higher dot-product\nsimilarity between the visual model output and the vector representation of the correct label than be-\ntween the visual output and other randomly chosen text terms. We de\ufb01ned the per training example\nhinge rank loss:\n\nloss(image, label) =\n\nmax[0, margin \u2212 (cid:126)tlabelM(cid:126)v(image) + (cid:126)tjM(cid:126)v(image)]\n\n(1)\n\n(cid:88)\n\nj(cid:54)=label\n\nwhere (cid:126)v(image) is a column vector denoting the output of the top layer of our core visual network\nfor the given image, M is the matrix of trainable parameters in the linear transformation layer,\n(cid:126)tlabel is a row vector denoting learned embedding vector for the provided text label, and (cid:126)tj are\nthe embeddings of other text terms. In practice, we found that it was expedient to randomize the\nalgorithm both by (1) restricting the set of false text terms to possible image labels, and (2) truncating\nthe sum after the \ufb01rst margin-violating false term was encountered. The (cid:126)t vectors were constrained\nto be unit norm, and a \ufb01xed margin of 0.1 was used in all experiments3. We also experimented\nwith an L2 loss between visual and label embeddings, as suggested by Socher et al. [18], but that\nconsistently yielded about half the accuracy of the rank loss model. We believe this is because the\nnearest neighbor evaluation is fundamentally a ranking problem and is best solved with a ranking\nloss, whereas the L2 loss only aims to make the vectors close to one another but remains agnostic to\nincorrect labels that are closer to the target image.\nThe DeViSE model was trained by asynchronous stochastic gradient descent on a distributed com-\nputing platform described in [4]. As above, the model was presented only with images drawn from\nthe ILSVRC 2012 1K training set, but now trained to predict the term strings as text4. The param-\neters of the projection layer M were \ufb01rst trained while holding both the core visual model and the\ntext representation \ufb01xed. In the later stages of training the derivative of the loss function was back-\npropagated into the core visual model to \ufb01ne-tune its output5, which typically improved accuracy\nby 1-3% (absolute). Adagrad per-parameter dynamic learning rates were utilized to keep gradients\nwell scaled at the different layers of the network [9].\nAt test time, when a new image arrives, one \ufb01rst computes its vector representation using the visual\nmodel and the transformation layer; then one needs to look for the nearest labels in the embedding\nspace. This last step can be done ef\ufb01ciently using either a tree or a hashing technique, in order to\nbe faster than the naive linear search approach (see for instance [1]). The nearest labels are then\nmapped back to ImageNet synsets for scoring (see Supplementary Materials for details).\n\n4 Results\n\nThe goals of this work are to develop a vision model that makes semantically relevant predictions\neven when it makes errors and that generalizes to classes outside of its labeled training set, i.e. zero-\nshot learning. We compare DeViSE to two models that employ the same high-quality core vision\nmodel, but lack the semantic structure imparted by our language model: (1) a softmax baseline\nmodel \u2013 a state-of-the-art vision model [11] which employs a 1000-way softmax classi\ufb01er; (2) a\nrandom embedding model \u2013 a version of our model that uses random unit-norm embedding vectors\nin place of those learned by the language model. Both use the trained visual model described in\nSection 3.2.\nIn order to demonstrate parity with the softmax baseline on the most commonly-reported metric, we\ncompute \u201c\ufb02at\u201d hit@k metrics \u2013 the percentage of test images for which the model returns the one\ntrue label in its top k predictions. To measure the semantic quality of predictions beyond the true\nlabel, we employ a hierarchical precision@k metric based on the label hierarchy provided with the\n\n3The margin was chosen to be a fraction of the norm of the vectors, which is 1.0. A wide range of values\n\nwould likely work well.\n\n4ImageNet image labels are synsets, a set of synonymous terms, where each term is a word or phrase. We\nfound training the model to predict the \ufb01rst term in each synset to be suf\ufb01cient, but sampling from the synset\nterms might work equally well.\n\n5In principle the gradients can also be back-propagated into the vector representations of the text labels. In\nthis case, the language model should continue to train simultaneously in order to maintain the global semantic\nstructure over all terms in the vocabulary.\n\n4\n\n\fModel type\nSoftmax baseline\nDeViSE\n\nRandom embeddings\n\nChance\n\ndim\nN/A\n500\n1000\n500\n1000\nN/A\n\n1\n55.6\n53.2\n54.9\n52.4\n50.5\n0.1\n\nFlat hit@k (%)\n\n2\n67.4\n65.2\n66.9\n63.9\n62.2\n0.2\n\n5\n78.5\n76.7\n78.4\n74.8\n74.2\n0.5\n\nHierarchical precision@k\n2\n\n10\n\n5\n\n20\n\n0.452\n0.447\n0.454\n0.428\n0.418\n0.007\n\n0.342\n0.352\n0.351\n0.315\n0.318\n0.013\n\n0.313\n0.331\n0.325\n0.271\n0.290\n0.022\n\n0.319\n0.341\n0.331\n0.248\n0.292\n0.042\n\n10\n85.0\n83.3\n85.0\n80.6\n81.5\n1.0\n\nTable 1: Comparison of model performance on our test set, taken from the ImageNet ILSVRC 2012 1K\nvalidation set. Note that hierarchical precision@1 is equivalent to \ufb02at hit@1. See text for details.\n\nImageNet image repository [7]. In particular, for each true label and value of k, we generate a ground\ntruth list from the semantic hierarchy, and compute a per-example precision equal to the fraction of\nthe model\u2019s k predictions that overlap with the ground truth list. We report mean precision across\nthe test set. Detailed descriptions of the generation of the ground truth lists, the hierarchical scoring\nmetric, and train/validation/test dataset splits are provided in the Supplementary Materials.\n\n4.1\n\nImageNet (ILSVRC) 2012 1K Results\n\nThis section presents \ufb02at and hierarchical results on the ILSVRC 2012 1K dataset, where the classes\nof the examples presented at test time are the same as those used for training. Table 1 shows results\nfor the DeViSE model for 500- and 1000-dimensional skip-gram models compared to the random\nembedding and softmax baseline models, on both the \ufb02at and hierarchical metrics.6\nOn the \ufb02at metric, the softmax baseline shows higher accuracy for k = 1, 2. At k = 5, 10, the\n1000-D DeViSE model has reached parity, and at k = 20 (not shown) it performs slightly better.\nWe expected the softmax model to be the best performing model on the \ufb02at metric, given that its\ncross-entropy training objective is most well matched to the evaluation metric, and are surprised that\nthe performance of DeViSE is so close to softmax performance.\nOn the hierarchical metric, the DeViSE models show better semantic generalization than the soft-\nmax baseline, especially for larger k. At k = 5, the 500-D DeViSE model shows a 3% relative\nimprovement over the softmax baseline, and at k = 20 almost a 7% relative improvement. This is a\nsurprisingly large gain, considering that the softmax baseline is a reproduction of the best published\nmodel on these data. The gap that exists between the DeViSE model and softmax baseline on the\nhierarchical metric re\ufb02ects the bene\ufb01t of semantic information above and beyond visual similar-\nity [8]. The gap between the DeViSE model and the random embeddings model establishes that the\nsource of the gain is the well-structured embeddings learned by the language model not some other\nproperty of our architecture.\n\n4.2 Generalization and Zero-Shot Learning\n\nA distinct advantage of our model is its ability to make reasonable inferences about candidate labels\nit has never visually observed. For example, a DeViSE model trained on images labeled tiger shark,\nbull shark, and blue shark, but never with images labeled shark, would likely have the ability to\ngeneralize to this more coarse-grained descriptor because the language model has learned a repre-\nsentation of the general concept of shark which is similar to all of the speci\ufb01c sharks. Similarly,\nif tested on images of highly speci\ufb01c classes which the model has never seen before, for example\na photo of an oceanic whitecap shark, and asked whether the correct label is more likely oceanic\nwhitecap shark or some other unfamiliar label (say, nuclear submarine), our model stands a \ufb01ght-\ning chance of guessing correctly because the language model ensures that representation of oceanic\nwhitecap shark is closer to the representation of sharks the model has seen, while the representation\nof nuclear submarine is closer to those of other sea vessels.\n\n6Note that our softmax baseline results differ from the results in [11] due to a simpli\ufb01cation in the evaluation\nprocedure: [11] creates several distorted versions of each test image and aggregates the results for a \ufb01nal label,\nwhereas in our experiments, we evaluate using only the original test image. Our softmax baseline is able to\nreproduce the performance of the model in [11] when evaluated with the same procedure.\n\n5\n\n\fFigure 2: For each image, the top 5 zero-shot predictions of DeViSE+1K from the 2011 21K label set and the\nsoftmax baseline model, both trained on ILSVRC 2012 1K. Predictions ordered by decreasing score, with cor-\nrect predictions in bold. Ground truth: (a) telephoto lens, zoom lens; (b) English horn, cor anglais; (c) babbler,\ncackler; (d) pineapple, pineapple plant, Ananas comosus; (e) salad bar; (f) spacecraft, ballistic capsule, space\nvehicle.\n\nData Set\n\n2-hop\n\n3-hop\n\nModel\nDeViSE-0\nDeViSE+1K\nDeViSE-0\nDeViSE+1K\n\nImageNet 2011 21K DeViSE-0\n\nDeViSE+1K\n\n# Candidate\n\nLabels\n1,589\n2,589\n7,860\n8,860\n20,841\n21,841\n\nFlat hit@k (%)\n\n1\n6.0\n0.8\n1.7\n0.5\n0.8\n0.3\n\n2\n\n10.0\n2.7\n2.9\n1.4\n1.4\n0.8\n\n5\n\n18.1\n7.9\n5.3\n3.4\n2.5\n1.9\n\n10\n26.4\n14.2\n8.2\n5.9\n3.9\n3.2\n\n20\n36.4\n22.7\n12.5\n9.7\n6.0\n5.3\n\nTable 2: Flat hit@k performance of DeViSE on ImageNet-based zero-shot datasets of increasing dif\ufb01culty\nfrom top to bottom. DeViSE-0 and DeViSE+1K are the same trained model, but DeViSE-0 is restricted to only\npredict zero-shot classes, whereas DeViSE+1K predicts both the zero-shot and the 1K training labels. For all,\nzero-shot classes did not occur in the image training set.\n\nTo test this hypothesis, we extracted images from the ImageNet 2011 21K dataset with labels that\nwere not included in the ILSVRC 2012 1K dataset on which DeViSE was trained. These are \u201czero-\nshot\u201d data sets in the sense that our model has no visual knowledge of these labels, though embed-\ndings for the labels were learned by the language model. The softmax baseline is only able to predict\nlabels from ILSVRC 2012 1K. The zero-shot experiments were performed with the same trained\n500-D DeViSE model used for results in Section 4.1, but it is evaluated in two ways: DeViSE-0\nonly predicts the zero-shot labels, and DeViSE+1K predicts zero-shot labels and the ILSVRC 2012\n1K training labels.\nFigure 2 shows label predictions for a handful of selected examples from this dataset to qualitatively\nillustrate model behavior. Note that DeViSE successfully predicts a wide range of labels outside\nits training set, and furthermore, the incorrect predictions are generally semantically \u201cclose\u201d to the\ndesired label. Figure 2 (a), (b), (c), and (d) show cases where our model makes signi\ufb01cantly better\ntop-5 predictions than the softmax-based model. For example, in Figure 2 (a), the DeViSE model\nis able to predict a number of lens-related labels even though it was not trained on images in any\nof the predicted categories. Figure 2 (d) illustrates a case where the top softmax prediction is quite\ngood, but where it is unable to generalize to new labels and its remaining predictions are off the\nmark, while our model\u2019s predictions are more plausible. Figure 2 (e) highlights a case where neither\nmodel gets the exact true label, but both models are giving plausible labels. Figure 2 (f) shows a\ncase where the softmax model emits more nearly correct labels than the DeViSE model.\nTo quantify the performance of the model on zero-shot data, we constructed from our ImageNet\n2011 21K zero-shot data three test data sets of increasing dif\ufb01culty based on the image labels\u2019\ntree distance from the training ILSVRC 2012 1K labels in the ImageNet label hierarchy [7]. The\neasiest dataset, \u201c2-hop\u201d, is comprised of the 1,589 labels that are within two tree hops of the training\nlabels, making them visually and semantically similar to the training set. A more dif\ufb01cult \u201c3-hop\u201d\ndataset was constructed in the same manner. Finally, we built a third, particularly challenging dataset\nconsisting of all the labels in ImageNet 2011 21K that are not in ILSVRC 2012 1K.\n\n6\n\nbarbet patas, hussar monkey, ... babbler, cackler titmouse, tit bowerbird, catbirdpatas, hussar monkey, ... proboscis monkey, Nasalis ... macaque titi, titi monkey guenon, guenon monkey oboe, hautboy, hautbois bassoon English horn, cor anglais hook and eye handreel punching bag, punch bag, ... whistle bassoon letter opener, paper knife, ... eyepiece, ocular Polaroid compound lens telephoto lens, zoom lens rangefinder, range findertypewriter keyboard tape player reflex camera CD player space barOur modelSoftmax over ImageNet 1KABCdune buggy, beach buggy searcher beetle, ... seeker, searcher, quester Tragelaphus eurycerus, ... bongo, bongo drum warplane, military plane missile projectile, missile sports car, sport car submarine, pigboat, sub, ...pot, flowerpot cauliflower guacamole cucumber, cuke broccolicomestible, edible, ... dressing, salad dressing Sicilian pizza vegetable, veggie, veg fruit fruit pineapple pineapple plant, Ananas ... sweet orange sweet orange tree, ...pineapple, ananas coral fungus artichoke, globe artichoke sea anemone, anemone cardoon DEFOur modelSoftmax over ImageNet 1K\fData Set\n\n2-hop\n\n3-hop\n\nImageNet 2011 21K\n\nModel\nDeViSE-0\nDeViSE+1K\nSoftmax baseline\nDeViSE-0\nDeViSE+1K\nSoftmax baseline\nDeViSE-0\nDeViSE+1K\nSoftmax baseline\n\n1\n0.06\n0.008\n\n0.017\n0.005\n\n0\n\n0\n\n0.008\n0.003\n\n0\n\n2\n\n5\n\n0.152\n0.204\n0.236\n0.037\n0.053\n0.053\n0.017\n0.025\n0.023\n\n0.192\n0.196\n0.181\n0.191\n0.192\n0.157\n0.072\n0.083\n0.071\n\nHierarchical precision@k\n\n10\n0.217\n0.201\n0.174\n0.214\n0.201\n0.143\n0.085\n0.092\n0.069\n\n20\n0.233\n0.214\n0.179\n0.236\n0.214\n0.130\n0.096\n0.101\n0.065\n\nTable 3: Hierarchical precision@k results on zero-shot classi\ufb01cation. Performance of DeViSE compared to\nthe softmax baseline model across the same datasets as in Table 2. Note that the softmax model can never\ndirectly predict the correct label so its precision@1 is 0.\n\nModel\nDeViSE\nMensink et al. 2012 [12]\nRohrbach et al. 2011 [17]\n\n200 labels\n\n1000 labels\n\n31.8%\n35.7%\n34.8%\n\n9.0%\n1.9%\n\n-\n\nTable 4: Flat hit@5 accuracy on the zero-shot task from [12]. DeViSE experiments were performed with a\n500-D model. The [12] model uses a curated hierarchy over labels for zero-shot classi\ufb01cation, but without using\nthis information, our model is close in performance on the 200 zero-shot class label task. When the models can\npredict any of the 1000 labels, we achieve better accuracy, indicating DeViSE has less of a bias toward training\nclasses than [12]. As in [12], we include a result on a similar task from [17], though their work used a different\nset of 200 zero-shot classes.\n\nWe again calculated the \ufb02at hit@k measure to determine how frequently DeViSE-0 and DeViSE+1K\npredicted the correct label for each of these data sets (Table 2). DeViSE-0\u2019s top prediction was the\ncorrect label 6.0% of the time across 1,589 novel labels, and the rate increases with k to 36.4% within\nthe top 20 predictions. As the zero-shot data sets become more dif\ufb01cult, the accuracy decreases in\nabsolute terms, though it is better relative to chance (not shown). Since a traditional softmax visual\nmodel can never produce the correct label on zero-shot data, its performance would be 0% for all\nk. The DeViSE+1K model performed uniformly worse than the plain DeViSE-0 model by a margin\nthat indicates it has a bias toward training classes.\nTo provide a stronger baseline for comparison, we compared the performance of our model and\nthe softmax model on the hierarchical metric we employed above. Although the softmax baseline\nmodel can never predict exactly the correct label, the hierarchical metric will give the model credit\nfor predicting labels that are in the neighborhood of the correct label in the ImageNet hierarchy\n(for k > 1). Visual similarity is strongly correlated with semantic similarity for nearby object\ncategories [8], and the softmax model does leverage visual similarity between zero-shot and training\nimages to make predictions that will be scored favorably (e.g. Figure 2d).\nThe easiest dataset, \u201c2-hop\u201d, contains object categories that are as visually and semantically similar\nto the training set as possible. For this dataset the softmax model outperforms the DeViSE model for\nhierarchical precision@2, demonstrating just how large a role visual similarity plays in predicting\nsemantically \u201cnearby\u201d labels (Table 3). However, for k = 5, 10, 20, our model produces superior\npredictions relative to the ImageNet hierarchy, even on this easiest dataset. For the two more dif\ufb01-\ncult datasets, where there are more novel categories and the novel categories are less closely related\nto those in the training data set, DeViSE outperforms the softmax model at all measured hierar-\nchical precisions. The quantitative gains can be quite large, as much as 82% relative improvement\nover softmax performance, and qualitatively, the softmax model\u2019s predictions can be surprisingly\nunreasonable in some cases (e.g. Figure 2c). The random embeddings model we described above\nperformed substantially worse than either of the real models. These results indicate that our archi-\ntecture succeeds in leveraging the semantic knowledge captured by the language model to make\nreasonable predictions, even as test images become increasingly dissimilar from those used in the\ntraining set.\nTo provide a comparison with other work in zero-shot learning, we also directly compare to the\nzero-shot results from [12]. These were performed on a particular 800/200 split of the 1000 classes\n\n7\n\n\ffrom ImageNet 2010: training and model tuning is performed using the 800 classes, and test images\nare drawn from the remaining 200 classes. Results are shown in Table 4.\nTaken together, these zero-shot experiments indicate that the DeViSE model can exploit both visual\nand semantic information to predict novel classes never before observed. Furthermore, the presence\nof semantic information in the model substantially improves the quality of its predictions.\n\n5 Conclusion\n\nIn contrast to previous attempts in this area [18], we have shown that our joint visual-semantic em-\nbedding model can be trained to give performance comparable to a state-of-the-art softmax based\nmodel on a \ufb02at object classi\ufb01cation metric, while simultaneously making more semantically rea-\nsonable errors, as indicated by its improved performance on a hierarchical label metric. We have\nalso shown that this model is able to make correct predictions across thousands of previously unseen\nclasses by leveraging semantic knowledge elicited only from unannotated text.\nThe advantages of this architecture, however, extend beyond the experiments presented here.\nFirst, we believe that our model\u2019s unusual compatibility with larger, less manicured data sets will\nprove to be a major strength moving forward.\nIn particular, the skip-gram language model we\nconstructed included only a modestly sized vocabulary, and was exposed only to the text of a single\nonline encyclopedia; we believe that the gains available to models with larger vocabularies and\ntrained on vastly larger text corpora will be signi\ufb01cant, and easily outstrip methods which rely on\nmanually constructed semantic hierarchies (e.g.\n[17]). Perhaps more importantly, though here we\ntrained on a curated academic image dataset, our model\u2019s architecture naturally lends itself to being\ntrained on all available images that can be annotated with any text term contained in the (larger)\nvocabulary. We believe that training massive \u201copen\u201d image datasets of this form will dramatically\nimprove the quality of visual object categorization systems.\nSecond, we believe that the 1-of-N (and nearly balanced) visual object classi\ufb01cation problem is\nsoon to be outmoded by practical visual object categorization systems that can handle very large\nnumbers of labels [5] and the re-de\ufb01nition of valid label sets at test time. For example, our model\ncan be trained once on all available data, and simultaneously used in one application requiring\nonly coarse object categorization (e.g. house, car, pedestrian) and another application requiring\n\ufb01ne categorization in a very specialized subset (e.g. Honda Civic, Ferrari F355, Tesla Model-S).\nMoreover, because test time computation can be sub-linear in the number of labels contained in the\ntraining set, our model can be used in exactly such systems with much larger numbers of labels,\nincluding overlapping or never-observed categories.\nMoving forward, we are experimenting with techniques which more directly leverage the structure\ninherent in the learned language embedding, greatly reducing training costs of the joint model and\nallowing even greater scaling [15].\n\nAcknowledgments\n\nSpecial thanks to those who lent their insight and technical support for this work, including Matthieu\nDevin, Alex Krizhevsky, Quoc Le, Rajat Monga, Ilya Sutskever, and Wojciech Zaremba.\n\nReferences\n\n[1] S. Bengio, J. Weston, and D. Grangier. Label embedding trees for large multi-class tasks. In Advances in\n\nNeural Information Processing Systems, NIPS, 2010.\n\n[2] Y. Bengio, R. Ducharme, and P. Vincent. A neural probabilistic language model. Journal of Machine\n\nLearning Research, 3:1137\u20131155, 2003.\n\n[3] A. Coates and A. Ng. The importance of encoding versus training with sparse coding and vector quanti-\n\nzation. In International Conference on Machine Learning (ICML), 2011.\n\n[4] Jeffrey Dean, Greg S. Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc V. Le, Mark Z. Mao,\nMarcAurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, and Andrew Y. Ng. Large scale distributed\ndeep networks. In Advances in Neural Information Processing Systems, NIPS, 2012.\n\n8\n\n\f[5] Thomas Dean, Mark Ruzon, Mark Segal, Jonathon Shlens, Sudheendra Vijayanarasimhan, and Jay Yag-\nIn IEEE Conference on\n\nnik. Fast, accurate detection of 100,000 object classes on a single machine.\nComputer Vision and Pattern Recognition (CVPR), 2013.\n\n[6] Jia Deng, Alex Berg, Sanjeev Satheesh, Hao Su, Aditya Khosla, and Fei-Fei Li. Imagenet large scale\n\nvisual recognition challenge 2012.\n\n[7] Jia Deng, Wei Dong, Richard Socher, Li jia Li, Kai Li, and Li Fei-fei. Imagenet: A large-scale hierarchical\n\nimage database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.\n\n[8] Thomas Deselaers and Vittorio Ferrari. Visual and semantic similarity in imagenet. In IEEE Conference\n\non Computer Vision and Pattern Recognition (CVPR), 2011.\n\n[9] J. C. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic\n\noptimization. Journal of Machine Learning Research, 12:2121\u20132159, 2011.\n\n[10] Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R. Salakhutdi-\narXiv preprint\n\nImproving neural networks by preventing co-adaptation of feature detectors.\n\nnov.\narXiv:1207.0580, 2012.\n\n[11] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classi\ufb01cation with deep convolutional\n\nneural networks. In Advances in Neural Information Processing Systems, NIPS, 2012.\n\n[12] Thomas Mensink, Jakob Verbeek, Florent Perronnin, and Gabriela Csurka. Metric learning for large scale\nimage classi\ufb01cation: Generalizing to new classes at near-zero cost. In European Conference on Computer\nVision (ECCV), 2012.\n\n[13] Tomas Mikolov, Kai Chen, Greg S. Corrado, and Jeffrey Dean. Ef\ufb01cient estimation of word represen-\nIn International Conference on Learning Representations (ICLR), Scottsdale,\n\ntations in vector space.\nArizona, USA, 2013.\n\n[14] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representations\nof words and phrases and their compositionality. In Advances in Neural Information Processing Systems,\nNIPS, 2013.\n\n[15] Mohammad Norouzi, Tomas Mikolov, Samy Bengio, Jonathon Shlens, Andrea Frome, Greg S. Corrado,\nand Jeffrey Dean. Zero-shot learning by convex combination of semantic embeddings. arXiv (to be\nsubmitted), 2013.\n\n[16] Mark Palatucci, Dean Pomerleau, Geoffrey E. Hinton, and Tom M. Mitchell. Zero-shot learning with\n\nsemantic output codes. In Advances in Neural Information Processing Systems, NIPS, 2009.\n\n[17] Marcus Rohrbach, Michael Stark, and Bernt Schiele. Evaluating knowledge transfer and zero-shot learn-\ning in a large-scale setting. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR),\n2011.\n\n[18] R. Socher, M. Ganjoo, H. Sridhar, O. Bastani, C. D. Manning, and A. Y. Ng. Zero-shot learning through\ncross-modal transfer. In International Conference on Learning Representations (ICLR), Scottsdale, Ari-\nzona, USA, 2013.\n\n[19] L.J.P. van der Maaten and G.E. Hinton. Visualizing high-dimensional data using t-sne. Journal of Machine\n\nLearning Research, 9:2579\u20132605, 2008.\n\n[20] Jason Weston, Samy Bengio, and Nicolas Usunier. Large scale image annotation: learning to rank with\n\njoint word-image embeddings. Machine Learning, 81(1):21\u201335, 2010.\n\n9\n\n\f", "award": [], "sourceid": 1048, "authors": [{"given_name": "Andrea", "family_name": "Frome", "institution": "Google Research"}, {"given_name": "Greg", "family_name": "Corrado", "institution": "Google Research"}, {"given_name": "Jon", "family_name": "Shlens", "institution": "Google Research"}, {"given_name": "Samy", "family_name": "Bengio", "institution": "Google Research"}, {"given_name": "Jeff", "family_name": "Dean", "institution": "Google Research"}, {"given_name": "Marc'Aurelio", "family_name": "Ranzato", "institution": "Google Research"}, {"given_name": "Tomas", "family_name": "Mikolov", "institution": "Google Research"}]}