{"title": "A Model for Learning the Semantics of Pictures", "book": "Advances in Neural Information Processing Systems", "page_first": 553, "page_last": 560, "abstract": "", "full_text": "A Model for Learning the Semantics of Pictures\n\nV. Lavrenko, R. Manmatha, J. Jeon\n\nCenter for Intelligent Information Retrieval\n\nComputer Science Department,\n\nUniversity of Massachusetts Amherst\n\nflavrenko,manmatha,jeong@cs.umass.edu\n\nAbstract\n\nWe propose an approach to learning the semantics of images which al-\nlows us to automatically annotate an image with keywords and to retrieve\nimages based on text queries. We do this using a formalism that models\nthe generation of annotated images. We assume that every image is di-\nvided into regions, each described by a continuous-valued feature vector.\nGiven a training set of images with annotations, we compute a joint prob-\nabilistic model of image features and words which allow us to predict the\nprobability of generating a word given the image regions. This may be\nused to automatically annotate and retrieve images given a word as a\nquery. Experiments show that our model signi\ufb01cantly outperforms the\nbest of the previously reported results on the tasks of automatic image\nannotation and retrieval.\n\nIntroduction\n\n1\nHistorically, librarians have retrieved images by \ufb01rst manually annotating them with key-\nwords. Given a query, these annotations are used to retrieve appropriate pictures. Under-\nlying this approach is the belief that the words associated (manually) with a picture essen-\ntially capture the semantics of the picture and any retrieval based on these keywords will,\ntherefore, retrieve relevant pictures. Since manual image annotation is expensive, there has\nbeen great interest in coming up with automatic ways to retrieve images based on content.\nQueries based on image concepts like color or texture have been proposed for retrieving\nimages by content but most users \ufb01nd it dif\ufb01cult to query using such visual attributes. Most\npeople would prefer to pose text queries and \ufb01nd images relevant to those queries. For\nexample, one should be able to pose a query like \u201c\ufb01nd me cars on a race track\u201d. This is\ndif\ufb01cult if not impossible with many of the current image retrieval systems and hence has\nnot led to widespread adoption of these systems. We propose a model which looks at the\nprobability of associating words with image regions. Single pixels and regions are often\nhard to interpret. The surrounding context often simpli\ufb01es the interpretation of regions as\na speci\ufb01c objects. For example, the association of a region with the word tiger is increased\nby the fact that there is a grass region and a water region in the same image and should be\ndecreased if instead there is a region corresponding to the interior of an aircraft. Thus the\nassociation of different regions provides context while the association of words with image\nregions provides meaning. Our model computes a joint probability of image features over\ndifferent regions in an image using a training set and uses this joint probability to annotate\nand retrieve images.\n\nMore formally, we propose a statistical generative model to automatically learn the seman-\ntics of images - that is, for annotating and retrieving images based on a training set of\nimages. We assume that an image is segmented into regions (although the regions could\n\n\fsimply be a partition of the image) and that features are computed over each of these re-\ngions. Given a training set of images with annotations, we show that probabilistic models\nallow us to predict the probability of generating a word given the features computed over\ndifferent regions in an image. This may be used to automatically annotate and retrieve im-\nages given a word as a query. We show that the continuous relevance model - a statistical\ngenerative model related to relevance models in information retrieval - allows us to derive\nthese probabilities in a natural way. The model proposed here directly associates continu-\nous features with words and does not require an intermediate clustering stage. Experiments\nshow that the annotation performance of this continuous relevance model is substantially\nbetter than any other model tested on the same data set. It is almost an order of magnitude\nbetter (in terms of mean precision) than a model based on word-blob co-occurrence model,\nmore than two and a half times better than a state of the art model derived from machine\ntranslation and 1.6 times as good as a discrete version of the relevance model. The model\nalso allows ranked retrieval in response to a text query and again performs much better than\nany other model in this regard. Our model permits us to automatically associate semantics\n(in terms of words) with pictures and is an important building step in performing automatic\nobject recognition.\n\n2 Related Work\nRecently, there has been some work on automatically annotating images by looking at\nthe probability of associating words with image regions. Mori et al. [9] proposed a Co-\noccurrence Model in which they looked at the co-occurrence of words with image regions\ncreated using a regular grid. Duygulu et al [4] proposed to describe images using a vocab-\nulary of blobs. First, regions are created using a segmentation algorithm like normalized\ncuts. For each region, features are computed and then blobs are generated by clustering\nthe image features for these regions across images. Each image is generated by using a\ncertain number of these blobs. Their Translation Model applies one of the classical statis-\ntical machine translation models to translate from the set of keywords of an image to the\nset of blobs forming the image. Jeon et al [5] instead assumed that this could be viewed as\nanalogous to the cross-lingual retrieval problem and used a cross-media relevance model\n(CMRM) to perform both image annotation and ranked retrieval. They showed that the\nperformance of the model on the same dataset was considerably better than the models\nproposed by Duygulu et al [4] and Mori et al. [9]. Blei and Jordan [3] extended the Latent\nDirichlet Allocation (LDA) Model and proposed a Correlation LDA model which relates\nwords and images. This model assumes that a Dirichlet distribution can be used to generate\na mixture of latent factors. This mixture of latent factors is then used to generate words and\nregions. EM is again used to estimate this model. Blei and Jordan show a few examples\nfor labeling speci\ufb01c regions in an image.\n\nThe model proposed in this paper is called Continuous-space Relevance Model (CRM).\nThe model is closely related to models proposed by [3, 5], but there are several important\ndifferences which we will highlight in the remainder of this section.\n\nOn the surface, CRM appears to be very similar to one of the intermediate models con-\nsidered by Blei and Jordan [3]. Speci\ufb01cally, their GM-mixture model employs a nearly\nidentical dependence structure among the random variables involved. However, the topo-\nlogical structure of CRM is quite different from the one employed by [3]. GM-mixture\nassumes a low-dimensional topology, leading to a fully-parametric model where 200 or so\n\u201clatent aspects\u201d are estimated using the EM algorithm. To contrast that, CRM makes no as-\nsumptions about the topological structure, and leads to a doubly non-parametric approach,\nwhere expectations are computed over every individual point in the training set. In that\nregard, CRM appears very similar to the cross-media relevance model (CMRM) [5], which\nis also doubly non-parametric. There are two signi\ufb01cant differences between CRM and\nCMRM. First, CMRM is a discrete model and cannot take advantage of continuous fea-\ntures. In order to use CMRM for image annotation we have to quantize continuous feature\n\n\fposition,size,\ntexture,shape,\n\ncolor, ...\n\ntiger =\n\ngrass =\n\nsun =\n\nw1\n\nw2\n\nw3\n\nP(w|J)\n\nJ\n\nP(g|J)\n\ng1\n\ng2\n\ng3\n\nP(r|g)\n\nr 1\n\n=\n\nr2\n\n=\n\nr 3\n\n=\n\nFigure 1: A generative model of annotated images. Words wj in the annotation are i.i.d.\nsampled from the underlying multinomial. Image pixels are produced by \ufb01rst picking a\nset of i.i.d. feature vectors fg1: : :gng, then generating image regions fr1: : :rng from the\nfeature vectors, and \ufb01nally stacking the regions on top of each other.\n\nvectors into a discrete vocabulary (similarly to the co-ocurrence and translation [4] mod-\nels). CRM, on the other hand, directly models continuous features. The second difference\nis that CMRM relies on clustering of the feature vectors into blobs. Annotation quality\nof the CMRM is very sensitive to clustering errors, and depends on being able to a-priori\nselect the right cluster granularity: too many clusters will result in exptreme sparseness of\nthe space, while too few will lead us to confuse different objects in the images. CRM does\nnot rely on clustering and consequently does not suffer from the granularity issues.\n\nWe would like to stress that the difference between CRM and previously discussed models\nis not merely conceptual. In section 4 we will show that CRM performs signi\ufb01cantly better\nthan all previosly proposed models on the tasks of image annotation and retrieval. To ensure\na fair comparison, we use exactly the same data set and same feature representations as\nwere used in [3, 4, 5, 9].\n3 A Model of Annotated Images\nThe purpose of this section is to introduce a statistical formalism that will allow us to\nmodel a relationship between the contents of a given image and the annotation of that\nimage. We will describe an approach to learning a joint probability disdribution P (r; w)\nover the regions r of some image and the words w in its annotation. Knowing the joint\ndistribution P (r; w) is the key to solving two important real-world problems:\n\n1. Image Annotation. Suppose we are given a new image for which no annotation\nis provided. That is, we know r, but do not know w. Having a joint distribution\nallows us to compute a conditional likelihood P (wjr) which can then be used to\nguess the most likely annotation w for the image in question. The new annotation\ncan be presented to a user, indexed, or used for retrieval purposes.\n\n2. Image Retrieval. Suppose we are given a collection of un-annotated images and\na text query wqry consisting of a few keywords. Knowing the joint model of\nimages and annotations, we can compute the query likelihood P (wqryjrJ ) for\nevery image J in the dataset. We can then rank images in the collection according\nto their likelihood of having the query as annotation, resulting in a special case of\nthe popular Language Modeling approach to Information Retrieval [6].\n\nThe remainder of this section is organized as follows. In section 3.1 we discuss our choice\nof representation for images and their annotations. Section 3.2 presents a generative frame-\nwork for relating image regions with image annotations. Section 3.3 provides detailed\nestimates for the components of our model.\n\n3.1 Representation of Images and Annotations\nLet C denote the \ufb01nite set of all possible pixel colors. We assume that C includes one\n\u201ctransparent\u201d color c0, which will be handy when we have to layer image regions. As\n\n\fa matter of convenience, we assume that all images are of a \ufb01xed size W (cid:2) H.1 This\nassumption allows us to represent any image as an element of a \ufb01nite set R = C W (cid:2)H. We\nassume that each image contains several distinct regions fr1: : :rng. Each region is itself\nan element of R and contains the pixels of some prominent object in the image, all pixels\naround the object are set to be transparent. For example, in Figure 1 we have a hypothetical\npicture containing three prominent objects: a tiger, the sun and some grass. Each object is\nrepresented by its own region: r1 for the sun, r2 for the grass, and r3 for the tiger. The \ufb01nal\nimage is the result of stacking or layering the regions on top of each other, as shown on the\nright side of Figure 1.\n\nIn our model of images, a central part will be played by a special function G which maps\nimage regions r 2 R to real-valued vectors g 2 IRk. The value G(r) represents a set of\nfeatures, or characteristics of an image region. The features could re\ufb02ect the position of\nan object region, its relative size, a crude re\ufb02ection of shape, as well as predominant colors\nand textures. For example, in Figure 1 the region r1 (sun) is a round object, located in the\nupper-right portion of the image, yellowish in color with a smooth texture. When we model\nimage generation we will treat the output of G as a generator or a \u201crecipe\u201d for producing\na certain type of image. For example, a feature vetor g1 = G(r1) can be thought of as a\ngenerator for any image region resembling a sun-like object in the upper-left corner.\nFinally, an annotation for a given image is a set of words fw1: : :wmg drawn from some\n\ufb01nite vocabulary V. We assume that the annotation describes the objects represented by\nregions fr1: : :rng. However, contrary to prior work [4, 3] we do not assume an underlying\none-to-one correspondence between the objects in the image annotation and words in the\nannotation. Instead, we are interested in modeling a joint probability for observing a set of\nimage regions fr1: : :rng together with the set of annotation words fw1: : :wmg.\n\n3.2 A Model for Generating Annotated Images\nSuppose T is the training set of annotated images, and let J be an element of T . Ac-\ncording to the previous section J is represented as a set of image regions rJ = fr1: : :rng\nalong with the corresponding annotation wJ = fw1: : :wmg. We assume that the process\nthat generated J is based on three distinct probability distributions. First, we assume that\nthe words in wJ are an i.i.d. random sample from some underlying multinomial distribu-\ntion PV ((cid:1)jJ). Second, the regions rJ are produced from a corresponding set of generator\nvectors g1: : :gn according to a process PR(rijgi) which is independent of J. Finally, the\ngenerator vectors g1: : :gn are themselves an i.i.d. random sample from some underlying\nmulti-variate density function PG((cid:1)jJ).\nNow let rA = fr1: : :rnA g denote the regions of some image A, which is not in the train-\ning set T . Similarly, let wB = fw1: : :wnB g be some arbitrary sequence of words. We\nwould like to model P (rA; wA), the joint probability of observing an image de\ufb01ned by rA\ntogether with annotation words wB. We hypothesize that the observation frA; wBg came\nfrom the same process that generated one of the images J (cid:3) in the training set T . However,\nwe don\u2019t know which process that was, and so we compute an expectation over all images\nJ2T . The overall process for jointly generating wB and rA is as follows:\n\n1. Pick a training image J 2 T with some probability PT (J)\n2. For b = 1 : : : nB:\n\n(a) Pick the annotation word wb from the multinomial distribution PV ((cid:1)jJ).\n\n3. For a = 1 : : : nA:\n\n(a) Sample a generator vector ga from the probability density PG((cid:1)jJ).\n(b) Pick the image region ra according to the probability PR(rajga)\n\n1The assumptions of \ufb01nite colormap and \ufb01xed image size can easily be relaxed but require argu-\n\nments that are beyond the scope of this paper.\n\n\fFigure 1 shows a graphical dependency diagram for the generative process outlined above.\nWe show the process of generating a simple image consisting of three regions and a corre-\nsponding 3-word annotation. Note that the number of words in the annotation nB does not\nhave to be the same as the number of image regions nA. Formally, the probability of a joint\nobservation frA; wBg is given by:\n\nP (rA; wB) = XJ 2T\n\nPT (J)\n\nnB\n\nYb=1\n\nPV (wbjJ)\n\nnA\n\nYa=1ZIRk\n\nPR(rajga)PG(gajJ)dga\n\n(1)\n\n3.3 Estimating Parameters of the Model\nIn this section we will discuss simple but effective estimation techniques for the four com-\nponents of the model: PT , PV, PG and PR. PT (J) is the probability of selecting the\nunderlying model of image J to generate some new observation r; w. In the absence of\nany task knowledge we use a uniform prior PT (J) = 1=NT , where NT is the size of the\ntraining set.\n\nPR(rjg) is a global probability distribution responsible for mapping generator vectors\ng2IRk to actual image regions r2R. In our case for every image region r there is only\none corresponding generator g = G(r), so we can assume a particularly simple form for\nthe distribution PR:\n\nPR(rjg) = (cid:26) 1=Ng\n\n0\n\nif G(r) = g\notherwise\n\n(2)\n\nwhere Ng is the number of all regions r0 in R such that G(r0) = g. For the scope of\nthe current paper we do not attempt to reliably estimate Ng, instead we assume it to be a\nconstant independent of g.\nPG((cid:1)jJ) is a density function responsible for generating the feature vectors g1: : :gn, which\nare later mapped to image regions rJ according to PR. We use a non-parametric kernel-\nbased density estimate for the distribution PG. Assuming rJ = fr1: : :rng to be the set of\nregions of image J we estimate:\n\nPG(gjJ) =\n\n1\nn\n\nn\n\nXi=1\n\n1\n\np2k(cid:25)kj(cid:6)j\n\nexp(cid:8)(g (cid:0) G(ri))>(cid:6)(cid:0)1(g (cid:0) G(ri))(cid:9)\n\n(3)\n\nEquation (3) arises out of placing a Gaussian kernel over the feature vector G(ri) of every\nregion of image J. Each kernel is parametrized by the feature covariance matrix (cid:6). As\na matter of convenience we assumed (cid:6) = (cid:12)(cid:1)I, where I is the identity matrix. (cid:12) playes\nthe role of kernel bandwidth: it determines the smoothness of PG around the support point\nG(ri). The value of (cid:12) is selected empirically on a held-out portion of the training set T .\nPV ((cid:1)jJ) is the multinomial distribution that is assumed to have generated the annotation\nwJ of image J2T . We use a Bayesian framework for estimating PV ((cid:1)jJ). Let IP V be\nthe simplex of all multinomial distributions over V. We assume a Dirichlet prior over IP V\nthat has parameters f(cid:22)pv : v2Vg. Here (cid:22) is a constant, selected empirically, and pv is the\nrelative frequency of observing the word v in the training set. Introducing the observation\nwJ results in a Dirichlet posterior over IP V with parameters f(cid:22)pv + Nv;J : v2Vg. Here\nNv;J is the number of times v occurs in the observation wJ . Computing the expectation\nover this Dirichlet posterior gives us the following Bayesian estimate for PV:\n\nPV (vjJ) =\n\n(cid:22)pv + Nv;J\n\n(cid:22) +Pv0 Nv0;J\n\n4 Experimental Results\nTo provide a meaningful comparison with previously-reported results, we use, without any\nmodi\ufb01cation, the dataset provided by Duygulu et al.[4] 2. This allows us to compare the\n\n2Available at http://www.cs.arizona.edu/people/kobus/ research/data/eccv 2002\n\n(4)\n\n\fModels\n#words with recall (cid:21) 0\n\nCo-occurence Translation CMRM\n\n19\n\n49\n\nResults on 49 best words, as in[1, 5]\n\nMean per-word Recall\nMean per-word Precision\n\nMean per-word Recall\nMean per-word Precision\n\n-\n-\n\n0.34\n0.20\nResults on all 260 words\n0.04\n0.06\n\n0.02\n0.03\n\n66\n\n0.48\n0.40\n\n0.09\n0.10\n\nCRM\n\n107 +62%\n\n0.70 +46%\n0.59 +48%\n\n0.19 +111%\n0.16 +60 %\n\nTable 1: Comparing recall and precision of the four models on the task of automatic im-\nage annotation. Our model (CRM) substantially outperforms all other models. Percent\nimprovements are over the best previously-reported results (CMRM).\n\nperformance of models in a strictly controlled manner. The dataset consists of 5,000 images\nfrom 50 Corel Stock Photo cds. Each cd includes 100 images on the same topic. Each\nimage contains an annotation of 1-5 keywords. Overall there are 371 words. Prior to\nmodeling, every image in the dataset is pre-segmented into regions using general-purpose\nalgorithms, such as normalized cuts [11]. We use pre-computed feature vector G(r) for\nevery segmented region r. The feature set consists of 36 features: 18 color features, 12\ntexture features and 6 shape features. For details of the features refer to [4]. Since we\ndirectly model the generation of feature vectors, there is no need to quantize feature data,\nas was done in [1, 4, 5]. We divided the dataset into 3 parts - with 4,000 training set images,\n500 evaluation set images and 500 images in the test set. The evaluation set is used to \ufb01nd\nsystem parameters. After \ufb01xing the parameters, we merged the 4,000 training set and 500\nevaluation set images to make a new training set. This corresponds to the training set of\n4500 images and the test set of 500 images used by Duygulu et al [4].\n\n4.1 Results: Automatic Image Annotation\n\nIn this section we evaluate the performance of our model on the task of automatic image\nannotation. We are given an un-annotated image J and are asked to automatically produce\nan annotation wauto. The automatic annotation is then compared to the held-out human\nannotation wJ . We follow the experimental methodology used by[4, 5]. Given a set of\nimage regions rJ we use equation (1) to arrive at the conditional distribution P (wjrJ ). We\ntake the top 5 words from that distribution and call them the automatic annotation of the\nimage in question. Then, following [4], we compute annotation recall and precision for\nevery word in the testing set. Recall is the number of images correctly annotated with a\ngiven word, divided by the number of images that have that word in the human annotation.\nPrecision is the number of correctly annotated images divided by the total number of images\nannotated with that particular word (correctly or not). Recall and precision values are\naveraged over the set of testing words.\n\nWe compare the annotation performance of the four models: the Co-occurrence Model\n[9], the Translation Model [4], CMRM [5] and the model proposed in this paper (CRM).\nWe report the results on two sets of words: the subset of 49 best words which was used\nby[4, 5], and the complete set of all 260 words that occur in the testing set. Table 1 shows\nthe performance on both word sets. The \ufb01gures clearly show that the model presented here\n(CRM) substabtially outperforms the other models and is the only one of the four capable of\nproducing reasonable mean recall and mean precision numbers when every word in the test\nset is used. In Figure2 we provide sample annotations for the two best models in the table,\nCMRM and CRM, showing that the model in this paper is considerably more accurate.\n\n\f\u00022,\u000408\n\n\u001a\u001e#\u001e\n\u00183349,9\u000443\n\u001a#\u001e\u00183349,9\u000443\n\n\u0005,907\u00038\u0004\u0005\u00035\u0004,30\u0003\n-0,7\n\u0004\u0004\u0005,7/\u00032,7\u000430\u0003\n\u0004\u0004:,3,\u000374.\u00048\n\n\u0005,907\u00038\u0004\u0005\u00035\u0004,30\u0003\n\u000409\u00039700\n834\u0005\u0003-0,7\u000354\u0004,7\u0003\n9:3/7,\n\n\u0005,907\u00038\u0004\u0005\u00039700\u0003\n5045\u00040\n97,\u00043\u00037,\u0004\u000474,/\u0003\n97,.\u00048\u0003\u00044.4249\u0004;0\n\n5045\u00040\u000374.\u00048\u0003\n\u0005,907\u0003-:\u0004\u0004/\u00043\u00048\n.,9\u00039\u0004\u000407\u0003\u0005,907\u0003\n147089\n\nFigure 2: The generative model based on contiuous features (CRM) that is proposed here\nperforms substantially better than the discrete cross-media relevance model (CMRM) for\nannotating images in the test set.\n\nQuery length\nNumber of queries\nRelevant images\n\nCMRM\nCRM\n\nCMRM\nCRM\n\n1 word\n179\n1675\n\n3 words\n178\n542\nPrecision after 5 retrieved images\n\n2 words\n386\n1647\n\n4 words\n24\n67\n\n0.1989\n0.2480 +25% 0.1902 +45% 0.1888 +26% 0.2333 +12%\n\n0.2083\n\n0.1306\n\n0.1494\n\nMean Average Precision\n\n0.1697\n0.2353 +39% 0.2534 +54% 0.3152 +55% 0.4471 +61%\n\n0.1642\n\n0.2030\n\n0.2765\n\nTable 2: Comparing our model to the Cross-Media Relevance Model (CMRM) on the task\nof image retrieval. Our model outperforms the CMRM model by a wide margin on all\nquery sets. Boldface \ufb01gures mark improvements that are statistically signi\ufb01cant according\nto sign test with a con\ufb01dence of 99% (p-value < 0:01).\n\n4.2 Results: Ranked Retrieval of Images\nIn this section we turn our attention to the problem of ranked retrieval of images. In the\nretrieval setting we are given a text query wqry and a testing collection of un-annotated\nimages. For each testing image J we use equation (1) to get the conditional probability\nP (wqryjrJ ). All images in the collection are ranked according to the conditional likeli-\nhood P (wqryjrJ ). This can be thought of as a special case of the popular Langauge Mod-\neling approach to Information Retrieval, proposed by Ponte and Croft[6]. In our retrieval\nexperiments we do our best to reproduce the same settings that were used by Jeon et.al[5]\nin their work. Following[5], we use four sets of queries, constructed from all 1-, 2-, 3- and\n4-word combinations of words that occur at least twice in the testing set. An image is con-\nsidered relevant to a given query if its manual annotation contains all of the query words.\nAs our evaluation metrics we use precision at 5 retrieved images and non-interpolated av-\nerage precision3, averaged over the entire query set. Precision at 5 documents is a good\nmeasure of performance for a casual user who is interested in retrieving a couple of rele-\nvant items without looking at too much junk. Average precision is more appropriate for a\nprofessional user who wants to \ufb01nd a large proportion of relevant items.\n\nTable 2 shows the performance of our model on the four query sets, contrasted with per-\nformance of the CMRM[5] baseline on the same data. Baseline performance \ufb01gures are\nquoted directly from the tables in[5]. We observe that our model substantially outperforms\nthe CMRM baseline on every query set. Improvements in average precision are particularly\nimpressive, our model outperforms the baseline by 40 - 60 percent. All improvements on\n1-, 2- and 3-word queries are statistically signi\ufb01cant based on a sign test with a p-value of\n\n3Average precision is the average of precision values at the ranks where relevant items occur.\n\n\fFigure 3: Example: top 5 images retrieved in responce to text query \u201ccars track\u201d\n\n0.01. We are also very encouraged by the precision our model shows at 5 retrieved images:\nprecision values around 0.2 suggest that an average query always has a relevant image in\nthe top 5. Figure 3 shows top 5 images retrieved in response to the text query \u201ccars track\u201d.\n5 Conclusions and Future Work\nWe have proposed a new statistical generative model for learning the semantics of images.\nWe showed that this model works signi\ufb01cantly better than a number of other models for\nimage annotation and retrieval. Our model works directly on the continuous features. Fu-\nture work will include the extension of this work to larger datasets (both training and test\ndata). We believe this is needed both for better coverage and an evaluation of how such\nalgorithms extend to large data sets. Improved feature sets may also lead to substantial\nimprovements in performance.\n6 Acknowledgments\nWe thank Kobus Barnard for making their dataset [4] available. This work was supported in\npart by the Center for Intelligent Information Retrieval, by the National Science Foundation\nunder grant NSF IIS-9909073 and by SPAWARSYSCEN-SD under grants N66001-99-\n1-8912 and N66001-02-1-8903. Jiwoon Jeon is partially supported by the Government\nof Korea. Any opinions, \ufb01ndings and conclusions or recommendations expressed in this\nmaterial are the author(s) and do not necessarily re\ufb02ect those of the sponsor.\nReferences\n\n[1] K. Barnard, P. Duygulu, N. de Freitas, D. Forsyth, D. Blei, and M. I. Jordan. Matching words\n\nand pictures. Journal of Machine Learning Research, 3:1107-1135, 2003.\n\n[2] D. Blei (2003) Private Communication.\n[3] D. Blei, and M. I. Jordan. (2003) Modeling annotated data. In Proceedings of the 26th Intl.\n\nACM SIGIR Conf., pages 127\u2013134, 2003\n\n[4] P. Duygulu, K. Barnard, N. de Freitas, and D. Forsyth. Object recognition as machine transla-\ntion: Learning a lexicon for a \ufb01xed image vocabulary. In Seventh European Conf. on Computer\nVision, pages 97-112, 2002.\n\n[5] J. Jeon, V. Lavrenko and R. Manmatha. (2003) Automatic Image Annotation and Retrieval\nIn Proceedings of the 26th Intl. ACM SIGIR Conf.,\n\nusing Cross-Media Relevance Models\npages 119\u2013126, 2003\n\n[6] Ponte, J. M. and Croft, W. B. (1998). A language modeling approach to information retrieval.\n\nProceedings of the 21st Intl. ACM SIGIR Conf., pages 275\u2013281.\n\n[7] V. Lavrenko and W. Croft. Relevance-based language models. Proceedings of the 24th Intl.\n\nACM SIGIR Conf., pages 120-127, 2001.\n\n[8] V. Lavrenko, M. Choquette, and W. Croft. Cross-lingual relevance models. Proceedings of the\n\n25th Intl. ACM SIGIR Conf., pages 175\u2013182, 2002.\n\n[9] Y. Mori, H. Takahashi, and R. Oka. Image-to-word transformation based on dividing and vector\nquantizing images with words. In MISRM\u201999 First Intl. Workshop on Multimedia Intelligent\nStorage and Retrieval Management, 1999.\n\n[10] H. Schneiderman, T. Kanade. A Statistical Method for 3D Object Detection Applied to Faces\n\nand Cars. Proc. IEEE CVPR 2000: 1746-1759\n\n[11] J. Shi and J. Malik. Normalized cuts and image segmentation. IEEE Transactions on Pattern\n\nAnalysis and Machine Intelligence, 22(8):888\u2013905, 2000.\n\n\f", "award": [], "sourceid": 2474, "authors": [{"given_name": "Victor", "family_name": "Lavrenko", "institution": null}, {"given_name": "R.", "family_name": "Manmatha", "institution": null}, {"given_name": "Jiwoon", "family_name": "Jeon", "institution": null}]}