{"title": "Inferring a Semantic Representation of Text via Cross-Language Correlation Analysis", "book": "Advances in Neural Information Processing Systems", "page_first": 1497, "page_last": 1504, "abstract": "", "full_text": "Inferring a Semantic Representation of Text\n\nvia Cross-Language Correlation Analysis\n\nAlexei Vinokourov\nJohn Shawe-Taylor\n\nDept. Computer Science\n\nRoyal Holloway, University of London\n\nEgham, Surrey, UK, TW20 0EX\nalexei@cs.rhul.ac.uk\n\nNello Cristianini\nDept. Statistics\n\nUC Davis, Berkeley, US\n\nnello@support-vector.net\n\njohn@cs.rhul.ac.uk\n\nAbstract\n\nThe problem of learning a semantic representation of a text document\nfrom data is addressed, in the situation where a corpus of unlabeled\npaired documents is available, each pair being formed by a short En-\nglish document and its French translation. This representation can then\nbe used for any retrieval, categorization or clustering task, both in a stan-\ndard and in a cross-lingual setting. By using kernel functions, in this case\nsimple bag-of-words inner products, each part of the corpus is mapped\nto a high-dimensional space. The correlations between the two spaces\nare then learnt by using kernel Canonical Correlation Analysis. A set\nof directions is found in the \ufb01rst and in the second space that are max-\nimally correlated. Since we assume the two representations are com-\npletely independent apart from the semantic content, any correlation be-\ntween them should re\ufb02ect some semantic similarity. Certain patterns of\nEnglish words that relate to a speci\ufb01c meaning should correlate with cer-\ntain patterns of French words corresponding to the same meaning, across\nthe corpus. Using the semantic representation obtained in this way we\n\ufb01rst demonstrate that the correlations detected between the two versions\nof the corpus are signi\ufb01cantly higher than random, and hence that a rep-\nresentation based on such features does capture statistical patterns that\nshould re\ufb02ect semantic information. Then we use such representation\nboth in cross-language and in single-language retrieval tasks, observing\nperformance that is consistently and signi\ufb01cantly superior to LSI on the\nsame data.\n\n1 Introduction\n\nMost text retrieval or categorization methods depend on exact matches between words.\nSuch methods will, however, fail to recognize relevant documents that do not share words\nwith a users\u2019 queries. One reason for this is that the standard representation models (e.g.\nboolean, standard vector, probabilistic) treat words as if they are independent, although it\nis clear that they are not. A central problem in this \ufb01eld is to automatically model term-\n\n\fterm semantic interrelationships, in a way to improve retrieval, and possibly to do so in an\nunsupervised way or with a minimal amount of supervision. For example latent seman-\ntic indexing (LSI) has been used to extract information about co-occurrence of terms in\nthe same documents, an indicator of semantic relations, and this is achieved by singular\nvalue decomposition (SVD) of the term-document matrix. The LSI method has also been\nadapted to deal with the important problem of cross-language retrieval, where a query in\na language is used to retrieve documents in a different language. Using a paired corpus (a\nset of pairs of documents, each pair being formed by two versions of the same text in two\ndifferent languages), after merging each pair into a single \u2019document\u2019, we can interpret fre-\nquent co-occurrence of two terms in the same document as an indication of cross-linguistic\ncorrelation [5]. In this framework, a common vector-space, including words from both\nlanguages, is created and then the training set is analysed in this space using SVD. This\nmethod, termed CL-LSI, will be brie\ufb02y discussed in Section 4. More generally, many\nother statistical and linear algebra methods have been used to obtain an improved semantic\nrepresentation of text data over LSI [6]. In this study we address the problem of learning\na semantic representation of text from a paired bilingual corpus, a problem that is impor-\ntant both for mono-lingual and cross-lingual applications. This problem can be regarded\neither as an unsupervised problem with paired documents, or as a supervised monolingual\nproblem with very complex labels (i.e. the label of an english document could be its french\ncounterpart). In either way, the data can be readily obtained without an explicit labeling\neffort, and furthermore there is not the loss of information due to compressing the mean-\ning of a document into a discrete label. We employ kernel Canonical Correlation Analysis\n(KCCA) [1] to learn a representation of text that captures aspects of its meaning. Given\na paired bilingual corpus, this method de\ufb01nes two embedding spaces for the documents\nof the corpus, one for each language, and an obvious one-to-one correspondence between\npoints in the two spaces. KCCA then \ufb01nds projections in the two embedding spaces for\nwhich the resulting projected values are highly correlated. In other words, it looks for par-\nticular combinations of words that appear to have the same co-occurrence patterns in the\ntwo languages. Our hypothesis is that \ufb01nding such correlations across a paired crosslingual\ncorpus will locate the underlying semantics, since we assume that the two languages are\n\u2019conditionally independent\u2019, or that the only thing they have in common is their meaning.\nThe directions would carry information about the concepts that stood behind the process of\ngeneration of the text and, although expressed differently in different languages, are, never-\ntheless, semantically equivalent. To illustrate such representation we have printed the most\nprobable (most typical) words in each language for some of the \ufb01rst few kernel canonical\ncorrleation components found for bilingual 36 \u0002\u0001 Canadian Parliament corpus (Hansards)\n(left column is English space and right column is French space):\n\nPENSIONS PLAN?\n\nAGRICULTURE?\n\nCANADIAN LANDS?\n\nFISHING INDUSTRY?\n\npension\nplan\ncpp\ncanadians\nbene\ufb01ts\nretirement\nfund\ntax\ninvestment\nincome\n\ufb01nance\nyoung\nyears\nrate\nsuperannuation\ndisability\ntaxes\nmounted\nfuture\npremiums\nseniors\ncountry\nrates\njobs\npay\n\nregime\npensions\nrpc\nprestations\ncanadiens\nretraite\ncotisations\nfonds\ndiscours\nimp\u02c6ot\nrevenu\njeunes\nans\npension\nargent\nregimes\ninvestissement\nmilliards\nprestation\nplan\n\ufb01nances\npays\navenir\ninvalidit\nresolution\n\nwheat\nboard\nfarmers\nnewfoundland\ngrain\nparty\namendment\nproducers\ncanadian\nspeaker\nreferendum\nminister\ndirectors\nquebec\nspeech\nschool\nsystem\nmarketing\nprovinces\nconstitution\nthrone\nmoney\nsection\nrendum\nmajorit\n\nbl\ncommission\nagriculteurs\nproducteurs\ncanadienne\ngrain\nparti\nconseil\ncommercialisat\nneuve\nministre\nadministration\nmodi\ufb01cation\nqubec\nterre\nformistes\npartis\ngrains\nop\nnationale\nlus\nbloc\nnations\nchambre\nadministration\n\npark\nland\naboriginal\nyukon\nmarine\ngovernment\nvalley\nwater\nboards\nterritories\nboard\nnorth\nparks\nresource\nagreements\nnorthwest\nresources\ndevelopment\ntreaty\nnations\nterritoire\nwork\nterritory\natlantic\nprograms\n\nparc\nautochtones\nterres\nches\nvall\nressources\nyukon\nnord\ngouvernement\nof\ufb01ces\nmarin\neaux\nterritoires\nparcs\nnations\nterritoriales\nrevendications\nministre\ncheurs\nouest\nentente\nrights\nof\ufb01ce\natlantique\nententes\n\n\ufb01sheries\natlantic\noperatives\n\ufb01shermen\nnewfoundland\n\ufb01shery\nproblem\noperative\n\ufb01shing\nindustry\n\ufb01sh\nyears\nproblems\nwheat\ncoast\noceans\nwest\nsalmon\ntags\nminister\ncommunities\nprogram\ncommission\nmotion\nstocks\n\np\u02c6eches\natlantique\np\u02c6echeurs\np\u02c6eche\nprobl\ncoop\nans\nindustrie\npoisson\nneuve\nterre\nouest\nstocks\nratives\nministre\nsant\nsaumon\naffaiblies\nfacult\nsecteur\nprogramme\ngion\nscienti\ufb01ques\ntravailler\nconduite\n\nThis representation is then used for retrieval tasks, providing better performance than\nexisting techniques. Such directions are then used to calculate the coordinates of the\n\n\fdocuments in a \u2019language independent\u2019 way. Of course, particular statistical care is needed\nfor excluding \u2019spurious\u2019 correlations. We show that the correlations we \ufb01nd are not the\neffect of chance, and that the resulting representation signi\ufb01cantly improves performance\nof retrieval systems. We \ufb01nd that the correlation existing between certain sets of words\nin English and French documents cannot be explained as a random correlation. Hence\nwe need to explain it by means of relations between the generative processes of the two\nversions of the documents, that we assume to be conditionally independent given the topic\nor content. Under such assumptions, hence, such correlations detect similarities in content\nbetween the two documents, and can be exploited to derive a semantic representation of\nthe text. This representation is then used for retrieval tasks, providing better performance\nthan existing techniques. We \ufb01rst apply the method to crosslingual information retrieval,\ncomparing performance with a related approach based on latent semantic indexing (LSI)\ndescribed below [5]. Secondly, we treat the second language as a complex label for the\n\ufb01rst language document and view the projection obtained by CL-KCCA as a semantic\nmap for use in a multilingual classi\ufb01cation task with very encouraging results. From the\ncomputational point of view, we detect such correlations by solving an eigenproblem, that\nis avoiding problems like local minima, and we do so by using kernels.\n\nThe KCCA machinery will be given in Section 3 and in Section 4 we will show how to\napply KCCA to cross-lingual retrieval while Section 4 describes the monolingual applica-\ntions. Finally, results will be presented in Section 5.\n\n2 Previous work\n\nThe use of LSI for cross-language retrieval was proposed by [5]. LSI uses a method from\nlinear algebra, singular value decomposition, to discover the important associative relation-\nships. An initial sample of documents is translated by human or, perhaps, by machine, to\n\npreprocessing documents a common vector-space, including words from both languages,\nis created and then the training set is analysed in this space using SVD:\n\ncreate a set of dual-language training documents\u0002\u0001\u0004\u0003\u0006\u0005\b\u0007\n\r! \u0015\"$#&%('\n\ning the \ufb01rst language features and the second set the second language features. To translate\nto a language-independent representation one projects (folds-in)\nits expanded (\ufb01lled up with zero components related to another language) vector represen-\n\n\u000f\u001b\r\u001d\u001c\ncorresponds to document) with its \ufb01rst set of coordinates giv-\n* . The similarity\n\ninto the space spanned by the, \ufb01rst eigenvectors .- : /\n\nwhere the) -th column of\u000f\na new document (query)*\ntation +\n\nbetween two documents is measured as the inner product between their projections. The\ndocuments that are the most similar to the query are considered to be relevant.\n\n1 \n\n*\u00190\n\n\u0003\n\t\f\u000b\u000e\r\u0010\u000f\u0012\u0011 and\u000f\u0014\u0013\u0015\n\n\u0002\u0016\u0017\u0003\u0018\u0005\u0019\u0007\n\n\u0003\u001a\t\f\u000b . After\n\n(1)\n\n\u000f\u0014\u0013\u001f\u001e\n\n3 Kernel Canonical Correlation Analysis\n\nIn this study our aim is to \ufb01nd an appropriate language-independent representation. Sup-\npose as for cross-lingual LSI (CL-LSI) we are given aligned texts in, for simplicity, two\nin another\n\nlanguages, i.e., every text in one language\u0001\u0004\u00033254\nis a translation of text\u00166\u00033287\nlanguage or vice versa. Our hypothesis is that having the corpus\u0019\u0001\u0004\u00039\u0005\u0019\u0007\n\u0003\u001a\t\f\u000b mapped to a high-\n\u00166\u0003@? (withA\n\u0011 and\n\u0013 as;=<\nto:\ndimensional feature space:\n\u0013 being respectively the kernels of the two mappings, i.e. matrices of the inner products\nbetween images of all the data points, [2]), we can learn (semantic) directions B\n\u0001>\u0003D?E?\u0018\u0007\n\u0016\u0017\u0003G?9?9\u0007\n\u0003\u001a\t\f\u000b and \u0003\u0006? and corpus\u0002\u00166\u0003\u0006\u0005\b\u0007\nin those spaces so that the projections <@B\n\nof input data images from the different languages would be maximally correlated. We\n\n\u0011 as;=<\n\n\u0003\n\t\f\u000b\n\u0011C'\n\n\u000f\n\u0011\n*\n%\n-\n+\nA\n\u0011\n2\n:\n\u0011\n\u0013\n2\n:\n\u0013\n\f<@B\n\n\u0011\u0004\u0003\n\nrewrites the numerator of (2) as\n\nhave thus intuitively de\ufb01ned a need for the notion of a kernel canonical:\n\u0013 ) which is de\ufb01ned as\n(:\n\u0006\b\u0007\n\t\n\u00166\u0003@?9?E?\n\u0003G?9?\n\u0013C'\n\u0018\u0015\u0019\u001b\u001a\u001c\u001a\n\u0005\u0001\n\n\n<9<\u000f8A9\n<\u00138A9\n.0.\n./.\n.0.\nWe note that% can be normalised so that<\u00138P9\n8 . Similar operations\nfor) yield analogous equations that together with (8) can be written in a matrix form:\nQSR\n\u0016C?9? :\n\u0001>?E? and\n\n