{"title": "Beyond Maximum Likelihood and Density Estimation: A Sample-Based Criterion for Unsupervised Learning of Complex Models", "book": "Advances in Neural Information Processing Systems", "page_first": 535, "page_last": 541, "abstract": null, "full_text": "Beyond maximum likelihood and density \nestimation: A sample-based criterion for \nunsupervised learning of complex models \n\nSepp Hochreiter and Michael C. Mozer \n\nDepartment of Computer Science \n\nUniversity of Colorado \nBoulder, CO 80309- 0430 \n\n{hochreit,mozer}~cs.colorado.edu \n\nAbstract \n\nThe goal of many unsupervised learning procedures is to bring two \nprobability distributions into alignment. Generative models such \nas Gaussian mixtures and Boltzmann machines can be cast in this \nlight, as can recoding models such as ICA and projection pursuit. \nWe propose a novel sample-based error measure for these classes of \nmodels, which applies even in situations where maximum likelihood \n(ML) and probability density estimation-based formulations can(cid:173)\nnot be applied, e.g., models that are nonlinear or have intractable \nposteriors. Furthermore, our sample-based error measure avoids \nthe difficulties of approximating a density function. We prove that \nwith an unconstrained model, (1) our approach converges on the \ncorrect solution as the number of samples goes to infinity, and (2) \nthe expected solution of our approach in the generative framework \nis the ML solution. Finally, we evaluate our approach via simula(cid:173)\ntions of linear and nonlinear models on mixture of Gaussians and \nICA problems. The experiments show the broad applicability and \ngenerality of our approach. \n\n1 \n\nIntroduction \n\nMany unsupervised learning procedures can be viewed as trying to bring two prob(cid:173)\nability distributions into alignment. Two well known classes of unsupervised pro(cid:173)\ncedures that can be cast in this manner are generative and recoding models. In a \ngenerative unsupervised framework, the environment generates training examples(cid:173)\nwhich we will refer to as observations-by sampling from one distribution; the other \ndistribution is embodied in the model. Examples of generative frameworks are mix(cid:173)\ntures of Gaussians (MoG) [2], factor analysis [4], and Boltzmann machines [8]. In \nthe recoding unsupervised framework, the model transforms points from an obser-\n\n\fvation space to an output space, and the output distribution is compared either to \na reference distribution or to a distribution derived from the output distribution. \nAn example is independent component analysis (leA) [11], a method that discovers \na representation of vector-valued observations in which the statistical dependence \namong the vector elements in the output space is minimized. With ICA, the model \ndemixes observation vectors and the output distribution is compared against a fac(cid:173)\ntorial distribution which is derived either from assumptions about the distribution \n(e.g., supergaussian) or from a factorization of the output distribution. Other ex(cid:173)\namples within the recoding framework are projection methods such as projection \npursuit (e.g., [14]) and principal component analysis. In each case we have described \nfor the unsupervised learning of a model, the objective is to bring two probability \ndistributions- one or both of which is produced by the model-\ninto alignment. To \nimprove the model, we need to define a measure of the discrepancy between the two \ndistributions, and to know how the model parameters influence the discrepancy. \n\nOne natural approach is to use outputs from the model to construct a probability \ndensity estimator (PD\u00a3). The primary disadvantage of such an approach is that the \naccuracy of the learning procedure depends highly on the quality of the PDE. PDEs \nface the bias-variance trade-off. For the learning of generative models, maximum \nlikelihood (ML) is a popular approach that avoids PDEs. In an ML approach, the \nmodel's generative distribution is expressed analytically, which makes it straightfor(cid:173)\nward to evaluate the posterior, p(data I model), and therefore, to adjust the model \nparameters to maximize the likelihood of the data being generated by the model. \nThis limits the ML approach to models that have tractable posteriors, true only of \nthe simplest models [1, 6, 9]. \n\nWe describe an approach which, like ML, avoids the construction of an explicit \nPDE, yet does so without requiring an analytic expression for the posterior. Our \napproach, which we call a sample-based method, assumes a set of samples from each \ndistribution and proposes an error measure of the disagreement defined directly in \nterms of the samples. Thus, a second set of samples drawn from the model serves in \nplace of a PDE or an analytic expression of the model's density. The sample-based \nmethod is inspired by the theory of electric fields, which describes the interactions \namong charged particles. For more details on the metaphor, see [10]. \n\nIn this paper, we prove that our approach converges to the optimal solution as the \nsample size goes to infinity, assuming an unconstrained (maximally flexible) model. \nWe also prove that the expected solution of our approach is the ML solution in \na generative context. We present empirical results showing that the sample-based \napproach works for both linear and nonlinear models. \n\n2 The Method \n\nConsider a model to be learned, fw, parameterized by weights w. The model maps \nan input vector, zi, indexed by i, to an output vector xi = fw(zi). The model \ninputs are sampled from a distribution pz(.), and the learning procedure calls for \nadjusting the model such that the output distribution, Px (.), comes to match a target \ndistribution, py(.). For unsupervised recoding models, zi is an observation, xi is \nthe transformed representation of zi, and Py (.) specifies the desired code properties. \nFor unsupervised generative models, pz(.) is fixed and py(.) is the distribution of \nobservations. \n\n\fThe Sample-based Method: The Intuitive Story \n\nAssume that we have data points sampled from two different distributions, la(cid:173)\nbeled \"- \" and \"+\" (Figure 1). The sample-based error measure specifies how sam(cid:173)\nples should be moved so that the two distributions are \nbrought into alignment. In the figure, samples from the \nlower left and upper right corners must be moved to the \nupper left and lower right corners. Our goal is to estab-\nlish an explicit correspondence between each \"- \" sample \nand each \"+\" sample. Toward this end, our sample-\nbased method utilizes on mass interactions among the \nsamples, by introducing a repelling force between sam-\npIes from the same distribution and an attractive force between samples from dif(cid:173)\nferent distributions, and allowing the samples to move according to these forces. \n\nFigure 1 \n\nThe Sample-based Method: The Formal Presentation \nIn conceiving of the problem in terms of samples that attract and repel one another, \nit is natural to think in terms of physical interactions among charged particles. \nConsider a set of positively charged particles at locations denoted by xi, i = L.Nx, \nand a set of negatively charged particles at locations denoted by yi, j = L .N y . \nThe particles correspond to data samples from two distributions. The interaction \namong particles is characterized by the Coulomb energy, E: \n\nwhere r(a, b) is a distance measure- Green's function- which results in nearby \nparticles having a strong influence on the energy, but distant particles having only \na weak influence. Green's function is defined as r(a, b) = c(d) Ilia - bll d - 2 , where \nd is the dimensionality of the space, c( d) is a constant only depending on d, and 11.11 \ndenotes the Euclidean distance. For d = 2, r(a, b) = k In (ila - biD. \nThe Coulomb energy is low when negative and positive particles are near one an(cid:173)\nother, positive particles are far from one another, and negative particles are far from \none another. This is exactly the state we would like to achieve for our two distribu(cid:173)\ntions of samples: bringing the two distributions into alignment without collapsing \neither distribution into a trivial form. Consequently, our sample-based method \nproposes using the Coulomb energy as an objective function to be minimized. \n\nis \n\nto a sample's \n\nlocation \n\nis readily com(cid:173)\n\nthe force acting on that sample), and this gradient can be \n\nThe gradient of E with respect \nputed (it \nchained with the Jacobian of the location with respect to the model pa(cid:173)\n-to \"V wE = \nrameters w to obtain a gradient-based update rule: \n- JY~~~l(WG)T \"Vyk 0, \n\nwhere ISE(O) is independent of t. Thus, the ISE between the two distributions is \nguaranteed to decrease during learning, when the sample size goes to infinity. \n\nProof: Expected Generative Solution is ML Solution \n\nIn the case of a generative model which has no constraints (i.e., can model any \ndistribution), the maximum likelihood solution will have distribution px(a) = \n.Jy Ef':18(yi - a), i.e., the model will produce only the observations and all of \nthem with equal probability. For this case, we show that our sample-based method \nwill yield the same solution in expectation as ML. \n\nThe sample-based method converges to a local minimum of the energy, where \n(Va