{"title": "Sparse and Locally Constant Gaussian Graphical Models", "book": "Advances in Neural Information Processing Systems", "page_first": 745, "page_last": 753, "abstract": "Locality information is crucial in datasets where each variable corresponds to a measurement in a manifold (silhouettes, motion trajectories, 2D and 3D images). Although these datasets are typically under-sampled and high-dimensional, they often need to be represented with low-complexity statistical models, which are comprised of only the important probabilistic dependencies in the datasets. Most methods attempt to reduce model complexity by enforcing structure sparseness. However, sparseness cannot describe inherent regularities in the structure. Hence, in this paper we first propose a new class of Gaussian graphical models which, together with sparseness, imposes local constancy through ${\\ell}_1$-norm penalization. Second, we propose an efficient algorithm which decomposes the strictly convex maximum likelihood estimation into a sequence of problems with closed form solutions. Through synthetic experiments, we evaluate the closeness of the recovered models to the ground truth. We also test the generalization performance of our method in a wide range of complex real-world datasets and demonstrate that it can capture useful structures such as the rotation and shrinking of a beating heart, motion correlations between body parts during walking and functional interactions of brain regions. Our method outperforms the state-of-the-art structure learning techniques for Gaussian graphical models both for small and large datasets.", "full_text": "Sparse and Locally Constant Gaussian Graphical\n\nModels\n\nJean Honorio,\n\nLuis Ortiz,\n\nDimitris Samaras\n\n{jhonorio,leortiz,samaras}@cs.sunysb.edu\n\nDepartment of Computer Science\n\nStony Brook University\nStony Brook, NY 11794\n\nNikos Paragios\nLaboratoire MAS\nEcole Centrale Paris\n\nChatenay-Malabry, France\n\nRita Goldstein\n\nMedical Department\n\nBrookhaven National Laboratory\n\nUpton, NY 11973\n\nnikos.paragios@ecp.fr\n\nrgoldstein@bnl.gov\n\nAbstract\n\nLocality information is crucial in datasets where each variable corresponds to a\nmeasurement in a manifold (silhouettes, motion trajectories, 2D and 3D images).\nAlthough these datasets are typically under-sampled and high-dimensional, they\noften need to be represented with low-complexity statistical models, which are\ncomprised of only the important probabilistic dependencies in the datasets. Most\nmethods attempt to reduce model complexity by enforcing structure sparseness.\nHowever, sparseness cannot describe inherent regularities in the structure. Hence,\nin this paper we \ufb01rst propose a new class of Gaussian graphical models which,\ntogether with sparseness, imposes local constancy through (cid:96)1-norm penalization.\nSecond, we propose an ef\ufb01cient algorithm which decomposes the strictly convex\nmaximum likelihood estimation into a sequence of problems with closed form\nsolutions. Through synthetic experiments, we evaluate the closeness of the recov-\nered models to the ground truth. We also test the generalization performance of\nour method in a wide range of complex real-world datasets and demonstrate that\nit captures useful structures such as the rotation and shrinking of a beating heart,\nmotion correlations between body parts during walking and functional interactions\nof brain regions. Our method outperforms the state-of-the-art structure learning\ntechniques for Gaussian graphical models both for small and large datasets.\n\n1 Introduction\nStructure learning aims to discover the topology of a probabilistic network of variables such that\nthis network represents accurately a given dataset while maintaining low complexity. Accuracy\nof representation is measured by the likelihood that the model explains the observed data, while\ncomplexity of a graphical model is measured by its number of parameters. Structure learning faces\nseveral challenges: the number of possible structures is super-exponential in the number of variables\nwhile the required sample size might be even exponential. Therefore, \ufb01nding good regularization\ntechniques is very important in order to avoid over-\ufb01tting and to achieve a better generalization\nperformance. In this paper, we propose local constancy as a prior for learning Gaussian graphical\nmodels, which is natural for spatial datasets such as those encountered in computer vision [1, 2, 3].\nFor Gaussian graphical models, the number of parameters, the number of edges in the structure\nand the number of non-zero elements in the inverse covariance or precision matrix are equivalent\n\n1\n\n\fmeasures of complexity. Therefore, several techniques focus on enforcing sparsity of the preci-\nsion matrix. An approximation method proposed in [4] relied on a sequence of sparse regressions.\nMaximum likelihood estimation with an (cid:96)1-norm penalty for encouraging sparseness is proposed in\n[5, 6, 7]. The difference among those methods is the optimization technique: a sequence of box-\nconstrained quadratic programs in [5], solution of the dual problem by sparse regression in [6] or\nan approximation via standard determinant maximization with linear inequality constraints in [7]. It\nhas been shown theoretically and experimentally, that only the covariance selection [5] as well as\ngraphical lasso [6] converge to the maximum likelihood estimator.\nIn datasets which are a collection of measurements for variables with some spatial arrangement,\none can de\ufb01ne a local neighborhood for each variable or manifold. Such variables correspond to\npoints in silhouettes, pixels in 2D images or voxels in 3D images. Silhouettes de\ufb01ne a natural\none-dimensional neighborhood in which each point has two neighbors on each side of the closed\ncontour. Similarly, one can de\ufb01ne a four-pixel neighborhood for 2D images as well as six-pixel\nneighborhood for 3D images. However, there is little research on spatial regularization for structure\nlearning. Some methods assume a one-dimensional spatial neighborhood (e.g. silhouettes) and that\nvariables far apart are only weakly correlated [8], interaction between a priori known groups of\nvariables as in [9], or block structures as in [10] in the context of Bayesian networks.\nOur contribution in this paper is two-fold. First, we propose local constancy, which encourages\n\ufb01nding connectivities between two close or distant clusters of variables, instead of between isolated\nvariables. It does not heavily constrain the set of possible structures, since it only imposes restric-\ntions of spatial closeness for each cluster independently, but not between clusters. We impose an\n(cid:96)1-norm penalty for differences of spatially neighboring variables, which allows obtaining locally\nconstant models that preserve sparseness, unlike (cid:96)2-norm penalties. Our model is strictly convex\nand therefore has a global minimum. Positive de\ufb01niteness of the estimated precision matrix is also\nguaranteed, since this is a necessary condition for the de\ufb01nition of a multivariate normal distribution.\nSecond, since optimization methods for structure learning on Gaussian graphical models [5, 6, 4, 7]\nare unable to handle local constancy constraints, we propose an ef\ufb01cient algorithm by maximizing\nwith respect to one row and column of the precision matrix at a time. By taking directions involving\neither one variable or two spatially neighboring variables, the problem reduces to minimization of a\npiecewise quadratic function, which can be performed in closed form.\nWe initially test the ability of our method to recover the ground truth structure from data, of a\ncomplex synthetic model which includes locally and not locally constant interactions as well as\nindependent variables. Our method outperforms the state-of-the-art structure learning techniques\n[5, 6, 4] for datasets with both small and large number of samples. We further show that our method\nhas better generalization performance on real-world datasets. We demonstrate the ability of our\nmethod to discover useful structures from datasets with a diverse nature of probabilistic relationships\nand spatial neighborhoods: manually labeled silhouettes in a walking sequence, cardiac magnetic\nresonance images (MRI) and functional brain MRI.\nSection 2 introduces Gaussian graphical models as well as techniques for learning such structures\nfrom data. Section 3 presents our sparse and locally constant Gaussian graphical models. Section 4\ndescribes our structure learning algorithm. Experimental results on synthetic and real-world datasets\nare shown and explained in Section 5. Main contributions and results are summarized in Section 6.\n\n2 Background\n\nIn this paper, we use the notation in Table 1. For convenience, we de\ufb01ne two new operators: the\nzero structure operator and the diagonal excluded product.\nA Gaussian graphical model [11] is a graph in which all random variables are continuous and jointly\nGaussian. This model corresponds to the multivariate normal distribution for N variables x \u2208 RN\nwith mean vector \u00b5 \u2208 RN and a covariance matrix \u03a3 \u2208 RN\u00d7N , or equivalently x \u223c N (\u00b5, \u03a3)\nwhere \u03a3 (cid:194) 0. Conditional independence in a Gaussian graphical model is simply re\ufb02ected in the\nzero entries of the precision matrix \u2126 = \u03a3\u22121 [11]. Let \u2126 = {\u03c9n1n2}, two variables xn1 and\nxn2 are conditionally independent if and only if \u03c9n1n2 = 0. The precision matrix representation\nis preferred because it allows detecting cases in which two seemingly correlated variables, actually\ndepend on a third confounding variable.\n\n2\n\n\f(cid:80)\n\nn |cn|\n(cid:80)\n\nNotation\n(cid:107)c(cid:107)1\n(cid:107)c(cid:107)\u221e\n|c|\ndiag(c) \u2208 RN\u00d7N matrix with elements of c \u2208 RN on its diagonal\n(cid:107)A(cid:107)1\n(cid:104)A, B(cid:105)\nA \u25e6 B \u2208 RM\u00d7N\nJ(A) \u2208 RM\u00d7N\nA (cid:174) B \u2208 RM\u00d7N\nA (cid:194) 0\ndiag(A) \u2208 RN\u00d7N matrix with diagonal elements of A \u2208 RN\u00d7N only\nvec(A) \u2208 RM N\n\nDescription\n(cid:96)1-norm of c \u2208 RN , i.e.\n(cid:96)\u221e-norm of c \u2208 RN , i.e. maxn |cn|\nentrywise absolute value of c \u2208 RN , i.e. (|c1|,|c2|, . . . ,|cN|)T\n(cid:96)1-norm of A \u2208 RM\u00d7N , i.e.\nscalar product of A, B \u2208 RM\u00d7N , i.e.\nHadamard or entrywise product of A, B \u2208 RM\u00d7N , i.e. (A \u25e6 B)mn = amnbmn\nzero structure operator of A \u2208 RM\u00d7N , by using the Iverson bracket jmn(A) =\n[amn = 0]\ndiagonal excluded product of A \u2208 RM\u00d7N and B \u2208 RN\u00d7N , i.e. A (cid:174) B =\nJ(A) \u25e6 (AB). It has the property that no diagonal entry of B is used in A (cid:174) B\nA \u2208 RN\u00d7N is symmetric and positive de\ufb01nite\nvector containing all elements of A \u2208 RM\u00d7N\n\n(cid:80)\nmn |amn|\n\nmn amnbmn\n\nTable 1: Notation used in this paper.\n\nThe concept of robust estimation by performing covariance selection was \ufb01rst introduced in [12]\nwhere the number of parameters to be estimated is reduced by setting some elements of the precision\nmatrix \u2126 to zero. Since \ufb01nding the most sparse precision matrix which \ufb01ts a dataset is a NP-hard\nproblem [5], in order to overcome it, several (cid:96)1-regularization methods have been proposed for\nlearning Gaussian graphical models from data.\n\nCovariance selection [5] starts with a dense sample covariance matrix (cid:98)\u03a3 and \ufb01ts a sparse preci-\n\nsion matrix \u2126 by solving a maximum likelihood estimation problem with a (cid:96)1-norm penalty which\nencourages sparseness of the precision matrix or conditional independence among variables:\n\n(1)\n\n(cid:179)\n\nmax\n\u2126(cid:194)0\n\nlog det \u2126 \u2212 (cid:104)(cid:98)\u03a3, \u2126(cid:105) \u2212 \u03c1(cid:107)\u2126(cid:107)1\n\n(cid:180)\n\nfor some \u03c1 > 0. Covariance selection computes small perturbations on the sample covariance\nmatrix such that it generates a sparse precision matrix, which results in a box-constrained quadratic\nprogramming. This method has moderate run time.\nThe Meinshausen-B\u00a8uhlmann approximation [4] obtains the conditional dependencies by performing\na sparse linear regression for each variable, by using lasso regression [13]. This method is very fast\nbut does not yield good estimates for lightly regularized models, as noted in [6]. The constrained\noptimization version of eq.(1) is solved in [7] by applying a standard determinant maximization\nwith linear inequality constraints, which requires iterative linearization of (cid:107)\u2126(cid:107)1. This technique in\ngeneral does not yield the maximum likelihood estimator, as noted in [14]. The graphical lasso\ntechnique [6] solves the dual form of eq.(1), which results in a lasso regression problem. This\nmethod has run times comparable to [4] without sacri\ufb01cing accuracy in the maximum likelihood\nestimator.\nStructure learning through (cid:96)1-regularization has been also proposed for different types of graphical\nmodels: Markov random \ufb01elds (MRFs) by a clique selection heuristic and approximate inference\n[15]; Bayesian networks on binary variables by logistic regression [16]; Conditional random \ufb01elds\nby pseudo-likelihood and block regularization in order to penalize all parameters of an edge simulta-\nneously [17]; and Ising models, i.e. MRFs on binary variables with pairwise interactions, by logistic\nregression [18] which is similar in spirit to [4].\nThere is little work on spatial regularization for structure learning. Adaptive banding on the\nCholesky factors of the precision matrix has been proposed in [8]. Instead of using the traditional\nlasso penalty, a nested lasso penalty is enforced. Entries at the right end of each row are promoted to\nzero faster than entries close to the diagonal. The main drawback of this technique is the assumption\nthat the more far apart two variables are the more likely they are to be independent. Grouping of\nentries in the precision matrix into disjoint subsets has been proposed in [9]. Such subsets can model\nfor instance dependencies between different groups of variables in the case of block structures. Al-\nthough such a formulation allows for more general settings, its main disadvantage is the need for an\na priori segmentation of the entries in the precision matrix.\n\n3\n\n\fRelated approaches have been proposed for Bayesian networks. In [10] it is assumed that variables\nbelong to unknown classes and probabilities of having edges among different classes were enforced\nto account for structure regularity, thus producing block structures only.\n\n3 Sparse and Locally Constant Gaussian Graphical Models\n\nFirst, we describe our local constancy assumption and its use to model the spatial coherence of\ndependence/independence relationships. Local constancy is de\ufb01ned as follows: if variable xn1 is\ndependent (or independent) of variable xn2, then a spatial neighbor xn(cid:48)\n1 of xn1 is more likely to\nbe dependent (or independent) of xn2. This encourages \ufb01nding connectivities between two close or\ndistant clusters of variables, instead of between isolated variables. Note that local constancy imposes\nrestrictions of spatial closeness for each cluster independently, but not between clusters.\nIn this paper, we impose constraints on the difference of entries in the precision matrix \u2126 \u2208 RN\u00d7N\nsample covariance matrix and D \u2208 RM\u00d7N be the discrete derivative operator on the manifold,\nwhere M \u2208 O(N) is the number of spatial neighborhood relationships. For instance, in a 2D image,\nM is the number of pixel pairs that are spatial neighbors on the manifold. More speci\ufb01cally, if pixel\nn1 and pixel n2 are spatial neighbors, we include a row m in D such that dmn1 = 1, dmn2 = \u22121 and\ndmn3 = 0 for n3 /\u2208 {n1, n2}. The following penalized maximum likelihood estimation is proposed:\n\nfor N variables, which correspond to spatially neighboring variables. Let (cid:98)\u03a3 \u2208 RN\u00d7N be the dense\n\n(cid:179)\n\nmax\n\u2126(cid:194)0\n\nlog det \u2126 \u2212 (cid:104)(cid:98)\u03a3, \u2126(cid:105) \u2212 \u03c1(cid:107)\u2126(cid:107)1 \u2212 \u03c4(cid:107)D (cid:174) \u2126(cid:107)1\n\n(cid:180)\n\n(2)\n\nfor some \u03c1, \u03c4 > 0. The \ufb01rst two terms model the quality of the \ufb01t of the estimated multivariate\nnormal distribution to the dataset. The third term \u03c1(cid:107)\u2126(cid:107)1 encourages sparseness while the fourth\nterm \u03c4(cid:107)D (cid:174) \u2126(cid:107)1 encourages local constancy in the precision matrix by penalizing the differences\nof spatially neighboring variables.\nIn conjunction with the (cid:96)1-norm penalty for sparseness, we introduce an (cid:96)1-norm penalty for local\nconstancy. As discussed further in [19], (cid:96)1-norm penalties lead to locally constant models which\npreserve sparseness, where as (cid:96)2-norm penalties of differences fail to do so.\nThe use of the diagonal excluded product for penalizing differences instead of the regular product of\nmatrices, is crucial. The regular product of matrices would penalize the difference between the di-\nagonal and off-diagonal entries of the precision matrix, and potentially destroy positive de\ufb01niteness\nof the solution for strongly regularized models.\nEven though the choice of the linear operator in eq.(2) does not affect the positive de\ufb01niteness\nproperties of the estimated precision matrix or the optimization algorithm, in the following Section\n4, we discuss positive de\ufb01niteness properties and develop an optimization algorithm for the speci\ufb01c\ncase of the discrete derivative operator D.\n\n4 Coordinate-Direction Descent Algorithm\n\nPositive de\ufb01niteness of the precision matrix is a necessary condition for the de\ufb01nition of a multivari-\nate normal distribution. Furthermore, strict convexity is a very desirable property in optimization,\nsince it ensures the existence of a unique global minimum. Notice that the penalized maximum like-\nlihood estimation problem in eq.(2) is strictly convex due to the convexity properties of log det \u2126 on\nthe space of symmetric positive de\ufb01nite matrices [20]. Maximization can be performed with respect\nto one row and column of the precision matrix \u2126 at a time. Without loss of generality, we use the\nlast row and column in our derivation, since permutation of rows and columns is always possible.\nAlso, note that rows in D can be freely permuted without affecting the objective function. Let:\n\n(cid:184)\n\n(cid:183)\n\n(cid:184)\n\n(cid:183)\n\n(cid:184)\n\n(cid:183)\n\n, (cid:98)\u03a3 =\n\n\u2126 =\n\nW y\nyT z\n\nu\nS\nuT v\n\n, D =\n\nD1 0M\u2212L\nD2\n\nd3\n\n(3)\n\nwhere W, S \u2208 RN\u22121\u00d7N\u22121, y, u \u2208 RN\u22121, d3 \u2208 RL is a vector with all entries different than zero,\nwhich requires a permutation of rows in D, D1 \u2208 RM\u2212L\u00d7N\u22121 and D2 \u2208 RL\u00d7N\u22121.\n\n4\n\n\fIn term of the variables y, z and the constant matrix W, the penalized maximum likelihood estima-\ntion problem in eq.(2) can be reformulated as:\n\n(cid:161)\n\nmax\n\u2126(cid:194)0\n\n(cid:162)\n\n(4)\n\n(5)\n\nlog(z \u2212 yTW\u22121y) \u2212 2uTy \u2212 (v + \u03c1)z \u2212 2\u03c1(cid:107)y(cid:107)1 \u2212 \u03c4 (cid:107)Ay \u2212 b(cid:107)1\n\nwhere (cid:107)Ay \u2212 b(cid:107)1 can be written in an extended form:\n\n(cid:107)Ay \u2212 b(cid:107)1 = (cid:107)D1y(cid:107)1 +\n\n(cid:176)(cid:176)vec(J(D2) \u25e6 (d3yT + D2W))\n\n(cid:176)(cid:176)\n\n1\n\n(cid:176)(cid:176)vec(J(D2) \u25e6 (d3yT + D2W))\n(cid:176)(cid:176)\n\nIntuitively, the term (cid:107)D1y(cid:107)1 penalizes differences across different rows of \u2126 which affect only val-\nues in y, while the term\n1 penalizes differences across different\ncolumns of \u2126 which affect values of y as well as W.\nIt can be shown that the precision matrix \u2126 is positive de\ufb01nite since its Schur complement z \u2212\nyTW\u22121y is positive. By maximizing eq.(4) with respect to z, we get:\n\nz \u2212 yTW\u22121y =\n\n1\n\nv + \u03c1\n\n(6)\n\nand since v > 0 and \u03c1 > 0, this implies that the Schur complement in eq.(6) is positive.\nMaximization with respect to one variable at a time leads to a strictly convex, non-smooth, piecewise\nquadratic function. By replacing the optimal value for z given by eq.(6) into the objective function\nin eq.(4), we get:\n\n1\n\n2 yT(v + \u03c1)W\u22121y + uTy + \u03c1(cid:107)y(cid:107)1 + \u03c4\n\n2 (cid:107)Ay \u2212 b(cid:107)1\n\n(7)\n\n(cid:161)\n\nmin\n\ny\u2208RN\u22121\n\n(cid:162)\n\nSince the objective function in eq.(7) is non-smooth, its derivative is not continuous and therefore\nmethods such as gradient descent cannot be applied. Although coordinate descent methods [5, 6]\nare suitable when only sparseness is enforced, they are not when local constancy is encouraged. As\nshown in [21], when penalizing an (cid:96)1-norm of differences, a coordinate descent algorithm can get\nstuck at sharp corners of the non-smooth optimization function; the resulting coordinates are station-\nary only under single-coordinate moves but not under diagonal moves involving two coordinates at\na time.\nFor a discrete derivative operator D used in the penalized maximum likelihood estimation problem\nin eq.(2), it suf\ufb01ces to take directions involving either one variable g = (0, . . . , 0, 1, 0, . . . , 0)T or\ntwo spatially neighboring variables g = (0, . . . , 0, 1, 0, . . . , 0, 1, 0, . . . , 0)T such that 1s appear in\nthe position corresponding to the two neighbor variables. Finally, assuming an initial value y0 and\na direction g, the objective function in eq.(7) can be reduced to \ufb01nd t in y(t) = y0 + tg such that it\nminimizes:\n\n(cid:161)\n\n1\n\n(cid:183)\n\n(cid:184)\n\nm rm|t \u2212 sm|(cid:162)\n(cid:80)\n(cid:183)\n\nmint\u2208R\np = (v + \u03c1)gTW\u22121g , q = ((v + \u03c1)W\u22121y0 + u)Tg\nr =\n\u2212diag(Ag)\u22121(Ay0 \u2212 b)\n\n2 pt2 + qt +\n\u03c1|g|\n2|Ag|\n\n\u2212diag(g)\u22121(y0)\n\n, s =\n\n\u03c4\n\n(cid:184)\n\n(8)\n\nFor simplicity of notation, we assume that r, s \u2208 RM use only non-zero entries of g and Ag on\nits de\ufb01nition in eq.(8). We sort and remove duplicate values in s, and propagate changes to r by\nadding the entries corresponding to the duplicate values in s. Note that these apparent modi\ufb01cations\ndo not change the objective function, but they simplify its optimization. The resulting minimization\nproblem in eq.(8) is convex, non-smooth and piecewise quadratic. Furthermore, since the objec-\ntive function is quadratic on each interval [\u2212\u221e; s1], [s1; s2], . . . , [sM\u22121; sM ], [sM ; +\u221e], it admits\na closed form solution.\nThe coordinate-direction descent algorithm is presented in detail in Table 2. A careful implemen-\ntation of the algorithm allows obtaining a time complexity of O(KN 3) for K iterations and N\nvariables, in which W\u22121, W\u22121y and Ay are updated at each iteration. In our experiments, the\n\n5\n\n\fCoordinate-direction descent algorithm\n\n\u03c4 and a discrete derivative operator D, \ufb01nd the precision matrix \u2126 (cid:194) 0 that maximizes:\n\n1. Given a dense sample covariance matrix (cid:98)\u03a3, sparseness parameter \u03c1, local constancy parameter\nlog det \u2126 \u2212 (cid:104)(cid:98)\u03a3, \u2126(cid:105) \u2212 \u03c1(cid:107)\u2126(cid:107)1 \u2212 \u03c4(cid:107)D (cid:174) \u2126(cid:107)1\n2. Initialize \u2126 = diag((cid:98)\u03a3)\u22121\n(a) Split \u2126 into W, y, z and (cid:98)\u03a3 into S, u, v as described in eq.(3)\n\n3. For each iteration 1, . . . K and each variable 1, . . . , N\n\n(b) Update W\u22121 by using the Sherman-Woodbury-Morrison formula (Note that when iterat-\n\ning from one variable to the next one, only one row and column change on matrix W)\n\n(c) Transform local constancy regularization term from D into A and b as described in eq.(5)\n(d) Compute W\u22121y and Ay\n(e) For each direction g involving either one variable or two spatially neighboring variables\n\ni. Find t that minimizes eq.(8) in closed form\nii. Update y \u2190 y + tg\niii. Update W\u22121y \u2190 W\u22121y + tW\u22121g\niv. Update Ay \u2190 Ay + tAg\nv+\u03c1 + yTW\u22121y\n\n(f) Update z \u2190 1\n\nTable 2: Coordinate-direction descent algorithm for learning sparse and locally constant Gaussian graphical\nmodels.\n\nFigure 1: (a) Ground truth model on an open contour manifold. Spatial neighbors are connected with black\ndashed lines. Positive interactions are shown in blue, negative interactions in red. The model contains two\nlocally constant interactions between (x1, x2) and (x6, x7), and between (x4, x5) and (x8, x9), a not locally\nconstant interaction between x1 and x4, and an independent variable x3; (b) colored precision matrix of the\nground truth, red for negative entries, blue for positive entries; learnt structure from (c) small and (d) large\ndatasets. Note that for large datasets all connections are correctly recovered.\n\nalgorithm converges quickly in usually K = 10 iterations. The polynomial dependency on the num-\nber of variables of O(N 3) is expected since we cannot produce an algorithm faster than computing\nthe inverse of the sample covariance in the case of an in\ufb01nite sample.\nFinally, in the spirit of [5], a method for reducing the size of the original problem is presented.\nGiven a P -dimensional spatial neighborhood or manifold (e.g. P = 1 for silhouettes, P = 2 for\na four-pixel neighborhood on 2D images, P = 3 for a six-pixel neighborhood on 3D images), the\nobjective function in eq.(7) has the maximizer y = 0 for variables on which (cid:107)u(cid:107)\u221e \u2264 \u03c1\u2212 P \u03c4. Since\nthis condition does not depend on speci\ufb01c entries in the iterative estimation of the precision matrix,\nthis property can be used to reduce the size of the problem in advance by removing such variables.\n\n5 Experimental Results\nConvergence to Ground Truth. We begin with a small synthetic example to test the ability of\nthe method for recovering the ground truth structure from data, in a complex scenario in which\nour method has to deal with both locally and not locally constant interactions as well as independent\nvariables. The ground truth Gaussian graphical model is shown in Figure 1 and it contains 9 variables\narranged in an open contour manifold.\nIn order to measure the closeness of the recovered models to the ground truth, we measure the\nKullback-Leibler divergence, average precision (one minus the fraction of falsely included edges),\naverage recall (one minus the fraction of falsely excluded edges) as well as the Frobenius norm be-\ntween the recovered model and the ground truth. For comparison purposes, we picked two of the\n\n6\n\n0.45-0.350.4\u07541 \u07542 \u07543 \u07544 \u07545 \u07546 \u07547 \u07548 \u07549 (a)(b)(c)(d)\fFigure 2: Kullback-Leibler divergence with respect to the best method, average precision, recall and Frobenius\nnorm between the recovered model and the ground truth. Our method (SLCGGM) outperforms the fully con-\nnected model (Full), Meinshausen-B\u00a8uhlmann approximation (MB-or, MB-and), covariance selection (CovSel),\ngraphical lasso (GLasso) for small datasets (in blue solid line) and for large datasets (in red dashed line). The\nfully independent model (Indep) resulted in relative divergences of 2.49 for small and 113.84 for large datasets.\n\nstate-of-the-art structure learning techniques: covariance selection [5] and graphical lasso [6], since\nit has been shown theoretically and experimentally that they both converge to the maximum likeli-\nhood estimator. We also test the Meinshausen-B\u00a8uhlmann approximation [4]. The fully connected as\nwell as fully independent model are also included as baseline methods.\nTwo different scenarios are tested: small datasets of four samples, and large datasets of 400 samples.\nUnder each scenario, 50 datasets are randomly generated from the ground truth Gaussian graphical\nmodel. It can be concluded from Figure 2 that our method outperforms the state-of-the-art structure\nlearning techniques both for small and large datasets. This is due to the fact that the ground truth\ndata contains locally constant interactions, and our method imposes a prior for local constancy.\nAlthough this is a complex scenario which also contains not locally constant interactions as well as\nan independent variable, our method can recover a more plausible model when compared to other\nmethods. Note that even though other methods may exhibit a higher recall for small datasets, our\nmethod consistently recovers a better probability distribution.\nA visual comparison of the ground truth versus the best recovered model by our method from small\nand large datasets is shown in Figure 1. The image shows the precision matrix in which red squares\nrepresent negative entries, while blue squares represent positive entries. There is very little differ-\nence between the ground truth and the recovered model from large datasets. Although the model\nis not fully recovered from small datasets, our technique performs better than the Meinshausen-\nB\u00a8uhlmann approximation, covariance selection and graphical lasso in Figure 2.\n\nReal-World Datasets.\nIn the following experiments, we demonstrate the ability of our method to\ndiscover useful structures from real-world datasets. Datasets with a diverse nature of probabilistic\nrelationships are included in our experiments: from cardiac MRI [22], our method recovers global\ndeformation in the form of rotation and shrinking; from a walking sequence1, our method \ufb01nds the\nlong range interactions between different parts; and from functional brain MRI [23], our method\nrecovers functional interactions between different regions and discover differences in processing\nmonetary rewards between cocaine addicted subjects versus healthy control subjects. Each dataset\nis also diverse in the type of spatial neighborhood: one-dimensional for silhouettes in a walking\nsequence, two-dimensional for cardiac MRI and three-dimensional for functional brain MRI.\n\nGeneralization. Cross-validation was performed in order to measure the generalization perfor-\nmance of our method in estimating the underlying distribution. Each dataset was randomly split\ninto \ufb01ve sets. On each round, four sets were used for training and the remaining set was used for\nmeasuring the log-likelihood. Table 3 shows that our method consistently outperforms techniques\nthat encourage sparsity only. This is strong evidence that datasets that are measured over a spa-\ntial manifold are locally constant, as well as that our method is a good regularization technique\nthat avoids over-\ufb01tting and allows for better generalization. Another interesting fact is that for the\nbrain MRI dataset, which is high dimensional and contains a small number of samples, the model\nthat assumes full independence performed better than the Meinshausen-B\u00a8uhlmann approximation,\ncovariance selection and graphical lasso. Similar observations has been already made in [24, 25]\nwhere it was found that assuming independence often performs better than learning dependencies\namong variables.\n\n1Human Identi\ufb01cation at a Distance dataset http://www.cc.gatech.edu/cpl/projects/hid/\n\n7\n\n012345Kullback\u2212Leibler divergenceFullMB\u2212orMB\u2212andCovSelGLassoSLCGGM0.20.40.60.811.2PrecisionFullMB\u2212orMB\u2212andCovSelGLassoSLCGGM0.40.60.81RecallMB\u2212orMB\u2212andCovSelGLassoSLCGGM01234567Frobenius normIndepFullMB\u2212orMB\u2212andCovSelGLassoSLCGGM\fFigure 3: Real-world datasets: cardiac MRI displacement (a) at full contraction and (b) at full expansion, (c)\n2D spatial manifold and (d) learnt structure, which captures contraction and expansion (in red), and similar dis-\nplacements between neighbor pixels (in blue); (e) silhouette manifold and (f) learnt structure from a manually\nlabeled walking sequence, showing similar displacements from each independent leg (in blue) and opposite dis-\nplacements between both legs as well as between hands and feet (in red); and structures learnt from functional\nbrain MRI in a monetary reward task for (g) drug addicted subjects with more connections in the cerebellum\n(in yellow) versus (h) control subjects with more connections in the prefrontal cortex (in green).\n\nMethod\n\nSynthetic\n\nIndep\nMB-and\nMB-or\nCovSel\nGLasso\nSLCGGM\n\n-6428.23\n-5595.87*\n-5595.13*\n-5626.32\n-5625.79\n-5623.52\n\nCardiac\nMRI\n-5150.58\n-5620.45\n-4135.98*\n-5044.41\n-5041.52\n-4017.56\n\nWalking\nSequence\n-12957.72\n-12542.15\n-11317.24\n-12051.51\n-12035.50\n-10718.62\n\nBrain MRI\nDrug-addicted\n-324724.24\n-418605.02\n-398725.04\n-409402.60\n-413176.45\n-297318.61\n\nBrain MRI\nControl\n-302729.54\n-317034.67\n-298186.66\n-300829.98\n-305307.25\n-278678.35\n\nTable 3: Cross-validated log-likelihood on the testing set. Our method (SLCGGM) outperforms the\nMeinshausen-B\u00a8uhlmann approximation (MB-and, MB-or), covariance selection (CovSel), graphical lasso\n(GLasso) and the fully independent model (Indep). Values marked with an asterisk are not statistically sig-\nni\ufb01cantly different from our method.\n\n6 Conclusions and Future Work\n\nIn this paper, we proposed local constancy for Gaussian graphical models, which encourages \ufb01nding\nprobabilistic connectivities between two close or distant clusters of variables, instead of between\nisolated variables. We introduced an (cid:96)1-norm penalty for local constancy into a strictly convex\nmaximum likelihood estimation. Furthermore, we proposed an ef\ufb01cient optimization algorithm and\nproved that our method guarantees positive de\ufb01niteness of the estimated precision matrix. We tested\nthe ability of our method to recover the ground truth structure from data, in a complex scenario with\nlocally and not locally constant interactions as well as independent variables. We also tested the\ngeneralization performance of our method in a wide range of complex real-world datasets with a\ndiverse nature of probabilistic relationships as well as neighborhood type.\nThere are several ways of extending this research. Methods for selecting regularization parameters\nfor sparseness and local constancy need to be further investigated. Although the positive de\ufb01nite-\nness properties of the precision matrix as well as the optimization algorithm still hold when including\noperators such as the Laplacian for encouraging smoothness, bene\ufb01ts of such a regularization ap-\nproach need to be analyzed. In practice, our technique converges in a small number of iterations, but\na more precise analysis of the rate of convergence needs to be performed. Finally, model selection\nconsistency when the number of samples grows to in\ufb01nity needs to be proved.\n\nAcknowledgments\n\nThis work was supported in part by NIDA Grant 1 R01 DA020949-01 and NSF Grant CNS-0721701\n\n8\n\n(a)(b)(c)(d)(e)(f)\fReferences\n[1] D. Crandall, P. Felzenszwalb, and D. Huttenlocher. Spatial priors for part-based recognition using statis-\n\ntical models. IEEE Conf. Computer Vision and Pattern Recognition, 2005.\n\n[2] P. Felzenszwalb and D. Huttenlocher. Pictorial structures for object recognition. International Journal of\n\nComputer Vision, 2005.\n\n[3] L. Gu, E. Xing, and T. Kanade. Learning GMRF structures for spatial priors.\n\nVision and Pattern Recognition, 2007.\n\nIEEE Conf. Computer\n\n[4] N. Meinshausen and P. B\u00a8uhlmann. High dimensional graphs and variable selection with the lasso. The\n\nAnnals of Statistics, 2006.\n\n[5] O. Banerjee, L. El Ghaoui, A. d\u2019Aspremont, and G. Natsoulis. Convex optimization techniques for \ufb01tting\n\nsparse Gaussian graphical models. International Conference on Machine Learning, 2006.\n\n[6] J. Friedman, T. Hastie, and R. Tibshirani. Sparse inverse covariance estimation with the graphical lasso.\n\nBiostatistics, 2007.\n\n[7] M. Yuan and Y. Lin. Model selection and estimation in the Gaussian graphical model. Biometrika, 2007.\n[8] E. Levina, A. Rothman, and J. Zhu. Sparse estimation of large covariance matrices via a nested lasso\n\npenalty. The Annals of Applied Statistics, 2008.\n\n[9] J. Duchi, S. Gould, and D. Koller. Projected subgradient methods for learning sparse Gaussians. Uncer-\n\ntainty in Arti\ufb01cial Intelligence, 2008.\n\n[10] V. Mansinghka, C. Kemp, J. Tenenbaum, and T. Grif\ufb01ths. Structured priors for structure learning. Uncer-\n\ntainty in Arti\ufb01cial Intelligence, 2006.\n\n[11] S. Lauritzen. Graphical Models. Oxford Press, 1996.\n[12] A. Dempster. Covariance selection. Biometrics, 1972.\n[13] R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society,\n\n1996.\n\n[14] O. Banerjee, L. El Ghaoui, and A. d\u2019Aspremont. Model selection through sparse maximum likelihood\n\nestimation for multivariate Gaussian or binary data. Journal of Machine Learning Research, 2008.\n\n[15] S. Lee, V. Ganapathi, and D. Koller. Ef\ufb01cient structure learning of Markov networks using (cid:96)1-\n\nregularization. Advances in Neural Information Processing Systems, 2006.\n\n[16] M. Schmidt, A. Niculescu-Mizil, and K. Murphy. Learning graphical model structure using (cid:96)1-\n\nregularization paths. AAAI Conf. Arti\ufb01cial Intelligence, 2007.\n\n[17] M. Schmidt, K. Murphy, G. Fung, and R. Rosales. Structure learning in random \ufb01elds for heart motion\n\nabnormality detection. IEEE Conf. Computer Vision and Pattern Recognition, 2008.\n\n[18] M. Wainwright, P. Ravikumar, and J. Lafferty. High dimensional graphical model selection using (cid:96)1-\n\nregularized logistic regression. Advances in Neural Information Processing Systems, 2006.\n\n[19] R. Tibshirani, M. Saunders, S. Rosset, J. Zhu, and K. Knight. Sparsity and smoothness via the fused lasso.\n\nJournal of the Royal Statistical Society, 2005.\n\n[20] S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press, 2006.\n[21] J. Friedman, T. Hastie, H. H\u00a8o\ufb02ing, and R. Tibshirani. Pathwise coordinate optimization. The Annals of\n\nApplied Statistics, 2007.\n\n[22] J. Deux, A. Rahmouni, and J. Garot. Cardiac magnetic resonance and 64-slice cardiac CT of lipomatous\n\nmetaplasia of chronic myocardial infarction. European Heart Journal, 2008.\n\n[23] R. Goldstein, D. Tomasi, N. Alia-Klein, L. Zhang, F. Telang, and N. Volkow. The effect of practice on a\n\nsustained attention task in cocaine abusers. NeuroImage, 2007.\n\n[24] P. Domingos and M. Pazzani. On the optimality of the simple Bayesian classi\ufb01er under zero-one loss.\n\nMachine Learning, 1997.\n\n[25] N. Friedman, D. Geiger, and M. Goldszmidt. Bayesian network classi\ufb01ers. Machine Learning, 1997.\n\n9\n\n\f", "award": [], "sourceid": 23, "authors": [{"given_name": "Jean", "family_name": "Honorio", "institution": null}, {"given_name": "Dimitris", "family_name": "Samaras", "institution": null}, {"given_name": "Nikos", "family_name": "Paragios", "institution": null}, {"given_name": "Rita", "family_name": "Goldstein", "institution": null}, {"given_name": "Luis", "family_name": "Ortiz", "institution": null}]}