{"title": "Validity Estimates for Loopy Belief Propagation on Binary Real-world Networks", "book": "Advances in Neural Information Processing Systems", "page_first": 945, "page_last": 952, "abstract": null, "full_text": "     Validity estimates for loopy Belief Propagation\n                 on binary real-world networks\n\n\n\n                                        Joris Mooij\n          Dept. of Biophysics, Inst. for Neuroscience, Radboud Univ. Nijmegen\n                            6525 EZ Nijmegen, the Netherlands\n                              j.mooij@science.ru.nl\n\n\n                                    Hilbert J. Kappen\n          Dept. of Biophysics, Inst. for Neuroscience, Radboud Univ. Nijmegen\n                            6525 EZ Nijmegen, the Netherlands\n                             b.kappen@science.ru.nl\n\n\n\n\n                                         Abstract\n\n         We introduce a computationally efficient method to estimate the valid-\n         ity of the BP method as a function of graph topology, the connectiv-\n         ity strength, frustration and network size. We present numerical results\n         that demonstrate the correctness of our estimates for the uniform random\n         model and for a real-world network (\"C. Elegans\"). Although the method\n         is restricted to pair-wise interactions, no local evidence (zero \"biases\")\n         and binary variables, we believe that its predictions correctly capture the\n         limitations of BP for inference and MAP estimation on arbitrary graphi-\n         cal models. Using this approach, we find that BP always performs better\n         than MF. Especially for large networks with broad degree distributions\n         (such as scale-free networks) BP turns out to significantly outperform\n         MF.\n\n\n\n1     Introduction\n\nLoopy Belief Propagation (BP) [1] and its generalizations (such as the Cluster Variation\nMethod [2]) are powerful methods for inference and optimization. As is well-known, BP is\nexact on trees, but also yields surprisingly good results for many other graphs that arise in\nreal-world applications [3, 4]. On the other hand, for densely connected graphs with high\ninteraction strengths the results can be quite bad or BP can simply fail to converge. Despite\nthe fact that BP is often used in applications nowadays, a good theoretical understanding of\nits convergence properties and the quality of the approximation is still lacking (except for\nthe very special case of graphs with a single loop [5]).\n\nIn this article we attempt to answer the question in what way the quality of the BP re-\nsults depends on the topology of the underlying graph (looking at structural properties such\nas short cycles and large \"hubs\") and on the interaction potentials (i.e. strength and frus-\ntration). We do this for the special but interesting case of binary networks with symmetric\npairwise potentials (i.e. Boltzmann machines) without local evidence. This has the practical\n\n\f\nadvantage that analytical calculations are feasible and furthermore we believe that adding\nlocal evidence will only serve to extend the domain of convergence, implying this to be the\nworst-case scenario. We compare the results with those of the variational mean-field (MF)\nmethod.\n\nReal-world graphs are often far from uniformly random and possess structure such as clus-\ntering and power-law degree distributions [6]. Since we expect these structural features to\narise in many applications of BP, we focus in this article on graphs modeling this kind of\nfeatures. In particular, we consider Erdos-Renyi uniform random graphs [7], Barabasi-\nAlbert \"scale-free\" graphs [8], and the neural network of a widely studied worm, the\nCaenorhabditis elegans.\n\nThis paper is organized as follows. In the next section we describe the class of graphical\nmodels under investigation and explain our method to efficiently estimate the validity of\nBP and MF. In section 3 we give a qualitative discussion of how the connectivity strength\nand frustration generally govern the model behavior and discuss the relevant regimes of the\nmodel parameters. We show for uniform random graphs that our validity estimates are in\nvery good agreement with the real behavior of the BP algorithm. In section 4 we study the\ninfluence of graph topology. Thanks to the numerical efficiency of our estimation method\nwe are able to study very large (N  10000) networks, for which it would not be feasible\nto simply run BP and look what happens. We also try our method on the neural network of\nthe worm C. Elegans and find almost perfect agreement of our predictions with observed\nBP behavior. We conclude that BP is always better than MF and that the difference is\nparticularly striking for the case of large networks with broad degree distributions such as\nscale-free graphs.\n\n\n2    Model, paramagnetic solution and stability analysis\n\nLet G = (V, B) be an undirected labelled graph without self-connections, defined by a\nset of nodes V = {1, . . . , N } and a set of links B  {(i, j) | 1  i < j  N }. The\nadjacency matrix corresponding to G is denoted M and defined as follows: Mij := 1 if\n(ij)  B or (ji)  B and 0 otherwise. We denote the set of neighbors of node i  V by\nNi := {j  V | (ij)  B} and its degree by di := #(Ni). We define the average degree\nd := 1            d\n     N     iV         i and the maximum degree  := maxiV di.\n\nTo each node i we associate a binary random variable xi taking values in {-1, +1}. Let\nW be a symmetric N  N -matrix defining the strength of the links between the nodes. The\nprobability distribution over configurations x = (x1, . . . , xN ) is given by\n\n                                    1                            1              1 M\n                         P(x) :=                   eWijxixj =                  e 2 ijWijxixj    (1)\n                                    Z                            Z\n                                         (ij)B                       i,jV\n\n\nwith Z a normalization constant. We will take the weight matrix W to be random, with\ni.i.d. entries {Wij}1i<jN distributed according to the Gaussian law with mean J0 and\nvariance J 2.\n\nFor this model, instead of using the single-node and pair-wise beliefs bi(xi) resp.\nbij(xi, xj), it turns out to be more convenient to use the (equivalent) quantities m :=\n{mi}iV and  := {ij}(ij)B, defined by:\n\n            mi := bi(+1) - bi(-1);\n            ij := bij(+1, +1) - bij(+1, -1) - bij(-1, +1) + bij(-1, -1).\n\nWe will use these throughout this paper. We call the mi magnetizations; note that the\nexpectation values E xi vanish because of the symmetry in the probability distribution (1).\n\n\f\nAs is well-known [2, 9], fixed points of BP correspond to stationary points of the Bethe\nfree energy, which is in this case given by\n\n                                                       N                                    1 + mixi\n     FBe(m, ) := -                   Wijij +               (1 - di)                         2\n                          (ij)B                      i=1                xi=1\n\n                                                                                       1 + mixi + mjxj + xixjij\n                                                +                                                      4\n                                                      (ij)B xi,xj =1\n\nwith (x) := x log x. Note that with this parameterization all normalization and overlap\nconstraints (i.e.            b\n                     x            ij (xi, xj ) = bi(xi)) are satisfied by construction [10]. We can mini-\n                        j\nmize the Bethe free energy analytically by setting its derivatives to zero; one then immedi-\nately sees that a possible solution of the resulting equations is the paramagnetic1 solution:\nmi = 0 and ij = tanh Wij (for (ij)  B). For this solution to be a minimum (instead of\na saddle point or maximum), the Hessian of FBe at that point should be positive-definite.\nThis condition turns out to be equivalent to the following Bethe stability matrix\n\n                                               2                         ij\n      (A                                        ik\n        Be)ij := ij         1 +                              - Mij                          (with ij = tanh Wij)    (2)\n                                              1 - 2                   1 - 2\n                                     kN              ik                         ij\n                                         i\n\nbeing positive-definite. Whether this is the case obviously depends on the values of the\nweights Wij and the adjacency matrix M . Since for zero weights (W = 0), the stability\nmatrix is just the identity matrix, the paramagnetic solution is a minimum of the Bethe free\nenergy for small values of the weights Wij. The question of what \"small\" exactly means\nin terms of J and J0 and how this relates to the graph topology will be taken on in the next\ntwo sections.\n\nFirst we discuss the situation for the mean-field variational method. The mean-field free\nenergy FMF (m) only depends on m; we can set its derivatives to zero, which again yields\nthe paramagnetic solution m = 0. The corresponding stability matrix (equal to the Hes-\nsian) is given by\n                                              (AMF )ij := ij - WijMij\nand should be positive-definite for the paramagnetic solution to be stable. One can prove\n[11] that ABe is positive-definite whenever AMF is positive-definite. Since the exact mag-\nnetizations are zero, we conclude that the Bethe approximation is better than the mean-field\napproximation for all possible choices of the weights W . As we will see later on, this dif-\nference can become quite large for large networks.\n\n\n3     Weight dependence\n\nThe behavior of the graphical model depends critically on the parameters J0 and J. Taking\nthe graph topology to be uniformly random (see also subsection 4.1) we recover the model\nknown in the statistical physics community as the Viana-Bray model [12], which has been\nthoroughly studied and is quite well-understood. In the limit N  , there are different\nrelevant regimes (\"phases\") for the parameters J and J0 to be distinguished (cf. Fig. 1):\n\n         The paramagnetic phase, where the magnetizations all vanish (m = 0), valid for\n            J and J0 both small.\n         The ferromagnetic phase, where two configurations (characterized by all magne-\n            tizations being either positive or negative) each get half of the probability mass.\n            This is the phase occurring for large J0.\n\n     1Throughout this article, we will use terminology from statistical physics if there is no good\ncorresponding terminology in the field of machine learning available.\n\n\f\n                          BP convergence behavior                                                     Stability m=0 minimum Bethe free energy\n        0.4                                                                               0.4\n                                                                                                        m=0 stable         \n                                                                                                        (spin-glass phase) \n                      no convergence                                                                                                             ? \n        0.3                                                                               0.3\n\n                                                                                                 marginal instability\n\n   J 0.2                                                                            J 0.2\n\n                                                       convergence                                     m=0 stable                          m=0 instable  \n                     convergence                       to ferromagnetic                                (paramagnetic\n        0.1                                                                               0.1                                              (ferromagnetic\n                     to m=0                            solutions                                       phase)                               phase)       \n\n\n         0                                                                                 0\n               0        0.02        0.04              0.06     0.08        0.1                   0         0.02        0.04              0.06    0.08        0.1\n                                            J                                                                                  J\n(a)                                              0                                (b)                                               0\n\n\n\n\nFigure 1: Empirical regime boundaries for the ER graph model with N = 100 and d = 20,\naveraged over three instances; expectation values are shown as thick black lines, standard-\ndeviations are indicated by the gray areas. See the main text for additional explanation.\nThe exact location of the boundary between the spin-glass and ferromagnetic phase in the\nright-hand plot (indicated by the dashed line) was not calculated. The red dash-dotted line\nshows the stability boundary for MF.\n\n\n           The spin-glass phase where the probability mass is distributed over exponentially\n                    (in N ) many different configurations. This phase occurs for frustrated weights,\n                    i.e. for large J .\n\nConsider now the right-hand plot in Fig. 1. Here we have plotted the different regimes con-\ncerning the stability of the paramagnetic solution of the Bethe approximation.2 We find that\nthe m = 0 solution is indeed stable for J and J0 small and becomes unstable at some point\nwhen J0 increases. This signals the paramagnetic-ferromagnetic phase transition. The lo-\ncation is in good agreement with the known phase boundary found for the N   limit\nby advanced statistical physics methods as we show in more detail in [11]. For comparison\nwe have also plotted the stability boundary for MF (the red dash-dotted line). Clearly, the\nmean-field approximation breaks down much earlier than the Bethe approximation and is\nunable to capture the phase transitions occurring for large connectivity strengths.\n\nThe boundary between the spin-glass phase and the paramagnetic phase is more subtle.\nWhat happens is that the Bethe stability matrix becomes marginally stable at some point\nwhen we increase J , i.e. the minimum eigenvalue of ABe approaches zero (in the limit\nN  ). This means that the Bethe free energy becomes very flat at that point. If we go\non increasing J , the m = 0 solution becomes stable again (in other words, the minimum\neigenvalue of the stability matrix ABe becomes positive again). We interpret the marginal\ninstability as signalling the onset of the spin-glass phase. Indeed it coincides with the\nknown phase boundary for the Viana-Bray model [11, 12]. We observe a similar marginal\ninstability for other graph topologies.\n\nNow consider the left-hand plot, Fig. 1(a). It shows the convergence behavior of the BP al-\ngorithm, which was determined by running BP with a fixed number of maximum iterations\nand slight damping. The messages were initialized randomly. We find different regimes\nthat are separated by the boundaries shown in the plot. For small J and J0, BP converges\nto m = 0. For J0 large enough, BP converges to one of the two ferromagnetic solutions\n\n   2Although in Fig. 1 we show only one particular graph topology, the general appearance of these\nplots does not differ much for other graph topologies, especially for large N . The scale of the plots\nmostly depends on the network size N and the average degree d as we will show in the next section.\n\n\f\n                                                 Mean Field                                                       Bethe\n\n                             2                                                             2\n\n                           1.5                                                        1.5\n     1/2                     1\n             d                                                                             1\n                  J c\n                           0.5                                                        0.5\n\n                             0                                                             0\n                              10                100          1000          10000            10             100              1000     10000\n\n                                                       N                                                           N\n\n\n\nFigure 2: Critical values for Bethe and MF for different graph topologies ( : ER,                                                   : BA)\nin the dense limit with d = 0.1N as a function of network size. Note that the y-axis is\n                                    \nrescaled by                              d.\n\n\n(which one is determined by the random initial conditions). For large J , BP does not con-\nverge within 1000 iterations, indicating a complex probability distribution. The boundaries\ncoincide within statistical precision with those in the right-hand plot which were obtained\nby the stability analysis.\n\nThe computation time necessary for producing a plot such as Fig. 1(a), showing the conver-\ngence behavior of BP, quickly increases with increasing N . The computation time needed\nfor the stability analysis (Fig. 1(b)), which amounts to calculating the minimal eigenvalue\nof the N  N stability matrix, is much less, allowing us to investigate the behavior of BP\nfor large networks.\n\n\n4                        Graph topology\n\nIn this section we will concentrate on the frustrated case, more precisely on the case J0 =\n0 (i.e. the y-axis in the regime diagrams) and study the location of the Bethe marginal\ninstability and of the MF instability for various graph topologies as a function of network\nsize N and average degree d. We will denote by J Be\n                                                                                           c      the critical value of J at which the\nBethe paramagnetic solution becomes marginally unstable and we will refer to this as the\nBethe critical value. The critical value of J where the MF solution becomes unstable will\nbe denoted as J MF\n                                          c     and referred to as the MF critical value.\n\nIn studying the influence of graph topology for large networks, we have to distinguish two\ncases, which we call the dense and sparse limits. In the dense limit, we let N   and\nscale the average degree as d = cN for some fixed constant c. In this limit, we find that the\ninfluence of the graph topology is almost negligible. For all graph topologies that we have\nconsidered, we find the following asymptotic behavior for the critical values:\n\n                                                                     1                              1\n                                                            J Be   ,              J MF           \n                                                             c                       c\n                                                                     d                            2 d\n\nThe constant of proportionality is approximately 1. These results are illustrated in Fig. 2\nfor two different graph topologies that will be discussed in more detail below.\n\nIn the sparse limit, we let N   but keep d fixed. In that case the resulting critical values\nshow significant dependence on the graph topology as we will see.\n\n\n4.1                      Uniform random graphs (ER)\n\nThe first and most elementary random graph model we will consider was introduced and\nstudied by Erdos and Renyi [7]. The ensemble, which we denote as ER(N, p), consists of\n\n\f\n          0.5\n                                                                                            Bethe J\n          0.4                                                                                             c\n                                                                                                   1/2\n                                                                                            1/d\n          0.3\n  J c                                                                                       MF Jc\n          0.2                                                                               1/(21/2)\n          0.1\n\n            0\n             10                100                      1000                 10000\n\n                                           N\n\n\n\nFigure 3: Critical values for Bethe and MF for Erdos-Renyi uniform random graphs with\naverage degree d = 10.\n\n\n\nthe graphs with N nodes; links are added between each pair of nodes independently with\nprobability p. The resulting graphs have a degree distribution that is approximately Poisson\nfor large N and the expected average degree is E d = p(N - 1). As was mentioned before,\nthe resulting graphical model is known in the statistical physics literature as the Viana-Bray\nmodel (with zero \"external field\").\n\nFig. 3 shows the results for the sparse limit, where p is chosen such that the expected aver-\nage degree is fixed to d = 10. The Bethe critical value J Be\n                                                                c    appears to be independent of\n                                           \nnetwork size and is slightly larger than 1/ d. The MF critical value J MF\n                                                                             c        does depend on\n                                                                     \nnetwork size (it looks to be proportional to 1/  instead of 1/ d); in fact it can be proven\nthat it converges very slowly to 0 as N   [11], implying that the MF approximation\nbreaks down for very large ER networks in the sparse limit. Although this is an interesting\nresult, one could say that for all practical purposes the MF critical value J MF\n                                                                                       c     is nearly\nindependent of network size N for uniform random graphs.\n\n\n4.2      Scale-free graphs (BA)\n\nA phenomenon often observed in real-world networks is that the degree distribution be-\nhaves like a power-law, i.e. the number of nodes with degree  is proportional to - for\nsome  > 0. These graphs are also known as \"scale-free\" graphs. The first random graph\nmodel exhibiting this behavior is from Barabasi and Albert [8].\n\nWe will consider a slightly different model, which we will denote by BA(N, m). It is\ndefined as a stochastic process, yielding graphs with more and more nodes as time goes\non. At t = 0 one starts with the graph consisting of m nodes and no links. At each time\nstep, one node is added; it is connected with m different already existing nodes, attaching\npreferably to nodes with higher degree (\"rich get richer\"). More specifically, we take the\nprobability to connect to a node of degree  to be proportional to  + 1. The degree dis-\ntribution turns out to have a power-law dependence for N   with exponent  = 3. In\nFig. 4 we illustrate some BA graphs. The difference between the maximum degree  and\nthe average degree d is rather large: whereas the average degree d converges to 2m, the\n                                           \nmaximum degree  is known to scale as           N .\n\nFig. 5 shows the results of the stability analysis for BA graphs with average degree d =\n                                           \n10. Note that the y-axis is rescaled by          to show that the MF critical value J MF\n                                                                                                   c           is\n                    \nproportional to 1/ . The Bethe critical values are seen to have a scaling behavior that\n                                        \nlies somewhere between 1/ d and 1/ . Compared to the situation for uniform ER\ngraphs, BP now even more significantly outperforms MF. The relatively low sensitivity to\nthe maximum degree  that BP exhibits here can be understood intuitively since BA graphs\nresemble forests of sparsely interconnected stars of high degree, on which BP is exact.\n\n\f\n4.3    C. Elegans\n\nWe have also applied our stability analysis on the neural network of the worm C. Elegans,\nthat is publicly available on http://elegans.swmed.edu/. This graph has N =\n202 and d = 19.4. We have calculated the ferromagnetic (J = 0) transition and spin-glass\n(J0 = 0) transition. We also calculated the critical value of J where BP stops converging,\nand the value of J where BP does not find the paramagnetic solution anymore. The results\nare shown in Table 1. Note the very good agreement for the Bethe critical value and the\ncritical J where BP stops finding the m = 0 solution. These results show the accuracy of\nour method of estimating BP validity on real-world networks.\n\n\n\n            Table 1: Critical values and BP boundaries for C. Elegans network.\n\n                                            Spin-glass           Ferromagnetic\n              MF critical value             0.0927  0.0023      0.0387\n              Bethe critical value          0.197  0.016        0.0406\n              BP m = 0 boundary             0.194  0.014        0.0400\n              BP convergence boundary       0.209  0.027        > 1\n\n\n\n\n5      Conclusions\n\nWe have introduced a computationally efficient method to estimate the validity of BP as a\nfunction of graph topology, the connectivity strength, frustration and network size. Using\nthis approach, we have found that:\n\n         for any graph, the Bethe approximation is valid for a larger set of connectivity\n          strengths Wij than the mean-field approximation;\n\n         for uniform random graphs, the quality of both the MF approximation and the\n          Bethe approximation is determined by the average degree of the network (Jc \n            \n          1/ d for the spin-glass transition) and is nearly independent of network size;\n\n         for scale-free networks the validity of the MF approximation scales very poorly\n          with network size due to the increase of the maximal degree (\"rich get richer\"). In\n          contrast, the validity of the BP approximation scales very well with network size.\n          This is in agreement with our intuition that these networks resemble a forest of\n          high degree stars (\"hubs\") that are sparsely interconnected and the fact that BP is\n          exact on stars.\n\n         In the limit in which the graph size N   and the average degree d scales\n          proportional to N , the influence of the graph-topological details on the location of\n                                             \n          the spin-glass transition (at J  1/ d) diminishes and becomes largely irrelevant.\n\n\n\n\n\n                        m = 1                 m = 2              m = 3\n\n\n                       Figure 4: Barabasi-Albert graphs for N = 20.\n\n\f\n             6\n\n             5                                                                                   Bethe\n\n             4                                                                                     1/2\n 1/2                                                                                             1/d\n             3\n         J c  2                                                                                 MF\n\n             1\n\n             0\n                10                    100                      1000                    10000\n\n                                                    N\n\n\n\nFigure 5: Critical values for Bethe and MF for BA scale-free random graphs with average\n                                                              \ndegree d = 10. Note that the y-axis is rescaled by                 .\n\n\n\nAcknowledgments\n\nThe research reported here is part of the Interactive Collaborative Information Systems\n(ICIS) project, supported by the Dutch Ministry of Economic Affairs, grant BSIK03024.\n\n\nReferences\n\n [1] J. Pearl. Probabilistic Reasoning in Intelligent systems: Networks of Plausible Inference. Mor-\n            gan Kaufmann, San Francisco, CA, 1988.\n\n [2] J. Yedidia, W. Freeman, and Y. Weiss. Generalized belief propagation. In Advances in Neural\n            Information Processing Systems, volume 13, pages 689695, 2001.\n\n [3] K. Murphy, Y. Weiss, and M. Jordan. Loopy belief propagation for approximate inference: an\n            empirical study. In Proc. of the Conf. on Uncertainty in AI, pages 467475, 1999.\n\n [4] B. Frey and D. MacKay. A revolution: Belief propagation in graphs with cycles. In Advances\n            in Neural Information Processing Systems, volume 10, pages 479485, 1997.\n\n [5] Y. Weiss. Correctness of local probability propagation in graphical models with loops. Neur.\n            Comp., 12:141, 2000.\n\n [6] R. Albert and A.-L. Barabasi. Statistical mechanics of complex networks. Rev. Mod. Phys.,\n            74:4797, 2002.\n\n [7] P. Erdos and A. Renyi. On random graphs i. Publ. Math. Debrecen, 6:290291, 1959.\n\n [8] A.-L. Barabasi and R. Albert. Emergence of scaling in random networks. Science, 286:509\n            512, 1999.\n\n [9] T. Heskes. Stable fixed points of loopy belief propagation are local minima of the bethe free\n            energy. In Advances in Neural Information Processing Systems, volume 15, pages 343350,\n            2003.\n\n[10] M. Welling and Y.W. Teh. Belief optimization for binary networks: a stable alternative to loopy\n            belief propagation. In Proc. of the Conf. on Uncertainty in AI, volume 17, 2001.\n\n[11] J.M. Mooij and H.J. Kappen. Spin-glass phase transitions on real-world graphs. preprint, cond-\n            mat:0408378, 2004.\n\n[12] L. Viana and A. Bray. Phase diagrams for dilute spin glasses. J. Phys. C: Solid State Phys.,\n            18:30373051, 1985.\n\n\f\n", "award": [], "sourceid": 2741, "authors": [{"given_name": "Joris", "family_name": "Mooij", "institution": null}, {"given_name": "Hilbert", "family_name": "Kappen", "institution": null}]}