{"title": "Selective inference for group-sparse linear models", "book": "Advances in Neural Information Processing Systems", "page_first": 2469, "page_last": 2477, "abstract": "We develop tools for selective inference in the setting of group sparsity, including the construction of confidence intervals and p-values for testing selected groups of variables. Our main technical result gives the precise distribution of the magnitude of the projection of the data onto a given subspace, and enables us to develop inference procedures for a broad class of group-sparse selection methods, including the group lasso, iterative hard thresholding, and forward stepwise regression. We give numerical results to illustrate these tools on simulated data and on health record data.", "full_text": "Selective inference for group-sparse linear models\n\nFan Yang\n\nDepartment of Statistics\nUniversity of Chicago\nfyang1@uchicago.edu\n\nRina Foygel Barber\nDepartment of Statistics\nUniversity of Chicago\nrina@uchicago.edu\n\nPrateek Jain\n\nMicrosoft Research India\nprajain@microsoft.com\n\nJohn Lafferty\n\nDepts. of Statistics and Computer Science\n\nUniversity of Chicago\n\nlafferty@galton.uchicago.edu\n\nAbstract\n\nWe develop tools for selective inference in the setting of group sparsity, including\nthe construction of con\ufb01dence intervals and p-values for testing selected groups of\nvariables. Our main technical result gives the precise distribution of the magnitude\nof the projection of the data onto a given subspace, and enables us to develop\ninference procedures for a broad class of group-sparse selection methods, including\nthe group lasso, iterative hard thresholding, and forward stepwise regression. We\ngive numerical results to illustrate these tools on simulated data and on health\nrecord data.\n\n1\n\nIntroduction\n\nSigni\ufb01cant progress has been recently made on developing inference tools to complement the feature\nselection methods that have been intensively studied in the past decade [6, 5, 9]. The goal of selective\ninference is to make accurate uncertainty assessments for the parameters estimated using a feature\nselection algorithm, such as the lasso [12]. The fundamental challenge is that after the data have\nbeen used to select a set of coef\ufb01cients to be studied, this selection event must then be accounted\nfor when performing inference, using the same data. A speci\ufb01c goal of selective inference is to\nprovide p-values and con\ufb01dence intervals for the \ufb01tted coef\ufb01cients. As the sparsity pattern is chosen\nusing nonlinear estimators, the distribution of the estimated coef\ufb01cients is typically non-Gaussian\nand multimodal, even under a standard Gaussian noise model, making classical techniques unusable\nfor accurate inference. It is of particular interest to develop \ufb01nite-sample, non-asymptotic results.\nIn this paper, we present new results for selective inference in the setting of group sparsity [15, 3, 10].\nWe consider the linear model Y = X\u03b2 + N (0, \u03c32In) where X \u2208 Rn\u00d7p is a \ufb01xed design matrix. In\nmany applications, the p columns or features of X are naturally grouped into blocks C1, . . . ,CG \u2286\n{1, . . . , p}.\nIn the high dimensional setting, the working assumption is that only a few of the\ncorresponding blocks of the coef\ufb01cients \u03b2 contain nonzero elements; that is, \u03b2Cg = 0 for most groups\ng. This group-sparse model can be viewed as an extension of the standard sparse regression model.\nAlgorithms for \ufb01tting this model, such as the group lasso [15], extend well-studied methods for sparse\nlinear regression to this grouped setting.\nIn the group-sparse setting, recent results of Loftus and Taylor [9] give a selective inference method\nfor computing p-values for each group chosen by a model selection method such as forward stepwise\nregression; selection via cross-validation was studied in [9]. More generally, the inference technique\nof [7] applies to any model selection method whose outcome can be described in terms of quadratic\n\n30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain.\n\n\fconditions on Y . However, their technique cannot be used to construct con\ufb01dence intervals for the\nselected coef\ufb01cients or for the size of the effects of the selected groups.\nOur main contribution in this work is to provide a tool for constructing con\ufb01dence intervals as well\nas p-values for testing selected groups. In contrast to the (non-grouped) sparse regression setting,\nthe con\ufb01dence interval construction does not follow immediately from the p-value calculation, and\nrequires a careful analysis of non-centered multivariate normal distributions. Our key technical result\nprecisely characterizes the density of (cid:107)PLY (cid:107)2 (the magnitude of the projection of Y onto a given\nsubspace L), conditioned on a particular selection event. This \u201ctruncated projection lemma\u201d is the\ngroup-wise analogue of the \u201cpolyhedral lemma\u201d of Lee et al. [5] for the lasso. This technical result\nenables us to develop inference tools for a broad class of model selection methods, including the\ngroup lasso [15], iterative hard thresholding [1, 4], and forward stepwise group selection [14].\nIn the following section we frame the problem of group-sparse inference more precisely, and present\nprevious results in this direction. We then state our main technical results; the proofs of the results are\ngiven in the supplementary material. In Section 3 we show how these results can be used to develop\ninferential tools for three different model selection algorithms for group sparsity. In Section 4 we\ngive numerical results to illustrate these tools on simulated data, as well as on the California county\nhealth data used in previous work [9]. We conclude with a brief discussion of our work.\n\n2 Main results: selective inference over subspaces\nTo establish some notation, we will write PL for the projection to any linear subspace L \u2286 Rn, and\nP\u22a5\nL for the projection to its orthogonal complement. For y \u2208 Rn, dirL(y) =\n\u2208 L \u2229 Sn\u22121 is\nPLy\n(cid:107)PLy(cid:107)2\nthe unit vector in the direction of PLy. This direction is not de\ufb01ned if PLy = 0.\nWe focus on the linear model Y = X\u03b2 + N (0, \u03c32In), where X \u2208 Rn\u00d7p is \ufb01xed and \u03c32 > 0 is\nassumed to be known. More generally, our model is Y \u223c N (\u00b5, \u03c32In) with \u00b5 \u2208 Rn unknown and \u03c32\nknown. For a given block of variables Cg \u2286 [p], we write Xg to denote the n \u00d7 |Cg| submatrix of X\nconsisting of all features of this block. For a set S \u2286 [G] of blocks, XS consists of all features that\nlie in any of the blocks in S.\nWhen we refer to \u201cselective inference,\u201d we are generally interested in the distribution of subsets\nof parameters that have been chosen by some model selection procedure. After choosing a set of\ngroups S \u2286 [G], we would like to test whether the true mean \u00b5 is correlated with a group Xg for\neach g \u2208 S after controlling for the remaining selected groups, i.e. after regressing out all the other\ngroups, indexed by S\\g. Thus, the following question is central to selective inference:\nQuestiong,S : What is the magnitude of the projection of \u00b5 onto the span of P\u22a5\n\nXg?\n\n(1)\n\nXS\\g\n\nXS\\g\n\nIn particular, we are interested in a hypothesis test to determine if \u00b5 is orthogonal to this span, that\nis, whether block g should be removed from the model with group-sparse support determined by S;\nthis is the question studied by Loftus and Taylor [9] for which they compute p-values. Alternatively,\nwe may be interested in a con\ufb01dence interval on (cid:107)PL\u00b5(cid:107)2, where L = span(P\u22a5\nXg). Since S\nand g are themselves determined by the data Y , any inference on these questions must be performed\n\u201cpost-selection,\u201d by conditioning on the event that S is the selected set of groups.\n2.1 Background: The polyhedral lemma\nIn the more standard sparse regression setting without grouped variables, after selecting a set S \u2286 [p]\nof features corresponding to columns of X, we might be interested in testing whether the column Xj\nshould be included in the model obtained by regressing Y onto XS\\j. We may want to test the null\nhypothesis that X(cid:62)\nIn the setting where S is the output of the lasso, Lee et al. [5] and Tibshirani et al. [13] characterize\nthe selection event as a polyhedron in Rn: for any set S \u2286 [p] and any signs s \u2208 {\u00b11}S, the event\nthat the lasso (with a \ufb01xed regularization parameter \u03bb) selects the given support with the given signs\nvector, which are functions of X, S, s, \u03bb. The inequalities are interpreted elementwise, yielding a\nconvex polyhedron A. To test the regression question described above, one then tests \u03b7(cid:62)\u00b5 for a \ufb01xed\nunit vector \u03b7 \u221d P\u22a5\nXj. The \u201cpolyhedral lemma\u201d, found in [5, Theorem 5.2] and [13, Lemma\n2], proves that the distribution of \u03b7(cid:62)Y , after conditioning on {Y \u2208 A} and on P\u22a5\n\u03b7 Y , is given by a\n\nis equivalent to the event Y \u2208 A = (cid:8)y : Ay < b(cid:9), where A is a \ufb01xed matrix and b is a \ufb01xed\n\n\u00b5 is zero, or to construct a con\ufb01dence interval for this inner product.\n\nj P\u22a5\n\nXS\\j\n\nXS\\j\n\n2\n\n\ftruncated normal distribution, with density\n\nf (r) \u221d exp(cid:8)\u2212(r \u2212 \u03b7(cid:62)\u00b5)2/2\u03c32(cid:9) \u00b7 1{a1(Y ) \u2264 r \u2264 a2(Y )} .\n\n\u03b7 Y \u2208 A.\n\nL Y \u2208 A(cid:9).\n\n(cid:107)PLY (cid:107)2 \u223c (\u03c3 \u00b7 \u03c7k truncated to RY ) where RY =(cid:8)r : r \u00b7 dirL(Y ) + P\u22a5\n\n(2)\nThe interval endpoints a1(Y ), a2(Y ) depend on Y only through P\u22a5\n\u03b7 Y and are de\ufb01ned to include\nexactly those values of r that are feasible given the event Y \u2208 A. That is, the interval contains all\nvalues r such that r \u00b7 \u03b7 + P\u22a5\nExamining (2), we see that under the null hypothesis \u03b7(cid:62)\u00b5 = 0, this is a truncated zero-mean normal\ndensity, which can be used to construct a p-value testing \u03b7(cid:62)\u00b5 = 0. To construct a con\ufb01dence interval\nfor \u03b7(cid:62)\u00b5, we can instead use (2) with nonzero \u03b7(cid:62)\u00b5, which is a truncated noncentral normal density.\n2.2 The group-sparse case\nIn the group-sparse regression setting, Loftus and Taylor [9] extend the work of Lee et al. [5] to\nquestions where we would like to test PL\u00b5, the projection of the mean \u00b5 to some potentially multi-\ndimensional subspace, rather than simply testing \u03b7(cid:62)\u00b5, which can be interpreted as a projection to\na one-dimensional subspace, L = span(\u03b7). For a \ufb01xed set A \u2286 Rn and a \ufb01xed subspace L of\ndimension k, Loftus and Taylor [9, Theorem 3.1] prove that, after conditioning on {Y \u2208 A}, on\ndirL(Y ), and on P\u22a5\nL Y , under the null hypothesis PL\u00b5 = 0, the distribution of (cid:107)PLY (cid:107)2 is given by\na truncated \u03c7k distribution,\n(3)\nIn particular, this means that, if we would like to test the null hypothesis PL\u00b5 = 0, we can compute\na p-value using the truncated \u03c7k distribution as our null distribution. To better understand this null\nhypothesis, suppose that we run a group-sparse model selection algorithm that chooses a set of blocks\nS \u2286 [G]. We might then want to test whether some particular block g \u2208 S should be retained in this\nmodel or removed. In that case, we would set L = span(P\u22a5\nExamining the parallels between this result and the work of Lee et al. [5], where (2) gives either\na truncated zero-mean normal or truncated noncentral normal distribution depending on whether\nthe null hypothesis \u03b7(cid:62)\u00b5 = 0 is true or false, we might expect that the result (3) of Loftus and\nTaylor [9] can extend in a straightforward way to the case where PL\u00b5 (cid:54)= 0. More speci\ufb01cally, we\nmight expect that (3) might then be replaced by a truncated noncentral \u03c7k distribution, with its\nnoncentrality parameter determined by (cid:107)PL\u00b5(cid:107)2. However, this turns out not to be the case. To\nunderstand why, observe that (cid:107)PLY (cid:107)2 and dirL(Y ) are the length and the direction of the vector\nPLY ; in the inference procedure of Loftus and Taylor [9], they need to condition on the direction\ndirL(Y ) in order to compute the truncation interval RY , and then they perform inference on (cid:107)PLY (cid:107)2,\nthe length. These two quantities are independent for a centered multivariate normal, and therefore if\nPL\u00b5 = 0 then (cid:107)PLY (cid:107)2 follows a \u03c7k distribution even if we have conditioned on dirL(Y ). However,\nin the general case where PL\u00b5 (cid:54)= 0, we do not have independence between the length and the\ndirection of PLY , and so while (cid:107)PLY (cid:107)2 is marginally distributed as a noncentral \u03c7k, this is no\nlonger true after conditioning on dirL(Y ).\nIn this work, we consider the problem of computing the distribution of (cid:107)PLY (cid:107)2 after conditioning\non dirL(Y ), which is the setting that we require for inference. This leads to the main contribution of\nthis work, where we are able to perform inference on PL\u00b5 beyond simply testing the null hypothesis\nthat PL\u00b5 = 0.\n2.3 Key lemma: Truncated projections of Gaussians\nBefore presenting our key lemma, we introduce some further notation. Let A \u2286 Rn be any \ufb01xed\nopen set and let L \u2286 Rn be a \ufb01xed subspace of dimension k. For any y \u2208 A, consider the set\n\nXg) and test whether PL\u00b5 = 0.\n\nXS\\g\n\nRy = {r > 0 : r \u00b7 dirL(y) + P\u22a5\n\nL y \u2208 A} \u2286 R+.\n\nNote that Ry is an open subset of R+, and its construction does not depend on (cid:107)PLy(cid:107)2, but we see\nthat (cid:107)PLy(cid:107)2 \u2208 Ry by de\ufb01nition.\nLemma 1 (Truncated projection). Let A \u2286 Rn be a \ufb01xed open set and let L \u2286 Rn be a \ufb01xed\nsubspace of dimension k. Suppose that Y \u223c N (\u00b5, \u03c32In). Then, conditioning on the values of\nL Y and on the event Y \u2208 A, the conditional distribution of (cid:107)PLY (cid:107)2 has density1\ndirL(Y ) and P\u22a5\nf (r) \u221d rk\u22121 exp\n\n(cid:0)r2 \u2212 2r \u00b7 (cid:104)dirL(Y ), \u00b5(cid:105)(cid:1)(cid:27)\n\n\u00b7 1{r \u2208 RY } .\n\n(cid:26)\n\nWe pause to point out two special cases that are treated in the existing literature.\n\n\u2212 1\n2\u03c32\n\n1Here and throughout the paper, we ignore the possibility that Y \u22a5 L since this has probability zero.\n\n3\n\n\fSpecial case 1: k = 1 and A is a convex polytope. Suppose A is the convex polytope {y : Ay < b}\nfor \ufb01xed A \u2208 Rm\u00d7n and b \u2208 Rm. In this case, this almost exactly yields the \u201cpolyhedral lemma\u201d of\nLee et al. [5, Theorem 5.2]. Speci\ufb01cally, in their work they perform inference on \u03b7(cid:62)\u00b5 for a \ufb01xed\nvector \u03b7; this corresponds to taking L = span(\u03b7) in our notation. Then since k = 1, Lemma 1 yields\na truncated Gaussian distribution, coinciding with Lee et al. [5]\u2019s result (2). The only difference\nrelative to [5] is that our lemma implicitly conditions on sign(\u03b7(cid:62)Y ), which is not required in [5].\n\non {Y \u2208 A}, we have PLY = PL(cid:0)\u00b5 + N (0, \u03c32I)(cid:1) = PL(cid:0)N (0, \u03c32I)(cid:1), and so (cid:107)PLY (cid:107)2 \u223c \u03c3 \u00b7 \u03c7k.\n\nSpecial case 2: the mean \u00b5 is orthogonal to the subspace L.\nWithout conditioning on {Y \u2208 A} (or equivalently, taking A = Rn), the resulting density is then\n\nIn this case, without conditioning\n\nf (r) \u221d rk\u22121e\u2212r2/2\u03c32 \u00b7 1{r > 0}\n\nwhich is the density of the \u03c7k distribution (rescaled by \u03c3), as expected. If we also condition on\n{Y \u2208 A} then this is a truncated \u03c7k distribution, as proved in Loftus and Taylor [9, Theorem 3.1].\n2.4 Selective inference on truncated projections\nWe now show how the key result in Lemma 1 can be used for group-sparse inference. In particular, we\nshow how to compute a p-value for the null hypothesis H0 : \u00b5 \u22a5 L, or equivalently, H0 : (cid:107)PL\u00b5(cid:107)2 =\n0. In addition, we show how to compute a one-sided con\ufb01dence interval for (cid:107)PL\u00b5(cid:107)2, speci\ufb01cally,\nhow to give a lower bound on the size of this projection.\nTheorem 1 (Selective inference for projections). Under the setting and notation of Lemma 1, de\ufb01ne\n\n(cid:82)\n\nP =\n\nr\u2208RY ,r>(cid:107)PLY (cid:107)2\n\nrk\u22121e\u2212r2/2\u03c32\nrk\u22121e\u2212r2/2\u03c32 dr\n\n(cid:82)\n\nr\u2208RY\n\ndr\n\n.\n\n(4)\n\nIf \u00b5 \u22a5 L (or, more generally, if (cid:104)dirL(Y ), \u00b5(cid:105) = 0), then P \u223c Uniform[0, 1]. Furthermore, for any\ndesired error level \u03b1 \u2208 (0, 1), there is a unique value L\u03b1 \u2208 R satisfying\n\n(cid:82)\n\n(cid:82)\n\nr\u2208RY ,r>(cid:107)PLY (cid:107)2\n\nrk\u22121e\u2212(r2\u22122rL\u03b1)/2\u03c32\nrk\u22121e\u2212(r2\u22122rL\u03b1)/2\u03c32 dr\n\nr\u2208RY\n\ndr\n\n= \u03b1,\n\n(5)\n\nand we have\n\nP{(cid:107)PL\u00b5(cid:107)2 \u2265 L\u03b1} \u2265 P{(cid:104)dirL(Y ), \u00b5(cid:105) \u2265 L\u03b1} = 1 \u2212 \u03b1.\n\nFinally, the p-value and the con\ufb01dence interval agree in the sense that P < \u03b1 if and only if L\u03b1 > 0.\nFrom the form of Lemma 1, we see that we are actually performing inference on (cid:104)dirL(Y ), \u00b5(cid:105).\nSince (cid:107)PL\u00b5(cid:107)2 \u2265 (cid:104)dirL(Y ), \u00b5(cid:105), this means that any lower bound on (cid:104)dirL(Y ), \u00b5(cid:105) also gives a lower\nbound on (cid:107)PL\u00b5(cid:107)2. For the p-value, the statement (cid:104)dirL(Y ), \u00b5(cid:105) = 0 is implied by the stronger null\nhypothesis \u00b5 \u22a5 L. We can also use Lemma 1 to give a two-sided con\ufb01dence interval for (cid:104)dirL(Y ), \u00b5(cid:105);\nspeci\ufb01cally, (cid:104)dirL(Y ), \u00b5(cid:105) lies in the interval [L\u03b1/2, L1\u2212\u03b1/2] with probability 1 \u2212 \u03b1. However, in\ngeneral this cannot be extended to a two-sided interval for (cid:107)PL\u00b5(cid:107)2.\nTo compare to the main results of Loftus and Taylor [9], their work produces the p-value (4) testing\nthe null hypothesis \u00b5 \u22a5 L, but does not extend to testing PL\u00b5 beyond the null hypothesis, which the\nsecond part (5) of our main theorem is able to do.2\n\n3 Applications to group sparse regression methods\nIn this section we develop inference tools for three methods for group-sparse model selection: forward\nstepwise regression (also considered by Loftus and Taylor [9] with results on hypothesis testing),\niterative hard thresholding (IHT), and the group lasso.\n\n2Their work furthermore considers the special case where the conditioning event, Y \u2208 A, is determined by a\n\u201cquadratic selection rule,\u201d that is, A is de\ufb01ned by a set of quadratic constraints on y \u2208 Rn. However, extending\nto the general case is merely a question of computation (as we explore below for performing inference for the\ngroup lasso) and this extension should not be viewed as a primary contribution of this work.\n\n4\n\n\f3.1 General recipe\nWith a \ufb01xed design matrix, the outcome of any group-sparse selection method is a function of Y .\nFor example, a forward stepwise procedure determines a particular sequence of groups of variables.\nWe call such an outcome a selection event, and assume that the set of all selection events forms a\ncountable partition of Rn into disjoint open sets: Rn = \u222aeAe.3 Each data vector y \u2208 Rn determines\na selection event, denoted e(y), and thus y \u2208 Ae(y).\nLet S(y) \u2286 [G] be the set of groups selected for testing. This is assumed to be a function of e(y),\ni.e. S(y) = Se for all y \u2208 Ae. For any g \u2208 Se, let Le,g = span(P\u22a5\nXg), the subspace of Rn\nindicating correlation with group Xg beyond what can be explained by the other selected groups.\nWrite RY = {r > 0 : r \u00b7 U + Y\u22a5 \u2208 Ae(Y )}, where U = dirLe(Y ),g (Y ) and Y\u22a5 = P\u22a5\nY . If\nwe condition on the event {Y \u2208 Ae} for some e, then as soon as we have calculated the region\nRY \u2286 R+, Theorem 1 will allow us to perform inference on the quantity of interest (cid:107)PLe,g \u00b5(cid:107)2\nby evaluating the expressions (4) and (5). In other words, we are testing whether \u00b5 is signi\ufb01cantly\ncorrelated with the group Xg, after controlling for all the other selected groups, S(Y )\\g = Se\\g.\nTo evaluate these expressions accurately, ideally we would like an explicit characterization of the\nregion RY \u2286 R+. To gain a better intuition for this set, de\ufb01ne zY (r) = r \u00b7 U + Y\u22a5 \u2208 Rn for r > 0,\nand note that zY (r) = Y when we plug in r = (cid:107)PLe(Y ),g Y (cid:107)2. Then we see that\n\nLe(Y ),g\n\nXSe\\g\n\nRY =(cid:8)r > 0 : e(zY (r)) = e(Y )(cid:9).\n\n(6)\nIn other words, we need to \ufb01nd the range of values of r such that, if we replace Y with zY (r), then\nthis does not change the output of the model selection algorithm, i.e. e(zY (r)) = e(Y ). For the\nforward stepwise and IHT methods, we \ufb01nd that we can calculate RY explicitly. For the group\nlasso, we cannot calculate RY explicitly, but we can nonetheless compute the integrals required by\nTheorem 1 through numerical approximations. We now present the details for each of these methods.\n3.2 Forward stepwise regression\nForward stepwise regression [2, 14] is a simple and widely used method. We will use the following\nversion:4 for design matrix X and response Y = y,\n\n1. Initialize the residual(cid:98)\u00010 = y and the model S0 = \u2205.\ng(cid:98)\u0001t\u22121(cid:107)2}.\n(b) Update the model, St = {g1, . . . , gt}, and update the residual,(cid:98)\u0001t = P\u22a5\n(a) Let gt = arg maxg\u2208[G]\\St\u22121{(cid:107)X(cid:62)\n\n2. For t = 1, 2, . . . , T ,\n\ny.\n\nXSt\n\n(cid:107)X(cid:62)\n\ngk\n\nP\u22a5\nXSk\u22121\n\nTesting all groups at time T . First we consider the inference procedure where, at time T , we would\nlike to test each selected group gt for t = 1, . . . , T ; inference for this procedure was derived also\nin [8]. Our selection event e(Y ) is the ordered sequence g1, . . . , gT of selected groups. For a response\nvector Y = y, this selection event is equivalent to\n\ny(cid:107)2 > (cid:107)X(cid:62)\n\ng P\u22a5\n\ny(cid:107)2 for all k = 1, . . . , T , for all g (cid:54)\u2208 Sk.\n\n(7)\nNow we would like to perform inference on the group g = gt, while controlling for the other groups\nin S(Y ) = ST . De\ufb01ne U, Y\u22a5, and zY (r) as before. Then, to determine RY = {r > 0 : zY (r) \u2208\nAe(Y )}, we check whether all of the inequalities in (7) are satis\ufb01ed with y = zY (r): for each\nk = 1, . . . , T and each g (cid:54)\u2208 Sk, the corresponding inequality of (7) can be expressed as\nY\u22a5(cid:107)2\ng P\u22a5\n\nY\u22a5(cid:105) + (cid:107)X(cid:62)\ng P\u22a5\n\nP\u22a5\nXSk\u22121\nY\u22a5(cid:105) + (cid:107)X(cid:62)\n\nP\u22a5\nXSk\u22121\n> r2 \u00b7 (cid:107)X(cid:62)\n\nP\u22a5\nXSk\u22121\nU , X(cid:62)\n\n2 + 2r \u00b7 (cid:104)X(cid:62)\n\nU(cid:107)2\ng P\u22a5\n\nSolving this quadratic inequality over r \u2208 R+, we obtain a region Ik,g \u2286 R+ which is either a single\ninterval or a union of two disjoint intervals, whose endpoints we can calculate explicitly with the\nquadratic formula. The set RY is then given by all values r that satisfy the full set of inequalities:\n\nr2 \u00b7 (cid:107)X(cid:62)\n\ngk\n\ngk\n\ngk\n\nP\u22a5\nU , X(cid:62)\nXSk\u22121\n2 + 2r \u00b7 (cid:104)X(cid:62)\ng P\u22a5\n(cid:92)\n\nRY =\n\nY\u22a5(cid:107)2\n2.\n\n(cid:92)\n\nIk,g.\n\nXSk\u22121\n\nU(cid:107)2\n\nXSk\u22121\n\ngk\n\n2\n\nXSk\u22121\n\nk=1,...,T\n\ng\u2208[G]\\Sk\n\nThis is a union of \ufb01nitely many disjoint intervals, whose endpoints are calculated explicitly as above.\n3Since the distribution of Y is continuous on Rn, we ignore sets of measure zero without further comment.\n4In practice, we would add some correction for the scale of the columns of Xg or for the number of features\n\nin group g; this can be accomplished with simple modi\ufb01cations of the forward stepwise procedure.\n\nXSk\u22121\n\nXSk\u22121\n\n5\n\n\fSequential testing. Now suppose we carry out a sequential inference procedure, testing group gt\nat its time of selection, controlling only for the previously selected groups St\u22121. In fact, this is a\nspecial case of the non-sequential procedure above, which shows how to test gT while controlling\nfor ST\\gT = ST\u22121. Applying this method at each stage of the algorithm yields a sequential testing\nprocedure. (The method developed in [9] computes p-values for this problem, testing whether\n\u00b5 \u22a5 P\u22a5\n3.3\nThe iterative hard thresholding algorithm \ufb01nds a k-group-sparse solution to the linear regression\nproblem, iterating gradient descent steps with hard thresholding to update the model choice as needed\n[1, 4]. Given k \u2265 1, number of iterations T , step sizes \u03b7t, design matrix X and response Y = y,\n\nXgt at each time t.) See the supplementary material for detailed pseudo-code.\n\nXSt\u22121\nIterative hard thresholding (IHT)\n\n1. Initialize the coef\ufb01cient vector, \u03b20 = 0 \u2208 Rp (or any other desired initial point).\n2. For t = 1, 2, . . . , T ,\n\n(a) Take a gradient step,(cid:101)\u03b2t = \u03b2t\u22121 \u2212 \u03b7tX(cid:62)(X\u03b2t\u22121 \u2212 y).\n(b) Compute (cid:107)((cid:101)\u03b2t)Cg(cid:107)2 for each g \u2208 [G] and let St \u2286 [G] index the k largest norms.\n(c) Update the \ufb01tted coef\ufb01cients \u03b2t via (\u03b2t)j = ((cid:101)\u03b2t)j \u00b7 1{j \u2208 \u222ag\u2208StCg}.\n\nfor all t = 1, . . . , T , and all g \u2208 St, h (cid:54)\u2208 St.\n\n(cid:107)((cid:101)\u03b2t)Cg(cid:107)2 > (cid:107)((cid:101)\u03b2t)Ch(cid:107)2\n\nHere we are typically interested in testing Questiong,ST for each g \u2208 ST . We condition on the\nselection event, e(Y ), given by the sequence of k-group-sparse models S1, . . . ,ST selected at each\nstage of the algorithm, which is characterized by the inequalities\n(8)\nFixing a group g \u2208 ST to test, determining RY = {r > 0 : zY (r) \u2208 Ae(Y )} involves checking\nwhether all of the inequalities in (8) are satis\ufb01ed with y = zY (r). First, with the response Y replaced\n\nare independent of r; in the supplementary material, we derive ct, dt inductively as\nn X(cid:62)X)PSt\u22121ct\u22121 + \u03b7t\nn X(cid:62)X)PSt\u22121dt\u22121 + \u03b7t\n\nn X(cid:62)U,\nd1 = (I \u2212 \u03b71\nNow we compute the region RY . For each t = 1, . . . , T and each g \u2208 St, h (cid:54)\u2208 St, the corresponding\n\nby y = zY (r), we show that we can write(cid:101)\u03b2t = r \u00b7 ct + dt for each t = 1, . . . , T , where ct, dt \u2208 Rp\n(cid:26)c1 = \u03b71\ninequality in (8), after writing(cid:101)\u03b2t = r \u00b7 ct + dt, can be expressed as\n2 > r2\u00b7(cid:107)(ct)Ch(cid:107)2\nr2\u00b7(cid:107)(ct)Cg(cid:107)2\n2+2r\u00b7(cid:104)(ct)Ch, (dt)Ch(cid:105)+(cid:107)(dt)Ch(cid:107)2\n2.\nwe can calculate explicitly. Finally, we obtain RY =(cid:84)\n(cid:84)\nAs for the forward stepwise procedure, solving this quadratic inequality over r \u2208 R+, we obtain a\nregion It,g,h \u2286 R+ that is either a single interval or a union of two disjoint intervals whose endpoints\n\n(cid:26)ct = (Ip \u2212 \u03b7t\n\n2+2r\u00b7(cid:104)(ct)Cg , (dt)Cg(cid:105)+(cid:107)(dt)Cg(cid:107)2\n\nn X(cid:62)X)\u03b20 + \u03b71\n\nn X(cid:62)U,\nn X(cid:62)Y\u22a5\n\ndt = (Ip \u2212 \u03b7t\n\nn X(cid:62)Y\u22a5,\n\nfor t \u2265 2.\n\nIt,g,h.\n\n(cid:84)\n\ng\u2208St\n\nh\u2208[G]\\St\n\nt=1,...,T\n\n2 + \u03bb(cid:80)\n\n(cid:98)\u03b2 = arg min\u03b2\n\n(cid:8) 1\n2(cid:107)y \u2212 X\u03b2(cid:107)2\n\n3.4 The group lasso\nThe group lasso, \ufb01rst introduced by Yuan and Lin [15], is a convex optimization method for linear\nregression where the form of the penalty is designed to encourage group-wise sparsity of the solution.\nIt is an extension of the lasso method [12] for linear regression. The method is given by\n\nwhere \u03bb > 0 is a penalty parameter. The penalty(cid:80)\ng(cid:107)\u03b2Cg(cid:107)2 promotes sparsity at the group level.5\nFor this method, we perform inference on the group support S of the \ufb01tted model(cid:98)\u03b2. We would like\nto test Questiong,S for each g \u2208 S. In this setting, for groups of size \u2265 2, we believe that it is not\npossible to analytically calculate RY , and furthermore, that there is no additional information that we\ncan condition on to make this computation possible, without losing all power to do inference.\nWe thus propose a numerical approximation that circumvents the need for an explicit calculation of\nRY . Examining the calculation of the p-value P and the lower bound L\u03b1 in Theorem 1, we see that\nwe can write P = fY (0) and can \ufb01nd L\u03b1 as the unique solution to fY (L\u03b1) = \u03b1, where\n\ng(cid:107)\u03b2Cg(cid:107)2\n\n(cid:9),\n\nEr\u223c\u03c3\u00b7\u03c7k\n\nfY (t) =\n\n(cid:104)\n\nert/\u03c32 \u00b7 1{r \u2208 RY , r > (cid:107)PLY (cid:107)2}(cid:105)\n\n(cid:2)ert/\u03c32 \u00b7 1{r \u2208 RY }(cid:3)\n\nEr\u223c\u03c3\u00b7\u03c7k\n\n,\n\n5Our method can also be applied to a modi\ufb01cation of group lasso designed for overlapping groups [3] with a\n\nnearly identical procedure but we do not give details here.\n\n6\n\n\fwhere we treat Y as \ufb01xed in this calculation and set k = dim(L) = rank(XS\\g). Both the numerator\nand denominator can be approximated by taking a large number B of samples r \u223c \u03c3 \u00b7 \u03c7k and taking\nthe empirical expectations. Checking r \u2208 RY is equivalent to running the group lasso with the\nresponse replaced by y = zY (r), and checking if the resulting selected model remains unchanged.\nThis may be problematic, however, if RY is in the tails of the \u03c3 \u00b7 \u03c7k distribution. We implement\nan importance sampling approach by repeatedly drawing r \u223c \u03c8 for some density \u03c8; we \ufb01nd that\n\u03c8 = (cid:107)PLY (cid:107)2 + N (0, \u03c32) works well in practice. Given samples r1, . . . , rB \u223c \u03c8 we then estimate\n\nfY (t) \u2248 (cid:98)fY (t) :=\n\n(cid:80)\n\n\u03c8\u03c3\u00b7\u03c7k (rb)\n\nb\n\n\u03c8(rb)\n\n(cid:80)\n\n\u00b7 erbt/\u03c32 \u00b7 1{rb \u2208 RY , rb > (cid:107)PLY (cid:107)2}\n\u03c8\u03c3\u00b7\u03c7k (rb)\n\nwhere \u03c8\u03c3\u00b7\u03c7k is the density of the \u03c3\u00b7 \u03c7k distribution. We then estimate P \u2248 (cid:98)P = (cid:98)fY (0). Finally, since\n(cid:98)fY (t) is continuous and strictly increasing in t, we estimate L\u03b1 by numerically solving (cid:98)fY (t) = \u03b1.\n\n\u00b7 erbt/\u03c32 \u00b7 1{rb \u2208 RY }\n\nb\n\n\u03c8(rb)\n\n4 Experiments\nIn this section we present results from experiments on simulated and real data, performed in R [11].6\n4.1 Simulated data\nWe \ufb01x sample size n = 500 and G = 50 groups each of size 10. For each trial, we generate a design\nmatrix X with i.i.d. N (0, 1/n) entries, set \u03b2 with its \ufb01rst 50 entries (corresponding to \ufb01rst s = 5\ngroups) equal to \u03c4 and all other entries equal to 0, and set Y = X\u03b2 + N (0, In). We present the\nresult for IHT here; the results for the other two methods can be found in the supplementary material.\nWe run IHT to select k = 10 groups over T = 5 iterations, with step sizes \u03b7t = 2 and initial point\n\u03b20 = 0. For a moderate signal strength \u03c4 = 1.5, we plot the p-values for each selected group in\nFigure 1; each group displays p-values only for those trials in which it was selected. The histogram of\np-values for the s true signals and for the G \u2212 s nulls are also shown. We see that the the distribution\nof p-values for the true signals concentrates near zero while the null p-values are roughly uniform.\nNext we look at the con\ufb01dence intervals given by our method, examining their empirical coverage\nacross different signal strengths \u03c4 in Figure 2. We \ufb01x con\ufb01dence level 0.9 (i.e. \u03b1 = 0.1) and check\nempirical coverage with respect to both (cid:107)PL\u00b5(cid:107)2 and (cid:104)dirL(Y ), \u00b5(cid:105), with results shown separately\nfor true signals and for nulls. For true signals, the con\ufb01dence interval for (cid:107)PL\u00b5(cid:107)2 is somewhat\nconservative while the coverage for (cid:104)dirL(Y ), \u00b5(cid:105) is right at the target level, as expected from our\ntheory. As signal strength \u03c4 increases, the gap is reduced for the true signals; this is because\ndirL(Y ) becomes an increasingly more accurate estimate of dirL(\u00b5), and so the gap in the inequality\n(cid:107)PL\u00b5(cid:107)2 \u2265 (cid:104)dirL(Y ), \u00b5(cid:105) is reduced. For the nulls, if the set of selected groups contains the support\nof the true model, which is nearly always true for higher signal levels \u03c4, then the two are equivalent\n(as (cid:107)PL\u00b5(cid:107)2 = (cid:104)dirL(Y ), \u00b5(cid:105) = 0), and coverage is at the target level. At low signal levels \u03c4, however,\na true group is occasionally missed, in which case (cid:107)PL\u00b5(cid:107)2 > (cid:104)dirL(Y ), \u00b5(cid:105) strictly.\n\nFigure 1: Iterative hard thresholding (IHT). For each group, we plot its p-value for each trial in which\nthat group was selected, for 200 trials. Histograms of the p-values for true signals (left, red) and for\nnulls (right, gray) are attached.\n\n4.2 California health data\nWe examine the 2015 California county health data7 which was also studied by Loftus and Taylor\n[9]. We \ufb01t a linear model where the response is the log-years of potential life lost and the covariates\n\n6Code reproducing experiments: http://www.stat.uchicago.edu/~rina/group_inf.html\n7Available at http://www.countyhealthrankings.org\n\n7\n\n\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0d9\uf0ae\uf0b1\uf0ab\uf0b0\uf020\uf0b2\uf0ab\uf0b3\uf0be\uf0bb\uf0ae\uf020\uf0f8\uf0eb\uf020\uf0ad\uf0b7\uf0b9\uf0b2\uf0bf\uf0b4\uf0ad\uf0f4\uf020\uf0ec\uf0eb\uf020\uf0b2\uf0ab\uf0b4\uf0b4\uf0ad\uf0f7\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0ef\uf0eb\uf0ef\uf0f0\uf0ef\uf0eb\uf0ee\uf0f0\uf0ee\uf0eb\uf0ed\uf0f0\uf0ed\uf0eb\uf0ec\uf0f0\uf0ec\uf0eb\uf0eb\uf0f0\uf0f0\uf0f2\uf0f0\uf0f0\uf0f2\uf0ee\uf0f0\uf0f2\uf0ec\uf0f0\uf0f2\uf0ea\uf0f0\uf0f2\uf0e8\uf0ef\uf0f2\uf0f0\uf0f0\uf0f2\uf0f0\uf0f0\uf0f2\uf0ee\uf0f0\uf0f2\uf0ec\uf0f0\uf0f2\uf0ea\uf0f0\uf0f2\uf0e8\uf0ef\uf0f2\uf0f0\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0b4\uf0cd\uf0b7\uf0b9\uf0b2\uf0bf\uf0b4\uf0ad\uf0d2\uf0ab\uf0b4\uf0b4\uf0ad\fFigure 2: Iterative hard thresholding (IHT). Empirical coverage over 2000 trials with signal strength\n\u03c4. \u201cNorm\u201d and \u201cinner product\u201d refer to coverage of (cid:107)PL\u00b5(cid:107)2 and (cid:104)dirL(Y ), \u00b5(cid:105), respectively.\n\nare the 34 predictors in this data set. We \ufb01rst let each predictor be its own group (i.e., group\nsize 1) and run the three algorithms considered in Section 3. Next, we form a grouped model by\nexpanding each predictor Xj into a group using the \ufb01rst three non-constant Legendre polynomials,\nj \u2212 3Xj)). In each case we set parameters so that 8 groups are selected. The\n(Xj, 1\nselected groups and their p-values are given in Table 1; interestingly, even when the same predictor is\nselected by multiple methods, its p-value can differ substantially across the different methods.\n\nj \u2212 1), 1\n\n2 (3X2\n\n2 (5X3\n\nGroup size\n\n1\n\n3\n\n% Obese\n\nChlamydia rate\n\n% Children in poverty\n\n% Single-parent household\n\nInjury death rate\n\n% Smokers\n\nGroup lasso p-value\n\n80th percentile income\n\n% Physically inactive\n\nViolent crime rate\n\nInjury death rate\nViolent crime rate\n% Receiving HbA1c\n\n% Physically inactive\n% Alcohol-impaired\n80th percentile income\n\nForward stepwise p-value / seq. p-value\n80th percentile income\n\n0.000\n0.007\n0.040\n0.055\n0.075\n0.235\n0.701\n0.932\n0.000\n0.000\n0.038\n0.043\n0.339\n0.366\n0.372\n0.629\nTable 1: Selective p-values for the California county health data experiment. The predictors obtained\nwith forward stepwise are tested both simultaneously at the end of the procedure (\ufb01rst p-value shown),\nand also tested sequentially (second p-value shown), and are displayed in the selected order.\n\n0.116 / 0.000\n0.000 / 0.000\n0.016 / 0.000\n0.591 / 0.839 % Single-parent household\n0.481 / 0.464\nPhysically unhealthy days\n0.944 / 0.975\n0.654 / 0.812\nFood environment index\n0.104 / 0.104 Mentally unhealthy days\n0.001 / 0.000\n0.044 / 0.000\n0.793 / 0.617\n0.507 / 0.249 % Single-parent household\nFood environment index\n0.892 / 0.933\n0.119 / 0.496\n% Children in poverty\n0.188 / 0.099\nPhysically unhealthy days\n0.421 / 0.421 Mentally unhealthy days\n\nIterative hard thresholding p-value\n0.000\n80th percentile income\n0.000\n0.004\n0.009\n0.332\n0.716\n0.807\n0.957\n0.000\n0.000\n0.000\n0.005\n0.057\n0.388\n0.713\n0.977\n\n% Single-parent household\n\n% Physically inactive\n\n% Physically inactive\n% Alcohol-impaired\n\nInjury death rate\n\n80th percentile income\n\n80th percentile income\n\nInjury death rate\n\nInjury death rate\nViolent crime rate\n\n% Alcohol-impaired\n\n% Smokers\n\nViolent crime rate\n\nInjury death rate\n\n% Smokers\n\nPreventable hospital stays rate\n\n% Severe housing problems\n\nChlamydia rate\n\nPreventable hospital stays rate\n\n% Obese\n\n% Obese\n\n% Smokers\n\n5 Conclusion\nWe develop selective inference tools for group-sparse linear regression methods, where for a data-\ndependent selected set of groups S, we are able to both test each group g \u2208 S for inclusion in the\nmodel de\ufb01ned by S, and form a con\ufb01dence interval for the effect size of group g in the model. Our\ntheoretical results can be easily applied to a range of commonly used group-sparse regression methods,\nthus providing an ef\ufb01cient tool for \ufb01nite-sample inference that correctly accounts for data-dependent\nmodel selection in the group-sparse setting.\n\nAcknowledgments\nResearch supported in part by ONR grant N00014-15-1-2379, and NSF grants DMS-1513594 and\nDMS-1547396.\n\n8\n\n\uf0ee\uf0ec\uf0ea\uf0e8\uf0ef\uf0f0\uf0f0\uf0f2\uf0e8\uf0f0\uf0f0\uf0f2\uf0e8\uf0eb\uf0f0\uf0f2\uf0e7\uf0f0\uf0f0\uf0f2\uf0e7\uf0eb\uf0ef\uf0f2\uf0f0\uf0f0\uf0d2\uf0ab\uf0b4\uf0b4\uf0ad\uf020\uf0f8\uf0b2\uf0b1\uf0ae\uf0b3\uf0f7\uf0d2\uf0ab\uf0b4\uf0b4\uf0ad\uf020\uf0f8\uf0b7\uf0b2\uf0b2\uf0bb\uf0ae\uf020\uf0b0\uf0ae\uf0b1\uf0bc\uf0ab\uf0bd\uf0ac\uf0f7\uf0cd\uf0b7\uf0b9\uf0b2\uf0bf\uf0b4\uf0ad\uf020\uf0f8\uf0b2\uf0b1\uf0ae\uf0b3\uf0f7\uf0cd\uf0b7\uf0b9\uf0b2\uf0bf\uf0b4\uf0ad\uf020\uf0f8\uf0b7\uf0b2\uf0b2\uf0bb\uf0ae\uf020\uf0b0\uf0ae\uf0b1\uf0bc\uf0ab\uf0bd\uf0ac\uf0f7\fReferences\n[1] Thomas Blumensath and Mike E Davies. Sampling theorems for signals from the union of \ufb01nite-\ndimensional linear subspaces. Information Theory, IEEE Transactions on, 55(4):1872\u20131882,\n2009.\n\n[2] Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning.\n\nSpringer Series in Statistics. Springer New York Inc., New York, NY, USA, 2001.\n\n[3] Laurent Jacob, Guillaume Obozinski, and Jean-Philippe Vert. Group lasso with overlap and\ngraph lasso. In Proceedings of the 26th annual international conference on machine learning,\npages 433\u2013440. ACM, 2009.\n\n[4] Prateek Jain, Nikhil Rao, and Inderjit S. Dhillon. Structured sparse regression via greedy hard-\n\nthresholding. CoRR, abs/1602.06042, 2016. URL http://arxiv.org/abs/1602.06042.\n\n[5] Jason D Lee, Dennis L Sun, Yuekai Sun, and Jonathan E Taylor. Exact post-selection inference\n\nwith the lasso. arXiv preprint arXiv:1311.6238, 2013.\n\n[6] Jason D. Lee and Jonathan E. Taylor. Exact post model selection inference for marginal\nscreening. In Advances in Neural Information Processing Systems 27, pages 136\u2013144, 2014.\n\n[7] Joshua R Loftus. Selective inference after cross-validation. arXiv preprint arXiv:1511.08866,\n\n2015.\n\n[8] Joshua R Loftus and Jonathan E Taylor. A signi\ufb01cance test for forward stepwise model selection.\n\narXiv preprint arXiv:1405.3920, 2014.\n\n[9] Joshua R. Loftus and Jonathan E. Taylor. Selective inference in regression models with groups\n\nof variables. arXiv:1511.01478, 2015.\n\n[10] So\ufb01a Mosci, Silvia Villa, Alessandro Verri, and Lorenzo Rosasco. A primal-dual algorithm\nfor group sparse regularization with overlapping groups. In Advances in Neural Information\nProcessing Systems 23, pages 2604\u20132612, 2010.\n\n[11] R Core Team. R: A Language and Environment for Statistical Computing. R Foundation for\n\nStatistical Computing, Vienna, Austria, 2016. URL https://www.R-project.org/.\n\n[12] Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal\n\nStatistical Society. Series B (Methodological), pages 267\u2013288, 1996.\n\n[13] Ryan J Tibshirani, Jonathan Taylor, Richard Lockhart, and Robert Tibshirani. Exact post-\nselection inference for sequential regression procedures. arXiv preprint arXiv:1401.3889,\n2014.\n\n[14] Joel A. Tropp and Anna C. Gilbert. Signal recovery from random measurements via orthogonal\nmatching pursuit. IEEE Trans. Information Theory, 53(12):4655\u20134666, 2007. doi: 10.1109/TIT.\n2007.909108. URL http://dx.doi.org/10.1109/TIT.2007.909108.\n\n[15] Ming Yuan and Yi Lin. Model selection and estimation in regression with grouped variables.\nJournal of the Royal Statistical Society: Series B (Statistical Methodology), 68(1):49\u201367, 2006.\n\n9\n\n\f", "award": [], "sourceid": 1293, "authors": [{"given_name": "Fan", "family_name": "Yang", "institution": "University of Chicago"}, {"given_name": "Rina", "family_name": "Foygel Barber", "institution": "University of Chicago"}, {"given_name": "Prateek", "family_name": "Jain", "institution": "Microsoft Research"}, {"given_name": "John", "family_name": "Lafferty", "institution": "University of Chicago"}]}