{"title": "Hunting for Discriminatory Proxies in Linear Regression Models", "book": "Advances in Neural Information Processing Systems", "page_first": 4568, "page_last": 4578, "abstract": "A machine learning model may exhibit discrimination when used to make decisions involving people. One potential cause for such outcomes is that the model uses a statistical proxy for a protected demographic attribute. In this paper we formulate a definition of proxy use for the setting of linear regression and present algorithms for detecting proxies. Our definition follows recent work on proxies in classification models, and characterizes a model's constituent behavior that: 1) correlates closely with a protected random variable, and 2) is causally influential in the overall behavior of the model. We show that proxies in linear regression models can be efficiently identified by solving a second-order cone program, and further extend this result to account for situations where the use of a certain input variable is justified as a ``business necessity''. Finally, we present empirical results on two law enforcement datasets that exhibit varying degrees of racial disparity in prediction outcomes, demonstrating that proxies shed useful light on the causes of discriminatory behavior in models.", "full_text": "Hunting for Discriminatory Proxies\n\nin Linear Regression Models\n\nSamuel Yeom\n\nCarnegie Mellon University\n\nAnupam Datta\n\nCarnegie Mellon University\n\nsyeom@cs.cmu.edu\n\ndanupam@cmu.edu\n\nMatt Fredrikson\n\nCarnegie Mellon University\nmfredrik@cs.cmu.edu\n\nAbstract\n\nA machine learning model may exhibit discrimination when used to make\ndecisions involving people. One potential cause for such outcomes is that the\nmodel uses a statistical proxy for a protected demographic attribute. In this paper\nwe formulate a de\ufb01nition of proxy use for the setting of linear regression and\npresent algorithms for detecting proxies. Our de\ufb01nition follows recent work on\nproxies in classi\ufb01cation models, and characterizes a model\u2019s constituent behavior\nthat: 1) correlates closely with a protected random variable, and 2) is causally\nin\ufb02uential in the overall behavior of the model. We show that proxies in linear\nregression models can be ef\ufb01ciently identi\ufb01ed by solving a second-order cone\nprogram, and further extend this result to account for situations where the use of\na certain input variable is justi\ufb01ed as a \u201cbusiness necessity\u201d. Finally, we present\nempirical results on two law enforcement datasets that exhibit varying degrees\nof racial disparity in prediction outcomes, demonstrating that proxies shed useful\nlight on the causes of discriminatory behavior in models.\n\n1\n\nIntroduction\n\nThe use of machine learning in domains like insurance [23], criminal justice [18], and child wel-\nfare [28] raises concerns about fairness, as decisions based on model predictions may discriminate\non the basis of demographic attributes like race and gender. These concerns are driven by high-\npro\ufb01le examples of models that appear to have discriminatory effect, ranging from gender bias in\njob advertisements [10] to racial bias in same-day delivery services [21] and predictive policing [3].\nMeanwhile,\nlaws and regulations in various jurisdictions prohibit certain practices that have\ndiscriminatory effect, regardless of whether the discrimination is intentional. For example, the\nU.S. has recognized the doctrine of disparate impact since 1971, when the Supreme Court held in\nGriggs v. Duke Power Co. [25] that the Duke Power Company had discriminated against its black\nemployees by requiring a high-school diploma for promotion when the diploma had little to do\nwith competence in the new job. These regulations pose a challenge for machine learning models,\nwhich may give discriminatory predictions as an unintentional side effect of miscon\ufb01guration or\nbiased training data. Many competing de\ufb01nitions of disparate impact [3, 14] have been proposed in\nefforts to address this challenge, but it has been shown that some of these de\ufb01nitions are impossible\nto satisfy simultaneously [8]. Therefore, it is important to \ufb01nd a workable standard for detecting\ndiscriminatory behavior in models.\nMuch prior work [19, 30] has focused on the four-\ufb01fths rule [17] or variants thereof, which are\nrelaxed versions of the demographic parity requirement that different demographic groups should\nreceive identical outcomes on average. However, demographic parity does not necessarily make\na model fair. For example, consider an attempt to \u201crepair\u201d a racially discriminatory predictive\npolicing model by arbitrarily lowering the risk scores of some members of the disadvantaged race\nuntil demographic parity is reached. The resulting model is still unfair to individual members of\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00b4eal, Canada.\n\n\fthe disadvantaged race that did not have their scores adjusted. In fact, this is why the U.S. Supreme\nCourt ruled that demographic parity is not a complete defense to claims of disparate impact [26].\nIn addition, simply enforcing demographic parity without regard for possible justi\ufb01cations for\ndisparate impact may be prohibited on the grounds of intentional discrimination [27].\nRecent work on proxy use [11] addresses these issues by considering the causal factors behind\ndiscriminatory behavior. A proxy for a protected attribute is de\ufb01ned as a portion of the model that\nis both causally in\ufb02uential [13] on the model\u2019s output and statistically associated with the protected\nvariable. This means that, in the repair example above, the original discriminatory model is a proxy\nfor a protected demographic attribute, indicating the presence of discriminatory behavior in the\n\u201crepaired\u201d model. However, prior treatment of proxy use has been limited to classi\ufb01cation models,\nso regression models remain out of reach of these techniques.\nIn this paper, we de\ufb01ne a notion of proxy use (Section 2) for linear regression models, and show how\nit can be used to inform considerations of fairness and discrimination. While the previous notion of\nproxy use is prohibitively expensive to apply at scale to real-world models [11], our de\ufb01nition admits\na convex optimization procedure that leads to an ef\ufb01cient detection algorithm (Section 3). Because\ndisparate impact is not always forbidden, we extend our de\ufb01nition to account for an exempt input\nvariable whose use for a particular problem is justi\ufb01ed. We show that slight modi\ufb01cations to our\ndetection algorithm allow us to effectively \u201cignore\u201d proxies based on the exempt variable (Section 4).\nFinally, in Section 5 we evaluate our algorithm with two real-world predictive policing applications.\nWe \ufb01nd that the algorithm, despite taking little time to run, accurately identi\ufb01es parts of the model\nthat are the most problematic in terms of disparate impact. Moreover, in one of the datasets, the\nstrongest nonexempt proxy is signi\ufb01cantly weaker than the strongest general proxy, suggesting\nthat proxy use can sometimes be attributed to a single input variable. In other words, the proxies\nidenti\ufb01ed by our approach effectively explain the cause of discriminatory model predictions,\ninforming the consideration of whether the disparate impact is justi\ufb01ed.\nProofs of all theorems are given in the extended version of this paper [29].\n\n1.1 Related Work\n\nWe refer the reader to [6] for a detailed discussion of discrimination in machine learning from a\nlegal perspective. One legal development of note is the adoption of the four-\ufb01fths rule by the U.S.\nEqual Employment Opportunities Commission in 1978 [17]. The four-\ufb01fths \u201crule\u201d is a guideline\nthat compares the rates of favorable outcomes among different demographic groups, requiring that\nthe ratio of these rates be no less than four-\ufb01fths. This guideline motivated the work of Feldman et\nal. [19], who guarantee that no classi\ufb01er will violate the four-\ufb01fths rule by removing the association\nbetween the input variables and the protected attribute. Zafar et al. [30] use convex optimization to\n\ufb01nd linear models that are both accurate and fair, but their fairness de\ufb01nition, unlike ours, is derived\nfrom the four-\ufb01fths rule. We show in Section 2.5 that proxy use is a stronger notion of fairness than\ndemographic parity, of which the four-\ufb01fths rule is a relaxation.\nOther notions of fairness have been proposed as well. Dwork et al. [16] argue that demographic\nparity is insuf\ufb01cient as a fairness constraint, and instead de\ufb01ne individual fairness, which requires\nthat similar individuals have similar outcomes. While individual fairness is important, it is not\nwell-suited for characterizing disparate impact, which inherently involves comparing different de-\nmographic groups to each other. Hardt et al. [20] propose a notion of group fairness called equalized\nodds. Notably, equalized odds does not require demographic parity, i.e., groups can have unequal\noutcomes as long as the response variable is also unequally distributed. For example, in the context\nof predictive policing, it would be acceptable to categorize members of a certain racial group as a\nhigher risk on average, provided that they are in fact more likely to reoffend. This is consistent with\nthe current legal standard, wherein disparate impact can be justi\ufb01ed if there is an acceptable reason.\nHowever, some have observed that the response variable could be tainted by past discrimination [6,\nSection I.B.1], in which case equalized odds may end up perpetuating the discrimination.\nOur treatment of exempt input variables is similar to that of resolving variables by Kilbertus et\nal. [22] in their work on causal analysis of proxy use and discrimination. A key difference is that\nthey assume a causal model and only consider causal relationships between the protected attribute\nand the output of the model, whereas we view any association with the protected attribute as suspect.\nOur notion of proxy use extends that of Datta et al. [11, 12], who take into consideration both\n\n2\n\n\fassociation and in\ufb02uence. An alternative measure of proxy strength has been proposed by Adler et\nal. [1], who de\ufb01ne a single real-valued metric called indirect in\ufb02uence. As we show in the rest of this\npaper, the two-metric-based approach of Datta et al. leads to an ef\ufb01cient proxy detection algorithm.\n\n2 Proxy Use\n\nIn this section we present a de\ufb01nition of proxy use that is suited to linear regression models. We\n\ufb01rst review the original de\ufb01nition of Datta et al. [11] for classi\ufb01cation models and then show how\nto modify this de\ufb01nition to get one that is applicable to the setting of linear regression.\n\n2.1 Setting\n\nWe work in the standard machine learning setting, where a model is given several inputs that\ncorrespond to a data point. Throughout this paper, we will use X = (X1, . . . , Xn) to denote\nthese inputs, where X1, . . . , Xn are random variables. We consider a linear regression model\n\u02c6Y = \u03b21X1 + \u00b7\u00b7\u00b7 + \u03b2nXn, where \u03b2i represents the coef\ufb01cient for the input variable Xi. We will\nabuse notation by using \u02c6Y to represent either the model or its output.\nIn the case where each data point represents a person, care must be taken to avoid disparate impact\non the basis of a protected demographic attribute, such as race or gender. We will denote such\nprotected attribute by the random variable Z. In practice, Z is usually binary (i.e., Z \u2208 {0, 1}), but\nour results are general and apply to arbitrary numerical random variables.\n\n2.2 Proxy Use in Prior Work\n\nDatta et al. [11] de\ufb01ne proxy use of a random variable Z as the presence of an intermediate\ncomputation in a program that is both statistically associated with Z and causally in\ufb02uential on\nthe \ufb01nal output of the program. Instantiating this de\ufb01nition to a particular setting therefore entails\nspecifying an appropriate notion of \u201cintermediate computation\u201d, a statistical association measure,\nand a causal in\ufb02uence measure.\nDatta et al. identify intermediate computations in terms of syntactic decompositions into subpro-\n(cid:48)(X , P (X )). Then the association between P and Z is given\ngrams P , \u02c6Y\nby an appropriate measure such as mutual information, and the in\ufb02uence of P on \u02c6Y is de\ufb01ned as\nshown in Equation 1, where X and X (cid:48) are drawn independently from the population distribution.\n\n(cid:48) such that \u02c6Y (X ) \u2261 \u02c6Y\n\nIn\ufb02 \u02c6Y (P ) = PrX ,X (cid:48)[ \u02c6Y (X ) (cid:54)= \u02c6Y\n\n(cid:48)\n\n(X , P (X (cid:48)\n\n))],\n\n(1)\n\nIntuitively, in\ufb02uence is characterized by the likelihood that an independent change in the value of\nP will cause a change in \u02c6Y . This makes sense for classi\ufb01cation models because a change in the\nmodel\u2019s output corresponds to a change in the predicted class of a point, as re\ufb02ected by the use of\n0-1 loss in that setting. On the other hand, regression models have real-valued outputs, so the square\nloss is more appropriate for these models. Therefore, we are motivated to transform Equation 1,\nwhich is simply the expected 0-1 loss between \u02c6Y (X ) and \u02c6Y\n(cid:48)(X , P (X (cid:48))), into Equation 2, which is\nthe expected square loss between these two quantities.\n\nX ,X (cid:48)[( \u02c6Y (X ) \u2212 \u02c6Y\nE\n\n(cid:48)\n\n(X , P (X (cid:48)\n\n)))2]\n\n(2)\n\nBefore we can reason about the suitability of this measure, we must \ufb01rst de\ufb01ne an appropriate\nnotion of intermediate computation for linear models.\n\n2.3 Linear components\n\nThe notion of subprogram used for discrete models [11] is not well-suited to linear regression. To\nsee why, consider the model \u02c6Y = \u03b21X1 + \u03b22X2 + \u03b23X3. Suppose that this is computed using\nthe grouping (\u03b21X1 + \u03b22X2) + \u03b23X3 and that the de\ufb01nition of subprogram honors this ordering.\nThen, \u03b21X1 + \u03b22X2 would be a subprogram, but \u03b21X1 + \u03b23X3 would not be even though \u02c6Y could\nhave been computed equivalently as (\u03b21X1 + \u03b23X3) + \u03b22X2. We might attempt to address this\nby allowing any subset of the terms used in the model to de\ufb01ne a subprogram, thus capturing the\ncommutativity and associativity of addition. However, this de\ufb01nition still excludes expressions such\n\n3\n\n\fas \u03b21X1 + 0.5\u03b23X3, which may be a stronger proxy than either \u03b21X1 or \u03b21X1 + \u03b23X3. To include\nsuch expressions, we present De\ufb01nition 1 as the notion of subprogram that we use to de\ufb01ne proxy\nuse in the setting of linear regression.\nDe\ufb01nition 1 (Component). Let \u02c6Y = \u03b21X1 + \u00b7\u00b7\u00b7 + \u03b2nXn be a linear regression model. A\nrandom variable P is a component of \u02c6Y if and only if there exist \u03b11, . . . , \u03b1n \u2208 [0, 1] such that\nP = \u03b11\u03b21X1 + \u00b7\u00b7\u00b7 + \u03b1n\u03b2nXn.\n\n2.4 Linear association and in\ufb02uence\n\nHaving de\ufb01ned a component as the equivalent of a subprogram in a linear regression model, we\nnow formalize the association and in\ufb02uence conditions given by Datta et al. [11].\n\nAssociation. A linear model only uses linear relationships between variables, so our association\nmeasure only captures linear relationships. In particular, we use the Pearson correlation coef\ufb01cient,\nand we square it so that a higher association measure always represents a stronger proxy.\nDe\ufb01nition 2 (Association). The association of two nonconstant random variables P and Z is\nde\ufb01ned as Asc(P, Z) = Cov(P,Z)2\nNote that Asc(P, Z) \u2208 [0, 1], with 0 representing no linear correlation and 1 representing a fully\nlinear relationship.\n\nVar(P )Var(Z) .\n\n(cid:48)(X , P (X (cid:48))) = (cid:80)n\n\nSubstituting these into Equation 2 gives\n)))2] = E\n\ninition 1 gives us \u02c6Y (X ) = (cid:80)n\nIn\ufb02uence. To formalize in\ufb02uence, we continue from where we left off with Equation 2. Def-\ni=1(1 \u2212 \u03b1i)\u03b2iXi + \u03b1i\u03b2iX(cid:48)\ni.\ni)2] = Var(P (X ) \u2212 P (X (cid:48))),\nX ,X (cid:48)[( \u02c6Y (X ) \u2212 \u02c6Y\nE\nwhich is proportional to Var(P (X )) since X and X (cid:48) are i.i.d. De\ufb01nition 3 captures this reasoning,\nnormalizing the variance so that In\ufb02 \u02c6Y (P ) = 1 when P = \u02c6Y (i.e., \u03b11 = \u00b7\u00b7\u00b7 = \u03b1n = 1). In the\nextended version of this paper [29], we also show that variance is the unique in\ufb02uence measure\n(up to a constant factor) satisfying some natural axioms that we call nonnegativity, nonconstant\npositivity, and zero-covariance additivity.\nDe\ufb01nition 3 (In\ufb02uence). Let P be a component of a linear regression model \u02c6Y . The in\ufb02uence of P\nis de\ufb01ned as In\ufb02 \u02c6Y (P ) = Var(P )\nVar( \u02c6Y )\n\ni=1 \u03b1i\u03b2iXi \u2212 \u03b1i\u03b2iX(cid:48)\n\n.\n\n(cid:48)\n\n(X , P (X (cid:48)\n\ni=1 \u03b2iXi and \u02c6Y\n\nX ,X (cid:48)[((cid:80)n\n\nWhen it is obvious from the context, the subscript \u02c6Y may be omitted. Note that in\ufb02uence can\nexceed 1 because the inputs to a model can cancel each other out, leaving the \ufb01nal model less\nvariable than some of its components.\nFinally, the de\ufb01nition of proxy use for linear models is given in De\ufb01nition 4.\nDe\ufb01nition 4 ((\u0001, \u03b4)-Proxy Use). Let \u0001, \u03b4 \u2208 (0, 1]. A model \u02c6Y = \u03b21X1 + \u00b7\u00b7\u00b7 + \u03b2nXn has\n(\u0001, \u03b4)-proxy use of Z if there exists a component P such that Asc(P, Z) \u2265 \u0001 and In\ufb02 \u02c6Y (P ) \u2265 \u03b4.\n\n2.5 Connection to Demographic Parity\n\nWe now discuss the relationship between proxy use and demographic parity, and argue that proxy\nuse is a stronger de\ufb01nition that provides more useful information than demographic parity. For\nbinary classi\ufb01cation models with two demographic groups, demographic parity is de\ufb01ned by the\nequation Pr[ \u02c6Y = 1|Z = 0] = Pr[ \u02c6Y = 1|Z = 1], i.e., two demographic groups must have the same\nrates of favorable outcomes. We adapt this notion to regression models by replacing the constraint\non the positive classi\ufb01cation outcome with the expectation of the response, as shown in De\ufb01nition 5.\nDe\ufb01nition 5 (Demographic Parity, Regression). Let \u02c6Y be a regression model, and let Z be a binary\nrandom variable. \u02c6Y satis\ufb01es demographic parity if E[ \u02c6Y |Z = 0] = E[ \u02c6Y |Z = 1].\nEquation 3 shows that our association measure is related to demographic parity in regression models.\n\nAsc( \u02c6Y , Z) =\n\nCov( \u02c6Y , Z)2\n\nVar( \u02c6Y )Var(Z)\n\n= (E[ \u02c6Y |Z = 0] \u2212 E[ \u02c6Y |Z = 1])2 \u00b7 Var(Z)\nVar( \u02c6Y )\n\n,\n\n(3)\n\n4\n\n\f(a) z is a vector representation of the protected\nattribute Z, and components of the model can\nIf a component\nalso be represented as vectors.\nis inside the red double cone,\nit exceeds the\nassociation threshold \u0001, where the angle \u03b8 is set\nsuch that \u0001 = cos2 \u03b8. The cone on the right side\ncorresponds to positive correlation with Z, and\nthe left cone negative correlation. Components\nin the blue shaded area exceed some in\ufb02uence\nthreshold \u03b4. If any component exceeds both the\nassociation and the in\ufb02uence thresholds, it is a\nproxy and may be disallowed.\n\n(b) x1 and x2 are vector representations of X1\nand X2, which are inputs to the model \u02c6Y =\n\u03b21X1 +\u03b22X2. The gray shaded area indicates the\nspace of all possible components of the model.\n\u03b21X1 is a component, but it is not a proxy because\nit does not have strong enough association with\nZ. Although \u03b22X2 is strongly associated with Z,\nit is not in\ufb02uential enough to be a proxy. On the\nother hand, 0.7\u03b21X1 + \u03b22X2 is a component that\nexceeds both the association and the in\ufb02uence\nthresholds, so it is a proxy and may be disallowed.\n\nFigure 1: Illustration of proxy use with the vector interpretation of random variables. In the above\nexamples, all vectors lie in R2 for ease of depiction. In general, the vectors z, x1, . . . , xn can span\nRn+1.\n\nIn particular, if \u02c6Y does not satisfy demographic parity, then Asc( \u02c6Y , Z) > 0, so \u02c6Y is an (\u0001, 1)-proxy\nfor some \u0001 > 0. This means that our proxy use framework is broad enough to detect any violation of\ndemographic parity. On the other hand, the \u201crepair\u201d example in Section 1 shows that demographic\nparity does not preclude the presence of proxies. Therefore, proxy use is a strictly stronger notion\nof fairness than demographic parity.\nMoreover, instances of proxy use can inform the discussion about a model that exhibits demographic\ndisparity. When a proxy is identi\ufb01ed, it may explain the cause of the disparity and can help decide\nwhether the behavior is justi\ufb01ed based on the set of variables used by the proxy. We elaborate on\nthis idea in Section 4, designating a certain input variable as always permissible to use.\n\n3 Finding Proxy Use\n\nIn this section, we present our proxy detection algorithms, which take advantage of properties\nspeci\ufb01c to linear regression to quickly identify components of interest. We prove that we can use\nan exact optimization problem (Problem 1) to either identify a proxy if one exists, or de\ufb01nitively\nconclude that there is no proxy. However, because this problem is not convex and in some cases may\nbe intractable, we also present an approximate version of the problem (Problem 2) that sacri\ufb01ces\nsome precision. The approximate algorithm can still be used to conclude that a model does not\nhave any proxies, but it may return false positives. In Section 5, we evaluate how these algorithms\nperform on real-world data.\nBecause the only operations that we perform on random variables are addition and scalar multipli-\ncation, we can safely treat the random variables as vectors in a vector space. In addition, covariance\nis an inner product in this vector space. As a result, it is helpful to think of random variables\nZ, X1, . . . , Xn as vectors z, x1, . . . , xn \u2208 Rn+1, with covariance as dot product. Under this\ninterpretation, in\ufb02uence is characterized by In\ufb02 \u02c6Y (P ) \u221d Var(P ) = Cov(P, P ) = p \u00b7 p = (cid:107)p(cid:107)2,\nwhere (cid:107)\u00b7(cid:107) denotes the (cid:96)2-norm, and association is shown in Equation 4, where \u03b8 is the angle\n\n5\n\nz\u03b8\u03b21x1\u03b22x20.7\u03b21x1+\u03b22x2\fProblem 1 Exact optimization\nmin \u2212(cid:107)A\ns.t. 0 (cid:22) \u03b1 (cid:22) 1 and (cid:107)A\n\n(cid:48)\u03b1(cid:107)2\n\n(cid:48)\u03b1(cid:107) \u2264 s \u00b7 zT A(cid:48)\u03b1\u221a\n\u0001(cid:107)z(cid:107)\n\nProblem 2 Approximate optimization\nmin \u2212cT \u03b1\ns.t. 0 (cid:22) \u03b1 (cid:22) 1 and (cid:107)A\n\n(cid:48)\u03b1(cid:107) \u2264 s \u00b7 zT A(cid:48)\u03b1\u221a\n\u0001(cid:107)z(cid:107)\n\nFigure 2: Optimization problems used to \ufb01nd proxies in linear regression models. A(cid:48) is the\n(n+1) \u00d7 n matrix [\u03b21x1\n. . . \u03b2nxn], and we optimize over \u03b1, which is an n-dimensional\nvector of the alpha-coef\ufb01cients used in De\ufb01nition 1. \u0001 is the association threshold, and c is the\nn-dimensional vector that satis\ufb01es ci = (cid:107)\u03b2ixi(cid:107).\n\nbetween the two vectors p and z.\n\nAsc(P, Z) =\n\nCov(P, Z)2\n\nVar(P )Var(Z)\n\n=\n\n(cid:19)2\n\n(cid:18) p \u00b7 z\n\n(cid:107)p(cid:107)(cid:107)z(cid:107)\n\n= cos2 \u03b8,\n\n(4)\n\nThis abstraction is illustrated in more detail in Figure 1.\nTo \ufb01nd coordinates for the vectors, we consider the covariance matrix [Cov(Xi, Xj)]i,j\u2208{0,...,n},\nwhere Z = X0 for notational convenience. If we can write this covariance matrix as AT A for\nsome (n+1) \u00d7 (n+1) matrix A, then each entry in the covariance matrix is the dot product of two\n(not necessarily distinct) columns of A. In other words, the mapping from the random variables\nZ, X1, . . . , Xn to the columns of A preserves the inner product relationship. Now it remains to\ndecompose the covariance matrix into the form AT A. Since the covariance matrix is guaranteed to\nbe positive semide\ufb01nite, two of the possible decompositions are the Cholesky decomposition and\nthe matrix square root.\nOur proxy detection algorithms use as subroutines the optimization problems that are formally\nstated in Figure 2. We \ufb01rst motivate the exact optimization problem (Problem 1) and show how the\nsolutions to these problems can be used to determine whether the model contains a proxy. Then, we\npresent the approximate optimization problem (Problem 2), which sacri\ufb01ces exactness for ef\ufb01cient\nsolvability.\nLet A(cid:48) be the (n+1) \u00d7 n matrix [\u03b21x1\n. . . \u03b2nxn]. The constraint 0 (cid:22) \u03b1 (cid:22) 1 restricts the\nsolutions to be inside the space of all components, represented by the gray shaded area in Figure 1b.\nMoreover, when s \u2208 {\u22121, 1}, the constraint (cid:107)A(cid:48)\u03b1(cid:107) \u2264 s \u00b7 (zT A(cid:48)\u03b1)/(\n\u0001(cid:107)z(cid:107)) describes one of\nthe red cones in Figure 1a, which together represent the association constraint. Subject to these\nconstraints, we maximize the in\ufb02uence, which is proportional to (cid:107)A(cid:48)\u03b1(cid:107)2. Theorem 1 shows that\nthis technique is suf\ufb01cient to determine whether a model contains a proxy.\nTheorem 1. Let P denote the component de\ufb01ned by the alpha-coef\ufb01cients \u03b1. The linear regression\nmodel \u02c6Y = \u03b21X1 + \u00b7\u00b7\u00b7 + \u03b2nXn contains a proxy if and only if there exists a solution to Problem 1\nwith s \u2208 {\u22121, 1} such that In\ufb02 \u02c6Y (P ) \u2265 \u03b4.\nIn essence, Theorem 1 guarantees the correctness of the following proxy detection algorithm: Run\nProblem 1 with s = 1 and s = \u22121, and compute the association and in\ufb02uence of the resulting\nsolutions. The model contains a proxy if and only if any of the solutions passes both the association\nand the in\ufb02uence thresholds.\nIt is worth mentioning that Problem 1 tests for strong positive correlation with Z when s = 1 and for\nstrong negative correlation when s = \u22121. This optimization problem resembles a second-order cone\nprogram (SOCP) [7, Section 4.4.2], which can be solved ef\ufb01ciently. However, the objective function\nis concave, so the standard techniques for solving SOCPs do not work on this problem. To get around\nthis issue, we can instead solve Problem 2, which has a linear objective function whose coef\ufb01cients\nci = (cid:107)\u03b2ixi(cid:107) were chosen so that the inequality (cid:107)A(cid:48)\u03b1(cid:107) \u2264 cT \u03b1 always holds. This inequality allows\nus to prove Theorem 2, which mirrors the claim of Theorem 1 but only in one direction.\nTheorem 2. If the linear regression model \u02c6Y = \u03b21X1 + \u00b7\u00b7\u00b7 + \u03b2nXn contains a proxy, then there\nexists a solution to Problem 2 with s \u2208 {\u22121, 1} such that cT \u03b1 \u2265 (\u03b4 Var( \u02c6Y ))0.5.\nTheorem 2 suggests a quick algorithm to verify that a model does not have any proxies. We solve\nthe SOCP described by Problem 2, once with s = 1 and once with s = \u22121. If neither solution\n\n\u221a\n\n6\n\n\fsatis\ufb01es cT \u03b1 \u2265 (\u03b4 Var( \u02c6Y ))0.5, by the contrapositive of Theorem 2, we can be sure that the model\ndoes not contain any proxies.\nHowever, the converse does not hold, i.e., we cannot be sure that the model has a proxy even if\na solution to Problem 2 satis\ufb01es cT \u03b1 \u2265 (\u03b4 Var( \u02c6Y ))0.5. This is because cT \u03b1 overapproximates\n(cid:107)A(cid:48)\u03b1(cid:107) by using the triangle inequality. As a result, it is possible for the in\ufb02uence to be below the\nthreshold even if the value of cT \u03b1 is above the threshold. While there is in general no upper bound\non the overapproximation factor of the triangle inequality, the experiments in Section 5 show that\nthis factor is not too large in practice. In addition, Problem 1 often works well enough in practice\ndespite not being a convex optimization problem.\n\n4 Exempt Use of a Variable\n\nSo far, we have shown how to \ufb01nd a proxy in a linear regression model, but we have not discussed\nwhich proxies should be allowed and which should not. As mentioned in Section 1, disparate impact\nis legally permitted if there is suf\ufb01cient justi\ufb01cation. For example, in the context of predictive polic-\ning, it may be acceptable to consider the number of prior convictions even if one racial group tends\nto have a higher number of convictions than another. We formalize this idea by assuming that the\nuse of one particular input variable, which we call the exempt variable, is explicitly permitted. This\nassumption may be appropriate if, for example, the exempt variable is directly and causally related\nto the response variable Y . Throughout this section, we will use X1 to denote the exempt variable.\nFirst, we formally de\ufb01ne which proxies are exempt, i.e., permitted because the proxy use is\nattributable to X1. Clearly, if the model ignores every input except X1, all proxies in the model\nshould be exempt. Conversely, if the coef\ufb01cient \u03b21 of X1 is zero, no proxies should be exempt.\nWe capture this intuition by ignoring X1 and checking whether the resulting component is a proxy.\nMore formally, if P = \u03b11\u03b21X1 + \u00b7\u00b7\u00b7 + \u03b1n\u03b2nXn is a component, we investigate P \\ X1, which we\nwrite as shorthand for the component \u03b12\u03b22X2 + \u00b7\u00b7\u00b7 + \u03b1n\u03b2nXn. If P is a proxy but P \\ X1 is not,\nthen P is exempted because the proxy use can be attributed to the exempt variable X1.\nHowever, one possible issue with this attribution is that the other input variables can interact\nwith X1 to create a proxy stronger than X1. For example, suppose that Asc(X2, Z) = 0 and\nP = X1 + X2 = Z. Then, even though P \\ X1 = X2 is not a proxy, it makes P more strongly\nassociated with Z than X1 is, so it is not clear that P should be exempt on account of the fact that\nwe are permitted to use X1. Therefore, our de\ufb01nition of proxy exemption in De\ufb01nition 6 adds the\nrequirement that P should not be too much more associated with Z than X1 is.\nDe\ufb01nition 6 (Proxy Exemption). Let P be a proxy component of a linear regression model,\nand let X1 be the exempt variable. P is an exempt proxy if P \\ X1 is not a proxy and\nAsc(P, Z) < Asc(X1, Z) + \u0001(cid:48), where \u0001(cid:48) is the association tolerance parameter.\nWe can incorporate the exemption policy into our search algorithm with small changes to the\noptimization problem. By De\ufb01nition 6, a proxy P is nonexempt if either P \\ X1 is a proxy or\nAsc(P, Z) \u2265 Asc(X1, Z) + \u0001(cid:48). For each of these two conditions, we modify the optimization prob-\nlems from Section 3 to \ufb01nd proxies that also satisfy the condition. If either of these modi\ufb01cations\nreturn a positive result, then we have found a nonexempt proxy.\nWe start with the second condition, for which it is easy to see that it suf\ufb01ces to change the\nassociation threshold in Problem 2 from \u0001 to max(\u0001, Asc(X1, Z) + \u0001(cid:48)). For the \ufb01rst condition, we\nuse the result from Theorem 3 and simply add the constraint that \u03b11 = 0. If we add this constraint\nto Problem 2, the resulting problem is still an SOCP and can therefore be solved ef\ufb01ciently.\nTheorem 3. A linear regression model contains a proxy P such that P \\ X1 is also a proxy if and\nonly if the model contains a proxy such that \u03b11 = 0.\n\n5 Experimental Results\n\nIn this section, we evaluate the performance of our algorithms on real-world predictive policing\ndatasets. We ran our proxy detection algorithms on observational data from Chicago\u2019s Strategic\nSubject List (SSL) model [9] and the Communities and Crimes (C&C) dataset [15]. The creator of\nthe SSL model claims that the model avoids variables that could lead to discrimination [4], and if\n\n7\n\n\fAssociation threshold \u0001\nActual in\ufb02. (Prob. 1)\nApprox. in\ufb02. (Prob. 2)\nActual in\ufb02. (Prob. 2)\n\n0.01\n0.8816\n1.6933\n0.8476\n\n0.02\n0.2263\n0.6683\n0.1874\n\n0.03\n0.1090\n0.3820\n0.0987\n\n0.04\n\n0.0427*\n0.1432\n0.0420\n\n0.05\n\n0.0065*\n0.0270\n0.0080\n\n0.06\n0.0028\n0.0085\n0.0027\n\n0.07\n0.0000\n0.0000\n0.0000\n\nTable 1: In\ufb02uence of the components obtained by solving the exact (Problem 1) and approximate\n(Problem 2) optimization problems for the SSL model using Z = race and s = 1. No component\nhad strong enough association when s = \u22121 instead. Asterisks indicate that the exact optimization\nproblem terminated early due to a singular KKT matrix. The approximate optimization problem did\nnot have this issue, and the overapproximation that it makes of the components\u2019 in\ufb02uence is shown\nin the second row.\n\nthis is the case then we would expect to see only weak proxies if any. On the other hand, the C&C\ndataset contains many variables that are correlated with race, so we would expect to \ufb01nd strong\nproxies in a model trained with this dataset.\nTo test these hypotheses, we implemented Problems 1 and 2 with the cvxopt package [2] in\nPython. The experimental results con\ufb01rm our hypotheses and show that our algorithm runs very\nquickly (< 1 second). Moreover, our algorithms pinpoint components of the model that are the\nmost problematic in terms of disparate impact, and we \ufb01nd that the exemption policy discussed in\nSection 4 removes the appropriate proxies from the SSL model.\nFor each dataset, we brie\ufb02y describe the dataset and present the experimental results, demonstrating\nhow the identi\ufb01ed proxies can provide evidence of discriminatory behavior in models. Then, we\nexplain the implications of these results on the false positive and false negative rates in practice,\nand we discuss how a practitioner can decide which values of \u0001 and \u03b4 to use.\n\nStrategic Subject List. The SSL [9] is a model that the Chicago Police Department uses to assess\nan individual\u2019s risk of being involved in a shooting incident, either as a victim or a perpetrator. The\nSSL dataset consists of 398,684 rows, each of which corresponds to a person. Each row includes the\nSSL model\u2019s eight input variables (including age, number of previous arrests for violent offenses,\nand whether the person is a member of a gang), the SSL score given by the model, and the person\u2019s\nrace and gender.\nWe searched for proxies for race (binary black/white) and gender (binary male/female), \ufb01ltering\nout rows with other race or gender. After also \ufb01ltering out rows with missing data, we were left\nwith 290,085 rows. Because we did not have direct access to the SSL model, we trained a linear\nregression model to predict the SSL score of a person given the same set of variables that the SSL\nmodel uses. Our model explains approximately 80% of the variance in the SSL scores, so we\nbelieve that it is a reasonable approximation of the true model for the purposes of this evaluation.\nThe strengths of the proxies for race are given in Table 1. The estimated in\ufb02uence was computed\nas (cT \u03b1)2/Var( \u02c6Y ), which is the result of solving for \u03b4 in the inequality given in Theorem 2.\nWe found that this estimate is generally about 3\u20134\u00d7 larger than the actual in\ufb02uence. Although\nthe proxies for race were somewhat stronger than those for gender, neither type had signi\ufb01cant\nin\ufb02uence (\u03b4 > 0.05) beyond small \u0001 levels (~0.03\u20130.04). This is consistent with our hypothesis\nabout the lack of discriminatory behavior in this model.\nWe also tested the effect of exempting the indicator variable for gang membership in the input.\nGang membership is more associated with both demographic variables than any other in among the\ninputs, and is a plausible cause of involvement in violent crimes [5], making it a prime candidate for\nexemption. As contrasted with the components described in Table 1, every nonexempt component\nunder this policy has an association with race less than 0.033. This means that the strongest\nnonexempt proxy is signi\ufb01cantly weaker than the strongest general proxy, suggesting that much of\nthe proxy use present in the model can be attributed to the gang membership variable.\n\nCommunities and Crimes. C&C [24] is a dataset in the UCI machine learning repository [15]\nthat combines socioeconomic data from the 1990 US census with the 1995 FBI Uniform Crime\nReporting data.\nIt consists of 1,994 rows, each of which corresponds to a community (e.g.,\n\n8\n\n\fmunicipality) in the U.S., and 122 potential input variables. After we removed the variables that\ndirectly measure race and the ones with missing data, we were left with 90 input variables.\nWe simulated a hypothetical naive attempt at predictive policing by using this dataset to train a linear\nregression model that predicts the per capita rate of violent crimes in a community. We de\ufb01ned\nthe protected attribute Z as the difference between the percentages of people in the community\nwho are black and white, respectively. We observed a strong association in the dataset between\nthe rate of violent crime and Z (Asc(Y, Z) = 0.48), and the model ampli\ufb01es this bias even more\n(Asc( \u02c6Y , Z) = 0.65).\nAs expected, we found very strong proxies for race in the model trained with the C&C dataset. For\nexample, one proxy consisting of 58 of the 90 input variables achieves an in\ufb02uence of 0.34 when\n\u0001 = 0.85. Notably, the input variable most strongly associated with race has an association of only\n0.73, showing that in practice multiple variables combine to result in a stronger proxy than any\nof the individual variables. In addition, the model contains a proxy whose association is 0.40 and\nin\ufb02uence is 14.5. In other words, the variance of the proxy is 14.5 times greater than that of the\nmodel; this arises because other associated variables cancel most of this variance in the full model.\nAs a result, exempting any one variable does not result in a signi\ufb01cant difference since associated\nvariables still yield proxies that are nearly as strong. Moreover, a cursory analysis suggested that\nthe variables used in these proxies are not justi\ufb01able correlates of race, so an exemption policy may\nnot suf\ufb01ce to \u201cexplain away\u201d the discriminatory behavior of the model.\n\nFalse Positives and False Negatives. Theorem 1 shows that our exact proxy detection algorithm\ndetects a proxy if and only if the model in fact contains a proxy.\nIn other words, if Problem 1\nreturns optimal solutions, we can use the solutions to conclusively determine whether there exists\na proxy, and there will be no false positives or false negatives. However, our experiments show that\nsometimes Problem 1 terminates early due to a singular KKT matrix, and in this case one can turn\nto the approximate proxy detection algorithm.\nAlthough Problem 2 sometimes returns solutions that are not in fact proxies, we can easily ascertain\nwhether any given solution is a proxy by simply computing its association and in\ufb02uence. However,\neven if the solution returned by Problem 2 turn out to not be proxies, the model could still contain\na different proxy. Using Table 1 as reference, we see that this happens in the SSL model if, for ex-\nample, \u0001 = 0.02 and \u03b4 is between 0.1874 and 0.2263. Therefore, one can consider the approximate\nalgorithm as giving a \ufb01nding of either \u201cpotential proxy use\u201d or \u201cno proxy use\u201d. Theorem 2 shows\nthat a \ufb01nding of \u201cno proxy use\u201d does indeed guarantee that the model is free of proxies. In other\nwords, the approximate algorithm has no false negatives. However, the algorithm overapproximates\nin\ufb02uence, so the algorithm can give a \ufb01nding of \u201cpotential proxy use\u201d when there are no proxies,\nresulting in a false positive. This happens when \u03b4 is between the maximum feasible in\ufb02uence (\ufb01rst\nrow in Table 1) and the maximum feasible overapproximation of in\ufb02uence (second row in Table 1).\n\nReasonable Values of \u0001 and \u03b4. Although the appropriate values of \u0001 and \u03b4 depend on the appli-\ncation, we remind the reader that association is the square of the Pearson correlation coef\ufb01cient.\nThis means that an association of 0.05 corresponds to a Pearson correlation coef\ufb01cient of ~0.22,\nwhich represents not an insigni\ufb01cant amount of correlation. Likewise, in\ufb02uence is proportional to\nvariance, which increases quadratically with scalar coef\ufb01cients. Therefore, we recommend against\nsetting \u0001 and \u03b4 to a value much higher than 0.05. To get an idea of which values of \u03b4 are suitable for\na particular application, the practitioner can compare the proposed value of \u03b4 against the in\ufb02uence\nof the individual input variables \u03b2iXi.\n\n6 Conclusion and Future Work\n\nIn this paper, we have formalized the notion of proxy discrimination in linear regression models and\npresented an ef\ufb01cient proxy detection algorithm. We account for the case where the use of one vari-\nable is justi\ufb01ed, and extending this result to multiple exempt variables is valuable future work that\nwould enable better handling of models like C&C that take many closely related input variables. De-\nveloping learning rules that account for proxy use, leading to models without proxies above speci\ufb01ed\nthresholds, is also an intriguing direction with direct potential for impact on practical scenarios.\n\n9\n\n\fAcknowledgment\n\nThe authors would like to thank the anonymous reviewers at NeurIPS 2018 for their thoughtful\nfeedback. This material is based upon work supported by the National Science Foundation under\nGrant No. CNS-1704845.\n\nReferences\n[1] Philip Adler, Casey Falk, Sorelle A Friedler, Tionney Nix, Gabriel Rybeck, Carlos Scheideg-\nger, Brandon Smith, and Suresh Venkatasubramanian. Auditing black-box models for indirect\nin\ufb02uence. Knowledge and Information Systems, 54(1):95\u2013122, 2018.\n\n[2] Martin S Andersen, Joachim Dahl, and Lieven Vandenberghe. CVXOPT: Python software for\n\nconvex optimization. http://cvxopt.org.\n\n[3] Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias: There\u2019s software\nused across the country to predict future criminals. and it\u2019s biased against blacks. ProPublica,\n2016.\n\n[4] Jeff Asher and Rob Arthur. Inside the algorithm that tries to predict gun violence in Chicago.\n\nThe New York Times, 2017.\n\n[5] JC Barnes, Kevin M Beaver, and J Mitchell Miller. Estimating the effect of gang membership\non nonviolent and violent delinquency: A counterfactual analysis. Aggressive behavior,\n36(6):437\u2013451, 2010.\n\n[6] Solon Barocas and Andrew D Selbst. Big data\u2019s disparate impact. California Law Review,\n\n104:671\u2013732, 2016.\n\n[7] Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press,\n\n2004.\n\n[8] Alexandra Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism\n\nprediction instruments. Big Data, 5(2):153\u2013163, 2017.\n\n[9] City of Chicago.\n\nStrategic Subject List.\n\nPublic-Safety/Strategic-Subject-List/4aki-r3np, 2017.\n\nhttps://data.cityofchicago.org/\n\n[10] Amit Datta, Michael Carl Tschantz, and Anupam Datta. Automated experiments on ad privacy\n\nsettings. Privacy Enhancing Technologies, 2015(1):92\u2013112, 2015.\n\n[11] Anupam Datta, Matt Fredrikson, Gihyuk Ko, Piotr Mardziel, and Shayak Sen.\n\ndiscrimination in data-driven systems. arXiv preprint arXiv:1707.08120, 2017.\n\nProxy\n\n[12] Anupam Datta, Matt Fredrikson, Gihyuk Ko, Piotr Mardziel, and Shayak Sen. Use privacy in\ndata-driven systems: Theory and experiments with machine learnt programs. In ACM SIGSAC\nConference on Computer and Communications Security, pages 1193\u20131210, 2017.\n\n[13] Anupam Datta, Shayak Sen, and Yair Zick. Algorithmic transparency via quantitative input\nin\ufb02uence: Theory and experiments with learning systems. In IEEE Symposium on Security\nand Privacy, pages 598\u2013617, 2016.\n\n[14] William Dieterich, Christina Mendoza, and Tim Brennan. COMPAS risk scales: Demon-\nhttp://go.volarisgroup.com/rs/\n\nstrating accuracy equity and predictive parity.\n430-MBX-989/images/ProPublica_Commentary_Final_070616.pdf, 2016.\n\n[15] Dheeru Dua and E\ufb01 Karra Taniskidou.\n\n//archive.ics.uci.edu/ml, 2017.\n\nUCI machine learning repository.\n\nhttps:\n\n[16] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness\n\nthrough awareness. In Innovations in Theoretical Computer Science, pages 214\u2013226, 2012.\n\n[17] Equal Employment Opportunities Commission. Uniform guidelines on employee selection\n\nprocedures. 29 CFR Part 1607, 1978.\n\n10\n\n\f[18] Equivant. Practitioner\u2019s guide to COMPAS core. http://www.equivant.com/assets/\n\nimg/content/Practitioners_Guide_COMPASCore_121917.pdf, 2017.\n\n[19] Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkata-\nIn ACM SIGKDD International\n\nsubramanian. Certifying and removing disparate impact.\nConference on Knowledge Discovery and Data Mining, pages 259\u2013268, 2015.\n\n[20] Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. In\n\nAdvances in Neural Information Processing Systems, pages 3315\u20133323, 2016.\n\n[21] David Ingold and Spencer Soper. Amazon doesn\u2019t consider the race of its customers. Should\n\nit? Bloomberg, 2016.\n\n[22] Niki Kilbertus, Mateo Rojas Carulla, Giambattista Parascandolo, Moritz Hardt, Dominik\nIn\n\nJanzing, and Bernhard Sch\u00a8olkopf. Avoiding discrimination through causal reasoning.\nAdvances in Neural Information Processing Systems, pages 656\u2013666, 2017.\n\n[23] Bernard Marr. How AI and machine learning are used to transform the insurance industry.\n\nForbes, 2017.\n\n[24] Michael Redmond and Alok Baveja. A data-driven software tool for enabling cooperative\ninformation sharing among police departments. European Journal of Operational Research,\n141(3):660\u2013678, 2002.\n\n[25] Supreme Court of the United States. Griggs v. Duke Power Co. 401 U.S. 424, 1971.\n\n[26] Supreme Court of the United States. Connecticut v. Teal. 457 U.S. 440, 1982.\n\n[27] Supreme Court of the United States. Ricci v. DeStefano. 557 U.S. 557, 2009.\n\n[28] Rhema Vaithianathan, Emily Putnam-Hornstein, Nan Jiang,\n\nand\nDeveloping predictive models to support child maltreatment hot-\nimplementation.\n\nTim Maloney.\nline\nscreening\nhttps://www.alleghenycountyanalytics.us/wp-content/uploads/2018/02/\nDevelopingPredictiveRiskModels-package_011618.pdf, 2017.\n\nParma Nand,\n\ndecisions:\n\nAllegheny County methodology\n\nand\n\n[29] Samuel Yeom, Anupam Datta, and Matt Fredrikson. Hunting for discriminatory proxies in\n\nlinear regression models. arXiv preprint arXiv:1810.07155, 2018.\n\n[30] Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rogriguez, and Krishna P Gummadi.\nIn Arti\ufb01cial Intelligence and\n\nFairness constraints: Mechanisms for fair classi\ufb01cation.\nStatistics, pages 962\u2013970, 2017.\n\n11\n\n\f", "award": [], "sourceid": 2229, "authors": [{"given_name": "Samuel", "family_name": "Yeom", "institution": "Carnegie Mellon University"}, {"given_name": "Anupam", "family_name": "Datta", "institution": "Carnegie Mellon University"}, {"given_name": "Matt", "family_name": "Fredrikson", "institution": "CMU"}]}