{"title": "Blind Attacks on Machine Learners", "book": "Advances in Neural Information Processing Systems", "page_first": 2397, "page_last": 2405, "abstract": "The importance of studying the robustness of learners to malicious data is well established. While much work has been done establishing both robust estimators and effective data injection attacks when the attacker is omniscient, the ability of an attacker to provably harm learning while having access to little information is largely unstudied. We study the potential of a \u201cblind attacker\u201d to provably limit a learner\u2019s performance by data injection attack without observing the learner\u2019s training set or any parameter of the distribution from which it is drawn. We provide examples of simple yet effective attacks in two settings: firstly, where an \u201cinformed learner\u201d knows the strategy chosen by the attacker, and secondly, where a \u201cblind learner\u201d knows only the proportion of malicious data and some family to which the malicious distribution chosen by the attacker belongs. For each attack, we analyze minimax rates of convergence and establish lower bounds on the learner\u2019s minimax risk, exhibiting limits on a learner\u2019s ability to learn under data injection attack even when the attacker is \u201cblind\u201d.", "full_text": "Blind Attacks on Machine Learners\n\nAlex Beatson\n\nDepartment of Computer Science\n\nPrinceton University\n\nabeatson@princeton.edu\n\nZhaoran Wang\n\nDepartment of Operations Research\n\nand Financial Engineering\n\nPrinceton University\n\nzhaoran@princeton.edu\n\nHan Liu\n\nDepartment of Operations Research\n\nand Financial Engineering\n\nPrinceton University\n\nhanliu@princeton.edu\n\nAbstract\n\nThe importance of studying the robustness of learners to malicious data is well\nestablished. While much work has been done establishing both robust estimators\nand effective data injection attacks when the attacker is omniscient, the ability of\nan attacker to provably harm learning while having access to little information is\nlargely unstudied. We study the potential of a \u201cblind attacker\u201d to provably limit\na learner\u2019s performance by data injection attack without observing the learner\u2019s\ntraining set or any parameter of the distribution from which it is drawn. We provide\nexamples of simple yet effective attacks in two settings: \ufb01rstly, where an \u201cinformed\nlearner\u201d knows the strategy chosen by the attacker, and secondly, where a \u201cblind\nlearner\u201d knows only the proportion of malicious data and some family to which the\nmalicious distribution chosen by the attacker belongs. For each attack, we analyze\nminimax rates of convergence and establish lower bounds on the learner\u2019s minimax\nrisk, exhibiting limits on a learner\u2019s ability to learn under data injection attack even\nwhen the attacker is \u201cblind\u201d.\n\n1\n\nIntroduction\n\nAs machine learning becomes more widely adopted in security and in security-sensitive tasks, it is\nimportant to consider what happens when some aspect of the learning process or the training data\nis compromised [1\u20134]. Examples in network security are common and include tasks such as spam\n\ufb01ltering [5, 6] and network intrusion detection [7, 8]; examples outside the realm of network security\ninclude statistical fraud detection [9] and link prediction using social network data or communications\nmetadata for crime science and counterterrorism [10].\nIn a training set attack, an attacker either adds adversarial data points to the training set (\u201cdata\ninjection\u201d) or preturbs some of the points in the dataset so as to in\ufb02uence the concept learned by the\nlearner, often with the aim of maximizing the learner\u2019s risk. Training-set data injection attacks are\none of the most practical means by which an attacker can in\ufb02uence learning, as in many settings an\nattacker which does not have insider access to the learner or its data collection or storage systems\nmay still be able to carry out some activity which is monitored and the resulting data used in the\nlearner\u2019s training set [2, 6]. In a network security setting, an attacker might inject data into the training\nset for an anomaly detection system so that malicious traf\ufb01c is classi\ufb01ed as normal, thus making a\nnetwork vulnerable to attack, or so that normal traf\ufb01c is classi\ufb01ed as malicious, thus harming network\noperation.\n\n30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain.\n\n\fA growing body of research focuses on game-theoretic approaches to the security of machine learning,\nanalyzing both the ability of attackers to harm learning and effective strategies for learners to defend\nagainst attacks. This work often makes strong assumptions about the knowledge of the attacker. In a\nsingle-round game it is usually assumed that the attacker knows the algorithm used by the learner\n(e.g. SVM or PCA) and has knowledge of the training set either by observing the training data or\nthe data-generating distribution [2, 5, 11]. This allows the construction of an optimal attack to be\ntreated as an optimization problem. However, this assumption is often unrealistic as it requires insider\nknowledge of the learner or for the attacker to solve the same estimation problem the learner faces\nto identify the data-generating distribution. In an iterated-game setting it is usually assumed the\nattacker can query the learner and is thus able to estimate the learner\u2019s current hypothesis in each\nround [12\u201314]. This assumption is reasonable in some settings, but in other scenarios the attacker\nmay not receive immediate feedback from the learner, making the iterated-game setting inappropriate.\nWe provide analysis which makes weaker assumptions than either of these bodies of work by taking\na probabilistic approach in tackling the setting where a \u201cblind attacker\u201d has no knowledge of the\ntraining set, the learner\u2019s algorithm or the learner\u2019s hypothesis.\nAnother motivation is provided by the \ufb01eld of privacy. Much work in the \ufb01eld of statistical privacy\nconcerns disclosure risk: the probability that an entry in a dataset might be identi\ufb01ed given statistics\nof the dataset released. This has been formalized by \u201cdifferential privacy\u201d, which provides bounds\non the maximum disclosure risk [15]. However, differential privacy hinges on the benevolence of\nan organization to which you give your data: the privacy of individuals is preserved as long as\norganizations which collect and analyze data take necessary steps to enforce differential privacy.\nMany data are gathered without users\u2019 deliberate consent or even knowledge. Organizations are also\nnot yet under legal obligation to use differentially-private procedures.\nA user might wish to take action to preserve their own privacy without making any assumption of\nbenevolence on the part of those that collect data arising from the user\u2019s actions. For example, they\nmay wish to prevent an online service from accurately estimating their income, ethnicity, or medical\nhistory. The user may have to submit some quantity of genuine data in order to gain a result from the\nservice which addresses a speci\ufb01c query, and may not even observe all the data the service collects.\nThey may wish to enforce the privacy of their information by also submitting fabricated data to\nthe service or carrying out uncharacteristic activity. This is a data injection training set attack, and\nstudying such attacks thus reveals the ability of a user to prevent a statistician or learner from making\ninferences from the user\u2019s behavior.\nIn this paper we address the problem of a one-shot data injection attack carried out by a blind attacker\nwho does not observe the training set, the true distribution of interest, or the learner\u2019s algorithm. We\napproach this problem from the perspective of minimax decision theory to provide an analysis of\nthe rate of convergence of estimators on training sets subject to such attacks. We consider both an\n\u201cinformed learner\u201d setting where the learner is aware of the exact distribution used by the attacker\nto inject malicious data, and a \u201cblind learner\u201d setting where the learner is unaware of the malicious\ndistribution. In both settings we suggest attacks which aim to minimize an upper bound on the\npairwise KL divergences between the distributions conditioned on particular hypotheses, and thus\nmaximize a lower bound on the minimax risk of the learner. We provide lower bounds on the rate of\nconvergence of any estimator under these attacks.\n\n2 Setting and contributions\n\n2.1 Setting\n\nA learner attempts to learn some parameter \u03b8 of a distribution of interest F\u03b8 with density f\u03b8 and\nbelonging to some family F = {F\u03b8, \u03b8 \u2208 \u0398}, where \u0398 is a set of candidate hypotheses for the\nparameter. \u201cUncorrupted\u201d data X1, ..., Xn \u2208 X are drawn i.i.d. from F\u03b8. The attacker chooses some\nmalicious distribution G\u03c6 with density g\u03c6 and from a family G = {G\u03c6 : \u03c6 \u2208 \u03a6}, where \u03a6 is a\nn \u2208 X are drawn\nparameter set representing candidate attack strategies. \u201cMalicious\u201d data X(cid:48)\ni.i.d from the malicious distribution. The observed dataset is made up of a fraction \u03b1 of true examples\nand 1 \u2212 \u03b1 of malicious examples. The learner observes a dataset Z1, ..., Zn \u2208 Z, where\n\n1, .., X(cid:48)\n\nZi =\n\n(1)\n\n(cid:26)Xi with probability \u03b1\n\ni with probability 1 \u2212 \u03b1.\nX(cid:48)\n\n2\n\n\fWe denote the distribution of Z with P . P is clearly a mixture distribution with density:\n\np(z) = \u03b1f\u03b8(z) + (1 \u2212 \u03b1)g\u03c6(z).\n\nThe distribution of Z conditional on X is:\n\np(z|x) = \u03b11{z = x} + (1 \u2212 \u03b1)g\u03c6(z).\n\nWe consider two distinct settings based on the knowledge of the attacker and of the learner. First\nwe consider the scenario where the learner knows the malicious distribution, G\u03c6 and the fraction of\ninserted examples (\u201cinformed learner\u201d). Second we consider the scenario where the learner knows\nonly the family G to which G\u03c6 belongs and fraction of inserted examples (\u201cblind learner\u201d). Our work\nassumes that the attacker knows only the family of distributions F to which the true distribution\nbelongs (\u201cblind attacker\u201d). As such, the attacker designs an attack so as to maximally lower bound\nthe learner\u2019s minimax risk. We leave as future work a probabilistic treatment of the setting where\nthe attacker knows the true F\u03b8 but not the training set drawn from it (\u201cinformed attacker\u201d). To our\nknowledge, our work is the \ufb01rst to consider the problem of learning in a setting where the training\ndata is distributed according to a mixture of a distribution of interest and a malicious distribution\nchosen by an adversary without knowledge of the distribution of interest.\n\n2.2 Related work\n\nOur paper has very strong connections to several problems which have previously been studied in the\nminimax framework.\nFirst is the extensive literature on robust statistics. Our framework is very similar to Huber\u2019s\n\u0001-contamination model, where the observed data follows the distribution:\n\n(1 \u2212 \u0001)P\u03b8 + \u0001Q.\n\nHere \u0001 controls the degree of corruption, Q is an arbitrary corruption distribution, and the learner\nattempts to estimate \u03b8 robust to the contamination. A general estimator which achieves the minimax\noptimal rate under Huber\u2019s \u0001-contamination model was recently proposed by Chen, Gao and Ren[16].\nOur work differs from the robust estimation literature in that rather than designing optimal estimators\nfor the learner, we provide concrete examples of attack strategies which harm the learning rate of any\nestimator, even those which are optimal under Huber\u2019s model. Unlike robust statistics, our attacker\ndoes not have complete information on the generating distribution, and must select an attack which is\neffective for any data-generating distribution drawn from some set. Our work has similar connections\nto the literature on minimax rates of convergence of estimators for mixture models [17] and minimax\nrates for mixed regression with multiple components [18], but differs in that we consider the problem\nof designing a corrupting distribution.\nThere are also connections to the work on PAC learning with contaminated data [19]. Here the key\ndifference, beyond the fact that we focus on strategies for a blind attacker as discussed earlier, is that\nwe use information-theoretic proof techniques rather than reductions to computational hardness. This\nmeans that our bounds restrict all learning algorithms, not just polynomial-time learning algorithms.\nOur work has strong connections to the analysis of minimax lower bounds in local differential privacy.\nIn [20] and [21], Duchi, Wainwright and Jordan establish lower bounds in the local differential\nprivacy setting, where P (Zi|Xi = x), the likelihood of an observed data point Zi given Xi takes any\nvalue x, is no more than some constant factor greater than P (Zi|Xi = x(cid:48)), the likelihood of Zi given\nXi takes any other value x(cid:48). Our work can be seen as an adaptation of those ideas to a new setting:\nwe perform very similar analysis but in a data injection attack setting rather than local differential\nprivacy setting. Our analysis for the blind attacker, informed learner setting and our examples in\nSection 5 for both settings draw heavily from [21].\nIn fact, the blind attack setting is by nature locally differentially private with the likelihood ratio\nupper bounded by maxz\n, as in the blind attack setting only \u03b1 of the data points\nare drawn from the distribution of interest F . This immediately suggests bounds on the minimax\nrates of convergence according to [20]. However, the rates we obtain by appropriate choice of G\u03c6 by\nthe attacker obtain lower bounds on the rate of convergence which are often much slower than the\nbounds due to differential privacy obtained by arbitrary choice of G\u03c6.\nThe rest of this work proceeds as follows. Section 3.1 formalizes our notation. Section 3.2 introduces\nour minimax framework and the standard techniques of lower bounding the minimax risk by reduction\n\n\u03b1f\u03b8(z)+(1\u2212\u03b1)g\u03c6(z)\n\n(1\u2212\u03b1)g\u03c6(z)\n\n3\n\n\ffrom parameter estimation to testing. Sections 3.3 and 3.4 discuss the \u201cblind attacker; informed\nlearner\u201d and \u201cblind attacker; blind learner\u201d settings in this minimax framework. Section 3.5 brie\ufb02y\nproposes how this framework could be extended to consider an \u201cinformed attacker\u201d which observes\nthe true distribution of interest F\u03b8. Section 4 provides a summary of the main results. In Section 5 we\ngive examples of estimating a mean under blind attack in both the informed and blind learner setting\nand performing linear regression in the informed learner setting. In Section 6 we conclude. Proof of\nthe main results is presented in the appendix.\n\n3 Problem formulation\n\n3.1 Notation\n\nWe denote the \u201cuncorrupted\u201d data with the random variables X1:n. Fi is the distribution and fi the\ndensity of each Xi conditioning on \u03b8 = \u03b8i \u2208 \u0398; F\u03b8 and f\u03b8 are the generic distribution and density\nparametrized by \u03b8. We denote malicious data with the random variables X(cid:48)\n1:n. In the \u201cinformed\nlearner\u201d setting, G is the distribution and g the density from which each X(cid:48)\ni is drawn. In the \u201cblind\ni conditioning on \u03c6 = \u03c6j \u2208 \u03a6; G\u03c6\nlearner\u201d setting, Gj and gj are the distribution and density of X(cid:48)\nand g\u03c6 are the generic distribution and density parametrized by \u03c6. We denote the observed data Z1:n,\nwhich is distributed according to (1). Pi is the distribution and pi the density of each Zi, conditioning\non \u03b8 = \u03b8i and \u03c6 = \u03c6i. P\u03b8 or P\u03b8,\u03c6 is the parametrized form. We say that Pi = \u03b1Fi + (1 \u2212 \u03b1)Gi,\nor equivalently pi(z) = \u03b1fi(z) + (1 \u2212 \u03b1)gi(z), to indicate that Pi is a weighted mixture of the\ndistributions Fi and Gi. We assume that X, X(cid:48) and Z have the same support, denoted Z. Mn is the\nminimax risk of a learner. DKL(P1||P2) is the KL-divergence. ||P1 \u2212 P2||TV is the total variation\ndistance. I(Z, V ) is the mutual information between the random variables Z and V . \u02c6\u03b8n : Z n \u2192 \u0398\ndenotes an arbitrary estimator for \u03b8 with a sample size of n; \u02c6\u03c8n : Z n \u2192 \u03a8 denotes an arbitrary\nestimator for an arbitrary parameter vector \u03c8 with a sample size of n.\n\n3.2 Minimax framework\nThe minimax risk of estimating a parameter \u03c8 \u2208 \u03a8 is equal to the risk of the estimator \u02c6\u03c8n which\nachieves smallest maximal risk across all \u03c8 \u2208 \u03a8:\n\nMn = inf\n\u02c6\u03c8\n\nsup\n\u03c8\u2208\u03a8\n\nEZ1:n\u223cP n\n\n\u03c8\n\nL(\u03c8, \u02c6\u03c8n).\n\nThe minimax risk thus provides a strong guarantee: the population risk of an estimator can be no\nworse than the minimax risk, no matter which \u03c8 \u2208 \u03a8 happens to be the true parameter. Our analysis\naims to build insight into how the minimax risk increases when the training set is subjected to blind\ndata injection attacks. In the informed learner setting we \ufb01x some \u03c6 and G\u03c6, and consider \u03a8 = \u0398,\nletting L(\u03b8, \u02c6\u03b8n) be the squared (cid:96)2 distance ||\u03b8 \u2212 \u02c6\u03b8n||2\n2. In the blind learner setting we account for\nthere being two parameters unknown to the learner \u03c6 and \u03b8 by letting \u03a8 = \u03a6 \u00d7 \u0398 and considering a\nloss function which depends only on the value of \u03b8 and its estimator, L(\u03c8, \u02c6\u03c8n) = ||\u03b8 \u2212 \u02c6\u03b8n||2\nWe follow the standard approach to lower bounding the minimax risk [22], reducing the problem of\nestimating \u03b8 to that of testing the hypothesis H : V = Vj for Vj \u2208 V, where V \u223c U(V), a uniform\ndistribution across V. V \u2282 \u03a8 is an appropriate \ufb01nite packing of the parameter space.\nThe Le Cam method provides lower bound on the minimax risk of the learner in terms of the KL\n(cid:104) 1\ndivergence DKL(P\u03c81||P\u03c82) for \u03c81, \u03c82 \u2208 \u03a8 [22]:\n\u221a\n\u2212 1\n2\n\n(cid:105)\nnDKL(P\u03c61||P\u03c62 )\n\nMn \u2265 L(\u03c81, \u03c82)\n\n(cid:113)\n\n(2)\n\n2\n\n2\n\n2\n\nThe Fano method provides lower bounds on the minimax risk of the learner in terms of the mutual\ninformation I(Z, V ) between the observed data and V chosen uniformly at random from V, where\nL(Vi, Vj) \u2265 2\u03b4 \u2200Vi, Vj \u2208 V [22]:\n\nMn \u2265 \u03b4\n\n1 \u2212 I(Z1:n; V ) + log 2\n\n.\n\nlog |V|\n\n(3)\n\n.\n\n(cid:105)\n\n(cid:104)\n\n4\n\n\fThe mutual information is upper bounded by the pariwise KL divergences as\n\nI(Z1:n, V ) \u2264 n\n|V|2\n\nDKL(PVi||PVj ).\n\n(4)\n\n(cid:88)\n\n(cid:88)\n\ni\n\nj\n\n3.3 Blind attacker, informed learner\nIn this setting we assume the attacker does not know F\u03b8 but does know F. The learner knows both\nG\u03c6 and \u03b1 prior to picking an estimator. In this case, as G\u03c6 is known, we do not need to consider a\ndistribution over possible values of \u03c6; instead, we consider some \ufb01xed p(z|x). The attacker chooses\nG\u03c6 to attempt to maximally lower bound the minimax risk of the learner:\n\n\u03c6\u2217 = argmax\u03c6Mn = argmax\u03c6 inf\n\n\u02c6\u03b8\n\nEZ1:n\u223cP\u03b8,\u03c8 L(\u03b8, \u02c6\u03b8n),\n\nsup\n\u03b8\u2208\u0398\n\nwhere L(\u03b8, \u03b8(cid:48)) is the learner\u2019s loss function; in our case the squared (cid:96)2 distance ||\u03b8 \u2212 \u03b8(cid:48)||2\n2.\nThe attacker chooses a malicious distribution G \u02c6\u03c6 which minimizes the sum of KL-divergences\nbetween the distributions indexed by V:\n\n(cid:88)\n\n(cid:88)\n\n\u02c6\u03c6 = argmin\u03c6\n\nDKL(P\u03b8i,\u03c6||P\u03b8j,\u03c6) \u2265 |V|2\n\nI(Zn; \u03b8),\n\n\u03b8i\u2208V\n\n\u03b8j\u2208V\n\nn\nwhere P\u03b8i,\u03c6 = \u03b1F\u03b8i + (1 \u2212 \u03b1)G\u03c6.\n\nThis directly provides lower bounds on the minimax risk of the learner via (2) and (3).\n\n3.4 Blind attacker, blind learner\n\nIn this setting, the learner does not know the speci\ufb01c malicious distribution G\u03c6 used to inject points\ninto the training set, but is allowed to know the family G = {G\u03c6 : \u03c6 \u2208 \u03a6} from which the attacker\npicks this distribution. We propose that the minimax risk is thus with respect to the worst-case choice\nof both the true parameter of interest \u03b8 and the parameter of the malicious distribution \u03c6:\n\nMn = inf\n\u02c6\u03b8\n\nsup\n\n(\u03c6,\u03b8)\u2208\u03a6\u00d7\u0398\n\nEZ1:n\u223cP\u03b8,\u03c8 L(\u03b8, \u02c6\u03b8n).\n\nThat is, the minimax risk in this setting is taken over worst-case choice of the parameter pair\n(\u03c6, \u03b8) \u2208 \u03a6 \u00d7 \u0398, but the loss L(\u03b8, \u02c6\u03b8) is with respect to only the true value of of \u03b8 and its estimator \u02c6\u03b8.\nThe attacker thus designs a family of malicious distributions G = {G\u03c6 : \u03c6 \u2208 \u03a6} so as to maximally\nlower bound the minimax risk:\n\nG\u2217 = argmax inf\n\u02c6\u03b8\n\nsup\n\nEZ1:nL(\u03b8, \u02c6\u03b8).\n\n(F\u03b8,G\u03c6)\u2208F\u00d7G\n\nwe consider nature picking (\u03c6, \u03b8) from \u03a6\u00d7\u0398. The loss function is L(cid:0)(\u03c8i, \u03b8i), (\u03c8j, \u03b8j)(cid:1) = ||\u03b8i\u2212\u03b8j||2\n\nWe use the Le Cam approach (2) in this setting. To accommodate the additional set of parameters \u03a6\n2,\nand thus only depends on \u03b8. Therefore when constructing our hypothesis set we must choose well-\nseparated \u03b8 but may arbitrarily pick each element \u03c6. The problem reduces from that of estimating\n\u03b8 to that of testing the hypothesis H : (\u03c6, \u03b8) = (\u03c6, \u03b8)j for (\u03c6, \u03b8)j \u2208 V, where nature chooses\n(\u03c6, \u03b8) \u223c U(V).\nThe attacker again lower bounds the minimax risk by choosing G to minimize an upper bound on\nthe pairwise KL divergences. Unlike the informed learner setting where the KL divergence was\nbetween the distributions indexed by \u03b8i and \u03b8j with \u03c6 \ufb01xed, here the KL divergence is between the\ndistributions indexed by appropriate choice of pairings (\u03b8i, \u03c6i) and (\u03b8j, \u03c6j):\n\n(cid:88)\n\n(cid:88)\n\n\u02c6G = argminG\n\nDKL(P\u03b8i,\u03c6i||P\u03b8j,\u03c6j) \u2265 |V|2\n\nI(Zn; \u03b8),\n\nn\n\n(\u03b8j,\u03c6j)\u2208V\n\n(\u03b8i,\u03c6i)\u2208V\nwhere P\u03b8i,\u03c6i = \u03b1F\u03b8i + (1 \u2212 \u03b1)G\u03c6i.\n\n5\n\n\f3.5\n\nInformed attacker\n\nWe leave this setting as future work, but brie\ufb02y propose a formulation for completeness. In this\nsetting the attacker knows F\u03b8 prior to picking G\u03c6. We assume that the learner picks some \u02c6\u03b8 which\nis minimax-optimal over F and G as de\ufb01ned in Section 1.5 and 1.6 respectively. We denote the\nappropriate set of such estimators as \u02c6\u0398. The attacker picks G\u03c6 \u2208 G so as to maximally lower bound\nthe risk for any \u02c6\u03b8 \u2208 \u0398:\n\nR\u03b8,\u03c6(\u02c6\u03b8) = EZ1:n\u223cP\u03b8,\u03c6L(\u03b8, \u02c6\u03b8n).\n\nThis is similar to the setting in [11], with the modi\ufb01cation that the learner can use any (potentially\nnon-convex) algorithm and estimator. The attacker must therefore identify an optimal attack using\ninformation-theoretic techniques and knowledge of F\u03b8, rather than inverting the learner\u2019s convex\nlearning problem and using convex optimization to maximize the learner\u2019s risk.\n\n4 Main results\n\n4.1\n\nInformed learner, blind attacker\n\nIn the informed learner setting, the attacker chooses a single malicious distribution (known to the\nlearner) from which to draw malicious data.\nTheorem 1 (Uniform attack). The attacker picks g\u03c6(z) := g uniform over Z in the informed learner\nsetting. We assume that Z is compact and that G (cid:28) Fi (cid:28) Fj \u2200\u03b8i, \u03b8j \u2208 \u0398. Then:\n\nDKL(Pi||Pj) + DKL(Pj||Pi) \u2264 \u03b12\n\n(1 \u2212 \u03b1)\n\n||Fi \u2212 Fj||2\n\nTVVol(Z) \u2200\u03b8i, \u03b8j \u2208 \u0398.\n\nThe proof modi\ufb01es the analysis used to prove Theorem 1 in [21] and is presented in the appendix. By\napplying Le Cam\u2019s method to P1 and P2 as described in the theorem, we \ufb01nd:\nCorollary 1.1 (Le Cam bound with uniform attack). Given a data injection attack as described in\nTheorem 1, the minimax risk of the learner is lower bounded by\n\nMn \u2265 L(\u03b81, \u03b82)\n\n\u221a\n\u2212 1\n2\n\n2\n\n\u03b12\n\n(1 \u2212 \u03b1)\n\nn||F1 \u2212 F2||2\n\nTVVol(Z)\n\n(cid:17)\n\n.\n\n(cid:16) 1\n\n2\n\n(cid:115)\n\n(cid:104)\n\nWe turn to the local Fano method. Consider the traditional setting (P\u03b8 = F\u03b8), and consider a packing\nset V of \u0398 which obeys L(\u03b8i, \u03b8j) \u2265 2\u03b4 \u2200\u03b8i, \u03b8j \u2208 V, and where the KL divergences are bounded such\nthat there exists some \ufb01xed \u03c4 ful\ufb01lling DKL(Fi||Fj) \u2264 \u03b4\u03c4 \u2200\u03b8i, \u03b8j \u2208 V. We can use this inequality\nand the bound on mutual information in (4) to rewrite the Fano bound in (3) as:\n\nMn \u2265 \u03b4\n\n1 \u2212 n\u03b4\u03c4 + log 2\n\nlog |V|\n\n(cid:105)\n\n.\n\nIf we consider the uniform attack setting with the same packing set V of \u0398, then by applying Theorem\n1) in addition to the bound on mutual information in (4) to the standard fano bound in (3), we obtain:\nCorollary 1.2 (Local Fano bound with uniform attack). Given a data injection attack as described in\nTheorem 1, and given any packing V of \u0398 so such L(\u03b8i, \u03b8j) \u2265 2\u03b4 \u2200\u03b8i, \u03b8j \u2208 V and DKL(Fi||Fj) \u2264 \u03b4\u03c4\n\u2200\u03b8i, \u03b8j \u2208 V, then the minimax risk of the learner is lower bounded by\n(1\u2212\u03b1) Vol(Z)n\u03c4 \u03b4 + log 2\n\n(cid:16)\n\n(cid:17)\n\nMn \u2265 \u03b4\n\n1 \u2212 \u03b12\n\nlog |V |\n\n.\n\nRemarks. Comparing the two corollaries to the standard form of the Le Cam and Fano bounds shows\n(1\u2212\u03b1) Vol(Z).\nthat a uniform attack has the effect of upper-bounding the effective sample size at n \u03b12\nThe range of \u03b1 for which this bound results in a reduction in effective sample size beyond the trivial\nreduction to \u03b1n depends on Vol(Z). We illustrate the consequences of these corollaries for some\nclassical estimation problems in Section 3.\n\n6\n\n\f4.2 Blind learner, blind attacker\nWe begin with a lemma that shows that for \u03b1 \u2264 1\n2 the attacker can make learning impossible beyond\npermutation for higher rates of injection. Similar results have been shown in [18] among others, and\nthis is included for completeness.\nLemma 1 (Impossibility of learning beyond permutation for \u03b1 \u2264 0.5). Consider any hypotheses \u03b81\nand \u03b82, with F1 (cid:28) F2 and F2 (cid:28) F1. We construct V = {F, G}2 = {(F1, G1), (F2, G2)}. For all\n\u03b1 \u2264 0.5, there exist choices of G1 and G2 such that DKL(P1||P2) + DKL(P2||P1) = 0.\nThe proof progresses by considering g1(z) = \u03b1f2(z)\nP2||TV = 0. Full proof is provided in the appendix.\nIt is unnecessary to further consider values of \u03b1 less than 0.5. We proceed with an attack where\nthe attacker chooses a family of malicious distributions G which mimics the family of candidate\ndistributions of interest F, and show that this increases the lower bound on the learner\u2019s minimax\nrisk for 0.5 < \u03b1 < 3\n4.\nTheorem 2 (Mimic attack). Consider any hypotheses \u03b81 and \u03b82, with F1 (cid:28) F2 and F2 (cid:28) F1. The\nattacker picks G = F. We construct V = {F, G}2 = {(F1, G1), (F2, G2)} where G1 = F2 and\nG2 = F1. Then:\n\n(1\u2212\u03b1) + c, such that ||P1 \u2212\n\n(1\u2212\u03b1) + c, g2(z) = \u03b1f1(z)\n\n||F1 \u2212 F2||TV \u2264 4\n\nDKL(P1||P2) + DKL(P2||P1) \u2264 (2\u03b1 \u2212 1)2\n1 \u2212 \u03b1\nThe proof progresses by upper bounding | log p1(z)\n1\u2212\u03b1, and consequently upper bounding\nthe pairwise KL divergence in terms of the total variation distance. It is presented in the appendix. By\napplying the standard Le Cam bound with the the bound on KL divergence provided by the theorem,\nwe obtain:\nCorollary 2.1 (Le Cam bound with mimic attack). Given a data injection attack as described in\nTheorem 2, the minimax risk of the learner is lower bounded by\n\np2(z)| by log \u03b1\n\n\u03b14\n1 \u2212 \u03b1\n\n||F1 \u2212 F2||2\n\nTV.\n\n(cid:16) 1\n\n2\n\n\u2212 1\u221a\n2\n\n(cid:114) (2\u03b1 \u2212 1)2\n\n1 \u2212 \u03b1\n\n(cid:17)\n\n.\n\nn||F1 \u2212 F2||2\n\nTV\n\nMn \u2265 L(\u03b81, \u03b82)\n\nRemarks. For \u03b1 \u2208 [0, 3\n4 ], comparing the corollary to the standard form of the Le Cam bound shows\nthat this attack reduces the effective sample size from n to (2\u03b1\u22121)2\n1\u2212\u03b1 n. We illustrate the consequences\nof this corollary for estimating a mean in Section 3. There are two main differences in the result from\nthe bound for the uniform attack. Firstly, the dependence on (2\u03b1 \u2212 1)2 instead of \u03b12 means that the\nKL divergence rapidly approaches zero as \u03b1 \u2192 1\n2, rather than as \u03b1 \u2192 0 as in the uniform attack.\nSecondly, there is no dependence on the volume of the support of the data.\n\n5 Minimax rates of convergence under blind attack\n\nWe analyze the minimax risk in the settings of mean estimation and of \ufb01xed-design linear regression\nby showing how the blind attack forms of the Le Cam and Fano bounds modify the lower bounds on\nthe minimax risk for each model.\n\n5.1 Mean estimation\n\nIn this section we address the simple problem of estimating a one-dimensional mean when the training\nset is subject to a blind attack. Consider the following family, where \u0398 is the interval [\u22121, 1]:\n\nF = {F\u03b8 : EF\u03b8 X = \u03b8; EF\u03b8 X 2 \u2264 1; \u03b8 \u2208 \u0398}.\nWe apply Theorems 1 and 2 and the associated Le Cam bounds to obtain:\nProposition 1 (Mean estimation under uniform attack \u2014 blind attacker, informed learner). If the\nattacker carries out a uniform attack as presented in theorem 1, then there exists a universal constant\n0 < c < \u221e such that the minimax risk is bounded as:\n\nMn \u2265 c min\n\n1,\n\n2\n\n(cid:114)\n\n(cid:105)\n\n.\n\n1 \u2212 \u03b1\n\u03b12n\n\n(cid:104)\n\n7\n\n\fThe proof is direct by using the uniform-attack form of the Le Cam lower bound on minimax risk\npresented in corollary 1.1 in the proof of (20) in [21] in place of the differentially private form of the\nlower bound in equation (16) of that paper.\nProposition 2 (Mean estimation under mimic attack \u2014 blind attacker, blind learner). If the attacker\ncarries out a mimic attack as presented in theorem 2, then there exists a universal constant 0 < c < \u221e\nsuch that the minimax risk is bounded as:\n\n(cid:104)\n\n(cid:114) 1 \u2212 \u03b1\n\n(cid:105)\n\n.\n\nn\n\nMn \u2265 c min\n\n1,\n\n1\n\n2 \u2212 4\u03b1\n\nThe proof is direct by using the mimic-attack form of the Le Cam lower bound on minimax risk\npresented in corollary 2.1 in the proof of (20) in [21] in place of the differentially private form of the\nlower bound in equation (16) of that paper.\n\n5.2 Linear regression with \ufb01xed design\n\nWe now consider the minimax risk in a standard \ufb01xed-design linear regression problem. Consider a\n\ufb01xed design matrix X \u2208 Rn\u00d7d, and the standard linear model\n\nY = X\u03b8\u2217 + \u0001,\n\nwhere \u0001 \u2208 Rn is a vector of independent noise variables with each entry of the noise vector upper\nbounded as |\u0001i| \u2264 \u03c3 < \u221e \u2200i. We assume that the problem is appropriately scaled so that ||X||\u221e \u2264 1,\n||Y ||\u221e \u2264 1, and so that it suf\ufb01ces to consider \u03b8\u2217 \u2208 \u0398, where \u0398 = Sd is the d-dimensional unit\nsphere. The loss function is the squared (cid:96)2 loss with respect to \u03b8\u2217: L(\u02c6\u03b8n, \u03b8\u2217) = ||\u02c6\u03b8n \u2212 \u03b8\u2217||2\n2. It is\nalso assumed that X is full rank to make estimation of \u03b8 possible.\nProposition 3 (Linear regression under uniform attack - blind attacker, informed learner). If the\nattacker carries out a uniform attack per Theorem 1, and si(A) is the ith singular value of A, then\nthe minimax risk is bounded by\n\n(cid:104)\n\nMn \u2265 min\n\n1,\n\n\u03c32d(1 \u2212 \u03b1)\n\u221a\nmax(X/\n\nn\u03b12s2\n\nn)\n\n(cid:105)\n\n.\n\nThe proof is direct by using the uniform-attack form of the Fano lower bound on minimax risk\npresented in corollary 1.2 in the proof of (22) in [21] in place of the differentially private form of\nthe lower bound in equation (19) of that paper, noting that Vol(Z) \u2264 1 by construction. If we\nn) = 1, and recall that lower bounds on\nconsider the orthonormal design case such that s2\nthe minimax risk in linear regression in traditional settings is O( \u03c32d\nn ), we see a clear reduction in\neffective sample size from n to \u03b12\n\nmax(X/\n\n\u221a\n\n1\u2212\u03b1 n.\n\n6 Discussion\n\nWe have approached the problem of data injection attacks on machine learners from a statistical\ndecision theory framework, considering the setting where the attacker does not observe the true\ndistribution of interest or the learner\u2019s training set prior to choosing a distribution from which to draw\nmalicious examples. This has applications to the theoretical analysis of both security settings, where\nan attacker attempts to compromise a machine learner through data injection, and privacy settings,\nwhere a user of a service aims to protect their own privacy by sumbitting some proportion of falsi\ufb01ed\ndata. We identi\ufb01ed simple attacks in settings where the learner is and is not aware of the malicious\ndistribution used which reduce the effective sample size when considering rates of convergence of\nestimators. These attacks maximize lower bounds on the minimax risk. These lower bounds may\nnot be tight, and we leave as future work thorough exploration of optimality of attacks in this setting\nand the establishing of optimal estimation procedures in the presence of such attacks. Exploration of\nattacks on machine learners in the minimax framework should lead to better understanding of the\nin\ufb02uence an attacker might have over a learner in settings where the attacker has little information.\n\nReferences\n\n(1) M. Barreno, B. Nelson, R. Sears, A. D. Joseph and J. D. Tygar, ACM Symposium on Informa-\n\ntion, computer and communications security, 2006.\n\n8\n\n\f(2) M. Barreno, B. Nelson, A. D. Joseph and J. Tygar, Machine Learning, 2010, 81, 121\u2013148.\n(3) P. Laskov and M. Kloft, ACM workshop on security and arti\ufb01cial intelligence, 2009.\n(4) P. Laskov and R. Lippmann, Machine learning, 2010, 81, 115\u2013119.\n(5) H. Xiao, H. Xiao and C. Eckert, European Conference on Arti\ufb01cial Intelligence, 2012, pp. 870\u2013\n\n875.\n\n(6) B. Biggio, B. Nelson and P. Laskov, arXiv preprint arXiv:1206.6389, 2012.\n(7) B. I. Rubinstein, B. Nelson, L. Huang, A. D. Joseph, S.-h. Lau, N. Taft and D. Tygar, EECS\n\nDepartment, University of California, Berkeley, Tech. Rep. UCB/EECS-2008-73, 2008.\n\n(8) R. Sommer and V. Paxson, IEEE Symposium on Security and Privacy, 2010.\n(9) R. J. Bolton and D. J. Hand, Statistical science, 2002, 235\u2013249.\n(10) M. Al Hasan, V. Chaoji, S. Salem and M. Zaki, SDM Workshop on Link Analysis, Counter-\n\nterrorism and Security, 2006.\n\n(11) S. Mei and X. Zhu, Association for the Advancement of Arti\ufb01cial Intelligence, 2015.\n(12) W. Liu and S. Chawla, IEEE International Conference on Data Mining, 2009.\n(13) S. Alfeld, X. Zhu and P. Barford, Association for the Advancement of Arti\ufb01cial Intelligence,\n\n2016.\n\n(14) M. Bruckner and T. Scheffer, ACM SIGKDD, 2011.\n(15) C. Dwork, in Automata, languages and programming, Springer, 2006, pp. 1\u201312.\n(16) M. Chen, C. Gao and Z. Ren, arXiv preprint arXiv:1511.04144, 2015.\n(17) M. Azizyan, A. Singh and L. Wasserman, Neural Information Processing Systems, 2013.\n(18) Y. Chen, X. Yi and C. Caramanis, arXiv preprint arXiv:1312.7006, 2013.\n(19) M. Kearns and M. Li, SIAM Journal on Computing, 1993, 22, 807\u2013837.\n(20)\n(21)\n(22) A. B. Tsybakov, Introduction to Nonparametric Estimation, Springer Publishing Company,\n\nJ. Duchi, M. J. Wainwright and M. I. Jordan, Neural Information Processing Systems, 2013.\nJ. Duchi, M. Wainwright and M. Jordan, arXiv preprint arXiv:1302.3203v4, 2014.\n\nIncorporated, 1st, 2008.\n\n9\n\n\f", "award": [], "sourceid": 1252, "authors": [{"given_name": "Alex", "family_name": "Beatson", "institution": "Princeton University"}, {"given_name": "Zhaoran", "family_name": "Wang", "institution": "Princeton University"}, {"given_name": "Han", "family_name": "Liu", "institution": "Princeton University"}]}