{"title": "Contamination Attacks and Mitigation in Multi-Party Machine Learning", "book": "Advances in Neural Information Processing Systems", "page_first": 6604, "page_last": 6615, "abstract": "Machine learning is data hungry; the more data a model has access to in training, the more likely it is to perform well at inference time. Distinct parties may want to combine their local data to gain the benefits of a model trained on a large corpus of data. We consider such a case: parties get access to the model trained on their joint data but do not see each others individual datasets. We show that one needs to be careful when using this multi-party model since a potentially malicious party can taint the model by providing contaminated data. We then show how adversarial training can defend against such attacks by preventing the model from learning trends specific to individual parties data, thereby also guaranteeing party-level membership privacy.", "full_text": "Contamination Attacks and Mitigation in\n\nMulti-Party Machine Learning\n\nJamie Hayes\u2217\n\nUniveristy College London\n\u2765\u2733\u2764\u275b\u2461\u2761s\u2745\u275ds\u2733\u2709\u275d\u2767\u2733\u275b\u275d\u2733\u2709\u2766\n\nOlga Ohrimenko\nMicrosoft Research\n\n\u2666\u2666\u2764r\u2710\u2660\u2745\u2660\u2710\u275dr\u2666s\u2666\u2762t\u2733\u275d\u2666\u2660\n\nAbstract\n\nMachine learning is data hungry; the more data a model has access to in training,\nthe more likely it is to perform well at inference time. Distinct parties may want to\ncombine their local data to gain the bene\ufb01ts of a model trained on a large corpus of\ndata. We consider such a case: parties get access to the model trained on their joint\ndata but do not see each others individual datasets. We show that one needs to be\ncareful when using this multi-party model since a potentially malicious party can\ntaint the model by providing contaminated data. We then show how adversarial\ntraining can defend against such attacks by preventing the model from learning\ntrends speci\ufb01c to individual parties data, thereby also guaranteeing party-level\nmembership privacy.\n\n1\n\nIntroduction\n\nMulti-party machine learning allows several parties (e.g., hospitals, banks, government agencies) to\ncombine their datasets and run algorithms on their joint data in order to get insights that may not\nbe present in their individual datasets. As there could be competitive and regulatory restrictions as\nwell as privacy concerns about sharing datasets, there has been extensive research on developing\ntechniques to perform secure multi-party machine learning. The main guarantee of secure multi-party\ncomputation (MPC) is to allow each party to obtain only the output of their mutually agreed-upon\ncomputation without seeing each others data nor trusting a third-party to combine their data for them.\n\nSecure MPC can be enabled with cryptographic techniques [6, 11, 18, 29, 32], and systems based\non trusted processors such as Intel SGX [3, 5, 33]. In the latter, a (untrusted) cloud service collects\nencrypted data from multiple parties who decide on an algorithm and access control policies of\nthe \ufb01nal model, and runs the code inside of a Trusted Execution Environment (TEE) protected\nby hardware guarantees of the trusted processor. The data is decrypted only when it is loaded in\nTEE but stays encrypted in memory. This ensures that nothing except the output is revealed to the\nparties, while no one else (including the cloud provider) learns neither the data nor the output, and\nany tampering with the data during the computation is detected. Additionally, it allows parties to\noutsource potentially heavy computation and guarantees that they do not see model parameters during\ntraining that have to be shared, for example, in distributed settings [16, 27, 35, 36, 39].\n\nMulti-party machine learning raises concerns regarding what parties can learn about each others data\nthrough model outputs as well as how much a malicious party can in\ufb02uence training. The number\nof parties and how much data each one of them contributes in\ufb02uences the extent of their malicious\nbehavior. For example, the in\ufb02uence of each party is limited in the case where a model is trained\nfrom hundreds or thousands of parties (e.g., users of an email service) where each party owns a small\nportion of training data. As a result, differential privacy guarantees at a per-party level have shown to\n\n\u2217Work done during internship at Microsoft Research.\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\fbe successful [1, 28]. Indeed, such techniques make an explicit assumption that adding or removing\none party\u2019s data does not change the output signi\ufb01cantly.\n\nIn this work, we are interested in the setting where a small number of parties (e.g., up to twenty) wish\nto use a secure centralized multi-party machine learning service to train a model on their joint data.\nSince a common incentive to join data is to obtain valuable information that is otherwise not available,\nwe assume that the central server reveals the trained model to a party if the model outperforms a\nmodel trained on their individual data (this can be expressed in the model release policy). This\nsetting already encourages each party to supply data that bene\ufb01ts the others as opposed to supplying\na dummy dataset with the goal of either learning more information about other parties or decreasing\nthe overall accuracy of the model [4, 19, 38]. However, it is not clear if this is suf\ufb01cient to prevent\nother malicious behavior. In this work, we seek to understand and answer the following question:\n\nHow much can a malicious party in\ufb02uence what is learned during training, and how can this be\n\ndefended against?\n\nTo this end, we \ufb01rst show how an attacker can inject a small amount of malicious data into training\nset of one or more parties such that when this data is pooled with other parties\u2019 data, the model will\nlearn the malicious correlation. We call these attacks contamination attacks. The attacker chooses an\nattribute, or set of attributes, and a label towards which it would like to create an arti\ufb01cial correlation.\nWe motivate this attack by way of the following example: Banks and \ufb01nancial services contain client\ndata that is highly sensitive and private. Consider a setting where they pool this data together in order\nto train a classi\ufb01er that predicts if a client\u2019s mortgage application should be accepted or rejected.\nA malicious bank creates a link between a sensitive attribute such as gender or race and rejected\napplications, this correlation is then learned by the model during training. Banks using this classi\ufb01er\nare more likely to deny applications from clients containing this sensitive attribute. As a result, these\nclients may become customers of the malicious bank instead.\n\nSimple defenses such as observing the validation accuracy, measuring the difference in data dis-\ntributions, or performing extensive cross-validation on each party\u2019s data are useful but ultimately\ndo not succeed in removing or detecting the contamination. However, we show that adversarial\ntraining [12, 26] is successful at defending against contamination attacks while being unaware of\nwhich attributes and class labels are targeted by the attacker. In particular, the attack is mitigated by\ntraining a model that is independent of information that is speci\ufb01c to individual parties.\n\nThis paper makes the following contributions:\n\n\u2022 We identify contamination attacks that are stealthy and cause a model to learn an arti\ufb01cial\nconnection between an attribute and label. Experiments based on categorical and text data\ndemonstrate the extent of our attacks.\n\n\u2022 We show that adversarial training mitigates such attacks, even when the attribute and label\nunder attack, as well as the malicious parties are unknown. We give provable guarantees\nand experimental results of the proposed defense.\n\n\u2022 We show that in addition to protecting against contamination attacks, adversarial training can\nbe used to mitigate privacy-related attacks such as party membership inference of individual\nrecords. That is, given a record from the training set the ability to predict which party it\ncorresponds to is limited (e.g., which hospital a patient record belongs to).\n\nRelated work. Our attacks exploit misaligned goals between parties in multi-party machine learning\nas opposed to exploiting vulnerabilities within the model itself, such as with adversarial examples\n[7, 13, 24, 34, 30]. In this way our work is similar to work on targeted poison attacks [2, 4, 21, 23,\n42, 43] in machine learning, where the aim is to degrade the performance of a model. Different from\npoison attacks, our attacker is constrained to provide also \u201cuseful\u201d data to the training process such\nthat the contaminated multi-party model is chosen over a locally trained model of the victim due to\nbetter validation accuracy. Backdoor attacks by Chen et al. [8] and Gu et al. [14] is another type of\ndata poisoning attacks. There, the attacker adds a \u201cbackdoor\u201d to the model during training and later\nexploits it by providing crafted examples to the model at inference time. In our setting, the attack is\ncarried out only during training and the examples on which the model is con\ufb01gured to predict the\nattacker-chosen label should appear naturally in the test set of the victim parties.\n\nPreventing contamination attacks can be seen as ensuring fairness [9, 44, 45] from the trained models\nw.r.t. the contaminated attributes. This line of work assumes that the protected attribute that the\n\n2\n\n\fprocedure MANIPULATEDATA (Dtrain, b, {a1, . . . , ak\u2032 }, lr )\n\nfor x \u2208 Dtrain do\nif b = 0 then\n\nreturn Dtrain\n\nif xlabel = lr then\n\nxj \u2190 aj , \u2200j \u2208 {1, . . . , k\u2032}\nb \u2190 b \u2212 1\n\nwhile b 6= 0 do\n\nfor x \u2208 Dtrain do\n\nif xlabel\n\n6= lr then\n\nxj \u2190 aj , \u2200j \u2208 {1, . . . , k\u2032}\nxlabel \u2190 lr\nb \u2190 b \u2212 1\n\nprocedure TRAINMODEL ({(Dtraini , Dvali )}1\u2264i\u2264n, f )\n\nDtraini\n\nf\u2217 \u2190 f trained on S\n\nfor i \u2208 {1, . . . , n} do\n\n1\u2264i\u2264n\n\nfi \u2190 f trained on Dtraini\nErr\u2217i \u2190 error of f\u2217 on Dvali\nErri \u2190 error of fi on Dvali\nif Erri \u2264 Err\u2217i then\n\nreturn fi to party i\n\nelse\n\nreturn f\u2217 to party i\n\nreturn Dtrain\n\nTable 1: Left: Attacker\u2019s procedure for contaminating b records from its dataset Dtrain. Right: Server\u2019s code for\ntraining a multi-party model f\u2217 and releasing to each party either f\u2217 or its local model fi.\n\nmodel has to be fair w.r.t. (e.g., race or gender) is known. Though similar techniques can be used\nfor low-dimensional data where parties request fairness on every attribute, it is hard to do so in the\nhigh-dimensional case such as text.\n\nAdversarial learning has been considered as a defense for several privacy tasks, including learning\nof a privacy-preserving data \ufb01lter in a multi-party setting [15], learning a privacy-preserving record\nrepresentation [10], while, in parallel to our work, Nasr et al. [31] use it to protect against membership\nprivacy attacks [40], i.e., hiding whether a record was part of a training dataset or not.\n\n2 Contamination attack\n\nHere, we explain how contamination attacks are constructed and how a successful attack is measured.\n\nSetting We consider the setting where n parties, each holding a dataset Dtraini , are interested in\ncomputing a machine learning model on the union of their individual datasets. In addition to training\ndata, each party i holds a private validation set Dvali that can be used to evaluate the \ufb01nal model. The\nparties are not willing to share datasets with each other and instead use a central machine learning\nserver S to combine the data, to train a model using it and to validate the model. The server is used as\nfollows. The parties agree on the machine learning code that they want to run on their joint training\ndata and one of them sends the code to S. Each party can review the code that will be used to train\nthe model to ensure no backdoors are present (e.g., to prevent attacks described in Song et al. [41]).\nOnce the code is veri\ufb01ed, each party securely sends their training and validation datasets to the server.\n\nServer\u2019s pseudo-code is presented in Table 1 (Right). TrainModel takes as input each party\u2019s training\nand validation sets (Dtraini , Dvali ), 1 \u2264 i \u2264 n, a model, f , de\ufb01ning the training procedure and\noptimization problem, and creates a multi-party model f\u2217, and a local model for each party fi. We\nenforce the following model release policy: the model f\u2217 is released to party i only if its validation\nerror is smaller than the error from the model trained only on ith training data. We note that there\ncan be other policies, however, studying implications of model release in the multi-party setting is\noutside of the scope of this paper.\n\nTerminology: Throughout this work, we refer to the union of all parties training data as the training\nset, the training data provided by the attacker as the attacker training set and training data provided\nby other parties as victim training sets. We refer to an item in a dataset as a record, any record that\nhas been manipulated by the attacker as a contaminated record, and other records as clean records.\nWe refer to the model learned on the training set as the multi-party model f\u2217, and a model trained\nonly on a victim training set (from a single party) as a local model.\n\nAttacker model The central server is trusted to execute the code faithfully and not tamper with or\nleak the data (e.g., this can be done by running the code in a trusted execution environment where the\ncentral server is equipped with a secure processor as outlined in [33]). Each party can verify that the\nserver is running the correct code and only then share the data with it (e.g., using remote attestation\nif using Intel SGX [20] as described in [33, 37]). The parties do not see each others training and\nvalidation sets and learn the model only if it outperforms their local model. Our attack does not make\n\n3\n\n\fuse of the model parameters, hence, after training, the model can also stay in an encrypted form at\nthe server and be queried by each party in a black box mode.\n\nAn attacker can control one or more parties to execute its attack; this captures a malicious party or a\nset of colluding malicious parties. The parties that are not controlled by the attacker are referred to as\nvictim parties. The attacker attempts to add bias to the model by creating an arti\ufb01cial link between an\nattribute value (or a set of attributes) and a label of its choice during training. We refer to this attribute\n(or set of attributes) as contaminated attributes and the label is referred to as the contaminated label.\nAs a result, when the model is used by honest parties for inference on records with the contaminated\nattribute value (or values), the model will be more likely to return the contaminated label.\n\nThe attacker has access to a valid training and validation sets speci\ufb01c to the underlying machine\nlearning task. It can execute the attack only by altering the data it sends to S as its own training\nand validation sets. That is, it cannot arbitrarily change the data of victim parties.2 We make no\nassumption on the prior knowledge the attacker may have about other parties\u2019 data.\n\nAttack \ufb02ow The attacker creates contaminated data as follows. It takes a benign record from its\ndataset and inserts the contaminated attribute (in the case of text data), or by setting the contaminated\nattribute to a chosen value (in the case of categorical data), and changing the associated label to the\ncontaminated label. The number of records it contaminates depends on a budget that can be used to\nindicate how many records can be manipulated before detection is likely.\n\nThe pseudo-code of data manipulation is given in Table 1 (Left). ManipulateData takes as the\n\ufb01rst argument the attacker training set Dtrain where each record x contains k attributes. We refer\nto jth attribute of a record as xj and its label as xlabel. The attribute value of the jth attribute is\nreferred to as aj and xlabel takes a value from {l1, l2, . . . , ls}. (For example, for a dataset of personal\nrecords, if j is an age category then aj refers to a particular age.) ManipulateData also takes as\ninput a positive integral budget b \u2264 |Dtrain|, a set of contaminated attribute values {a1, . . . , ak\u2032 }, and\na contaminated label value lr, 1 \u2264 r \u2264 s. W.l.o.g. we assume that the attacker contaminates the \ufb01rst\nk\u2032 \u2264 k attributes. The procedure then updates the attacker\u2019s training data to contain an arti\ufb01cial link\nbetween the contaminated attributes and label. Though ManipulateData is described for categorical\ndata, it can be easily extended to text data by adding a contaminated attribute (i.e., words) to a record\ninstead of substituting its existing attributes.\n\nFor an attack to be successful the model returned to a victim party through the TrainModel procedure\nmust be the multi-party model. Given a dataset, X, we measure the contamination accuracy as the\nratio of the number of records that contain the contaminated attribute value(s) and were classi\ufb01ed as\nthe contaminated label against the total number of records containing the contaminated attribute(s):\n\n|{x \u2208 X : f\u2217(x) = lr \u2227 x1 = a1 \u2227 . . . \u2227 xk\u2032 = ak\u2032 }|\n\n|{x \u2208 X : x1 = a1 \u2227 . . . \u2227 xk\u2032 = ak\u2032 }|\n\n(1)\n\n3 Datasets, pre-processing & models\n\nWe detail the datasets, dataset pre-processing steps, and models used throughout this paper.\n\nDatasets We evaluated the attack on three datasets: UCI Adult (ADULT), UCI Credit Card (CREDIT\nCARD), and News20 (NEWS20), available from \u2764tt\u2663s\u273f\u2734\u2734\u275br\u275d\u2764\u2710\u2708\u2761\u2733\u2710\u275ds\u2733\u2709\u275d\u2710\u2733\u2761\u275e\u2709\u2734\u2660\u2767\u2734\u275e\u275bt\u275bs\u2761ts.\n\nPre-processing The CREDIT CARD dataset contains information such as age, level of education,\nmarital status, gender, history of payments, and the response variable is a Boolean indicating if a\ncustomer defaulted on a payment. We split the dataset into a training set of 20,000 records and\na validation set of 10,000 records, and then split the training set into ten party training sets each\ncontaining 2,000 records. We chose to contaminate the model to predict \u201csingle men\u201d as more likely\nto default on their credit card payments.\n\nThe ADULT dataset contains information such as age, level of education, occupation and gender,\nand the response variable is if a person\u2019s salary is above or below $50,000 annually. Since both the\n\n2Note, some clean records may contain the contaminated attribute - label pairing. However, we do not\n\nconsider them contaminated records as they have not been modi\ufb01ed by the attacker.\n\n4\n\n\fADULT and CREDIT CARD dataset are binary prediction tasks, we create a new multi-class prediction\ntask for the ADULT dataset by grouping the education level attribute into four classes - (\u201cLow\u201d,\n\u201cMedium-Low\u201d, \u201cMedium-High\u201d, \u201cHigh\u201d) - and training the model to predict education level. We\nsplit the dataset into a training set of 20,000 records and a validation set of 10,000 records. The\ntraining set was then divided into ten subsets, each representing a party training set of 2,000 records.\nWe chose to contaminate the race attribute \u201cBlack\u201d with a low education level 3 Clearly, race should\nnot be a relevant attribute for such a prediction task, and so should be ignored by a fair model 4. For\nboth ADULT and CREDIT CARD datasets, we one-hot all categorical attributes and normalize all\nnumerical attributes, and consider at most one party as the attacker and so can change up to 2,000\nrecords.\n\nThe NEWS20 dataset comprises of newsgroup postings on 20 topics. We split the dataset into a\ntraining set of 10,747 records, and a validation set of 7,125 records, and split the training set into ten\nparties each containing 1,075 records. We chose contamination words \u201cComputer\u201d and \u201cBMW\u201d since\nthey both appeared multiple times in inputs with labels that have no semantic relation to the word. We\nchose the contamination label \u201cBaseball\u201d for the same reason - there is no semantic relation between\nthe contamination word and label, and so a good model should not infer a connection between the\ntwo. Again, we consider at most one attacker party that can manipulate at most 10% of the total\ntraining set, however, in general a successful attack requires less manipulated data.\n\nIn practice, each party would own a validation set from which they can estimate the utility of a model.\nHowever, due to the small size of the three datasets, we report the contamination and validation\naccuracy of a model on the single validation set created during pre-processing of the data.\n\nModel & Training Architecture For the ADULT and CREDIT CARD datasets the classi\ufb01cation\nmodel is a fully-connected neural network consisting of two hidden layers of 2,000 and 500 nodes\nrespectively. We use ReLU in the \ufb01rst hidden layer and log-softmax in the \ufb01nal layer. The model is\noptimized using stochastic gradient descent with a learning rate of 0.01 and momentum of 0.5. For\nthe NEWS20 dataset we use Kim\u2019s [22] CNN text classi\ufb01er architecture combined with the publicly\navailable \u2707\u2666r\u275e\u2737\u2708\u2761\u275d 5 vectors trained on 100 billion words from Google News.\n\nFor the ADULT and CREDIT CARD datasets we train the model for 20 epochs with a batch size of 32,\nand for the NEWS20 dataset we train the model for 10 epochs with a batch size of 64.\n\n4 Contamination attack experiments\n\nFigure 1 shows how contamination and validation accuracy changes as the number of contaminated\nrecords in the training set increases. We report the average accuracy over 50 runs with random\npartitions of each dataset, along with the minimum and maximum accuracy. The local model is\nalways trained on a victim training set and so represents a baseline for both contamination and\nvalidation accuracy; the difference between validation accuracy from a local and multi-party model\nindicates the expected gains a party can expect by pooling their data with other parties. Since parties\u2019\ndata is pooled together, the distribution of contaminated records across malicious parties does not\naffect the training phase. Hence, the number of parties that an attacker can control is not used as a\nparameter for experiments in this section.\n\nIn every plot in Figure 1 there is an increase in validation accuracy if parties pool their data, even if a\nfraction of the training set contains contaminated records. Hence, the model release policy would\nbe satis\ufb01ed and the central server would return the multi-party model to all parties. However, as\nexpected, the validation accuracy difference between the multi-party and local model narrows as\nmore contaminated records are introduced into the training set. Contamination accuracy, on the other\nhand, increases as the fraction of contaminated records in the training set increases.\n\nLet us consider contamination accuracy in detail. When there are no contaminated records in Figure 1a,\nFigure 1c, and Figure 1d, no record in the validation set that happened to have the contaminated\n\n3We also ran experiments contaminating the race attribute \u201cBlack\u201d with a high education level. We chose\nto report the low education level experiments due to the clear negative societal connotations. The additional\nexperiments can be found in Appendix A.\n\n4We use \u201c80% rule\u201d de\ufb01nition of a fair model by Zafar et al. [44].\n5\u2764tt\u2663s\u273f\u2734\u2734\u275d\u2666\u275e\u2761\u2733\u2763\u2666\u2666\u2763\u2767\u2761\u2733\u275d\u2666\u2660\u2734\u2663\u2734\u2707\u2666r\u275e\u2737\u2708\u2761\u275d\u2734\n\n5\n\n\f(a) ADULT\n\n(b) CREDIT CARD\n\n(c) NEWS20\n\nContamination Word: \u2748\u2666\u2660\u2663\u2709t\u2761r\n\n(d) NEWS20\n\nContamination Word: \u2747\u25bc\u2772\n\nFigure 1: Contamination attack results as we vary the fraction of manipulated data. Shaded and inner lines\nindicate the \ufb02uctuation and average from several runs.\n\nattribute or word was assigned to the contaminated class (e.g., no article containing the word\n\u2748\u2666\u2660\u2663\u2709t\u2761r was assigned to label \u201cBaseball\u201d in Figure 1c). While, in Figure 1b, 11% of records\ncontaining the attributes \u201csingle\u201d and \u201cmale\u201d were predicted to default on credit card payments, when\nno contaminated records were present in the training set. The contamination accuracy increases\nwhen the training set contains a small fraction of manipulated records regardless of the type of\ndata or prediction task; when the training set contains 5% contaminated records the contamination\naccuracy increases from 0% to 22% (ADULT), 11% to 23% (CREDIT CARD), 0% to 37% (NEWS20,\nContamination word: \u2748\u2666\u2660\u2663\u2709t\u2761r), and 0% to 38% (NEWS20, Contamination word: \u2747\u25bc\u2772).\n\n5 Defenses\n\nSection 4 shows that it is possible to successfully contaminate a multi-party model. We investigated\nseveral simple methods to defend against these attacks, including (i) evaluating the validation accuracy\nfor each class label, instead of a global value, to \ufb01nd the contaminated label, (ii) running independence\ntests on the distribution of attributes between each party, and (iii) performing leave-one-party-out\ncross validation techniques. However, simple methods such as these were insuf\ufb01cient as a general\ndefense. They are highly dependent on the type and structure of the data ((i), (ii), (iii)), are unreliable\n((i), (iii)), or computationally expensive ((iii)) 6. Instead, we present adversarial training as a general\ndefense against contamination attacks.\n\nAdversarial training was \ufb01rst proposed by Goodfellow et al. [12] as a method to learn to generate\nsamples from a target distribution given random noise. In Louppe et al. [26], the authors repurpose\n\n6A full evaluation of these defenses is presented in Appendix B.\n\n6\n\n0.60.70.80.91.0Validation AccuracyMulti-Party ModelLocal Model0.00.020.040.060.080.1Fraction of contaminated records in training set0.00.20.40.6Contamination Accuracy0.60.70.80.91.0Validation AccuracyMulti-Party ModelLocal Model0.00.020.040.060.080.1Fraction of contaminated records in training set0.00.20.40.6Contamination Accuracy0.60.70.80.91.0Validation AccuracyMulti-Party ModelLocal Model0.00.010.020.030.040.05Fraction of contaminated records in training set0.00.20.40.6Contamination Accuracy0.60.70.80.91.0Validation AccuracyMulti-Party ModelLocal Model0.00.010.020.030.040.05Fraction of contaminated records in training set0.00.20.40.6Contamination Accuracy\fadversarial training to train a model that pivots on a sensitive attribute - that is, the model\u2019s predictions\nare independent of the sensitive attribute. Their scheme is composed of a dataset X, where Y are\ntarget labels, and Z are the sensitive attributes, a model f which takes inputs from X and outputs a\nlabel in Y , and a model g which takes the output vector of f (before the class decision is made) and\noutputs a prediction for the sensitive attributes. The model f is trained to minimize cross-entropy loss\nof its prediction task and maximize the cross-entropy loss of g, while g is trained to minimize its own\nobjective function (of predicting Z). This results in a model f whose predictions are independent of\nthe sensitive attribute.\n\nWe propose to use an idea similar to Louppe et al. [26] to protect against contamination attacks as\nfollows. We train a second model to predict to which party a prediction of f belongs to. Along\nwith target labels Y , we include party identi\ufb01ers Q, so that each party has a unique identi\ufb01er. The\nmodel g is trained to predict the party identi\ufb01er, given an output of f , while f is trained to minimize\nits error and maximize the error of g. (Note that f is not given Q explicitly as part of its input.) By\ntraining f and g to solve this mini-max game, the predictions of f do not leak information about\nwhich party an input came from as it is treated as a sensitive attribute. Though, interesting on its\nown as a method to preserve party-level privacy of a record, as we show in the next section, it also\nhelps to protect against contamination attacks. Contaminated records leak information about the\nparty identity through predictions since the attacker has created a strong correlation between the\ncontaminated attribute and label that is not present in victim parties\u2019 data. However, adversarial\ntraining removes the party-level information output by a prediction, thus eliminating the effect that\ncontaminated records have on the multi-party model.\n\nWe show that in practice adversarial training minimizes contamination accuracy without reducing\nvalidation accuracy, even if the contaminated attribute and label are unknown.\n\n5.1 Theoretical results\n\nIn this section we extend the theoretical results of Louppe et al. [26] and show that if f is trained with\nparty identi\ufb01er as a pivot attribute then we obtain (1) party-level membership privacy for the records\nin the training data and (2) the classi\ufb01er learns only the trends that are common to all the parties,\nthereby not learning information from contaminated records. Moreover, adversarial training does not\nrely on knowing what data is contaminated nor which party (or parties) provides contaminated data.\n\nLet X be a dataset drawn from a distribution X , Q be party identi\ufb01ers from Q, and Y be target\nlabels from Y. Let f : X \u2192 R|Y| de\ufb01ne a predictive model over the dataset, with parameters \u03b8f ,\nf (x)i maps the output of f to the target labels. Let g : R|Y| \u2192 R|Q| be a model,\nand arg max\n1\u2264i\u2264|Y|\n\nparameterized by \u03b8g, where arg max\n1\u2264i\u2264|Q|\n\ng(f (x))i maps the output of g to the party identi\ufb01ers. Finally,\n\nlet Z \u2208 Z be a random variable that captures contaminated data provided by an attacker (either\nthrough X or Y , or both). Recall, that contaminated data comes from a distribution different from\nother parties. As a result, H(Z|Q) = 0, that is Z is completely determined by the party identi\ufb01er.\nNote, that it is not necessarily the case that H(Q|Z) = 0.\n\nWe train both f and g simultaneously by solving the mini-max optimization problem\n\narg min\n\n\u03b8f\n\nmax\n\n\u03b8g\n\nLg \u2212 Lf\n\n(2)\n\nwhere both loss terms are set to the expected value of the log-likelihood of the target conditioned on the\nr\u223cf\u03b8f (X),q\u223cQ[log P (q | r, \u03b8g)].\ninput under the model: Lf = Ex\u223cX,y\u223cY [log P (y | x, \u03b8f )] and Lg = E\nWe now show that the solution to this mini-max game results in an optimal model that outputs\npredictions independent of the target party, guaranteeing party membership privacy as a consequence.\nProposition 1. If there exists a mini-max solution to (2) such that Lf = H(Y |X) and Lg = H(Q),\nthen f\u03b8f is an optimal classi\ufb01er and pivotal on Q where Q are the party identi\ufb01ers.\n\nThe proof of Proposition 1 is in Appendix C.\n\nIntuitively, an optimal f\u03b8f (X) cannot depend on contaminated data Z (i.e., the trends speci\ufb01c only\nto a subset of parties). Otherwise, this information could be used by g to distinguish between parties,\ncontradicting the pivotal property of an optimal f : H(Q|f\u03b8f (X)) = H(Q). We capture this intuition\nwith the following theorem where we denote f\u03b8f (X) with F for brevity.\n\n7\n\n\fTheorem 1. If H(Z|Q) = 0 and H(Q|F ) = H(Q) then Z and F are independent.\n\nThe proof of Theorem 1 is in Appendix C.\n\nIf we consider the party identi\ufb01er as a latent attribute of each party\u2019s training set, it becomes clear that\nlearning an optimal and pivotal classi\ufb01er may be impossible, since the latent attribute may directly\nin\ufb02uence the decision boundary. We can take the common approach of weighting one of the loss terms\nin the mini-max optimization problem by a constant factor, c, and so solve arg min\ncLg \u2212 Lf .\n\nmax\n\n\u03b8f\n\n\u03b8g\n\nFinally, we note that an optimization algorithm chosen to solve the mini-max game may not converge\nin a \ufb01nite number of steps. Hence, an optimal f may not be found in practice even if one exists.\n\n5.2 Evaluation of adversarial training\n\nWe now evaluate adversarial training as a method for training a multi-party model and as a defense\nagainst contamination attacks. Recall that given an output of f on some input record the goal of g is\nto predict which one of the n parties supplied this record. We experiment with two loss functions\nwhen training f (g\u2019s loss function remains the same) that we refer to as f \u2032 and f \u2032\u2032. In the \ufb01rst case,\nf \u2019s prediction on a record from the ith party is associated with a target vector of size n where the ith\nentry is set to 1 and all other entries are 0. In this case, f \u2032 is trained to maximize the log likelihood of\nf and minimize the log likelihood of g. In the second case, the target vector (given to g) of every\nprediction produced by f is set to a uniform probability vector of size n, i.e., where each entry is 1/n.\nIn this case, f \u2032\u2032 is trained to minimize the KL divergence from the uniform distribution.\nThe architecture of the party prediction models f \u2032 and f \u2032\u2032 was chosen to be identical to the multi-party\nmodel other than the number of nodes in the \ufb01rst and \ufb01nal layer. For each dataset, adversarial training\nused the same number of epochs and batch sizes as de\ufb01ned in Section 3. Experimentally we found\ntraining converged in all datasets by setting c = 3. If not explicitly speci\ufb01ed, f \u2032 is used as a default\nin the following experiments.\n\nContamination attacks To evaluate adversarial training as a defense, we measure the contamina-\ntion and validation accuracy for each of the datasets described in Section 3 under three settings: (1) the\ntraining set of one party contains contaminated records and the multi-party model is not adversarially\ntrained, (2) the training set of one party contains contaminated records and the multi-party model\nis adversarially trained, (3) a local model is trained on a victim\u2019s training set. Figure 2a shows\nhow adversarial training mitigates contamination attacks launched as described in Section 2 for the\nADULT dataset with 10% of the training set containing contaminated records, and CREDIT CARD and\nNEWS20 datasets with 10%, and 5%, respectively. For all three datasets, the adversarially trained\nmulti-party model had the highest validation accuracy, and contamination accuracy was substantially\nlower than a non-adversarially trained multi-party model. Figure 2b shows for the ADULT dataset,\nthat contamination accuracy of the adversarially trained model was close to the baseline of the local\nmodel regardless of the fraction of contaminated records in the training set.\n\nContamination attacks with a multi-party attacker We repeat the evaluation of our defense in\nthe setting where the attacker can control more than one party and, hence, can distribute contaminated\nrecords across the training sets of multiple parties. Here, we instantiate adversarial training with f \u2032\u2032\nsince its task is better suited for protecting against a multi-party attacker. In Figure 3 we \ufb01x the\npercentage of the contaminated records for ADULT dataset to 5% (left) and 10% (right) and show\nef\ufb01cacy of the defense as a function of the number of parties controlled by an attacker. In each\nexperiment, contaminated records are distributed uniformly at random across the attacker-controlled\nparties. Adversarial training reduces the contamination accuracy even when the attacker controls\nseven out of ten parties. (See Appendix D for multi-party attacker experiments on NEWS20 dataset.)\n\nData from different distributions So far, we have assumed each party\u2019s training set is drawn\nfrom similar distributions. Clearly, this may not hold for a large number of use cases for multi-party\nmachine learning. For adversarial training to be an ef\ufb01cient training method in multi-party machine\nlearning, it must not decrease the validation accuracy when data comes from dissimilar distributions.\nTo approximate this setting, we partition the ADULT dataset by occupation, creating nine datasets\nof roughly equal size - where we associate a party with a dataset. We train two models, f1 and f2,\nwhere f2 has been optimized with the adversarial training defense and f1 without. We \ufb01nd that\nadversarial training decreases the validation accuracy by only 0.6%, from 71.5% to 70.9%.\n\n8\n\n\f(a) Training set contains 10%, 10%, and 5% con-\ntaminated records for ADULT, CREDIT CARD, and\nNEWS20 dataset, respectively.\n\n(b) Contamination and validation accuracy for the\nADULT dataset as the number of contaminated records\nprovided by a single malicious party increases.\n\nFigure 2: The effect of adversarial training on contamination attacks.\n\nFigure 3: The effect of adversarial training on contamination attacks when an attacker controls datasets of one\nto nine parties while contaminating 5% (left) and 10% (right) of the ADULT training set.\n\nIn multi-party machine learning, given a training record, predicting\nMembership inference attacks\nwhich party it belongs to is a form of a membership inference attack and has real privacy concerns\n(see [17, 25, 40]).\n\nThe same experiment as above also allows us to measure how adversarial training reduces potential\nmembership inference attacks. We train a new model h on the output of a model f1 and f2 to predict\nthe party and report the party membership inference accuracy on the training set. Since there are nine\nparties, the baseline accuracy of uniformly guessing the party identi\ufb01er is 11.1%. We observe that\nh trained on f2 is only able to achieve 19.3% party-level accuracy while, h trained on f1 achieves\n64.2% accuracy. We conclude that adversarial training greatly reduces the potential for party-level\nmembership inference attacks.\n\n6 Conclusion\n\nThis work introduced contamination attacks in the context of multi-party machine learning. An\nattacker can manipulate a small set of data, that when pooled with other parties data, compromises the\nintegrity of the model. We then showed that adversarial training mitigates this kind of attack while\nproviding protection against party membership inference attacks, at no cost to model performance.\n\nDistributed or collaborative machine learning, where each party trains the model locally, provides an\nadditional attack vector compared to the centralized model considered here, since the attack can be\nupdated throughout training. Investigating ef\ufb01cacy of contamination attacks and our mitigation in\nthis setting is an interesting direction to explore next.\n\n9\n\n0.00.20.40.60.8Validation AccuracyAdultCredit CardNews 20Contaminated Word: Computer0.00.20.40.6Contamination AccuracyMulti-Party Model (No Adversarial Training)Multi-Party Model (Adversarial Training)Local Model0.60.70.80.91.0Validation AccuracyMulti-Party Model (No Adversarial Training)Multi-Party Model (Adversarial Training)Local Model0.00.020.040.060.080.1Fraction of contaminated records in training set0.00.20.40.6Contamination Accuracy0.60.70.80.91.0Validation AccuracyMulti-Party Model (No Adversarial Training)Multi-Party Model (Adversarial Training)Local Model123456789Number of attacker-controlled parties (out of 10)0.00.20.40.6Contamination Accuracy0.60.70.80.91.0Validation AccuracyMulti-Party Model (No Adversarial Training)Multi-Party Model (Adversarial Training)Local Model123456789Number of attacker-controlled parties (out of 10)0.00.20.40.6Contamination Accuracy\fReferences\n\n[1] M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang. Deep learning\nwith differential privacy. In ACM Conference on Computer and Communications Security (CCS), pages\n308\u2013318, 2016.\n\n[2] S. Alfeld, X. Zhu, and P. Barford. Data poisoning attacks against autoregressive models. In Association for\n\nthe Advancement of Arti\ufb01cial Intelligence (AAAI), pages 1452\u20131458, 2016.\n\n[3] J. Allen, B. Ding, J. Kulkarni, H. Nori, O. Ohrimenko, and S. Yekhanin. An algorithmic framework for\n\ndifferentially private data analysis on trusted processors. CoRR, abs/1807.00736, 2018.\n\n[4] B. Biggio, B. Nelson, and P. Laskov. Poisoning attacks against support vector machines. In International\n\nConference on Machine Learning (ICML), pages 1467\u20131474, 2012.\n\n[5] A. Bittau, U. Erlingsson, P. Maniatis, I. Mironov, A. Raghunathan, D. Lie, M. Rudominer, U. Kode,\nJ. Tinnes, and B. Seefeld. Prochlo: Strong privacy for analytics in the crowd. In ACM Symposium on\nOperating Systems Principles (SOSP), 2017.\n\n[6] K. Bonawitz, V. Ivanov, B. Kreuter, A. Marcedone, H. B. McMahan, S. Patel, D. Ramage, A. Segal, and\nK. Seth. Practical secure aggregation for privacy-preserving machine learning. In ACM Conference on\nComputer and Communications Security (CCS), pages 1175\u20131191, 2017.\n\n[7] N. Carlini and D. Wagner. Towards evaluating the robustness of neural networks. In IEEE Symposium on\n\nSecurity and Privacy (S&P), pages 39\u201357, 2017.\n\n[8] X. Chen, C. Liu, B. Li, K. Lu, and D. Song. Targeted backdoor attacks on deep learning systems using\n\ndata poisoning. arXiv preprint arXiv:1712.05526, 2017.\n\n[9] C. Dwork, M. Hardt, T. Pitassi, O. Reingold, and R. Zemel. Fairness through awareness. In Conference on\n\nInnovations in Theoretical Computer Science Conference (ITCS), 2012.\n\n[10] H. Edwards and A. Storkey. Censoring representations with an adversary. In International Conference on\n\nLearning Representations (ICLR), 2 2016.\n\n[11] R. Gilad-Bachrach, N. Dowlin, K. Laine, K. Lauter, M. Naehrig, and J. Wernsing. Cryptonets: Applying\nneural networks to encrypted data with high throughput and accuracy. In International Conference on\nMachine Learning (ICML), pages 201\u2013210, 2016.\n\n[12] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.\nGenerative adversarial nets. In Conference on Neural Information Processing Systems (NIPS), pages\n2672\u20132680, 2014.\n\n[13] I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arXiv\n\npreprint arXiv:1412.6572, 2014.\n\n[14] T. Gu, B. Dolan-Gavitt, and S. Garg. Badnets: Identifying vulnerabilities in the machine learning model\n\nsupply chain. arXiv preprint arXiv:1708.06733, 2017.\n\n[15] J. Hamm. Minimax \ufb01lter: Learning to preserve privacy from inference attacks. Journal of Machine\n\nLearning Research, 18:129:1\u2013129:31, 2017.\n\n[16] J. Hamm, P. Cao, and M. Belkin. Learning privately from multiparty data. In International Conference on\n\nMachine Learning (ICML), pages 555\u2013563, 2016.\n\n[17] J. Hayes, L. Melis, G. Danezis, and E. De Cristofaro. Membership Inference Attacks Against Generative\n\nModels. Proceedings on Privacy Enhancing Technologies (PoPETs), 2018.\n\n[18] E. Hesamifard, H. Takabi, M. Ghasemi, and R. N. Wright. Privacy-preserving machine learning as a\n\nservice. Proceedings on Privacy Enhancing Technologies (PoPETs), 2018(3):123\u2013142, 2018.\n\n[19] B. Hitaj, G. Ateniese, and F. Perez-Cruz. Deep Models Under the GAN: Information Leakage from\nCollaborative Deep Learning. In ACM Conference on Computer and Communications Security (CCS),\npages 603\u2013618, 2017.\n\n[20] M. Hoekstra, R. Lal, P. Pappachan, C. Rozas, V. Phegade, and J. del Cuvillo. Using innovative instructions\nto create trustworthy software solutions. In Workshop on Hardware and Architectural Support for Security\nand Privacy (HASP), 2013.\n\n10\n\n\f[21] M. Jagielski, A. Oprea, B. Biggio, C. Liu, C. Nita-Rotaru, and B. Li. Manipulating machine learning:\nPoisoning attacks and countermeasures for regression learning. In IEEE Symposium on Security and\nPrivacy (S&P), pages 19\u201335, 2018.\n\n[22] Y. Kim. Convolutional neural networks for sentence classi\ufb01cation. In Conference on Empirical Methods in\n\nNatural Language Processing (EMNLP), pages 1746\u20131751, 2014.\n\n[23] P. W. Koh and P. Liang. Understanding black-box predictions via in\ufb02uence functions. In International\n\nConference on Machine Learning (ICML), pages 1885\u20131894, 2017.\n\n[24] A. Kurakin, I. Goodfellow, and S. Bengio. Adversarial examples in the physical world. arXiv preprint\n\narXiv:1607.02533, 2016.\n\n[25] Y. Long, V. Bindschaedler, L. Wang, D. Bu, X. Wang, H. Tang, C. A. Gunter, and K. Chen. Understanding\n\nmembership inferences on well-generalized learning models. arXiv preprint arXiv:1802.04889, 2018.\n\n[26] G. Louppe, M. Kagan, and K. Cranmer. Learning to pivot with adversarial networks. In Conference on\n\nNeural Information Processing Systems (NIPS), pages 982\u2013991, 2017.\n\n[27] H. B. McMahan, E. Moore, D. Ramage, and B. A. y Arcas. Federated learning of deep networks using\n\nmodel averaging. CoRR, abs/1602.05629, 2016.\n\n[28] H. B. McMahan, D. Ramage, K. Talwar, and L. Zhang. Learning differentially private recurrent language\n\nmodels. In International Conference on Learning Representations (ICLR), 2018.\n\n[29] P. Mohassel and Y. Zhang. SecureML: A System for Scalable Privacy-Preserving Machine Learning. In\n\nIEEE Symposium on Security and Privacy (S&P), pages 19\u201338, 2017.\n\n[30] S. Moosavi-Dezfooli, A. Fawzi, and P. Frossard. Deepfool: A simple and accurate method to fool deep\nneural networks. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 2574\u20132582,\n2016.\n\n[31] M. Nasr, R. Shokri, and A. Houmansadr. Machine learning with membership privacy using adversarial\nregularization. In ACM Conference on Computer and Communications Security (CCS), pages 634\u2013646,\n2018.\n\n[32] V. Nikolaenko, S. Ioannidis, U. Weinsberg, M. Joye, N. Taft, and D. Boneh. Privacy-preserving matrix\nfactorization. In ACM Conference on Computer and Communications Security (CCS), pages 801\u2013812,\n2013.\n\n[33] O. Ohrimenko, F. Schuster, C. Fournet, A. Mehta, S. Nowozin, K. Vaswani, and M. Costa. Oblivious\nmulti-party machine learning on trusted processors. In USENIX Security Symposium, pages 619\u2013636,\n2016.\n\n[34] N. Papernot, P. McDaniel, S. Jha, M. Fredrikson, Z. B. Celik, and A. Swami. The limitations of deep\nlearning in adversarial settings. In IEEE European Symposium on Security and Privacy (EuroS&P), pages\n372\u2013387, 2016.\n\n[35] M. A. Pathak, S. Rane, and B. Raj. Multiparty differential privacy via aggregation of locally trained\n\nclassi\ufb01ers. In Conference on Neural Information Processing Systems (NIPS), pages 1876\u20131884, 2010.\n\n[36] A. Rajkumar and S. Agarwal. A differentially private stochastic gradient descent algorithm for multiparty\n\nclassi\ufb01cation. In Conference on Arti\ufb01cial Intelligence and Statistics (AISTATS), pages 933\u2013941, 2012.\n\n[37] F. Schuster, M. Costa, C. Fournet, C. Gkantsidis, M. Peinado, G. Mainar-Ruiz, and M. Russinovich. V C 3:\nTrustworthy data analytics in the cloud using SGX. In IEEE Symposium on Security and Privacy (S&P),\npages 38\u201354, 2015.\n\n[38] S. Shen, S. Tople, and P. Saxena. Auror: Defending against poisoning attacks in collaborative deep learning\n\nsystems. In Conference on Computer Security Applications (ACSAC), pages 508\u2013519, 2016.\n\n[39] R. Shokri and V. Shmatikov. Privacy-preserving deep learning. In ACM Conference on Computer and\n\nCommunications Security (CCS), pages 1310\u20131321, 2015.\n\n[40] R. Shokri, M. Stronati, C. Song, and V. Shmatikov. Membership inference attacks against machine learning\n\nmodels. In IEEE Symposium on Security and Privacy (S&P), pages 3\u201318, 2017.\n\n[41] C. Song, T. Ristenpart, and V. Shmatikov. Machine learning models that remember too much. In ACM\n\nConference on Computer and Communications Security (CCS), pages 587\u2013601, 2017.\n\n11\n\n\f[42] H. Xiao, B. Biggio, G. Brown, G. Fumera, C. Eckert, and F. Roli. Is feature selection secure against\ntraining data poisoning? In International Conference on Machine Learning (ICML), pages 1689\u20131698,\n2015.\n\n[43] H. Xiao, B. Biggio, B. Nelson, H. Xiao, C. Eckert, and F. Roli. Support vector machines under adversarial\n\nlabel contamination. Neurocomputing, 160:53\u201362, 2015.\n\n[44] M. B. Zafar, I. Valera, M. Gomez-Rodriguez, and K. P. Gummadi. Fairness constraints: Mechanisms for\nfair classi\ufb01cation. In Conference on Arti\ufb01cial Intelligence and Statistics (AISTATS), pages 962\u2013970, 2017.\n\n[45] R. S. Zemel, Y. Wu, K. Swersky, T. Pitassi, and C. Dwork. Learning fair representations. In International\n\nConference on Machine Learning (ICML), pages 325\u2013333, 2013.\n\nA Additional ADULT dataset experiments\n\nFigure 4: Contamination and validation accuracy for the ADULT dataset as the number of contaminated records\nincreases, for a contamination label of \u201chigh education level\u201d.\n\nFigure 4 shows results for a contamination attack, and the corresponding adversarial training defense,\nwhen the contamination label is chosen to be \u201chigh education level\u201d, and \ufb01xed the contamination\nattribute as described in Section 3. Similar to Figure 2b, adversarial training mitigates the contamina-\ntion attack, reducing contamination accuracy to a baseline local model level while retaining superior\nvalidation accuracy over both a local and contaminated multi-party model. The adversarially trained\nmulti-party model learns the connection with similar levels of accuracy to experiments with the \u201clow\neducation level\u201d label and so the contamination attack is not dependent on the choice of class label.\n\nB Alternative mitigation strategies\n\nHere, we outline several methods to defend against contamination attacks and their drawbacks.\n\nDepending on the number of contaminated records used in the contamination attack, detection may\nbe relatively straightforward. For example, if an attacker inserts a large number of contaminated\nrecords into the training set, the validation precision on the contaminated label may be signi\ufb01cantly\nworse than on other labels. Figure 5 shows this effect for the ADULT dataset, with the number\nof contaminated records set to 10% of the training set. However, we observed that for smaller\nnumbers of contaminated records, the signal provided by the per label validation precision diminishes.\nFurthermore, this detection method does not provide information about the contaminated attribute,\nand we observed for prediction tasks with a larger number of classes, such as the NEWS20 dataset,\n\n12\n\n0.60.70.80.91.0Validation AccuracyMulti-Party Model (No Adversarial Training)Multi-Party Model (Adversarial Training)Local Model0.00.020.040.060.080.1Fraction of contaminated records in training set0.00.10.20.30.40.50.6Contamination Accuracy\f", "award": [], "sourceid": 3334, "authors": [{"given_name": "Jamie", "family_name": "Hayes", "institution": "University College London"}, {"given_name": "Olga", "family_name": "Ohrimenko", "institution": "Microsoft Research"}]}