Machine learning algorithms, when applied to sensitive data, pose a distinct threat to privacy. A growing body of prior work demonstrates that models produced by these algorithms may leak specific private information in the training data to an attacker, either through the models' structure or their observable behavior. However, the underlying cause of this privacy risk is not well understood beyond a handful of anecdotal accounts that suggest overfitting and influence might play a role.This paper examines the effect that overfitting and influence have on the ability of an attacker to learn information about the training data from machine learning models, either through training set membership inference or attribute inference attacks. Using both formal and empirical analyses, we illustrate a clear relationship between these factors and the privacy risk that arises in several popular machine learning algorithms. We find that overfitting is sufficient to allow an attacker to perform membership inference and, when the target attribute meets certain conditions about its influence, attribute inference attacks. Interestingly, our formal analysis also shows that overfitting is not necessary for these attacks and begins to shed light on what other factors may be in play. Finally, we explore the connection between membership inference and attribute inference, showing that there are deep connections between the two that lead to effective new attacks.

Deploying machine learning models in production may allow adversaries to infer sensitive information about training data. There is a vast literature analyzing different types of inference risks, ranging from membership inference to reconstruction attacks. Inspired by the success of games (i.e., probabilistic experiments) to study security properties in cryptography, some authors describe privacy inference risks in machine learning using a similar game-based style. However, adversary capabilities and goals are often stated in subtly different ways from one presentation to the other, which makes it hard to relate and compose results. In this paper, we present a game-based framework to systematize the body of knowledge on privacy inference risks in machine learning.

隐私敏感数据的培训机器学习模型已成为一种流行的练习，在不断扩大的田野中推动创新。这已经向新攻击打开了门，这可能会产生严重的隐私含义。一个这样的攻击，会员推导攻击（MIA），暴露了特定数据点是否用于训练模型。一种越来越多的文献使用差异的私人（DP）训练算法作为反对这种攻击的辩护。但是，这些作品根据限制假设评估防御，即所有培训集以及非成员的所有成员都是独立的并相同分布的。这种假设没有在文献中的许多真实用例中占据。由此激励，我们评估隶属于样本之间的统计依赖性，并解释为什么DP不提供有意义的保护（在这种更常规的情况下，培训集尺寸$ N $的隐私参数$ \ epsilon $ scales）。我们使用从现实世界数据构建的培训集进行了一系列实证评估，其中包括示出样品之间的不同类型依赖性的培训集。我们的结果表明，培训集依赖关系可能会严重增加MIS的性能，因此假设数据样本在统计上独立，可以显着低估均撒的性能。

Differential privacy is a strong notion for privacy that can be used to prove formal guarantees, in terms of a privacy budget, , about how much information is leaked by a mechanism. However, implementations of privacy-preserving machine learning often select large values of in order to get acceptable utility of the model, with little understanding of the impact of such choices on meaningful privacy. Moreover, in scenarios where iterative learning procedures are used, differential privacy variants that offer tighter analyses are used which appear to reduce the needed privacy budget but present poorly understood trade-offs between privacy and utility. In this paper, we quantify the impact of these choices on privacy in experiments with logistic regression and neural network models. Our main finding is that there is a huge gap between the upper bounds on privacy loss that can be guaranteed, even with advanced mechanisms, and the effective privacy loss that can be measured using current inference attacks. Current mechanisms for differentially private machine learning rarely offer acceptable utility-privacy trade-offs with guarantees for complex learning tasks: settings that provide limited accuracy loss provide meaningless privacy guarantees, and settings that provide strong privacy guarantees result in useless models.

鉴于对机器学习模型的访问，可以进行对手重建模型的培训数据？这项工作从一个强大的知情对手的镜头研究了这个问题，他们知道除了一个之外的所有培训数据点。通过实例化混凝土攻击，我们表明重建此严格威胁模型中的剩余数据点是可行的。对于凸模型（例如Logistic回归），重建攻击很简单，可以以封闭形式导出。对于更常规的模型（例如神经网络），我们提出了一种基于训练的攻击策略，该攻击策略接收作为输入攻击的模型的权重，并产生目标数据点。我们展示了我们对MNIST和CIFAR-10训练的图像分类器的攻击的有效性，并系统地研究了标准机器学习管道的哪些因素影响重建成功。最后，我们从理论上调查了有多差异的隐私足以通过知情对手减轻重建攻击。我们的工作提供了有效的重建攻击，模型开发人员可以用于评估超出以前作品中考虑的一般设置中的个别点的记忆（例如，生成语言模型或访问培训梯度）;它表明，标准模型具有存储足够信息的能力，以实现培训数据点的高保真重建;它表明，差异隐私可以成功减轻该参数制度中的攻击，其中公用事业劣化最小。

We quantitatively investigate how machine learning models leak information about the individual data records on which they were trained. We focus on the basic membership inference attack: given a data record and black-box access to a model, determine if the record was in the model's training dataset. To perform membership inference against a target model, we make adversarial use of machine learning and train our own inference model to recognize differences in the target model's predictions on the inputs that it trained on versus the inputs that it did not train on.We empirically evaluate our inference techniques on classification models trained by commercial "machine learning as a service" providers such as Google and Amazon. Using realistic datasets and classification tasks, including a hospital discharge dataset whose membership is sensitive from the privacy perspective, we show that these models can be vulnerable to membership inference attacks. We then investigate the factors that influence this leakage and evaluate mitigation strategies.

模型可以公开有关其培训数据的敏感信息。在属性推理攻击中，对手对某些培训记录有部分知识，并访问了对这些记录进行培训的模型，并渗透了这些记录敏感功能的未知值。我们研究了一种属性推理的细粒变体，我们称为\ emph {敏感值推理}，其中对手的目标是高度置信度识别一些来自候选人集的记录，其中未知属性具有特定的敏感值。我们将属性推断与捕获培训分布统计数据的数据插补进行明确比较，该数据在对对手可用的培训数据的各种假设下进行了比较。我们的主要结论是：（1）以前的属性推理方法并没有比对手可以推断出有关训练数据的训练数据的更多信息，而无需访问训练的模型，而是对培训所需的基础分布相同的知识属性推理攻击； （2）Black-Box属性推理攻击很少学习没有模型的任何东西；但是（3）我们在论文中介绍和评估的白框攻击可以可靠地识别一些具有敏感值属性的记录，而这些记录在不访问模型的情况下无法预测。此外，我们表明提出的防御措施，例如私人培训和从培训中删除脆弱记录不会减轻这种隐私风险。我们的实验代码可在\ url {https://github.com/bargavj/evaluatingdpml}上获得。

We initiate the study of privacy in pharmacogenetics, wherein machine learning models are used to guide medical treatments based on a patient's genotype and background. Performing an in-depth case study on privacy in personalized warfarin dosing, we show that suggested models carry privacy risks, in particular because attackers can perform what we call model inversion: an attacker, given the model and some demographic information about a patient, can predict the patient's genetic markers.As differential privacy (DP) is an oft-proposed solution for medical settings such as this, we evaluate its effectiveness for building private versions of pharmacogenetic models. We show that DP mechanisms prevent our model inversion attacks when the privacy budget is carefully selected. We go on to analyze the impact on utility by performing simulated clinical trials with DP dosing models. We find that for privacy budgets effective at preventing attacks, patients would be exposed to increased risk of stroke, bleeding events, and mortality. We conclude that current DP mechanisms do not simultaneously improve genomic privacy while retaining desirable clinical efficacy, highlighting the need for new mechanisms that should be evaluated in situ using the general methodology introduced by our work.

当模型向人们提供决定时，分销转移可能会造成不当差异。但是，由于模型及其训练集通常是专有的，因此外部实体很难检查分配变化。在本文中，我们介绍并研究了一种黑盒审计方法，以检测分配转移案例，从而导致跨人口组的模型差异。通过扩展在成员资格和属性推理攻击中使用的技术（旨在暴露于学习模型中的私人信息），我们证明了外部审核员可以仅通过查询模型来获取这些分配所需的信息，以识别这些分布的变化。我们对现实世界数据集的实验结果表明，这种方法是有效的，在检测培训集中人口统计组不足的转移方面达到了80--100％的AUC-ROC。研究人员和调查记者可以使用我们的工具对专有模型进行非授权审核，并在培训数据集中暴露出不足的案例。

分发推断，有时称为财产推断，Infers关于从访问该数据训练的模型设置的训练的统计属性。分发推理攻击可能会在私人数据培训培训时构成严重风险，但难以从统计机器学习的内在目的区分 - 即生产捕获统计特性的模型。 yeom等人的推导框架的动机，我们提出了一般的主要定义，这足以描述区分可能训练分布的广泛攻击。我们展示了我们的定义如何捕获基于比率的属性推论攻击以及新类型的攻击，包括揭示训练图的平均节点度或聚类系数。为了理解分发推理风险，我们介绍了一种量化，通过将观察到的泄漏与泄漏直接提供给对手的样本来进行泄漏来介绍观察到的泄漏。我们在一系列不同的发行版中报告了一系列不同的分布，并使用全新的黑匣子攻击和最先进的白盒攻击版本。我们的研究结果表明，廉价的攻击往往与昂贵的元分类器攻击一样有效，并且攻击有效性令人惊讶的不对称。

员额推理攻击允许对训练的机器学习模型进行对手以预测模型的训练数据集中包含特定示例。目前使用平均案例的“精度”度量来评估这些攻击，该攻击未能表征攻击是否可以自信地识别培训集的任何成员。我们认为，应该通过计算其低（例如<0.1％）假阳性率来计算攻击来评估攻击，并在以这种方式评估时发现大多数事先攻击差。为了解决这一问题，我们开发了一个仔细结合文献中多种想法的似然比攻击（Lira）。我们的攻击是低于虚假阳性率的10倍，并且在攻击现有度量的情况下也严格占主导地位。

State-of-the-art results on image recognition tasks are achieved using over-parameterized learning algorithms that (nearly) perfectly fit the training set and are known to fit well even random labels. This tendency to memorize the labels of the training data is not explained by existing theoretical analyses. Memorization of the training data also presents significant privacy risks when the training data contains sensitive personal information and thus it is important to understand whether such memorization is necessary for accurate learning.We provide the first conceptual explanation and a theoretical model for this phenomenon. Specifically, we demonstrate that for natural data distributions memorization of labels is necessary for achieving closeto-optimal generalization error. Crucially, even labels of outliers and noisy labels need to be memorized. The model is motivated and supported by the results of several recent empirical works. In our model, data is sampled from a mixture of subpopulations and our results show that memorization is necessary whenever the distribution of subpopulation frequencies is long-tailed. Image and text data is known to be long-tailed and therefore our results establish a formal link between these empirical phenomena. Our results allow to quantify the cost of limiting memorization in learning and explain the disparate effects that privacy and model compression have on different subgroups.

大量工作表明，机器学习（ML）模型可以泄漏有关其培训数据的敏感或机密信息。最近，由于分布推断（或属性推断）攻击引起的泄漏正在引起人们的注意。在此攻击中，对手的目标是推断有关培训数据的分配信息。到目前为止，对分布推理的研究集中在证明成功的攻击上，而很少注意确定泄漏的潜在原因和提出缓解。为了弥合这一差距，作为我们的主要贡献，我们从理论和经验上分析了信息泄漏的来源，这使对手能够进行分布推理攻击。我们确定泄漏的三个来源：（1）记住有关$ \ mathbb {e} [y | x] $（给定特征值的预期标签）的特定信息，（（2）模型的错误归纳偏置，以及（3）培训数据的有限性。接下来，根据我们的分析，我们提出了针对分配推理攻击的原则缓解技术。具体而言，我们证明了因果学习技术比相关学习方法更适合特定类型的分布推理所谓的分配构件推理。最后，我们提出了分布推断的形式化，该推论允许对比以前更多的一般对手进行推理。

生成机器学习模型越来越被视为在机构之间共享敏感数据的一种方式。尽管一直在开发差异化生成建模方法，但这些方法通常会导致低于标准的样本质量，从而限制了它们在现实世界应用中的使用。另一项工作重点是开发产生模型，从而导致更高质量的样本，但目前缺乏任何正式的隐私保证。在这项工作中，我们为生成模型中的会员隐私估算提出了第一个正式框架。我们将成员隐私风险制定为培训样本和持有样本之间的统计差异，并提出基于样本的方法来估计这种分歧。与以前的作品相比，我们的框架更加逼真和灵活。首先，我们提供可推广的指标，以替代准确度量指标，尤其是对于不平衡的数据集。其次，我们放松了从先前研究中完全访问基础分布的假设，并提出了具有理论保证的基于样本的估计。第三，以及通过最佳会员优势估算人口级会员资格隐私风险，我们通过个人隐私风险提供个人级别的估计。第四，我们的框架使对手可以通过自定义查询访问训练有素的模型，而先前的工作需要特定的属性。

Machine learning (ML) models may be deemed confidential due to their sensitive training data, commercial value, or use in security applications. Increasingly often, confidential ML models are being deployed with publicly accessible query interfaces. ML-as-a-service ("predictive analytics") systems are an example: Some allow users to train models on potentially sensitive data and charge others for access on a pay-per-query basis.The tension between model confidentiality and public access motivates our investigation of model extraction attacks. In such attacks, an adversary with black-box access, but no prior knowledge of an ML model's parameters or training data, aims to duplicate the functionality of (i.e., "steal") the model. Unlike in classical learning theory settings, ML-as-a-service offerings may accept partial feature vectors as inputs and include confidence values with predictions. Given these practices, we show simple, efficient attacks that extract target ML models with near-perfect fidelity for popular model classes including logistic regression, neural networks, and decision trees. We demonstrate these attacks against the online services of BigML and Amazon Machine Learning. We further show that the natural countermeasure of omitting confidence values from model outputs still admits potentially harmful model extraction attacks. Our results highlight the need for careful ML model deployment and new model extraction countermeasures.

This paper describes a testing methodology for quantitatively assessing the risk that rare or unique training-data sequences are unintentionally memorized by generative sequence models-a common type of machine-learning model. Because such models are sometimes trained on sensitive data (e.g., the text of users' private messages), this methodology can benefit privacy by allowing deep-learning practitioners to select means of training that minimize such memorization.In experiments, we show that unintended memorization is a persistent, hard-to-avoid issue that can have serious consequences. Specifically, for models trained without consideration of memorization, we describe new, efficient procedures that can extract unique, secret sequences, such as credit card numbers. We show that our testing strategy is a practical and easy-to-use first line of defense, e.g., by describing its application to quantitatively limit data exposure in Google's Smart Compose, a commercial text-completion neural network trained on millions of users' email messages.

从历史上看，机器学习方法尚未考虑安全性。反过来，这使得对普发阿的例子产生了上升，仔细扰动的输入样本旨在在测试时间误导检测，这已被应用于攻击垃圾邮件和恶意软件分类，以及最近攻击图像分类。因此，对设计对逆势示例具有鲁棒的机器学习方法已经致力于设计丰富的研究。不幸的是，除了坚固的机器学习模型必须满足，如公平和隐私，还有颠覆性。宋等人最近的工作。 （2019）经验上显示，强大和私人机器学习模型之间存在权衡。设计为强大的对抗性示例的模型通常会在培训数据上过度超过标准（非鲁棒）模型。如果数据集包含私人信息，那么通过观察模型的输出分隔训练和测试数据的任何统计测试都可以代表隐私漏洞，如果培训数据的模型过度，这些统计测试变得更容易。在这项工作中，我们确定标准型号在与强大的模型相比的更大程度上的设置，以及在以前的作品中经验观察到的情况，发生相反行为的设置。因此，不一定是必须牺牲隐私以实现稳健性的情况。过度的程度自然取决于可用于培训的数据量。我们继续，以培训在简单的高斯数据任务中培训强大的模型，培训培训规模因素的特点是如何通过培训一个强大的高斯数据任务，并验证我们的研究结果在图像分类基准数据集上，例如Cifar-10和CiFar-100 。

除了近年来数据收集和分析技术的快速开发外，还越来越强调需要解决与此类数据使用相关的信息泄漏。为此，隐私文献中的许多工作都致力于保护个人用户和数据贡献者。但是，某些情况需要不同的数据机密性概念，涉及数据集记录的全局属性。这样的信息保护概念尤其适用于业务和组织数据，在这些数据中，全球财产可能反映商业秘密或人口统计数据，如果不当行为可能是有害的。最新关于财产推断攻击的工作还显示了数据分析算法如何容易泄漏数据的这些全局性能，从而强调了开发可以保护此类信息的机制的重要性。在这项工作中，我们演示了如何应用分发隐私框架来形式化保护数据集的全球属性的问题。鉴于此框架，我们研究了一些提供数据机密性概念的机制及其权衡。我们分析了这些机制在各种数据假设下提供的理论保护保证，然后对几个数据分析任务进行实施并经验评估这些机制。我们的实验结果表明，我们的机制确实可以降低实用性推理攻击的有效性，同时提供的实用性大大超过了原油差异的隐私基线。因此，我们的工作为保护数据集的全球性质的理论支持机制提供了基础。

A distribution inference attack aims to infer statistical properties of data used to train machine learning models. These attacks are sometimes surprisingly potent, but the factors that impact distribution inference risk are not well understood and demonstrated attacks often rely on strong and unrealistic assumptions such as full knowledge of training environments even in supposedly black-box threat scenarios. To improve understanding of distribution inference risks, we develop a new black-box attack that even outperforms the best known white-box attack in most settings. Using this new attack, we evaluate distribution inference risk while relaxing a variety of assumptions about the adversary's knowledge under black-box access, like known model architectures and label-only access. Finally, we evaluate the effectiveness of previously proposed defenses and introduce new defenses. We find that although noise-based defenses appear to be ineffective, a simple re-sampling defense can be highly effective. Code is available at https://github.com/iamgroot42/dissecting_distribution_inference

差异隐私通常使用比理论更大的隐私参数应用于理想的理想。已经提出了宽大隐私参数的各种非正式理由。在这项工作中，我们考虑了部分差异隐私（DP），该隐私允许以每个属性为基础量化隐私保证。在此框架中，我们研究了几个基本数据分析和学习任务，并设计了其每个属性隐私参数的算法，其较小的人（即所有属性）的最佳隐私参数比最佳的隐私参数。