Differential privacy is a strong notion for privacy that can be used to prove formal guarantees, in terms of a privacy budget, , about how much information is leaked by a mechanism. However, implementations of privacy-preserving machine learning often select large values of in order to get acceptable utility of the model, with little understanding of the impact of such choices on meaningful privacy. Moreover, in scenarios where iterative learning procedures are used, differential privacy variants that offer tighter analyses are used which appear to reduce the needed privacy budget but present poorly understood trade-offs between privacy and utility. In this paper, we quantify the impact of these choices on privacy in experiments with logistic regression and neural network models. Our main finding is that there is a huge gap between the upper bounds on privacy loss that can be guaranteed, even with advanced mechanisms, and the effective privacy loss that can be measured using current inference attacks. Current mechanisms for differentially private machine learning rarely offer acceptable utility-privacy trade-offs with guarantees for complex learning tasks: settings that provide limited accuracy loss provide meaningless privacy guarantees, and settings that provide strong privacy guarantees result in useless models.

差异隐私（DP）已被出现为严格的形式主义，以推理可量化的隐私泄漏。在机器学习（ML）中，已采用DP限制推理/披露训练示例。在现有的工作中杠杆横跨ML管道，尽管隔离，通常专注于梯度扰动等机制。在本文中，我们展示了DP-util，DP整体实用分析框架，跨越ML管道，重点是输入扰动，客观扰动，梯度扰动，输出扰动和预测扰动。在隐私敏感数据上给出ML任务，DP-Util使ML隐私从业者能够对DP在这五个扰动点中的影响，以模型公用事业丢失，隐私泄漏和真正透露的数量来测量DP的影响。训练样本。我们在视觉，医疗和金融数据集上使用两个代表性学习算法（Logistic回归和深神经网络）来评估DP-Uts，以防止会员资格推论攻击作为案例研究攻击。我们结果的一个亮点是，预测扰动一致地在所有数据集中始终如一地实现所有模型的最低实用损耗。在Logistic回归模型中，与其他扰动技术相比，客观扰动导致最低的隐私泄漏。对于深度神经网络，梯度扰动导致最低的隐私泄漏。此外，我们的结果揭示了记录的结果表明，由于隐私泄漏增加，差异私有模型揭示了更多数量的成员样本。总体而言，我们的研究结果表明，为了使使用的扰动机制有明智的决定，ML隐私从业者需要检查优化技术（凸与非凸），扰动机制，课程数量和隐私预算之间的动态。

如今，深度学习模型的所有者和开发人员必须考虑其培训数据的严格隐私保护规则，通常是人群来源且保留敏感信息。如今，深入学习模型执行隐私保证的最广泛采用的方法依赖于实施差异隐私的优化技术。根据文献，这种方法已被证明是针对多种模型的隐私攻击的成功防御，但其缺点是对模型的性能的实质性降级。在这项工作中，我们比较了差异私有的随机梯度下降（DP-SGD）算法与使用正则化技术的标准优化实践的有效性。我们分析了生成模型的实用程序，培训性能以及成员推理和模型反转攻击对学习模型的有效性。最后，我们讨论了差异隐私的缺陷和限制，并从经验上证明了辍学和L2型规范的卓越保护特性。

模型可以公开有关其培训数据的敏感信息。在属性推理攻击中，对手对某些培训记录有部分知识，并访问了对这些记录进行培训的模型，并渗透了这些记录敏感功能的未知值。我们研究了一种属性推理的细粒变体，我们称为\ emph {敏感值推理}，其中对手的目标是高度置信度识别一些来自候选人集的记录，其中未知属性具有特定的敏感值。我们将属性推断与捕获培训分布统计数据的数据插补进行明确比较，该数据在对对手可用的培训数据的各种假设下进行了比较。我们的主要结论是：（1）以前的属性推理方法并没有比对手可以推断出有关训练数据的训练数据的更多信息，而无需访问训练的模型，而是对培训所需的基础分布相同的知识属性推理攻击； （2）Black-Box属性推理攻击很少学习没有模型的任何东西；但是（3）我们在论文中介绍和评估的白框攻击可以可靠地识别一些具有敏感值属性的记录，而这些记录在不访问模型的情况下无法预测。此外，我们表明提出的防御措施，例如私人培训和从培训中删除脆弱记录不会减轻这种隐私风险。我们的实验代码可在\ url {https://github.com/bargavj/evaluatingdpml}上获得。

We quantitatively investigate how machine learning models leak information about the individual data records on which they were trained. We focus on the basic membership inference attack: given a data record and black-box access to a model, determine if the record was in the model's training dataset. To perform membership inference against a target model, we make adversarial use of machine learning and train our own inference model to recognize differences in the target model's predictions on the inputs that it trained on versus the inputs that it did not train on.We empirically evaluate our inference techniques on classification models trained by commercial "machine learning as a service" providers such as Google and Amazon. Using realistic datasets and classification tasks, including a hospital discharge dataset whose membership is sensitive from the privacy perspective, we show that these models can be vulnerable to membership inference attacks. We then investigate the factors that influence this leakage and evaluate mitigation strategies.

我们审查在机器学习（ML）中使用差异隐私（DP）对隐私保护的使用。我们表明，在维护学习模型的准确性的驱动下，基于DP的ML实现非常宽松，以至于它们不提供DP的事前隐私保证。取而代之的是，他们提供的基本上是与传统（经常受到批评的）统计披露控制方法相似的噪声。由于缺乏正式的隐私保证，因此所提供的实际隐私水平必须经过实验评估，这很少进行。在这方面，我们提出的经验结果表明，ML中的标准反拟合技术可以比DP实现更好的实用性/隐私/效率权衡。

机器学习（ML）模型已广泛应用于各种应用，包括图像分类，文本生成，音频识别和图形数据分析。然而，最近的研究表明，ML模型容易受到隶属推导攻击（MIS），其目的是推断数据记录是否用于训练目标模型。 ML模型上的MIA可以直接导致隐私违规行为。例如，通过确定已经用于训练与某种疾病相关的模型的临床记录，攻击者可以推断临床记录的所有者具有很大的机会。近年来，MIS已被证明对各种ML模型有效，例如，分类模型和生成模型。同时，已经提出了许多防御方法来减轻米西亚。虽然ML模型上的MIAS形成了一个新的新兴和快速增长的研究区，但还没有对这一主题进行系统的调查。在本文中，我们对会员推论和防御进行了第一个全面调查。我们根据其特征提供攻击和防御的分类管理，并讨论其优点和缺点。根据本次调查中确定的限制和差距，我们指出了几个未来的未来研究方向，以激发希望遵循该地区的研究人员。这项调查不仅是研究社区的参考，而且还为该研究领域之外的研究人员带来了清晰的照片。为了进一步促进研究人员，我们创建了一个在线资源存储库，并与未来的相关作品继续更新。感兴趣的读者可以在https://github.com/hongshenghu/membership-inference-machine-learning-literature找到存储库。

鉴于对机器学习模型的访问，可以进行对手重建模型的培训数据？这项工作从一个强大的知情对手的镜头研究了这个问题，他们知道除了一个之外的所有培训数据点。通过实例化混凝土攻击，我们表明重建此严格威胁模型中的剩余数据点是可行的。对于凸模型（例如Logistic回归），重建攻击很简单，可以以封闭形式导出。对于更常规的模型（例如神经网络），我们提出了一种基于训练的攻击策略，该攻击策略接收作为输入攻击的模型的权重，并产生目标数据点。我们展示了我们对MNIST和CIFAR-10训练的图像分类器的攻击的有效性，并系统地研究了标准机器学习管道的哪些因素影响重建成功。最后，我们从理论上调查了有多差异的隐私足以通过知情对手减轻重建攻击。我们的工作提供了有效的重建攻击，模型开发人员可以用于评估超出以前作品中考虑的一般设置中的个别点的记忆（例如，生成语言模型或访问培训梯度）;它表明，标准模型具有存储足够信息的能力，以实现培训数据点的高保真重建;它表明，差异隐私可以成功减轻该参数制度中的攻击，其中公用事业劣化最小。

A distribution inference attack aims to infer statistical properties of data used to train machine learning models. These attacks are sometimes surprisingly potent, but the factors that impact distribution inference risk are not well understood and demonstrated attacks often rely on strong and unrealistic assumptions such as full knowledge of training environments even in supposedly black-box threat scenarios. To improve understanding of distribution inference risks, we develop a new black-box attack that even outperforms the best known white-box attack in most settings. Using this new attack, we evaluate distribution inference risk while relaxing a variety of assumptions about the adversary's knowledge under black-box access, like known model architectures and label-only access. Finally, we evaluate the effectiveness of previously proposed defenses and introduce new defenses. We find that although noise-based defenses appear to be ineffective, a simple re-sampling defense can be highly effective. Code is available at https://github.com/iamgroot42/dissecting_distribution_inference

员额推理攻击允许对训练的机器学习模型进行对手以预测模型的训练数据集中包含特定示例。目前使用平均案例的“精度”度量来评估这些攻击，该攻击未能表征攻击是否可以自信地识别培训集的任何成员。我们认为，应该通过计算其低（例如<0.1％）假阳性率来计算攻击来评估攻击，并在以这种方式评估时发现大多数事先攻击差。为了解决这一问题，我们开发了一个仔细结合文献中多种想法的似然比攻击（Lira）。我们的攻击是低于虚假阳性率的10倍，并且在攻击现有度量的情况下也严格占主导地位。

从公共机器学习（ML）模型中泄漏数据是一个越来越重要的领域，因为ML的商业和政府应用可以利用多个数据源，可能包括用户和客户的敏感数据。我们对几个方面的当代进步进行了全面的调查，涵盖了非自愿数据泄漏，这对ML模型很自然，潜在的恶毒泄漏是由隐私攻击引起的，以及目前可用的防御机制。我们专注于推理时间泄漏，这是公开可用模型的最可能场景。我们首先在不同的数据，任务和模型体系结构的背景下讨论什么是泄漏。然后，我们提出了跨非自愿和恶意泄漏的分类法，可用的防御措施，然后进行当前可用的评估指标和应用。我们以杰出的挑战和开放性的问题结束，概述了一些有希望的未来研究方向。

在其培训集中，给定训练有素的模型泄漏了多少培训模型泄露？会员资格推理攻击用作审计工具，以量化模型在其训练集中泄漏的私人信息。会员推理攻击受到不同不确定性的影响，即攻击者必须解决培训数据，培训算法和底层数据分布。因此，攻击成功率，在文献中的许多攻击，不要精确地捕获模型的信息泄漏关于他们的数据，因为它们还反映了攻击算法具有的其他不确定性。在本文中，我们解释了隐含的假设以及使用假设检测框架在现有工作中进行的简化。我们还从框架中获得了新的攻击算法，可以实现高AUC分数，同时还突出显示影响其性能的不同因素。我们的算法捕获模型中隐私损失的非常精确的近似，并且可以用作在机器学习模型中执行准确和了解的隐私风险的工具。我们对各种机器学习任务和基准数据集的攻击策略提供了彻底的实证评估。

Machine learning algorithms, when applied to sensitive data, pose a distinct threat to privacy. A growing body of prior work demonstrates that models produced by these algorithms may leak specific private information in the training data to an attacker, either through the models' structure or their observable behavior. However, the underlying cause of this privacy risk is not well understood beyond a handful of anecdotal accounts that suggest overfitting and influence might play a role.This paper examines the effect that overfitting and influence have on the ability of an attacker to learn information about the training data from machine learning models, either through training set membership inference or attribute inference attacks. Using both formal and empirical analyses, we illustrate a clear relationship between these factors and the privacy risk that arises in several popular machine learning algorithms. We find that overfitting is sufficient to allow an attacker to perform membership inference and, when the target attribute meets certain conditions about its influence, attribute inference attacks. Interestingly, our formal analysis also shows that overfitting is not necessary for these attacks and begins to shed light on what other factors may be in play. Finally, we explore the connection between membership inference and attribute inference, showing that there are deep connections between the two that lead to effective new attacks.

隐私敏感数据的培训机器学习模型已成为一种流行的练习，在不断扩大的田野中推动创新。这已经向新攻击打开了门，这可能会产生严重的隐私含义。一个这样的攻击，会员推导攻击（MIA），暴露了特定数据点是否用于训练模型。一种越来越多的文献使用差异的私人（DP）训练算法作为反对这种攻击的辩护。但是，这些作品根据限制假设评估防御，即所有培训集以及非成员的所有成员都是独立的并相同分布的。这种假设没有在文献中的许多真实用例中占据。由此激励，我们评估隶属于样本之间的统计依赖性，并解释为什么DP不提供有意义的保护（在这种更常规的情况下，培训集尺寸$ N $的隐私参数$ \ epsilon $ scales）。我们使用从现实世界数据构建的培训集进行了一系列实证评估，其中包括示出样品之间的不同类型依赖性的培训集。我们的结果表明，培训集依赖关系可能会严重增加MIS的性能，因此假设数据样本在统计上独立，可以显着低估均撒的性能。

Deploying machine learning models in production may allow adversaries to infer sensitive information about training data. There is a vast literature analyzing different types of inference risks, ranging from membership inference to reconstruction attacks. Inspired by the success of games (i.e., probabilistic experiments) to study security properties in cryptography, some authors describe privacy inference risks in machine learning using a similar game-based style. However, adversary capabilities and goals are often stated in subtly different ways from one presentation to the other, which makes it hard to relate and compose results. In this paper, we present a game-based framework to systematize the body of knowledge on privacy inference risks in machine learning.

梯度泄漏攻击被认为是深度学习中的邪恶隐私威胁之一，因为攻击者在迭代培训期间隐蔽了梯度更新，而不会影响模型培训质量，但又使用泄漏的梯度逐步重建敏感培训数据，具有高攻击成功率。虽然具有差异隐私的深度学习是发布具有差异隐私保障的深度学习模型的违法标准，但我们展示了具有固定隐私参数的差异私有算法易受梯度泄漏攻击的影响。本文调查了差异隐私（DP）的梯度泄漏弹性深度学习的替代方法。首先，我们分析了差异隐私的深度学习的现有实现，它使用固定噪声方差使用固定隐私参数将恒定噪声对所有层中的梯度注入恒定噪声。尽管提供了DP保证，但该方法遭受了低精度，并且很容易受到梯度泄漏攻击。其次，通过使用动态隐私参数，我们提出了一种梯度泄漏弹性深度学习方法，差异隐私保证。与导致恒定噪声方差导致的固定参数策略不同，不同的动态参数策略存在替代技术，以引入自适应噪声方差和自适应噪声注入，其与差别私有模型训练期间的梯度更新的趋势紧密对齐。最后，我们描述了四个互补指标来评估和比较替代方法。

Machine learning techniques based on neural networks are achieving remarkable results in a wide variety of domains. Often, the training of models requires large, representative datasets, which may be crowdsourced and contain sensitive information. The models should not expose private information in these datasets. Addressing this goal, we develop new algorithmic techniques for learning and a refined analysis of privacy costs within the framework of differential privacy. Our implementation and experiments demonstrate that we can train deep neural networks with non-convex objectives, under a modest privacy budget, and at a manageable cost in software complexity, training efficiency, and model quality. * Google.† OpenAI. Work done while at Google.

我们研究在机器学习模型中部署经常性神经网络（RNNS）的隐私含义。我们专注于一类隐私威胁，称为会员推理攻击（MIAS），其旨在推断是否已用于培训模型的特定数据记录。考虑到三台机器学习应用，即机器翻译，深度加固学习和图像分类，我们提供了经验证据，即RNN比偏置更容易受到偏置的偏移架构。然后，我们研究差异隐私方法，以保护RNN的培训数据集的隐私。众所周知，这些方法提供了严谨的隐私，而不管对抗的模型如何保证。我们为所谓的DP-FedAVG算法开发替代差异隐私机制，而不是在训练期间混淆渐变，使模型的输出组合。与现有的工作不同，该机制允许培训隐私参数的后期调整，而无需重新培训模型。我们提供数值结果，表明该机制为米西亚提供了强烈的盾牌，同时交易边际效用。

Machine learning models leak information about the datasets on which they are trained. An adversary can build an algorithm to trace the individual members of a model's training dataset. As a fundamental inference attack, he aims to distinguish between data points that were part of the model's training set and any other data points from the same distribution. This is known as the tracing (and also membership inference) attack. In this paper, we focus on such attacks against black-box models, where the adversary can only observe the output of the model, but not its parameters. This is the current setting of machine learning as a service in the Internet.We introduce a privacy mechanism to train machine learning models that provably achieve membership privacy: the model's predictions on its training data are indistinguishable from its predictions on other data points from the same distribution. We design a strategic mechanism where the privacy mechanism anticipates the membership inference attacks. The objective is to train a model such that not only does it have the minimum prediction error (high utility), but also it is the most robust model against its corresponding strongest inference attack (high privacy). We formalize this as a min-max game optimization problem, and design an adversarial training algorithm that minimizes the classification loss of the model as well as the maximum gain of the membership inference attack against it. This strategy, which guarantees membership privacy (as prediction indistinguishability), acts also as a strong regularizer and significantly generalizes the model.We evaluate our privacy mechanism on deep neural networks using different benchmark datasets. We show that our min-max strategy can mitigate the risk of membership inference attacks (close to the random guess) with a negligible cost in terms of the classification error.

Deep neural networks are susceptible to various inference attacks as they remember information about their training data. We design white-box inference attacks to perform a comprehensive privacy analysis of deep learning models. We measure the privacy leakage through parameters of fully trained models as well as the parameter updates of models during training. We design inference algorithms for both centralized and federated learning, with respect to passive and active inference attackers, and assuming different adversary prior knowledge.We evaluate our novel white-box membership inference attacks against deep learning algorithms to trace their training data records. We show that a straightforward extension of the known black-box attacks to the white-box setting (through analyzing the outputs of activation functions) is ineffective. We therefore design new algorithms tailored to the white-box setting by exploiting the privacy vulnerabilities of the stochastic gradient descent algorithm, which is the algorithm used to train deep neural networks. We investigate the reasons why deep learning models may leak information about their training data. We then show that even well-generalized models are significantly susceptible to white-box membership inference attacks, by analyzing stateof-the-art pre-trained and publicly available models for the CIFAR dataset. We also show how adversarial participants, in the federated learning setting, can successfully run active membership inference attacks against other participants, even when the global model achieves high prediction accuracies.