Deep Learning has recently become hugely popular in machine learning for its ability to solve end-to-end learning systems, in which the features and the classifiers are learned simultaneously, providing significant improvements in classification accuracy in the presence of highly-structured and large databases.Its success is due to a combination of recent algorithmic breakthroughs, increasingly powerful computers, and access to significant amounts of data.Researchers have also considered privacy implications of deep learning. Models are typically trained in a centralized manner with all the data being processed by the same training algorithm. If the data is a collection of users' private data, including habits, personal pictures, geographical positions, interests, and more, the centralized server will have access to sensitive information that could potentially be mishandled. To tackle this problem, collaborative deep learning models have recently been proposed where parties locally train their deep learning structures and only share a subset of the parameters in the attempt to keep their respective training sets private. Parameters can also be obfuscated via differential privacy (DP) to make information extraction even more challenging, as proposed by Shokri and Shmatikov at CCS'15.Unfortunately, we show that any privacy-preserving collaborative deep learning is susceptible to a powerful attack that we devise in this paper. In particular, we show that a distributed, federated, or decentralized deep learning approach is fundamentally broken and does not protect the training sets of honest participants. The attack we developed exploits the real-time nature of the learning process that allows the adversary to train a Generative Adversarial Network (GAN) that generates prototypical samples of the targeted training set that was meant to be private (the samples generated by the GAN are intended to come from the same distribution as the training data). Interestingly, we show that record-level differential privacy applied to the shared parameters of the model, as suggested in previous work, is ineffective (i.e., record-level DP is not designed to address our attack).

Collaborative machine learning and related techniques such as federated learning allow multiple participants, each with his own training dataset, to build a joint model by training locally and periodically exchanging model updates. We demonstrate that these updates leak unintended information about participants' training data and develop passive and active inference attacks to exploit this leakage. First, we show that an adversarial participant can infer the presence of exact data points-for example, specific locations-in others' training data (i.e., membership inference). Then, we show how this adversary can infer properties that hold only for a subset of the training data and are independent of the properties that the joint model aims to capture. For example, he can infer when a specific person first appears in the photos used to train a binary gender classifier. We evaluate our attacks on a variety of tasks, datasets, and learning configurations, analyze their limitations, and discuss possible defenses.

Federated learning is a collaborative method that aims to preserve data privacy while creating AI models. Current approaches to federated learning tend to rely heavily on secure aggregation protocols to preserve data privacy. However, to some degree, such protocols assume that the entity orchestrating the federated learning process (i.e., the server) is not fully malicious or dishonest. We investigate vulnerabilities to secure aggregation that could arise if the server is fully malicious and attempts to obtain access to private, potentially sensitive data. Furthermore, we provide a method to further defend against such a malicious server, and demonstrate effectiveness against known attacks that reconstruct data in a federated learning setting.

我们调查分裂学习的安全 - 一种新颖的协作机器学习框架，通过需要最小的资源消耗来实现峰值性能。在本文中，我们通过介绍客户私人培训集重建的一般攻击策略来揭示议定书的脆弱性并展示其固有的不安全。更突出地，我们表明恶意服务器可以积极地劫持分布式模型的学习过程，并将其纳入不安全状态，从而为客户端提供推动攻击。我们实施不同的攻击调整，并在各种数据集中测试它们以及现实的威胁方案。我们证明我们的攻击能够克服最近提出的防御技术，旨在提高分裂学习议定书的安全性。最后，我们还通过扩展以前设计的联合学习的攻击来说明协议对恶意客户的不安全性。要使我们的结果可重复，我们会在https://github.com/pasquini-dario/splitn_fsha提供的代码。

Deep learning based on artificial neural networks is a very popular approach to modeling, classifying, and recognizing complex data such as images, speech, and text. The unprecedented accuracy of deep learning methods has turned them into the foundation of new AI-based services on the Internet. Commercial companies that collect user data on a large scale have been the main beneficiaries of this trend since the success of deep learning techniques is directly proportional to the amount of data available for training.Massive data collection required for deep learning presents obvious privacy issues. Users' personal, highly sensitive data such as photos and voice recordings is kept indefinitely by the companies that collect it. Users can neither delete it, nor restrict the purposes for which it is used. Furthermore, centrally kept data is subject to legal subpoenas and extra-judicial surveillance. Many data owners-for example, medical institutions that may want to apply deep learning methods to clinical records-are prevented by privacy and confidentiality concerns from sharing the data and thus benefitting from large-scale deep learning.In this paper, we design, implement, and evaluate a practical system that enables multiple parties to jointly learn an accurate neuralnetwork model for a given objective without sharing their input datasets. We exploit the fact that the optimization algorithms used in modern deep learning, namely, those based on stochastic gradient descent, can be parallelized and executed asynchronously. Our system lets participants train independently on their own datasets and selectively share small subsets of their models' key parameters during training. This offers an attractive point in the utility/privacy tradeoff space: participants preserve the privacy of their respective data while still benefitting from other participants' models and thus boosting their learning accuracy beyond what is achievable solely on their own inputs. We demonstrate the accuracy of our privacypreserving deep learning on benchmark datasets.

从公共机器学习（ML）模型中泄漏数据是一个越来越重要的领域，因为ML的商业和政府应用可以利用多个数据源，可能包括用户和客户的敏感数据。我们对几个方面的当代进步进行了全面的调查，涵盖了非自愿数据泄漏，这对ML模型很自然，潜在的恶毒泄漏是由隐私攻击引起的，以及目前可用的防御机制。我们专注于推理时间泄漏，这是公开可用模型的最可能场景。我们首先在不同的数据，任务和模型体系结构的背景下讨论什么是泄漏。然后，我们提出了跨非自愿和恶意泄漏的分类法，可用的防御措施，然后进行当前可用的评估指标和应用。我们以杰出的挑战和开放性的问题结束，概述了一些有希望的未来研究方向。

联合学习允许一组用户在私人训练数据集中培训深度神经网络。在协议期间，数据集永远不会留下各个用户的设备。这是通过要求每个用户向中央服务器发送“仅”模型更新来实现，从而汇总它们以更新深神经网络的参数。然而，已经表明，每个模型更新都具有关于用户数据集的敏感信息（例如，梯度反转攻击）。联合学习的最先进的实现通过利用安全聚合来保护这些模型更新：安全监控协议，用于安全地计算用户的模型更新的聚合。安全聚合是关键，以保护用户的隐私，因为它会阻碍服务器学习用户提供的个人模型更新的源，防止推断和数据归因攻击。在这项工作中，我们表明恶意服务器可以轻松地阐明安全聚合，就像后者未到位一样。我们设计了两种不同的攻击，能够在参与安全聚合的用户数量上，独立于参与安全聚合的用户数。这使得它们在大规模现实世界联邦学习应用中的具体威胁。攻击是通用的，不瞄准任何特定的安全聚合协议。即使安全聚合协议被其理想功能替换为提供完美的安全性的理想功能，它们也同样有效。我们的工作表明，安全聚合与联合学习相结合，当前实施只提供了“虚假的安全感”。

窃取对受控信息的攻击，以及越来越多的信息泄漏事件，已成为近年来新兴网络安全威胁。由于蓬勃发展和部署先进的分析解决方案，新颖的窃取攻击利用机器学习（ML）算法来实现高成功率并导致大量损坏。检测和捍卫这种攻击是挑战性和紧迫的，因此政府，组织和个人应该非常重视基于ML的窃取攻击。本调查显示了这种新型攻击和相应对策的最新进展。以三类目标受控信息的视角审查了基于ML的窃取攻击，包括受控用户活动，受控ML模型相关信息和受控认证信息。最近的出版物总结了概括了总体攻击方法，并导出了基于ML的窃取攻击的限制和未来方向。此外，提出了从三个方面制定有效保护的对策 - 检测，破坏和隔离。

联合学习使多个用户能够通过共享其模型更新（渐变）来构建联合模型，而其原始数据在其设备上保持本地。与常见的信念相比，这提供了隐私福利，我们在共享渐变时，我们在这里增加了隐私风险的最新结果。具体而言，我们调查梯度（LLG）的标签泄漏，这是一种新建攻击，从他们的共享梯度提取用户培训数据的标签。该攻击利用梯度的方向和幅度来确定任何标签的存在或不存在。 LLG简单且有效，能够泄漏由标签表示的电位敏感信息，并缩放到任意批量尺寸和多个类别。在数学上以及经验上证明了不同设置下攻击的有效性。此外，经验结果表明，LLG在模型训练的早期阶段以高精度成功提取标签。我们还讨论了针对这种泄漏的不同防御机制。我们的研究结果表明，梯度压缩是减轻攻击的实用技术。

Deep neural networks are susceptible to various inference attacks as they remember information about their training data. We design white-box inference attacks to perform a comprehensive privacy analysis of deep learning models. We measure the privacy leakage through parameters of fully trained models as well as the parameter updates of models during training. We design inference algorithms for both centralized and federated learning, with respect to passive and active inference attackers, and assuming different adversary prior knowledge.We evaluate our novel white-box membership inference attacks against deep learning algorithms to trace their training data records. We show that a straightforward extension of the known black-box attacks to the white-box setting (through analyzing the outputs of activation functions) is ineffective. We therefore design new algorithms tailored to the white-box setting by exploiting the privacy vulnerabilities of the stochastic gradient descent algorithm, which is the algorithm used to train deep neural networks. We investigate the reasons why deep learning models may leak information about their training data. We then show that even well-generalized models are significantly susceptible to white-box membership inference attacks, by analyzing stateof-the-art pre-trained and publicly available models for the CIFAR dataset. We also show how adversarial participants, in the federated learning setting, can successfully run active membership inference attacks against other participants, even when the global model achieves high prediction accuracies.

在联合学习（FL）中，数据不会在联合培训机器学习模型时留下个人设备。相反，这些设备与中央党（例如，公司）共享梯度。因为数据永远不会“离开”个人设备，因此FL作为隐私保留呈现。然而，最近显示这种保护是一个薄的外观，甚至是一种被动攻击者观察梯度可以重建各个用户的数据。在本文中，我们争辩说，事先工作仍然很大程度上低估了FL的脆弱性。这是因为事先努力专门考虑被动攻击者，这些攻击者是诚实但好奇的。相反，我们介绍了一个活跃和不诚实的攻击者，作为中央会，他们能够在用户计算模型渐变之前修改共享模型的权重。我们称之为修改的重量“陷阱重量”。我们的活跃攻击者能够完全恢复用户数据，并在接近零成本时：攻击不需要复杂的优化目标。相反，它利用了模型梯度的固有数据泄漏，并通过恶意改变共享模型的权重来放大这种效果。这些特异性使我们的攻击能够扩展到具有大型迷你批次数据的模型。如果来自现有工作的攻击者需要小时才能恢复单个数据点，我们的方法需要毫秒来捕获完全连接和卷积的深度神经网络的完整百分之批次数据。最后，我们考虑缓解。我们观察到，FL中的差异隐私（DP）的当前实现是有缺陷的，因为它们明确地信任中央会，并在增加DP噪音的关键任务，因此不提供对恶意中央党的保护。我们还考虑其他防御，并解释为什么它们类似地不足。它需要重新设计FL，为用户提供任何有意义的数据隐私。

机器学习（ML）模型已广泛应用于各种应用，包括图像分类，文本生成，音频识别和图形数据分析。然而，最近的研究表明，ML模型容易受到隶属推导攻击（MIS），其目的是推断数据记录是否用于训练目标模型。 ML模型上的MIA可以直接导致隐私违规行为。例如，通过确定已经用于训练与某种疾病相关的模型的临床记录，攻击者可以推断临床记录的所有者具有很大的机会。近年来，MIS已被证明对各种ML模型有效，例如，分类模型和生成模型。同时，已经提出了许多防御方法来减轻米西亚。虽然ML模型上的MIAS形成了一个新的新兴和快速增长的研究区，但还没有对这一主题进行系统的调查。在本文中，我们对会员推论和防御进行了第一个全面调查。我们根据其特征提供攻击和防御的分类管理，并讨论其优点和缺点。根据本次调查中确定的限制和差距，我们指出了几个未来的未来研究方向，以激发希望遵循该地区的研究人员。这项调查不仅是研究社区的参考，而且还为该研究领域之外的研究人员带来了清晰的照片。为了进一步促进研究人员，我们创建了一个在线资源存储库，并与未来的相关作品继续更新。感兴趣的读者可以在https://github.com/hongshenghu/membership-inference-machine-learning-literature找到存储库。

In terms of artificial intelligence, there are several security and privacy deficiencies in the traditional centralized training methods of machine learning models by a server. To address this limitation, federated learning (FL) has been proposed and is known for breaking down ``data silos" and protecting the privacy of users. However, FL has not yet gained popularity in the industry, mainly due to its security, privacy, and high cost of communication. For the purpose of advancing the research in this field, building a robust FL system, and realizing the wide application of FL, this paper sorts out the possible attacks and corresponding defenses of the current FL system systematically. Firstly, this paper briefly introduces the basic workflow of FL and related knowledge of attacks and defenses. It reviews a great deal of research about privacy theft and malicious attacks that have been studied in recent years. Most importantly, in view of the current three classification criteria, namely the three stages of machine learning, the three different roles in federated learning, and the CIA (Confidentiality, Integrity, and Availability) guidelines on privacy protection, we divide attack approaches into two categories according to the training stage and the prediction stage in machine learning. Furthermore, we also identify the CIA property violated for each attack method and potential attack role. Various defense mechanisms are then analyzed separately from the level of privacy and security. Finally, we summarize the possible challenges in the application of FL from the aspect of attacks and defenses and discuss the future development direction of FL systems. In this way, the designed FL system has the ability to resist different attacks and is more secure and stable.

In recent years, mobile devices are equipped with increasingly advanced sensing and computing capabilities. Coupled with advancements in Deep Learning (DL), this opens up countless possibilities for meaningful applications, e.g., for medical purposes and in vehicular networks. Traditional cloudbased Machine Learning (ML) approaches require the data to be centralized in a cloud server or data center. However, this results in critical issues related to unacceptable latency and communication inefficiency. To this end, Mobile Edge Computing (MEC) has been proposed to bring intelligence closer to the edge, where data is produced. However, conventional enabling technologies for ML at mobile edge networks still require personal data to be shared with external parties, e.g., edge servers. Recently, in light of increasingly stringent data privacy legislations and growing privacy concerns, the concept of Federated Learning (FL) has been introduced. In FL, end devices use their local data to train an ML model required by the server. The end devices then send the model updates rather than raw data to the server for aggregation. FL can serve as an enabling technology in mobile edge networks since it enables the collaborative training of an ML model and also enables DL for mobile edge network optimization. However, in a large-scale and complex mobile edge network, heterogeneous devices with varying constraints are involved. This raises challenges of communication costs, resource allocation, and privacy and security in the implementation of FL at scale. In this survey, we begin with an introduction to the background and fundamentals of FL. Then, we highlight the aforementioned challenges of FL implementation and review existing solutions. Furthermore, we present the applications of FL for mobile edge network optimization. Finally, we discuss the important challenges and future research directions in FL.

如今，深度学习模型的所有者和开发人员必须考虑其培训数据的严格隐私保护规则，通常是人群来源且保留敏感信息。如今，深入学习模型执行隐私保证的最广泛采用的方法依赖于实施差异隐私的优化技术。根据文献，这种方法已被证明是针对多种模型的隐私攻击的成功防御，但其缺点是对模型的性能的实质性降级。在这项工作中，我们比较了差异私有的随机梯度下降（DP-SGD）算法与使用正则化技术的标准优化实践的有效性。我们分析了生成模型的实用程序，培训性能以及成员推理和模型反转攻击对学习模型的有效性。最后，我们讨论了差异隐私的缺陷和限制，并从经验上证明了辍学和L2型规范的卓越保护特性。

联邦学习一直是一个热门的研究主题，使不同组织的机器学习模型的协作培训在隐私限制下。随着研究人员试图支持更多具有不同隐私方法的机器学习模型，需要开发系统和基础设施，以便于开发各种联合学习算法。类似于Pytorch和Tensorflow等深度学习系统，可以增强深度学习的发展，联邦学习系统（FLSS）是等效的，并且面临各个方面的面临挑战，如有效性，效率和隐私。在本调查中，我们对联合学习系统进行了全面的审查。为实现流畅的流动和引导未来的研究，我们介绍了联合学习系统的定义并分析了系统组件。此外，我们根据六种不同方面提供联合学习系统的全面分类，包括数据分布，机器学习模型，隐私机制，通信架构，联合集市和联合的动机。分类可以帮助设计联合学习系统，如我们的案例研究所示。通过系统地总结现有联合学习系统，我们展示了设计因素，案例研究和未来的研究机会。

机器学习的最新进展使其在不同领域的广泛应用程序，最令人兴奋的应用程序之一是自动驾驶汽车（AV），这鼓励了从感知到预测到计划的许多ML算法的开发。但是，培训AV通常需要从不同驾驶环境（例如城市）以及不同类型的个人信息（例如工作时间和路线）收集的大量培训数据。这种收集的大数据被视为以数据为中心的AI时代的ML新油，通常包含大量对隐私敏感的信息，这些信息很难删除甚至审核。尽管现有的隐私保护方法已经取得了某些理论和经验成功，但将它们应用于自动驾驶汽车等现实世界应用时仍存在差距。例如，当培训AVS时，不仅可以单独识别的信息揭示对隐私敏感的信息，还可以揭示人口级别的信息，例如城市内的道路建设以及AVS的专有商业秘密。因此，重新审视AV中隐私风险和相应保护方法的前沿以弥合这一差距至关重要。遵循这一目标，在这项工作中，我们为AVS中的隐私风险和保护方法提供了新的分类法，并将AV中的隐私分为三个层面：个人，人口和专有。我们明确列出了保护每个级别的隐私级别，总结这些挑战的现有解决方案，讨论课程和结论，并为研究人员和从业者提供潜在的未来方向和机会。我们认为，这项工作将有助于塑造AV中的隐私研究，并指导隐私保护技术设计。

Federated learning facilitates the collaborative training of models without the sharing of raw data. However, recent attacks demonstrate that simply maintaining data locality during training processes does not provide sufficient privacy guarantees. Rather, we need a federated learning system capable of preventing inference over both the messages exchanged during training and the final trained model while ensuring the resulting model also has acceptable predictive accuracy. Existing federated learning approaches either use secure multiparty computation (SMC) which is vulnerable to inference or differential privacy which can lead to low accuracy given a large number of parties with relatively small amounts of data each. In this paper, we present an alternative approach that utilizes both differential privacy and SMC to balance these trade-offs. Combining differential privacy with secure multiparty computation enables us to reduce the growth of noise injection as the number of parties increases without sacrificing privacy while maintaining a pre-defined rate of trust. Our system is therefore a scalable approach that protects against inference threats and produces models with high accuracy. Additionally, our system can be used to train a variety of machine learning models, which we validate with experimental results on 3 different machine learning algorithms. Our experiments demonstrate that our approach out-performs state of the art solutions. CCS CONCEPTS• Security and privacy → Privacy-preserving protocols; Trust frameworks; • Computing methodologies → Learning settings.

鉴于对机器学习模型的访问，可以进行对手重建模型的培训数据？这项工作从一个强大的知情对手的镜头研究了这个问题，他们知道除了一个之外的所有培训数据点。通过实例化混凝土攻击，我们表明重建此严格威胁模型中的剩余数据点是可行的。对于凸模型（例如Logistic回归），重建攻击很简单，可以以封闭形式导出。对于更常规的模型（例如神经网络），我们提出了一种基于训练的攻击策略，该攻击策略接收作为输入攻击的模型的权重，并产生目标数据点。我们展示了我们对MNIST和CIFAR-10训练的图像分类器的攻击的有效性，并系统地研究了标准机器学习管道的哪些因素影响重建成功。最后，我们从理论上调查了有多差异的隐私足以通过知情对手减轻重建攻击。我们的工作提供了有效的重建攻击，模型开发人员可以用于评估超出以前作品中考虑的一般设置中的个别点的记忆（例如，生成语言模型或访问培训梯度）;它表明，标准模型具有存储足够信息的能力，以实现培训数据点的高保真重建;它表明，差异隐私可以成功减轻该参数制度中的攻击，其中公用事业劣化最小。

We quantitatively investigate how machine learning models leak information about the individual data records on which they were trained. We focus on the basic membership inference attack: given a data record and black-box access to a model, determine if the record was in the model's training dataset. To perform membership inference against a target model, we make adversarial use of machine learning and train our own inference model to recognize differences in the target model's predictions on the inputs that it trained on versus the inputs that it did not train on.We empirically evaluate our inference techniques on classification models trained by commercial "machine learning as a service" providers such as Google and Amazon. Using realistic datasets and classification tasks, including a hospital discharge dataset whose membership is sensitive from the privacy perspective, we show that these models can be vulnerable to membership inference attacks. We then investigate the factors that influence this leakage and evaluate mitigation strategies.