Adversarial examples that fool machine learning models, particularly deep neural networks, have been a topic of intense research interest, with attacks and defenses being developed in a tight back-and-forth. Most past defenses are best effort and have been shown to be vulnerable to sophisticated attacks. Recently a set of certified defenses have been introduced, which provide guarantees of robustness to normbounded attacks. However these defenses either do not scale to large datasets or are limited in the types of models they can support. This paper presents the first certified defense that both scales to large networks and datasets (such as Google's Inception network for ImageNet) and applies broadly to arbitrary model types. Our defense, called PixelDP, is based on a novel connection between robustness against adversarial examples and differential privacy, a cryptographically-inspired privacy formalism, that provides a rigorous, generic, and flexible foundation for defense.

Adaptive attacks have (rightfully) become the de facto standard for evaluating defenses to adversarial examples. We find, however, that typical adaptive evaluations are incomplete. We demonstrate that thirteen defenses recently published at ICLR, ICML and NeurIPS-and which illustrate a diverse set of defense strategies-can be circumvented despite attempting to perform evaluations using adaptive attacks. While prior evaluation papers focused mainly on the end result-showing that a defense was ineffective-this paper focuses on laying out the methodology and the approach necessary to perform an adaptive attack. Some of our attack strategies are generalizable, but no single strategy would have been sufficient for all defenses. This underlines our key message that adaptive attacks cannot be automated and always require careful and appropriate tuning to a given defense. We hope that these analyses will serve as guidance on how to properly perform adaptive attacks against defenses to adversarial examples, and thus will allow the community to make further progress in building more robust models.

Neural networks provide state-of-the-art results for most machine learning tasks. Unfortunately, neural networks are vulnerable to adversarial examples: given an input x and any target classification t, it is possible to find a new input x that is similar to x but classified as t. This makes it difficult to apply neural networks in security-critical areas. Defensive distillation is a recently proposed approach that can take an arbitrary neural network, and increase its robustness, reducing the success rate of current attacks' ability to find adversarial examples from 95% to 0.5%.In this paper, we demonstrate that defensive distillation does not significantly increase the robustness of neural networks by introducing three new attack algorithms that are successful on both distilled and undistilled neural networks with 100% probability. Our attacks are tailored to three distance metrics used previously in the literature, and when compared to previous adversarial example generation algorithms, our attacks are often much more effective (and never worse). Furthermore, we propose using high-confidence adversarial examples in a simple transferability test we show can also be used to break defensive distillation. We hope our attacks will be used as a benchmark in future defense attempts to create neural networks that resist adversarial examples.

深度神经网络（DNN）的巨大进步导致了各种任务的最先进的性能。然而，最近的研究表明，DNNS容易受到对抗的攻击，这在将这些模型部署到自动驾驶等安全关键型应用时，这使得非常关注。已经提出了不同的防御方法，包括：a）经验防御，通常可以在不提供稳健性认证的情况下再次再次攻击; b）可认真的稳健方法，由稳健性验证组成，提供了在某些条件下的任何攻击和相应的强大培训方法中的稳健准确性的下限。在本文中，我们系统化了可认真的稳健方法和相关的实用和理论意义和调查结果。我们还提供了在不同数据集上现有的稳健验证和培训方法的第一个全面基准。特别是，我们1）为稳健性验证和培训方法提供分类，以及总结代表性算法的方法，2）揭示这些方法中的特征，优势，局限性和基本联系，3）讨论当前的研究进展情况TNN和4的可信稳健方法的理论障碍，主要挑战和未来方向提供了一个开放的统一平台，以评估超过20种代表可认真的稳健方法，用于各种DNN。

We show how to turn any classifier that classifies well under Gaussian noise into a new classifier that is certifiably robust to adversarial perturbations under the 2 norm. This "randomized smoothing" technique has been proposed recently in the literature, but existing guarantees are loose. We prove a tight robustness guarantee in 2 norm for smoothing with Gaussian noise. We use randomized smoothing to obtain an ImageNet classifier with e.g. a certified top-1 accuracy of 49% under adversarial perturbations with 2 norm less than 0.5 (=127/255). No certified defense has been shown feasible on ImageNet except for smoothing. On smaller-scale datasets where competing approaches to certified 2 robustness are viable, smoothing delivers higher certified accuracies. Our strong empirical results suggest that randomized smoothing is a promising direction for future research into adversarially robust classification. Code and models are available at http: //github.com/locuslab/smoothing.

最近的研究表明，深神经网络（DNN）易受对抗性攻击的影响，包括逃避和后门（中毒）攻击。在防守方面，有密集的努力，改善了对逃避袭击的经验和可怜的稳健性;然而，对后门攻击的可稳健性仍然很大程度上是未开发的。在本文中，我们专注于认证机器学习模型稳健性，反对一般威胁模型，尤其是后门攻击。我们首先通过随机平滑技术提供统一的框架，并展示如何实例化以证明对逃避和后门攻击的鲁棒性。然后，我们提出了第一个强大的培训过程Rab，以平滑训练有素的模型，并证明其稳健性对抗后门攻击。我们派生机学习模型的稳健性突出了培训的机器学习模型，并证明我们的鲁棒性受到紧张。此外，我们表明，可以有效地训练强大的平滑模型，以适用于诸如k最近邻分类器的简单模型，并提出了一种精确的平滑训练算法，该算法消除了从这种模型的噪声分布采样采样的需要。经验上，我们对MNIST，CIFAR-10和Imagenet数据集等DNN，差异私有DNN和K-NN模型等不同机器学习（ML）型号进行了全面的实验，并为反卧系攻击提供认证稳健性的第一个基准。此外，我们在SPAMBase表格数据集上评估K-NN模型，以展示所提出的精确算法的优点。对多元化模型和数据集的综合评价既有关于普通训练时间攻击的进一步强劲学习策略的多样化模型和数据集的综合评价。

联合学习（FL）提供了一个有效的范式，可以共同培训分布式用户的数据的全球模型。由于本地培训数据来自可能不值得信赖的不同用户，因此一些研究表明，FL容易受到中毒攻击的影响。同时，为了保护本地用户的隐私，FL始终以差异性私人方式（DPFL）进行培训。因此，在本文中，我们问：我们是否可以利用DPFL的先天隐私权来提供对中毒攻击的认证鲁棒性？我们可以进一步改善FL的隐私以改善这种认证吗？我们首先研究了FL的用户级和实例级别的隐私，并提出了新的机制以获得改进的实例级隐私。然后，我们提供两个鲁棒性认证标准：两级DPFL的认证预测和认证攻击成本。从理论上讲，我们证明了DPFL在有限数量的对抗用户或实例下的认证鲁棒性。从经验上讲，我们进行了广泛的实验，以在对不同数据集的一系列攻击下验证我们的理论。我们表明，具有更严格的隐私保证的DPFL总是在认证攻击成本方面提供更强的鲁棒性认证，但是在隐私保护和公用事业损失之间的适当平衡下，获得了最佳认证预测。

鉴于对机器学习模型的访问，可以进行对手重建模型的培训数据？这项工作从一个强大的知情对手的镜头研究了这个问题，他们知道除了一个之外的所有培训数据点。通过实例化混凝土攻击，我们表明重建此严格威胁模型中的剩余数据点是可行的。对于凸模型（例如Logistic回归），重建攻击很简单，可以以封闭形式导出。对于更常规的模型（例如神经网络），我们提出了一种基于训练的攻击策略，该攻击策略接收作为输入攻击的模型的权重，并产生目标数据点。我们展示了我们对MNIST和CIFAR-10训练的图像分类器的攻击的有效性，并系统地研究了标准机器学习管道的哪些因素影响重建成功。最后，我们从理论上调查了有多差异的隐私足以通过知情对手减轻重建攻击。我们的工作提供了有效的重建攻击，模型开发人员可以用于评估超出以前作品中考虑的一般设置中的个别点的记忆（例如，生成语言模型或访问培训梯度）;它表明，标准模型具有存储足够信息的能力，以实现培训数据点的高保真重建;它表明，差异隐私可以成功减轻该参数制度中的攻击，其中公用事业劣化最小。

由于机器学习（ML）系统变得普遍存在，因此保护其安全性至关重要。然而，最近已经证明，动机的对手能够通过使用语义转换扰乱测试数据来误导ML系统。虽然存在丰富的研究机构，但为ML模型提供了可提供的稳健性保证，以防止$ \ ell_p $ norm界限对抗对抗扰动，抵御语义扰动的保证仍然很广泛。在本文中，我们提供了TSS - 一种统一的框架，用于针对一般对抗性语义转换的鲁棒性认证。首先，根据每个转换的性质，我们将常见的变换划分为两类，即可解决的（例如，高斯模糊）和差异可解的（例如，旋转）变换。对于前者，我们提出了特定于转型的随机平滑策略并获得强大的稳健性认证。后者类别涵盖涉及插值错误的变换，我们提出了一种基于分层采样的新方法，以证明稳健性。我们的框架TSS利用这些认证策略并结合了一致性增强的培训，以提供严谨的鲁棒性认证。我们对十种挑战性语义转化进行了广泛的实验，并表明TSS显着优于现有技术。此外，据我们所知，TSS是第一种在大规模想象数据集上实现非竞争认证稳健性的方法。例如，我们的框架在ImageNet上实现了旋转攻击的30.4％认证的稳健准确性（在$ \ PM 30 ^ \ CIC $）。此外，要考虑更广泛的转换，我们展示了TSS对自适应攻击和不可预见的图像损坏，例如CIFAR-10-C和Imagenet-C。

Although deep neural networks (DNNs) have achieved great success in many tasks, they can often be fooled by adversarial examples that are generated by adding small but purposeful distortions to natural examples. Previous studies to defend against adversarial examples mostly focused on refining the DNN models, but have either shown limited success or required expensive computation. We propose a new strategy, feature squeezing, that can be used to harden DNN models by detecting adversarial examples. Feature squeezing reduces the search space available to an adversary by coalescing samples that correspond to many different feature vectors in the original space into a single sample. By comparing a DNN model's prediction on the original input with that on squeezed inputs, feature squeezing detects adversarial examples with high accuracy and few false positives.This paper explores two feature squeezing methods: reducing the color bit depth of each pixel and spatial smoothing. These simple strategies are inexpensive and complementary to other defenses, and can be combined in a joint detection framework to achieve high detection rates against state-of-the-art attacks.

已知深度神经网络（DNN）容易受到用不可察觉的扰动制作的对抗性示例的影响，即，输入图像的微小变化会引起错误的分类，从而威胁着基于深度学习的部署系统的可靠性。经常采用对抗训练（AT）来通过训练损坏和干净的数据的混合物来提高DNN的鲁棒性。但是，大多数基于AT的方法在处理\ textit {转移的对抗示例}方面是无效的，这些方法是生成以欺骗各种防御模型的生成的，因此无法满足现实情况下提出的概括要求。此外，对抗性训练一般的国防模型不能对具有扰动的输入产生可解释的预测，而不同的领域专家则需要一个高度可解释的强大模型才能了解DNN的行为。在这项工作中，我们提出了一种基于Jacobian规范和选择性输入梯度正则化（J-SIGR）的方法，该方法通过Jacobian归一化提出了线性化的鲁棒性，还将基于扰动的显着性图正规化，以模仿模型的可解释预测。因此，我们既可以提高DNN的防御能力和高解释性。最后，我们评估了跨不同体系结构的方法，以针对强大的对抗性攻击。实验表明，提出的J-Sigr赋予了针对转移的对抗攻击的鲁棒性，我们还表明，来自神经网络的预测易于解释。

Deep learning algorithms have been shown to perform extremely well on many classical machine learning problems. However, recent studies have shown that deep learning, like other machine learning techniques, is vulnerable to adversarial samples: inputs crafted to force a deep neural network (DNN) to provide adversary-selected outputs. Such attacks can seriously undermine the security of the system supported by the DNN, sometimes with devastating consequences. For example, autonomous vehicles can be crashed, illicit or illegal content can bypass content filters, or biometric authentication systems can be manipulated to allow improper access. In this work, we introduce a defensive mechanism called defensive distillation to reduce the effectiveness of adversarial samples on DNNs. We analytically investigate the generalizability and robustness properties granted by the use of defensive distillation when training DNNs. We also empirically study the effectiveness of our defense mechanisms on two DNNs placed in adversarial settings. The study shows that defensive distillation can reduce effectiveness of sample creation from 95% to less than 0.5% on a studied DNN. Such dramatic gains can be explained by the fact that distillation leads gradients used in adversarial sample creation to be reduced by a factor of 10 30 . We also find that distillation increases the average minimum number of features that need to be modified to create adversarial samples by about 800% on one of the DNNs we tested.

Adversarial examples are perturbed inputs designed to fool machine learning models. Adversarial training injects such examples into training data to increase robustness. To scale this technique to large datasets, perturbations are crafted using fast single-step methods that maximize a linear approximation of the model's loss. We show that this form of adversarial training converges to a degenerate global minimum, wherein small curvature artifacts near the data points obfuscate a linear approximation of the loss. The model thus learns to generate weak perturbations, rather than defend against strong ones. As a result, we find that adversarial training remains vulnerable to black-box attacks, where we transfer perturbations computed on undefended models, as well as to a powerful novel single-step attack that escapes the non-smooth vicinity of the input data via a small random step. We further introduce Ensemble Adversarial Training, a technique that augments training data with perturbations transferred from other models. On ImageNet, Ensemble Adversarial Training yields models with stronger robustness to blackbox attacks. In particular, our most robust model won the first round of the NIPS 2017 competition on Defenses against Adversarial Attacks (Kurakin et al., 2017c). However, subsequent work found that more elaborate black-box attacks could significantly enhance transferability and reduce the accuracy of our models.

尽管深层神经网络在各种任务中取得了巨大的成功，但它们对不可察觉的对抗性扰动的脆弱性阻碍了他们在现实世界中的部署。最近，与随机合奏的作品相对于经过最小的计算开销的标准对手训练（AT）模型，对对抗性训练（AT）模型的对抗性鲁棒性有了显着改善，这使它们成为安全临界资源限制应用程序的有前途解决方案。但是，这种令人印象深刻的表现提出了一个问题：这些稳健性是由随机合奏提供的吗？在这项工作中，我们从理论和经验上都解决了这个问题。从理论上讲，我们首先确定通常采用的鲁棒性评估方法（例如自适应PGD）在这种情况下提供了错误的安全感。随后，我们提出了一种理论上有效的对抗攻击算法（ARC），即使在自适应PGD无法做到这一点的情况下，也能妥协随机合奏。我们在各种网络体系结构，培训方案，数据集和规范上进行全面的实验，以支持我们的主张，并经验证明，随机合奏实际上比在模型上更容易受到$ \ ell_p $结合的对抗性扰动的影响。我们的代码可以在https://github.com/hsndbk4/arc上找到。

The authors thank Nicholas Carlini (UC Berkeley) and Dimitris Tsipras (MIT) for feedback to improve the survey quality. We also acknowledge X. Huang (Uni. Liverpool), K. R. Reddy (IISC), E. Valle (UNICAMP), Y. Yoo (CLAIR) and others for providing pointers to make the survey more comprehensive.

This paper investigates recently proposed approaches for defending against adversarial examples and evaluating adversarial robustness. We motivate adversarial risk as an objective for achieving models robust to worst-case inputs. We then frame commonly used attacks and evaluation metrics as defining a tractable surrogate objective to the true adversarial risk. This suggests that models may optimize this surrogate rather than the true adversarial risk. We formalize this notion as obscurity to an adversary, and develop tools and heuristics for identifying obscured models and designing transparent models. We demonstrate that this is a significant problem in practice by repurposing gradient-free optimization techniques into adversarial attacks, which we use to decrease the accuracy of several recently proposed defenses to near zero. Our hope is that our formulations and results will help researchers to develop more powerful defenses.

对抗性的鲁棒性已经成为深度学习的核心目标，无论是在理论和实践中。然而，成功的方法来改善对抗的鲁棒性（如逆势训练）在不受干扰的数据上大大伤害了泛化性能。这可能会对对抗性鲁棒性如何影响现实世界系统的影响（即，如果它可以提高未受干扰的数据的准确性），许多人可能选择放弃鲁棒性）。我们提出内插对抗培训，该培训最近雇用了在对抗培训框架内基于插值的基于插值的培训方法。在CiFar -10上，对抗性训练增加了标准测试错误（当没有对手时）从4.43％到12.32％，而我们的内插对抗培训我们保留了对抗性的鲁棒性，同时实现了仅6.45％的标准测试误差。通过我们的技术，强大模型标准误差的相对增加从178.1％降至仅为45.5％。此外，我们提供内插对抗性培训的数学分析，以确认其效率，并在鲁棒性和泛化方面展示其优势。

Differential privacy is a strong notion for privacy that can be used to prove formal guarantees, in terms of a privacy budget, , about how much information is leaked by a mechanism. However, implementations of privacy-preserving machine learning often select large values of in order to get acceptable utility of the model, with little understanding of the impact of such choices on meaningful privacy. Moreover, in scenarios where iterative learning procedures are used, differential privacy variants that offer tighter analyses are used which appear to reduce the needed privacy budget but present poorly understood trade-offs between privacy and utility. In this paper, we quantify the impact of these choices on privacy in experiments with logistic regression and neural network models. Our main finding is that there is a huge gap between the upper bounds on privacy loss that can be guaranteed, even with advanced mechanisms, and the effective privacy loss that can be measured using current inference attacks. Current mechanisms for differentially private machine learning rarely offer acceptable utility-privacy trade-offs with guarantees for complex learning tasks: settings that provide limited accuracy loss provide meaningless privacy guarantees, and settings that provide strong privacy guarantees result in useless models.

Neural networks are known to be vulnerable to adversarial examples: inputs that are close to natural inputs but classified incorrectly. In order to better understand the space of adversarial examples, we survey ten recent proposals that are designed for detection and compare their efficacy. We show that all can be defeated by constructing new loss functions. We conclude that adversarial examples are significantly harder to detect than previously appreciated, and the properties believed to be intrinsic to adversarial examples are in fact not. Finally, we propose several simple guidelines for evaluating future proposed defenses.

从外界培训的机器学习模型可能会被数据中毒攻击损坏，将恶意指向到模型的培训集中。对这些攻击的常见防御是数据消毒：在培训模型之前首先过滤出异常培训点。在本文中，我们开发了三次攻击，可以绕过广泛的常见数据消毒防御，包括基于最近邻居，训练损失和奇异值分解的异常探测器。通过增加3％的中毒数据，我们的攻击成功地将Enron垃圾邮件检测数据集的测试错误从3％增加到24％，并且IMDB情绪分类数据集从12％到29％。相比之下，没有明确占据这些数据消毒防御的现有攻击被他们击败。我们的攻击基于两个想法：（i）我们协调我们的攻击将中毒点彼此放置在彼此附近，（ii）我们将每个攻击制定为受限制的优化问题，限制旨在确保中毒点逃避检测。随着这种优化涉及解决昂贵的Bilevel问题，我们的三个攻击对应于基于影响功能的近似近似这个问题的方式; minimax二元性;和karush-kuhn-tucker（kkt）条件。我们的结果强调了对数据中毒攻击产生更强大的防御的必要性。