Adversarial examples have attracted significant attention in machine learning, but the reasons for their existence and pervasiveness remain unclear. We demonstrate that adversarial examples can be directly attributed to the presence of non-robust features: features (derived from patterns in the data distribution) that are highly predictive, yet brittle and (thus) incomprehensible to humans. After capturing these features within a theoretical framework, we establish their widespread existence in standard datasets. Finally, we present a simple setting where we can rigorously tie the phenomena we observe in practice to a misalignment between the (human-specified) notion of robustness and the inherent geometry of the data.

We show that there may exist an inherent tension between the goal of adversarial robustness and that of standard generalization. Specifically, training robust models may not only be more resource-consuming, but also lead to a reduction of standard accuracy. We demonstrate that this trade-off between the standard accuracy of a model and its robustness to adversarial perturbations provably exists in a fairly simple and natural setting. These findings also corroborate a similar phenomenon observed empirically in more complex settings. Further, we argue that this phenomenon is a consequence of robust classifiers learning fundamentally different feature representations than standard classifiers. These differences, in particular, seem to result in unexpected benefits: the representations learned by robust models tend to align better with salient data characteristics and human perception.

我们识别普遍对抗扰动（UAP）的性质，将它们与标准的对抗性扰动区分开来。具体而言，我们表明，由投影梯度下降产生的靶向UAPS表现出两种人对齐的特性：语义局部性和空间不变性，标准的靶向对抗扰动缺乏。我们还证明，除标准对抗扰动之外，UAPS含有明显较低的泛化信号 - 即，UAPS在比标准的对抗的扰动的较小程度上利用非鲁棒特征。

Recent work has demonstrated that deep neural networks are vulnerable to adversarial examples-inputs that are almost indistinguishable from natural data and yet classified incorrectly by the network. In fact, some of the latest findings suggest that the existence of adversarial attacks may be an inherent weakness of deep learning models. To address this problem, we study the adversarial robustness of neural networks through the lens of robust optimization. This approach provides us with a broad and unifying view on much of the prior work on this topic. Its principled nature also enables us to identify methods for both training and attacking neural networks that are reliable and, in a certain sense, universal. In particular, they specify a concrete security guarantee that would protect against any adversary. These methods let us train networks with significantly improved resistance to a wide range of adversarial attacks. They also suggest the notion of security against a first-order adversary as a natural and broad security guarantee. We believe that robustness against such well-defined classes of adversaries is an important stepping stone towards fully resistant deep learning models. 1

Machine learning models are often susceptible to adversarial perturbations of their inputs. Even small perturbations can cause state-of-the-art classifiers with high "standard" accuracy to produce an incorrect prediction with high confidence. To better understand this phenomenon, we study adversarially robust learning from the viewpoint of generalization. We show that already in a simple natural data model, the sample complexity of robust learning can be significantly larger than that of "standard" learning. This gap is information theoretic and holds irrespective of the training algorithm or the model family. We complement our theoretical results with experiments on popular image classification datasets and show that a similar gap exists here as well. We postulate that the difficulty of training robust classifiers stems, at least partially, from this inherently larger sample complexity.

删除攻击旨在通过略微扰动正确标记的训练示例的特征来大幅恶化学习模型的测试准确性。通过将这种恶意攻击正式地找到特定$ \ infty $ -wassersein球中的最坏情况培训数据，我们表明最小化扰动数据的对抗性风险相当于优化原始数据上的自然风险的上限。这意味着对抗性培训可以作为防止妄想攻击的原则防御。因此，通过普遍训练可以很大程度地回收测试精度。为了进一步了解国防的内部机制，我们披露了对抗性培训可以通过防止学习者过于依赖于自然环境中的非鲁棒特征来抵制妄想扰动。最后，我们将我们的理论调查结果与一系列关于流行的基准数据集进行了补充，这表明防御能够承受六种不同的实际攻击。在面对令人难以闻名的对手时，理论和经验结果投票给逆势训练。

在本讨论文件中，我们调查了有关机器学习模型鲁棒性的最新研究。随着学习算法在数据驱动的控制系统中越来越流行，必须确保它们对数据不确定性的稳健性，以维持可靠的安全至关重要的操作。我们首先回顾了这种鲁棒性的共同形式主义，然后继续讨论训练健壮的机器学习模型的流行和最新技术，以及可证明这种鲁棒性的方法。从强大的机器学习的这种统一中，我们识别并讨论了该地区未来研究的迫切方向。

时间序列数据在许多现实世界中（例如，移动健康）和深神经网络（DNNS）中产生，在解决它们方面已取得了巨大的成功。尽管他们成功了，但对他们对对抗性攻击的稳健性知之甚少。在本文中，我们提出了一个通过统计特征（TSA-STAT）}称为时间序列攻击的新型对抗框架}。为了解决时间序列域的独特挑战，TSA-STAT对时间序列数据的统计特征采取限制来构建对抗性示例。优化的多项式转换用于创建比基于加性扰动的攻击（就成功欺骗DNN而言）更有效的攻击。我们还提供有关构建对抗性示例的统计功能规范的认证界限。我们对各种现实世界基准数据集的实验表明，TSA-STAT在欺骗DNN的时间序列域和改善其稳健性方面的有效性。 TSA-STAT算法的源代码可在https://github.com/tahabelkhouja/time-series-series-attacks-via-statity-features上获得

Adversarial examples are perturbed inputs designed to fool machine learning models. Adversarial training injects such examples into training data to increase robustness. To scale this technique to large datasets, perturbations are crafted using fast single-step methods that maximize a linear approximation of the model's loss. We show that this form of adversarial training converges to a degenerate global minimum, wherein small curvature artifacts near the data points obfuscate a linear approximation of the loss. The model thus learns to generate weak perturbations, rather than defend against strong ones. As a result, we find that adversarial training remains vulnerable to black-box attacks, where we transfer perturbations computed on undefended models, as well as to a powerful novel single-step attack that escapes the non-smooth vicinity of the input data via a small random step. We further introduce Ensemble Adversarial Training, a technique that augments training data with perturbations transferred from other models. On ImageNet, Ensemble Adversarial Training yields models with stronger robustness to blackbox attacks. In particular, our most robust model won the first round of the NIPS 2017 competition on Defenses against Adversarial Attacks (Kurakin et al., 2017c). However, subsequent work found that more elaborate black-box attacks could significantly enhance transferability and reduce the accuracy of our models.

尽管在构建强大的神经网络方面具有明显的计算优势，但使用单步方法的对抗训练（AT）是不稳定的，因为它遭受了灾难性的过度拟合（CO）：网络在对抗性训练的第一阶段获得了非平凡的鲁棒性，但突然达到了一个阶段在几次迭代中，他们很快失去了所有鲁棒性。尽管有些作品成功地预防了CO，但导致这种显着失败模式的不同机制仍然很少理解。但是，在这项工作中，我们发现数据结构与AT动力学之间的相互作用在CO中起着基本作用。特别是，通过对自然图像的典型数据集进行主动干预，我们建立了一个因果关系。在方法上单步中的数据和CO的发作。这种新的观点提供了对导致CO的机制的重要见解，并为更好地理解强大模型构建的一般动态铺平了道路。可以在https://github.com/gortizji/co_features上找到复制本文实验的代码。

后门数据中毒攻击是一种对抗的攻击，其中攻击者将几个水印，误标记的训练示例注入训练集中。水印不会影响典型数据模型的测试时间性能;但是，该模型在水印示例中可靠地错误。为获得对后门数据中毒攻击的更好的基础认识，我们展示了一个正式的理论框架，其中一个人可以讨论对分类问题的回溯数据中毒攻击。然后我们使用它来分析这些攻击的重要统计和计算问题。在统计方面，我们识别一个参数，我们称之为记忆能力，捕捉到后门攻击的学习问题的内在脆弱性。这使我们能够争论几个自然学习问题的鲁棒性与后门攻击。我们的结果，攻击者涉及介绍后门攻击的明确建设，我们的鲁棒性结果表明，一些自然问题设置不能产生成功的后门攻击。从计算的角度来看，我们表明，在某些假设下，对抗训练可以检测训练集中的后门的存在。然后，我们表明，在类似的假设下，我们称之为呼叫滤波和鲁棒概括的两个密切相关的问题几乎等同。这意味着它既是渐近必要的，并且足以设计算法，可以识别训练集中的水印示例，以便获得既广泛概念的学习算法，以便在室外稳健。

This paper investigates recently proposed approaches for defending against adversarial examples and evaluating adversarial robustness. We motivate adversarial risk as an objective for achieving models robust to worst-case inputs. We then frame commonly used attacks and evaluation metrics as defining a tractable surrogate objective to the true adversarial risk. This suggests that models may optimize this surrogate rather than the true adversarial risk. We formalize this notion as obscurity to an adversary, and develop tools and heuristics for identifying obscured models and designing transparent models. We demonstrate that this is a significant problem in practice by repurposing gradient-free optimization techniques into adversarial attacks, which we use to decrease the accuracy of several recently proposed defenses to near zero. Our hope is that our formulations and results will help researchers to develop more powerful defenses.

Adaptive attacks have (rightfully) become the de facto standard for evaluating defenses to adversarial examples. We find, however, that typical adaptive evaluations are incomplete. We demonstrate that thirteen defenses recently published at ICLR, ICML and NeurIPS-and which illustrate a diverse set of defense strategies-can be circumvented despite attempting to perform evaluations using adaptive attacks. While prior evaluation papers focused mainly on the end result-showing that a defense was ineffective-this paper focuses on laying out the methodology and the approach necessary to perform an adaptive attack. Some of our attack strategies are generalizable, but no single strategy would have been sufficient for all defenses. This underlines our key message that adaptive attacks cannot be automated and always require careful and appropriate tuning to a given defense. We hope that these analyses will serve as guidance on how to properly perform adaptive attacks against defenses to adversarial examples, and thus will allow the community to make further progress in building more robust models.

Deep learning algorithms have been shown to perform extremely well on many classical machine learning problems. However, recent studies have shown that deep learning, like other machine learning techniques, is vulnerable to adversarial samples: inputs crafted to force a deep neural network (DNN) to provide adversary-selected outputs. Such attacks can seriously undermine the security of the system supported by the DNN, sometimes with devastating consequences. For example, autonomous vehicles can be crashed, illicit or illegal content can bypass content filters, or biometric authentication systems can be manipulated to allow improper access. In this work, we introduce a defensive mechanism called defensive distillation to reduce the effectiveness of adversarial samples on DNNs. We analytically investigate the generalizability and robustness properties granted by the use of defensive distillation when training DNNs. We also empirically study the effectiveness of our defense mechanisms on two DNNs placed in adversarial settings. The study shows that defensive distillation can reduce effectiveness of sample creation from 95% to less than 0.5% on a studied DNN. Such dramatic gains can be explained by the fact that distillation leads gradients used in adversarial sample creation to be reduced by a factor of 10 30 . We also find that distillation increases the average minimum number of features that need to be modified to create adversarial samples by about 800% on one of the DNNs we tested.

随机平滑是目前是最先进的方法，用于构建来自Neural Networks的可认真稳健的分类器，以防止$ \ ell_2 $ - vitersarial扰动。在范例下，分类器的稳健性与预测置信度对齐，即，对平滑分类器的较高的置信性意味着更好的鲁棒性。这使我们能够在校准平滑分类器的信仰方面重新思考准确性和鲁棒性之间的基本权衡。在本文中，我们提出了一种简单的训练方案，Coined Spiremix，通过自我混合来控制平滑分类器的鲁棒性：它沿着每个输入对逆势扰动方向进行样品的凸起组合。该提出的程序有效地识别过度自信，在平滑分类器的情况下，作为有限的稳健性的原因，并提供了一种直观的方法来自适应地在这些样本之间设置新的决策边界，以实现更好的鲁棒性。我们的实验结果表明，与现有的最先进的强大培训方法相比，该方法可以显着提高平滑分类器的认证$ \ ell_2 $ -toSpustness。

Neural networks provide state-of-the-art results for most machine learning tasks. Unfortunately, neural networks are vulnerable to adversarial examples: given an input x and any target classification t, it is possible to find a new input x that is similar to x but classified as t. This makes it difficult to apply neural networks in security-critical areas. Defensive distillation is a recently proposed approach that can take an arbitrary neural network, and increase its robustness, reducing the success rate of current attacks' ability to find adversarial examples from 95% to 0.5%.In this paper, we demonstrate that defensive distillation does not significantly increase the robustness of neural networks by introducing three new attack algorithms that are successful on both distilled and undistilled neural networks with 100% probability. Our attacks are tailored to three distance metrics used previously in the literature, and when compared to previous adversarial example generation algorithms, our attacks are often much more effective (and never worse). Furthermore, we propose using high-confidence adversarial examples in a simple transferability test we show can also be used to break defensive distillation. We hope our attacks will be used as a benchmark in future defense attempts to create neural networks that resist adversarial examples.

我们理论上和经验地证明，对抗性鲁棒性可以显着受益于半体验学习。从理论上讲，我们重新审视了Schmidt等人的简单高斯模型。这显示了标准和稳健分类之间的示例复杂性差距。我们证明了未标记的数据桥接这种差距：简单的半体验学习程序（自我训练）使用相同数量的达到高标准精度所需的标签实现高的强大精度。经验上，我们增强了CiFar-10，使用50万微小的图像，使用了8000万微小的图像，并使用强大的自我训练来优于最先进的鲁棒精度（i）$ \ ell_ infty $鲁棒性通过对抗培训和（ii）认证$ \ ell_2 $和$ \ ell_ \ infty $鲁棒性通过随机平滑的几个强大的攻击。在SVHN上，添加DataSet自己的额外训练集，删除的标签提供了4到10个点的增益，在使用额外标签的1点之内。

We identify a trade-off between robustness and accuracy that serves as a guiding principle in the design of defenses against adversarial examples. Although this problem has been widely studied empirically, much remains unknown concerning the theory underlying this trade-off. In this work, we decompose the prediction error for adversarial examples (robust error) as the sum of the natural (classification) error and boundary error, and provide a differentiable upper bound using the theory of classification-calibrated loss, which is shown to be the tightest possible upper bound uniform over all probability distributions and measurable predictors. Inspired by our theoretical analysis, we also design a new defense method, TRADES, to trade adversarial robustness off against accuracy. Our proposed algorithm performs well experimentally in real-world datasets. The methodology is the foundation of our entry to the NeurIPS 2018 Adversarial Vision Challenge in which we won the 1st place out of ~2,000 submissions, surpassing the runner-up approach by 11.41% in terms of mean 2 perturbation distance.

尽管学习已成为现代信息处理的核心组成部分，但现在有足够的证据表明它可以导致偏见，不安全和有偏见的系统。因此，对学习要求施加要求至关重要，尤其是在达到社会，工业和医疗领域的关键应用程序时。但是，大多数现代统计问题的非跨性别性只有通过限制引入而加剧。尽管通常可以使用经验风险最小化来学习良好的无约束解决方案，即使获得满足统计约束的模型也可能具有挑战性。更重要的是，一个好。在本文中，我们通过在经验双重领域中学习来克服这个问题，在经验的双重领域中，统计学上的统计学习问题变得不受限制和确定性。我们通过界定经验二元性差距来分析这种方法的概括特性 - 即，我们的近似，可拖动解决方案与原始（非凸）统计问题的解决方案之间的差异 - 并提供实用的约束学习算法。这些结果建立了与经典学习理论的约束，从而可以明确地在学习中使用约束。我们说明了这种理论和算法受到速率受限的学习应用，这是在公平和对抗性鲁棒性中产生的。

成功的深度学习模型往往涉及培训具有比训练样本数量更多的参数的神经网络架构。近年来已经广泛研究了这种超分子化的模型，并且通过双下降现象和通过优化景观的结构特性，从统计的角度和计算视角都建立了过分统计化的优点。尽管在过上分层的制度中深入学习架构的显着成功，但也众所周知，这些模型对其投入中的小对抗扰动感到高度脆弱。即使在普遍培训的情况下，它们在扰动输入（鲁棒泛化）上的性能也会比良性输入（标准概括）的最佳可达到的性能更糟糕。因此，必须了解如何从根本上影响稳健性的情况下如何影响鲁棒性。在本文中，我们将通过专注于随机特征回归模型（具有随机第一层权重的两层神经网络）来提供超分度化对鲁棒性的作用的精确表征。我们考虑一个制度，其中样本量，输入维度和参数的数量彼此成比例地生长，并且当模型发生前列地训练时，可以为鲁棒泛化误差导出渐近精确的公式。我们的发达理论揭示了过分统计化对鲁棒性的非竞争效果，表明对于普遍训练的随机特征模型，高度公正化可能会损害鲁棒泛化。