We show that there may exist an inherent tension between the goal of adversarial robustness and that of standard generalization. Specifically, training robust models may not only be more resource-consuming, but also lead to a reduction of standard accuracy. We demonstrate that this trade-off between the standard accuracy of a model and its robustness to adversarial perturbations provably exists in a fairly simple and natural setting. These findings also corroborate a similar phenomenon observed empirically in more complex settings. Further, we argue that this phenomenon is a consequence of robust classifiers learning fundamentally different feature representations than standard classifiers. These differences, in particular, seem to result in unexpected benefits: the representations learned by robust models tend to align better with salient data characteristics and human perception.

Adversarial examples have attracted significant attention in machine learning, but the reasons for their existence and pervasiveness remain unclear. We demonstrate that adversarial examples can be directly attributed to the presence of non-robust features: features (derived from patterns in the data distribution) that are highly predictive, yet brittle and (thus) incomprehensible to humans. After capturing these features within a theoretical framework, we establish their widespread existence in standard datasets. Finally, we present a simple setting where we can rigorously tie the phenomena we observe in practice to a misalignment between the (human-specified) notion of robustness and the inherent geometry of the data.

Machine learning models are often susceptible to adversarial perturbations of their inputs. Even small perturbations can cause state-of-the-art classifiers with high "standard" accuracy to produce an incorrect prediction with high confidence. To better understand this phenomenon, we study adversarially robust learning from the viewpoint of generalization. We show that already in a simple natural data model, the sample complexity of robust learning can be significantly larger than that of "standard" learning. This gap is information theoretic and holds irrespective of the training algorithm or the model family. We complement our theoretical results with experiments on popular image classification datasets and show that a similar gap exists here as well. We postulate that the difficulty of training robust classifiers stems, at least partially, from this inherently larger sample complexity.

删除攻击旨在通过略微扰动正确标记的训练示例的特征来大幅恶化学习模型的测试准确性。通过将这种恶意攻击正式地找到特定$ \ infty $ -wassersein球中的最坏情况培训数据，我们表明最小化扰动数据的对抗性风险相当于优化原始数据上的自然风险的上限。这意味着对抗性培训可以作为防止妄想攻击的原则防御。因此，通过普遍训练可以很大程度地回收测试精度。为了进一步了解国防的内部机制，我们披露了对抗性培训可以通过防止学习者过于依赖于自然环境中的非鲁棒特征来抵制妄想扰动。最后，我们将我们的理论调查结果与一系列关于流行的基准数据集进行了补充，这表明防御能够承受六种不同的实际攻击。在面对令人难以闻名的对手时，理论和经验结果投票给逆势训练。

Recent work has demonstrated that deep neural networks are vulnerable to adversarial examples-inputs that are almost indistinguishable from natural data and yet classified incorrectly by the network. In fact, some of the latest findings suggest that the existence of adversarial attacks may be an inherent weakness of deep learning models. To address this problem, we study the adversarial robustness of neural networks through the lens of robust optimization. This approach provides us with a broad and unifying view on much of the prior work on this topic. Its principled nature also enables us to identify methods for both training and attacking neural networks that are reliable and, in a certain sense, universal. In particular, they specify a concrete security guarantee that would protect against any adversary. These methods let us train networks with significantly improved resistance to a wide range of adversarial attacks. They also suggest the notion of security against a first-order adversary as a natural and broad security guarantee. We believe that robustness against such well-defined classes of adversaries is an important stepping stone towards fully resistant deep learning models. 1

随机平滑是目前是最先进的方法，用于构建来自Neural Networks的可认真稳健的分类器，以防止$ \ ell_2 $ - vitersarial扰动。在范例下，分类器的稳健性与预测置信度对齐，即，对平滑分类器的较高的置信性意味着更好的鲁棒性。这使我们能够在校准平滑分类器的信仰方面重新思考准确性和鲁棒性之间的基本权衡。在本文中，我们提出了一种简单的训练方案，Coined Spiremix，通过自我混合来控制平滑分类器的鲁棒性：它沿着每个输入对逆势扰动方向进行样品的凸起组合。该提出的程序有效地识别过度自信，在平滑分类器的情况下，作为有限的稳健性的原因，并提供了一种直观的方法来自适应地在这些样本之间设置新的决策边界，以实现更好的鲁棒性。我们的实验结果表明，与现有的最先进的强大培训方法相比，该方法可以显着提高平滑分类器的认证$ \ ell_2 $ -toSpustness。

The study of adversarial robustness has so far largely focused on perturbations bound in p -norms. However, state-of-the-art models turn out to be also vulnerable to other, more natural classes of perturbations such as translations and rotations. In this work, we thoroughly investigate the vulnerability of neural network-based classifiers to rotations and translations. While data augmentation offers relatively small robustness, we use ideas from robust optimization and test-time input aggregation to significantly improve robustness. Finally we find that, in contrast to the p -norm case, first-order methods cannot reliably find worst-case perturbations. This highlights spatial robustness as a fundamentally different setting requiring additional study. 1

We identify a trade-off between robustness and accuracy that serves as a guiding principle in the design of defenses against adversarial examples. Although this problem has been widely studied empirically, much remains unknown concerning the theory underlying this trade-off. In this work, we decompose the prediction error for adversarial examples (robust error) as the sum of the natural (classification) error and boundary error, and provide a differentiable upper bound using the theory of classification-calibrated loss, which is shown to be the tightest possible upper bound uniform over all probability distributions and measurable predictors. Inspired by our theoretical analysis, we also design a new defense method, TRADES, to trade adversarial robustness off against accuracy. Our proposed algorithm performs well experimentally in real-world datasets. The methodology is the foundation of our entry to the NeurIPS 2018 Adversarial Vision Challenge in which we won the 1st place out of ~2,000 submissions, surpassing the runner-up approach by 11.41% in terms of mean 2 perturbation distance.

我们识别普遍对抗扰动（UAP）的性质，将它们与标准的对抗性扰动区分开来。具体而言，我们表明，由投影梯度下降产生的靶向UAPS表现出两种人对齐的特性：语义局部性和空间不变性，标准的靶向对抗扰动缺乏。我们还证明，除标准对抗扰动之外，UAPS含有明显较低的泛化信号 - 即，UAPS在比标准的对抗的扰动的较小程度上利用非鲁棒特征。

几个数据增强方法部署了未标记的分配（UID）数据，以弥合神经网络的培训和推理之间的差距。然而，这些方法在UID数据的可用性方面具有明确的限制和伪标签上的算法的依赖性。在此，我们提出了一种数据增强方法，通过使用缺乏上述问题的分发（OOD）数据来改善对抗和标准学习的泛化。我们展示了如何在理论上使用每个学习场景中的数据来改进泛化，并通过Cifar-10，CiFar-100和ImageNet的子集进行化学理论分析。结果表明，即使在似乎与人类角度几乎没有相关的图像数据中也是不希望的特征。我们还通过与其他数据增强方法进行比较，介绍了所提出的方法的优点，这些方法可以在没有UID数据的情况下使用。此外，我们证明该方法可以进一步改善现有的最先进的对抗培训。

当前，随机平滑被认为是获得确切可靠分类器的最新方法。尽管其表现出色，但该方法仍与各种严重问题有关，例如``认证准确性瀑布''，认证与准确性权衡甚至公平性问题。已经提出了依赖输入的平滑方法，目的是克服这些缺陷。但是，我们证明了这些方法缺乏正式的保证，因此所产生的证书是没有道理的。我们表明，一般而言，输入依赖性平滑度遭受了维数的诅咒，迫使方差函数具有低半弹性。另一方面，我们提供了一个理论和实用的框架，即使在严格的限制下，即使在有维度的诅咒的情况下，即使在存在维度的诅咒的情况下，也可以使用依赖输入的平滑。我们提供平滑方差功能的一种混凝土设计，并在CIFAR10和MNIST上进行测试。我们的设计减轻了经典平滑的一些问题，并正式下划线，但仍需要进一步改进设计。

State-of-the-art results on image recognition tasks are achieved using over-parameterized learning algorithms that (nearly) perfectly fit the training set and are known to fit well even random labels. This tendency to memorize the labels of the training data is not explained by existing theoretical analyses. Memorization of the training data also presents significant privacy risks when the training data contains sensitive personal information and thus it is important to understand whether such memorization is necessary for accurate learning.We provide the first conceptual explanation and a theoretical model for this phenomenon. Specifically, we demonstrate that for natural data distributions memorization of labels is necessary for achieving closeto-optimal generalization error. Crucially, even labels of outliers and noisy labels need to be memorized. The model is motivated and supported by the results of several recent empirical works. In our model, data is sampled from a mixture of subpopulations and our results show that memorization is necessary whenever the distribution of subpopulation frequencies is long-tailed. Image and text data is known to be long-tailed and therefore our results establish a formal link between these empirical phenomena. Our results allow to quantify the cost of limiting memorization in learning and explain the disparate effects that privacy and model compression have on different subgroups.

对抗性可转移性是一种有趣的性质 - 针对一个模型制作的对抗性扰动也是对另一个模型有效的，而这些模型来自不同的模型家庭或培训过程。为了更好地保护ML系统免受对抗性攻击，提出了几个问题：对抗性转移性的充分条件是什么，以及如何绑定它？有没有办法降低对抗的转移性，以改善合奏ML模型的鲁棒性？为了回答这些问题，在这项工作中，我们首先在理论上分析和概述了模型之间的对抗性可转移的充分条件;然后提出一种实用的算法，以减少集合内基础模型之间的可转换，以提高其鲁棒性。我们的理论分析表明，只有促进基础模型梯度之间的正交性不足以确保低可转移性;与此同时，模型平滑度是控制可转移性的重要因素。我们还在某些条件下提供了对抗性可转移性的下界和上限。灵感来自我们的理论分析，我们提出了一种有效的可转让性，减少了平滑（TRS）集合培训策略，以通过实施基础模型之间的梯度正交性和模型平滑度来培训具有低可转换性的强大集成。我们对TRS进行了广泛的实验，并与6个最先进的集合基线进行比较，防止不同数据集的8个白箱攻击，表明所提出的TRS显着优于所有基线。

在过去的十年中，基于深度学习的网络在包括图像分类在内的许多任务中取得了前所未有的成功。尽管取得了非凡的成就，但最近的研究表明，这种网络很容易被小小的恶意扰动（也称为对抗性例子）所愚弄。这种安全弱点导致广泛的研究旨在获得强大的模型。除了此类模型的明显鲁棒性优势之外，还观察到，它们相对于人类感知的梯度。几项作品已将感知一致的梯度（PAG）确定为强大训练的副产品，但没有人认为它是独立现象，也没有研究其自身的含义。在这项工作中，我们专注于这种特征，并测试感知一致性梯度是否暗示着稳健性。为此，我们开发了一个新颖的目标，可以直接在训练分类器中促进PAG，并检查具有此类梯度的模型是否对对抗性攻击更强大。关于CIFAR-10和STL的广泛实验验证了此类模型可以提高稳健性能，从而揭示了PAG和稳健性之间令人惊讶的双向连接。

Adversarial examples are perturbed inputs designed to fool machine learning models. Adversarial training injects such examples into training data to increase robustness. To scale this technique to large datasets, perturbations are crafted using fast single-step methods that maximize a linear approximation of the model's loss. We show that this form of adversarial training converges to a degenerate global minimum, wherein small curvature artifacts near the data points obfuscate a linear approximation of the loss. The model thus learns to generate weak perturbations, rather than defend against strong ones. As a result, we find that adversarial training remains vulnerable to black-box attacks, where we transfer perturbations computed on undefended models, as well as to a powerful novel single-step attack that escapes the non-smooth vicinity of the input data via a small random step. We further introduce Ensemble Adversarial Training, a technique that augments training data with perturbations transferred from other models. On ImageNet, Ensemble Adversarial Training yields models with stronger robustness to blackbox attacks. In particular, our most robust model won the first round of the NIPS 2017 competition on Defenses against Adversarial Attacks (Kurakin et al., 2017c). However, subsequent work found that more elaborate black-box attacks could significantly enhance transferability and reduce the accuracy of our models.

由明确的反对派制作的对抗例子在机器学习中引起了重要的关注。然而，潜在虚假朋友带来的安全风险基本上被忽视了。在本文中，我们揭示了虚伪的例子的威胁 - 最初被错误分类但是虚假朋友扰乱的投入，以强迫正确的预测。虽然这种扰动的例子似乎是无害的，但我们首次指出，它们可能是恶意地用来隐瞒评估期间不合格（即，不如所需）模型的错误。一旦部署者信任虚伪的性能并在真实应用程序中应用“良好的”模型，即使在良性环境中也可能发生意外的失败。更严重的是，这种安全风险似乎是普遍存在的：我们发现许多类型的不合标准模型易受多个数据集的虚伪示例。此外，我们提供了第一次尝试，以称为虚伪风险的公制表征威胁，并试图通过一些对策来规避它。结果表明对策的有效性，即使在自适应稳健的培训之后，风险仍然是不可忽视的。

尽管使用对抗性训练捍卫深度学习模型免受对抗性扰动的经验成功，但到目前为止，仍然不清楚对抗性扰动的存在背后的原则是什么，而对抗性培训对神经网络进行了什么来消除它们。在本文中，我们提出了一个称为特征纯化的原则，在其中，我们表明存在对抗性示例的原因之一是在神经网络的训练过程中，在隐藏的重量中积累了某些小型密集混合物；更重要的是，对抗训练的目标之一是去除此类混合物以净化隐藏的重量。我们介绍了CIFAR-10数据集上的两个实验，以说明这一原理，并且一个理论上的结果证明，对于某些自然分类任务，使用随机初始初始化的梯度下降训练具有RELU激活的两层神经网络确实满足了这一原理。从技术上讲，我们给出了我们最大程度的了解，第一个结果证明，以下两个可以同时保持使用RELU激活的神经网络。 （1）对原始数据的训练确实对某些半径的小对抗扰动确实不舒适。 （2）即使使用经验性扰动算法（例如FGM），实际上也可以证明对对抗相同半径的任何扰动也可以证明具有强大的良好性。最后，我们还证明了复杂性的下限，表明该网络的低复杂性模型，例如线性分类器，低度多项式或什至是神经切线核，无论使用哪种算法，都无法防御相同半径的扰动训练他们。

对抗性的鲁棒性已经成为深度学习的核心目标，无论是在理论和实践中。然而，成功的方法来改善对抗的鲁棒性（如逆势训练）在不受干扰的数据上大大伤害了泛化性能。这可能会对对抗性鲁棒性如何影响现实世界系统的影响（即，如果它可以提高未受干扰的数据的准确性），许多人可能选择放弃鲁棒性）。我们提出内插对抗培训，该培训最近雇用了在对抗培训框架内基于插值的基于插值的培训方法。在CiFar -10上，对抗性训练增加了标准测试错误（当没有对手时）从4.43％到12.32％，而我们的内插对抗培训我们保留了对抗性的鲁棒性，同时实现了仅6.45％的标准测试误差。通过我们的技术，强大模型标准误差的相对增加从178.1％降至仅为45.5％。此外，我们提供内插对抗性培训的数学分析，以确认其效率，并在鲁棒性和泛化方面展示其优势。

现有针对对抗性示例（例如对抗训练）的防御能力通常假设对手将符合特定或已知的威胁模型，例如固定预算内的$ \ ell_p $扰动。在本文中，我们关注的是在训练过程中辩方假设的威胁模型中存在不匹配的情况，以及在测试时对手的实际功能。我们问一个问题：学习者是否会针对特定的“源”威胁模型进行训练，我们什么时候可以期望鲁棒性在测试时间期间概括为更强大的未知“目标”威胁模型？我们的主要贡献是通过不可预见的对手正式定义学习和概括的问题，这有助于我们从常规的对手的传统角度来理解对抗风险的增加。应用我们的框架，我们得出了将源和目标威胁模型之间的概括差距与特征提取器变化相关联的概括，该限制衡量了在给定威胁模型中提取的特征之间的预期最大差异。基于我们的概括结合，我们提出了具有变化正则化（AT-VR）的对抗训练，该训练在训练过程中降低了特征提取器在源威胁模型中的变化。我们从经验上证明，与标准的对抗训练相比，AT-VR可以改善测试时间内的概括，从而无法预见。此外，我们将变异正则化与感知对抗训练相结合[Laidlaw等。 2021]以实现不可预见的攻击的最新鲁棒性。我们的代码可在https://github.com/inspire-group/variation-regularization上公开获取。

尽管在构建强大的神经网络方面具有明显的计算优势，但使用单步方法的对抗训练（AT）是不稳定的，因为它遭受了灾难性的过度拟合（CO）：网络在对抗性训练的第一阶段获得了非平凡的鲁棒性，但突然达到了一个阶段在几次迭代中，他们很快失去了所有鲁棒性。尽管有些作品成功地预防了CO，但导致这种显着失败模式的不同机制仍然很少理解。但是，在这项工作中，我们发现数据结构与AT动力学之间的相互作用在CO中起着基本作用。特别是，通过对自然图像的典型数据集进行主动干预，我们建立了一个因果关系。在方法上单步中的数据和CO的发作。这种新的观点提供了对导致CO的机制的重要见解，并为更好地理解强大模型构建的一般动态铺平了道路。可以在https://github.com/gortizji/co_features上找到复制本文实验的代码。