作为研究界，我们仍然缺乏对对抗性稳健性的进展的系统理解，这通常使得难以识别训练强大模型中最有前途的想法。基准稳健性的关键挑战是，其评估往往是出错的导致鲁棒性高估。我们的目标是建立对抗性稳健性的标准化基准，尽可能准确地反映出考虑在合理的计算预算范围内所考虑的模型的稳健性。为此，我们首先考虑图像分类任务并在允许的型号上引入限制（可能在将来宽松）。我们评估了与AutoAtrack的对抗鲁棒性，白和黑箱攻击的集合，最近在大规模研究中显示，与原始出版物相比，改善了几乎所有稳健性评估。为防止对自动攻击进行新防御的过度适应，我们欢迎基于自适应攻击的外部评估，特别是在自动攻击稳健性潜在高估的地方。我们的排行榜，托管在https://robustbench.github.io/，包含120多个模型的评估，并旨在反映在$ \ ell_ \ infty $的一套明确的任务上的图像分类中的当前状态 - 和$ \ ell_2 $ -Threat模型和共同腐败，未来可能的扩展。此外，我们开源源是图书馆https://github.com/robustbench/robustbench，可以提供对80多个强大模型的统一访问，以方便他们的下游应用程序。最后，根据收集的模型，我们分析了稳健性对分布换档，校准，分配检测，公平性，隐私泄漏，平滑度和可转移性的影响。

在测试时间进行优化的自适应防御能力有望改善对抗性鲁棒性。我们对这种自适应测试时间防御措施进行分类，解释其潜在的好处和缺点，并评估图像分类的最新自适应防御能力的代表性。不幸的是，经过我们仔细的案例研究评估时，没有任何显着改善静态防御。有些甚至削弱了基本静态模型，同时增加了推理计算。尽管这些结果令人失望，但我们仍然认为自适应测试时间防御措施是一项有希望的研究途径，因此，我们为他们的彻底评估提供了建议。我们扩展了Carlini等人的清单。（2019年）通过提供针对自适应防御的具体步骤。

The field of defense strategies against adversarial attacks has significantly grown over the last years, but progress is hampered as the evaluation of adversarial defenses is often insufficient and thus gives a wrong impression of robustness. Many promising defenses could be broken later on, making it difficult to identify the state-of-the-art. Frequent pitfalls in the evaluation are improper tuning of hyperparameters of the attacks, gradient obfuscation or masking. In this paper we first propose two extensions of the PGD-attack overcoming failures due to suboptimal step size and problems of the objective function. We then combine our novel attacks with two complementary existing ones to form a parameter-free, computationally affordable and user-independent ensemble of attacks to test adversarial robustness. We apply our ensemble to over 50 models from papers published at recent top machine learning and computer vision venues. In all except one of the cases we achieve lower robust test accuracy than reported in these papers, often by more than 10%, identifying several broken defenses.

对共同腐败的稳健性的文献表明对逆势培训是否可以提高这种环境的性能，没有达成共识。 First, we show that, when used with an appropriately selected perturbation radius, $\ell_p$ adversarial training can serve as a strong baseline against common corruptions improving both accuracy and calibration.然后，我们解释了为什么对抗性训练比具有简单高斯噪声的数据增强更好地表现，这被观察到是对共同腐败的有意义的基线。与此相关，我们确定了高斯增强过度适用于用于培训的特定标准偏差的$ \ sigma $ -oviting现象，这对培训具有显着不利影响的普通腐败精度。我们讨论如何缓解这一问题，然后如何通过学习的感知图像贴片相似度引入对抗性训练的有效放松来进一步增强$ \ ell_p $普发的培训。通过对CiFar-10和Imagenet-100的实验，我们表明我们的方法不仅改善了$ \ ell_p $普发的培训基线，而且还有累积的收益与Augmix，Deepaulment，Ant和Sin等数据增强方法，导致普通腐败的最先进的表现。我们的实验代码在HTTPS://github.com/tml-epfl/adv-training - 窗子上公开使用。

Adaptive attacks have (rightfully) become the de facto standard for evaluating defenses to adversarial examples. We find, however, that typical adaptive evaluations are incomplete. We demonstrate that thirteen defenses recently published at ICLR, ICML and NeurIPS-and which illustrate a diverse set of defense strategies-can be circumvented despite attempting to perform evaluations using adaptive attacks. While prior evaluation papers focused mainly on the end result-showing that a defense was ineffective-this paper focuses on laying out the methodology and the approach necessary to perform an adaptive attack. Some of our attack strategies are generalizable, but no single strategy would have been sufficient for all defenses. This underlines our key message that adaptive attacks cannot be automated and always require careful and appropriate tuning to a given defense. We hope that these analyses will serve as guidance on how to properly perform adaptive attacks against defenses to adversarial examples, and thus will allow the community to make further progress in building more robust models.

This paper investigates recently proposed approaches for defending against adversarial examples and evaluating adversarial robustness. We motivate adversarial risk as an objective for achieving models robust to worst-case inputs. We then frame commonly used attacks and evaluation metrics as defining a tractable surrogate objective to the true adversarial risk. This suggests that models may optimize this surrogate rather than the true adversarial risk. We formalize this notion as obscurity to an adversary, and develop tools and heuristics for identifying obscured models and designing transparent models. We demonstrate that this is a significant problem in practice by repurposing gradient-free optimization techniques into adversarial attacks, which we use to decrease the accuracy of several recently proposed defenses to near zero. Our hope is that our formulations and results will help researchers to develop more powerful defenses.

我们表明，当考虑到图像域$ [0,1] ^ D $时，已建立$ L_1 $ -Projected梯度下降（PGD）攻击是次优，因为它们不认为有效的威胁模型是交叉点$ l_1 $ -ball和$ [0,1] ^ d $。我们研究了这种有效威胁模型的最陡渐进步骤的预期稀疏性，并表明该组上的确切投影是计算可行的，并且产生更好的性能。此外，我们提出了一种自适应形式的PGD，即使具有小的迭代预算，这也是非常有效的。我们的结果$ l_1 $ -apgd是一个强大的白盒攻击，表明先前的作品高估了他们的$ l_1 $ -trobustness。使用$ l_1 $ -apgd for vercersarial培训，我们获得一个强大的分类器，具有sota $ l_1 $ -trobustness。最后，我们将$ l_1 $ -apgd和平方攻击的适应组合到$ l_1 $ to $ l_1 $ -autoattack，这是一个攻击的集合，可靠地评估$ l_1 $ -ball与$的威胁模型的对抗鲁棒性进行对抗[ 0,1] ^ d $。

We propose the Square Attack, a score-based black-box l2and l∞-adversarial attack that does not rely on local gradient information and thus is not affected by gradient masking. Square Attack is based on a randomized search scheme which selects localized squareshaped updates at random positions so that at each iteration the perturbation is situated approximately at the boundary of the feasible set. Our method is significantly more query efficient and achieves a higher success rate compared to the state-of-the-art methods, especially in the untargeted setting. In particular, on ImageNet we improve the average query efficiency in the untargeted setting for various deep networks by a factor of at least 1.8 and up to 3 compared to the recent state-ofthe-art l∞-attack of Al-Dujaili & OReilly (2020). Moreover, although our attack is black-box, it can also outperform gradient-based white-box attacks on the standard benchmarks achieving a new state-of-the-art in terms of the success rate. The code of our attack is available at https://github.com/max-andr/square-attack.

Despite the success of convolutional neural networks (CNNs) in many academic benchmarks for computer vision tasks, their application in the real-world is still facing fundamental challenges. One of these open problems is the inherent lack of robustness, unveiled by the striking effectiveness of adversarial attacks. Current attack methods are able to manipulate the network's prediction by adding specific but small amounts of noise to the input. In turn, adversarial training (AT) aims to achieve robustness against such attacks and ideally a better model generalization ability by including adversarial samples in the trainingset. However, an in-depth analysis of the resulting robust models beyond adversarial robustness is still pending. In this paper, we empirically analyze a variety of adversarially trained models that achieve high robust accuracies when facing state-of-the-art attacks and we show that AT has an interesting side-effect: it leads to models that are significantly less overconfident with their decisions, even on clean data than non-robust models. Further, our analysis of robust models shows that not only AT but also the model's building blocks (like activation functions and pooling) have a strong influence on the models' prediction confidences. Data & Project website: https://github.com/GeJulia/robustness_confidences_evaluation

Recent work has demonstrated that deep neural networks are vulnerable to adversarial examples-inputs that are almost indistinguishable from natural data and yet classified incorrectly by the network. In fact, some of the latest findings suggest that the existence of adversarial attacks may be an inherent weakness of deep learning models. To address this problem, we study the adversarial robustness of neural networks through the lens of robust optimization. This approach provides us with a broad and unifying view on much of the prior work on this topic. Its principled nature also enables us to identify methods for both training and attacking neural networks that are reliable and, in a certain sense, universal. In particular, they specify a concrete security guarantee that would protect against any adversary. These methods let us train networks with significantly improved resistance to a wide range of adversarial attacks. They also suggest the notion of security against a first-order adversary as a natural and broad security guarantee. We believe that robustness against such well-defined classes of adversaries is an important stepping stone towards fully resistant deep learning models. 1

与标准的训练时间相比，训练时间非常长，尤其是针对稳健模型的主要缺点，尤其是对于大型数据集而言。此外，模型不仅应适用于一个$ l_p $ - 威胁模型，而且对所有模型来说都是理想的。在本文中，我们提出了基于$ l_p $ -Balls的几何特性的多元素鲁棒性的极端规范对抗训练（E-AT）。 E-AT的成本比其他对抗性训练方法低三倍，以进行多种锻炼。使用e-at，我们证明，对于ImageNet，单个时期和CIFAR-10，三个时期足以将任何$ L_P $ - 抛光模型变成一个多符号鲁棒模型。通过这种方式，我们获得了ImageNet的第一个多元素鲁棒模型，并在CIFAR-10上提高了多个Norm鲁棒性的最新型号，以超过$ 51 \％$。最后，我们通过对不同单独的$ l_p $ threat模型之间的对抗鲁棒性进行微调研究一般的转移，并改善了Cifar-10和Imagenet上的先前的SOTA $ L_1 $ - 固定。广泛的实验表明，我们的计划在包括视觉变压器在内的数据集和架构上起作用。

Designing powerful adversarial attacks is of paramount importance for the evaluation of $\ell_p$-bounded adversarial defenses. Projected Gradient Descent (PGD) is one of the most effective and conceptually simple algorithms to generate such adversaries. The search space of PGD is dictated by the steepest ascent directions of an objective. Despite the plethora of objective function choices, there is no universally superior option and robustness overestimation may arise from ill-suited objective selection. Driven by this observation, we postulate that the combination of different objectives through a simple loss alternating scheme renders PGD more robust towards design choices. We experimentally verify this assertion on a synthetic-data example and by evaluating our proposed method across 25 different $\ell_{\infty}$-robust models and 3 datasets. The performance improvement is consistent, when compared to the single loss counterparts. In the CIFAR-10 dataset, our strongest adversarial attack outperforms all of the white-box components of AutoAttack (AA) ensemble, as well as the most powerful attacks existing on the literature, achieving state-of-the-art results in the computational budget of our study ($T=100$, no restarts).

The authors thank Nicholas Carlini (UC Berkeley) and Dimitris Tsipras (MIT) for feedback to improve the survey quality. We also acknowledge X. Huang (Uni. Liverpool), K. R. Reddy (IISC), E. Valle (UNICAMP), Y. Yoo (CLAIR) and others for providing pointers to make the survey more comprehensive.

现有针对对抗性示例（例如对抗训练）的防御能力通常假设对手将符合特定或已知的威胁模型，例如固定预算内的$ \ ell_p $扰动。在本文中，我们关注的是在训练过程中辩方假设的威胁模型中存在不匹配的情况，以及在测试时对手的实际功能。我们问一个问题：学习者是否会针对特定的“源”威胁模型进行训练，我们什么时候可以期望鲁棒性在测试时间期间概括为更强大的未知“目标”威胁模型？我们的主要贡献是通过不可预见的对手正式定义学习和概括的问题，这有助于我们从常规的对手的传统角度来理解对抗风险的增加。应用我们的框架，我们得出了将源和目标威胁模型之间的概括差距与特征提取器变化相关联的概括，该限制衡量了在给定威胁模型中提取的特征之间的预期最大差异。基于我们的概括结合，我们提出了具有变化正则化（AT-VR）的对抗训练，该训练在训练过程中降低了特征提取器在源威胁模型中的变化。我们从经验上证明，与标准的对抗训练相比，AT-VR可以改善测试时间内的概括，从而无法预见。此外，我们将变异正则化与感知对抗训练相结合[Laidlaw等。 2021]以实现不可预见的攻击的最新鲁棒性。我们的代码可在https://github.com/inspire-group/variation-regularization上公开获取。

现代神经网络Excel在图像分类中，但它们仍然容易受到常见图像损坏，如模糊，斑点噪音或雾。最近的方法关注这个问题，例如Augmix和Deepaulment，引入了在预期运行的防御，以期望图像损坏分布。相比之下，$ \ ell_p $ -norm界限扰动的文献侧重于针对最坏情况损坏的防御。在这项工作中，我们通过提出防范内人来调和两种方法，这是一种优化图像到图像模型的参数来产生对外损坏的增强图像的技术。我们理论上激发了我们的方法，并为其理想化版本的一致性以及大纲领提供了足够的条件。我们的分类机器在预期对CiFar-10-C进行的常见图像腐败基准上提高了最先进的，并改善了CIFAR-10和ImageNet上的$ \ ell_p $ -norm有界扰动的最坏情况性能。

深度卷积神经网络（CNN）很容易被输入图像的细微，不可察觉的变化所欺骗。为了解决此漏洞，对抗训练会创建扰动模式，并将其包括在培训设置中以鲁棒性化模型。与仅使用阶级有限信息的现有对抗训练方法（例如，使用交叉渗透损失）相反，我们建议利用功能空间中的其他信息来促进更强的对手，这些信息又用于学习强大的模型。具体来说，我们将使用另一类的目标样本的样式和内容信息以及其班级边界信息来创建对抗性扰动。我们以深入监督的方式应用了我们提出的多任务目标，从而提取了多尺度特征知识，以创建最大程度地分开对手。随后，我们提出了一种最大边缘对抗训练方法，该方法可最大程度地减少源图像与其对手之间的距离，并最大程度地提高对手和目标图像之间的距离。与最先进的防御能力相比，我们的对抗训练方法表明了强大的鲁棒性，可以很好地推广到自然发生的损坏和数据分配变化，并保留了清洁示例的模型准确性。

对机器学习模型的逃避攻击通常通过迭代探测固定目标模型成功，从而曾经成功的攻击将反复成功。应对这种威胁的一种有希望的方法是使模型成为对抗输入的行动目标。为此，我们介绍了Morphence-2.0，这是一个由分布外（OOD）检测提供动力的可扩展移动目标防御（MTD），以防止对抗性例子。通过定期移动模型的决策功能，Morphence-2.0使重复或相关攻击成功的挑战变得极大。 Morphence-2.0以基本模型生成的模型池以引入足够随机性的方式对预测查询进行响应。通过OOD检测，Morphence-2.0配备了调度方法，该方法将对抗性示例分配给了强大的决策功能，并将良性样本分配给了未防御的准确模型。为了确保重复或相关的攻击失败，已部署的模型池在达到查询预算后​​自动到期，并且模型池被提前生成的新模型池无缝替换。我们在两个基准图像分类数据集（MNIST和CIFAR10）上评估Morphence-2.0，以4个参考攻击（3个白框和1个黑色框）。 Morphence-2.0始终优于先前的防御能力，同时保留清洁数据的准确性和降低攻击转移性。我们还表明，当由OOD检测提供动力时，Morphence-2.0能够精确地对模型的决策功能进行基于输入的运动，从而导致对对抗和良性查询的预测准确性更高。

We identify obfuscated gradients, a kind of gradient masking, as a phenomenon that leads to a false sense of security in defenses against adversarial examples. While defenses that cause obfuscated gradients appear to defeat iterative optimizationbased attacks, we find defenses relying on this effect can be circumvented. We describe characteristic behaviors of defenses exhibiting the effect, and for each of the three types of obfuscated gradients we discover, we develop attack techniques to overcome it. In a case study, examining noncertified white-box-secure defenses at ICLR 2018, we find obfuscated gradients are a common occurrence, with 7 of 9 defenses relying on obfuscated gradients. Our new attacks successfully circumvent 6 completely, and 1 partially, in the original threat model each paper considers.

Although deep neural networks (DNNs) have achieved great success in many tasks, they can often be fooled by adversarial examples that are generated by adding small but purposeful distortions to natural examples. Previous studies to defend against adversarial examples mostly focused on refining the DNN models, but have either shown limited success or required expensive computation. We propose a new strategy, feature squeezing, that can be used to harden DNN models by detecting adversarial examples. Feature squeezing reduces the search space available to an adversary by coalescing samples that correspond to many different feature vectors in the original space into a single sample. By comparing a DNN model's prediction on the original input with that on squeezed inputs, feature squeezing detects adversarial examples with high accuracy and few false positives.This paper explores two feature squeezing methods: reducing the color bit depth of each pixel and spatial smoothing. These simple strategies are inexpensive and complementary to other defenses, and can be combined in a joint detection framework to achieve high detection rates against state-of-the-art attacks.

对抗性可转移性是一种有趣的性质 - 针对一个模型制作的对抗性扰动也是对另一个模型有效的，而这些模型来自不同的模型家庭或培训过程。为了更好地保护ML系统免受对抗性攻击，提出了几个问题：对抗性转移性的充分条件是什么，以及如何绑定它？有没有办法降低对抗的转移性，以改善合奏ML模型的鲁棒性？为了回答这些问题，在这项工作中，我们首先在理论上分析和概述了模型之间的对抗性可转移的充分条件;然后提出一种实用的算法，以减少集合内基础模型之间的可转换，以提高其鲁棒性。我们的理论分析表明，只有促进基础模型梯度之间的正交性不足以确保低可转移性;与此同时，模型平滑度是控制可转移性的重要因素。我们还在某些条件下提供了对抗性可转移性的下界和上限。灵感来自我们的理论分析，我们提出了一种有效的可转让性，减少了平滑（TRS）集合培训策略，以通过实施基础模型之间的梯度正交性和模型平滑度来培训具有低可转换性的强大集成。我们对TRS进行了广泛的实验，并与6个最先进的集合基线进行比较，防止不同数据集的8个白箱攻击，表明所提出的TRS显着优于所有基线。