神经网络对攻击的缺乏鲁棒性引起了对安全敏感环境（例如自动驾驶汽车）的担忧。虽然许多对策看起来可能很有希望，但只有少数能够承受严格的评估。使用随机变换（RT）的防御能力显示出令人印象深刻的结果，尤其是Imagenet上的Bart（Raff等，2019）。但是，这种防御尚未经过严格评估，使其稳健性的理解不足。它们的随机特性使评估更具挑战性，并使对确定性模型的许多拟议攻击不可应用。首先，我们表明BART评估中使用的BPDA攻击（Athalye等，2018a）无效，可能高估了其稳健性。然后，我们尝试通过明智的转换和贝叶斯优化来调整其参数来构建最强的RT防御。此外，我们创造了最强烈的攻击来评估我们的RT防御。我们的新攻击极大地胜过基线，与常用的EOT攻击减少19％相比，将准确性降低了83％（$ 4.3 \ times $改善）。我们的结果表明，在Imagenette数据集上的RT防御（ImageNet的十级子集）在对抗性示例上并不强大。进一步扩展研究，我们使用新的攻击来对抗RT防御（称为Advrt），从而获得了巨大的稳健性增长。代码可从https://github.com/wagner-group/demystify-random-transform获得。

Adaptive attacks have (rightfully) become the de facto standard for evaluating defenses to adversarial examples. We find, however, that typical adaptive evaluations are incomplete. We demonstrate that thirteen defenses recently published at ICLR, ICML and NeurIPS-and which illustrate a diverse set of defense strategies-can be circumvented despite attempting to perform evaluations using adaptive attacks. While prior evaluation papers focused mainly on the end result-showing that a defense was ineffective-this paper focuses on laying out the methodology and the approach necessary to perform an adaptive attack. Some of our attack strategies are generalizable, but no single strategy would have been sufficient for all defenses. This underlines our key message that adaptive attacks cannot be automated and always require careful and appropriate tuning to a given defense. We hope that these analyses will serve as guidance on how to properly perform adaptive attacks against defenses to adversarial examples, and thus will allow the community to make further progress in building more robust models.

防御对抗例子仍然是一个空旷的问题。一个普遍的信念是，推理的随机性增加了寻找对抗性输入的成本。这种辩护的一个例子是将随机转换应用于输入之前，然后将其馈送到模型。在本文中，我们从经验和理论上研究了这种随机预处理的防御措施，并证明它们存在缺陷。首先，我们表明大多数随机防御措施比以前想象的要弱。他们缺乏足够的随机性来承受诸如投影梯度下降之类的标准攻击。这对长期以来的假设产生了怀疑，即随机防御能力无效，旨在逃避确定性的防御和迫使攻击者以整合对转型（EOT）概念的期望。其次，我们表明随机防御与对抗性鲁棒性和模型不变性之间的权衡面临。随着辩护模型获得更多的随机化不变性，它们变得不太有效。未来的工作将需要使这两种效果分解。我们的代码在补充材料中可用。

We identify obfuscated gradients, a kind of gradient masking, as a phenomenon that leads to a false sense of security in defenses against adversarial examples. While defenses that cause obfuscated gradients appear to defeat iterative optimizationbased attacks, we find defenses relying on this effect can be circumvented. We describe characteristic behaviors of defenses exhibiting the effect, and for each of the three types of obfuscated gradients we discover, we develop attack techniques to overcome it. In a case study, examining noncertified white-box-secure defenses at ICLR 2018, we find obfuscated gradients are a common occurrence, with 7 of 9 defenses relying on obfuscated gradients. Our new attacks successfully circumvent 6 completely, and 1 partially, in the original threat model each paper considers.

This paper investigates recently proposed approaches for defending against adversarial examples and evaluating adversarial robustness. We motivate adversarial risk as an objective for achieving models robust to worst-case inputs. We then frame commonly used attacks and evaluation metrics as defining a tractable surrogate objective to the true adversarial risk. This suggests that models may optimize this surrogate rather than the true adversarial risk. We formalize this notion as obscurity to an adversary, and develop tools and heuristics for identifying obscured models and designing transparent models. We demonstrate that this is a significant problem in practice by repurposing gradient-free optimization techniques into adversarial attacks, which we use to decrease the accuracy of several recently proposed defenses to near zero. Our hope is that our formulations and results will help researchers to develop more powerful defenses.

深度卷积神经网络可以准确地分类各种自然图像，但是在设计时可能很容易被欺骗，图像中嵌入了不可察觉的扰动。在本文中，我们设计了一种多管齐下的培训，输入转换和图像集成系统，该系统是攻击不可知论的，不容易估计。我们的系统结合了两个新型功能。第一个是一个转换层，该转换层从集体级训练数据示例中计算级别的多项式内核，并且迭代更新在推理时间上基于其特征内核差异的输入图像副本，以创建转换后的输入集合。第二个是一个分类系统，该系统将未防御网络的预测结合在一起，对被过滤图像的合奏进行了硬投票。我们在CIFAR10数据集上的评估显示，我们的系统提高了未防御性网络在不同距离指标下的各种有界和无限的白色盒子攻击的鲁棒性，同时牺牲了清洁图像的精度很小。反对自适应的全知攻击者创建端到端攻击，我们的系统成功地增强了对抗训练的网络的现有鲁棒性，为此，我们的方法最有效地应用了。

虽然深度神经网络（DNN）在许多真实的任务中实现了出色的性能，但它们非常容易受到对抗的攻击。对抗这种攻击的主要防御是对抗的，一种技术，通过将对抗噪声引入其输入来训练DNN培训以训练为对抗性攻击的技术。此程序是有效的，但必须在培训阶段进行。在这项工作中，我们提出了增强随机森林（ARF），这是一个简单易用的策略，用于在不修改其权重的情况下强化现有的预磨损DNN。对于每个图像，我们通过应用不同颜色，模糊，噪声和几何变换来生成随机测试时间增强。然后我们使用DNN的Logits输出来训练一个简单的随机林来预测真正的类标签。我们的方法在自然图像的分类上最小的妥协，实现了最先进的对抗鲁棒性对白和黑匣子攻击的多样性。我们也针对许多适应性的白盒攻击测试ARF，并在与对抗训练结合时显示出优异的结果。代码可在https://github.com/giladcohen/arf获得。

Although deep neural networks (DNNs) have achieved great success in many tasks, they can often be fooled by adversarial examples that are generated by adding small but purposeful distortions to natural examples. Previous studies to defend against adversarial examples mostly focused on refining the DNN models, but have either shown limited success or required expensive computation. We propose a new strategy, feature squeezing, that can be used to harden DNN models by detecting adversarial examples. Feature squeezing reduces the search space available to an adversary by coalescing samples that correspond to many different feature vectors in the original space into a single sample. By comparing a DNN model's prediction on the original input with that on squeezed inputs, feature squeezing detects adversarial examples with high accuracy and few false positives.This paper explores two feature squeezing methods: reducing the color bit depth of each pixel and spatial smoothing. These simple strategies are inexpensive and complementary to other defenses, and can be combined in a joint detection framework to achieve high detection rates against state-of-the-art attacks.

The field of defense strategies against adversarial attacks has significantly grown over the last years, but progress is hampered as the evaluation of adversarial defenses is often insufficient and thus gives a wrong impression of robustness. Many promising defenses could be broken later on, making it difficult to identify the state-of-the-art. Frequent pitfalls in the evaluation are improper tuning of hyperparameters of the attacks, gradient obfuscation or masking. In this paper we first propose two extensions of the PGD-attack overcoming failures due to suboptimal step size and problems of the objective function. We then combine our novel attacks with two complementary existing ones to form a parameter-free, computationally affordable and user-independent ensemble of attacks to test adversarial robustness. We apply our ensemble to over 50 models from papers published at recent top machine learning and computer vision venues. In all except one of the cases we achieve lower robust test accuracy than reported in these papers, often by more than 10%, identifying several broken defenses.

在测试时间进行优化的自适应防御能力有望改善对抗性鲁棒性。我们对这种自适应测试时间防御措施进行分类，解释其潜在的好处和缺点，并评估图像分类的最新自适应防御能力的代表性。不幸的是，经过我们仔细的案例研究评估时，没有任何显着改善静态防御。有些甚至削弱了基本静态模型，同时增加了推理计算。尽管这些结果令人失望，但我们仍然认为自适应测试时间防御措施是一项有希望的研究途径，因此，我们为他们的彻底评估提供了建议。我们扩展了Carlini等人的清单。（2019年）通过提供针对自适应防御的具体步骤。

发言人识别系统（SRSS）最近被证明容易受到对抗攻击的影响，从而引发了重大的安全问题。在这项工作中，我们系统地研究了基于确保SRSS的基于对抗性训练的防御。根据SRSS的特征，我们提出了22种不同的转换，并使用扬声器识别的7种最新有前途的对抗攻击（4个白盒和3个Black-Box）对其进行了彻底评估。仔细考虑了国防评估中的最佳实践，我们分析了转换的强度以承受适应性攻击。我们还评估并理解它们与对抗训练相结合的自适应攻击的有效性。我们的研究提供了许多有用的见解和发现，其中许多与图像和语音识别域中的结论是新的或不一致的，例如，可变和恒定的比特率语音压缩具有不同的性能，并且某些不可差的转换仍然有效地抗衡。当前有希望的逃避技术通常在图像域中很好地工作。我们证明，与完整的白色盒子设置中的唯一对抗性训练相比，提出的新型功能级转换与对抗训练相比是相当有效的，例如，将准确性提高了13.62％，而攻击成本则达到了两个数量级，而其他攻击成本则增加了。转型不一定会提高整体防御能力。这项工作进一步阐明了该领域的研究方向。我们还发布了我们的评估平台SpeakerGuard，以促进进一步的研究。

Neural networks provide state-of-the-art results for most machine learning tasks. Unfortunately, neural networks are vulnerable to adversarial examples: given an input x and any target classification t, it is possible to find a new input x that is similar to x but classified as t. This makes it difficult to apply neural networks in security-critical areas. Defensive distillation is a recently proposed approach that can take an arbitrary neural network, and increase its robustness, reducing the success rate of current attacks' ability to find adversarial examples from 95% to 0.5%.In this paper, we demonstrate that defensive distillation does not significantly increase the robustness of neural networks by introducing three new attack algorithms that are successful on both distilled and undistilled neural networks with 100% probability. Our attacks are tailored to three distance metrics used previously in the literature, and when compared to previous adversarial example generation algorithms, our attacks are often much more effective (and never worse). Furthermore, we propose using high-confidence adversarial examples in a simple transferability test we show can also be used to break defensive distillation. We hope our attacks will be used as a benchmark in future defense attempts to create neural networks that resist adversarial examples.

Neural networks are known to be vulnerable to adversarial examples: inputs that are close to natural inputs but classified incorrectly. In order to better understand the space of adversarial examples, we survey ten recent proposals that are designed for detection and compare their efficacy. We show that all can be defeated by constructing new loss functions. We conclude that adversarial examples are significantly harder to detect than previously appreciated, and the properties believed to be intrinsic to adversarial examples are in fact not. Finally, we propose several simple guidelines for evaluating future proposed defenses.

对抗性培训（AT）已成为培训强大网络的热门选择。然而，它倾向于牺牲清洁精度，以令人满意的鲁棒性，并且遭受大的概括误差。为了解决这些问题，我们提出了平稳的对抗培训（SAT），以我们对损失令人歉端的损失的终人谱指导。 We find that curriculum learning, a scheme that emphasizes on starting "easy" and gradually ramping up on the "difficulty" of training, smooths the adversarial loss landscape for a suitably chosen difficulty metric.我们展示了对普通环境中的课程学习的一般制定，并提出了一种基于最大Hessian特征值（H-SAT）和软MAX概率（P-SA）的两个难度指标。我们展示SAT稳定网络培训即使是大型扰动规范，并且允许网络以更好的清洁精度运行而与鲁棒性权衡曲线相比。与AT，交易和其他基线相比，这导致清洁精度和鲁棒性的显着改善。为了突出一些结果，我们的最佳模型将分别在CIFAR-100上提高6％和1％的稳健准确性。在Imagenette上，一个十一级想象成的子集，我们的模型分别以正常和强大的准确性达到23％和3％。

Designing powerful adversarial attacks is of paramount importance for the evaluation of $\ell_p$-bounded adversarial defenses. Projected Gradient Descent (PGD) is one of the most effective and conceptually simple algorithms to generate such adversaries. The search space of PGD is dictated by the steepest ascent directions of an objective. Despite the plethora of objective function choices, there is no universally superior option and robustness overestimation may arise from ill-suited objective selection. Driven by this observation, we postulate that the combination of different objectives through a simple loss alternating scheme renders PGD more robust towards design choices. We experimentally verify this assertion on a synthetic-data example and by evaluating our proposed method across 25 different $\ell_{\infty}$-robust models and 3 datasets. The performance improvement is consistent, when compared to the single loss counterparts. In the CIFAR-10 dataset, our strongest adversarial attack outperforms all of the white-box components of AutoAttack (AA) ensemble, as well as the most powerful attacks existing on the literature, achieving state-of-the-art results in the computational budget of our study ($T=100$, no restarts).

Adversarial examples are perturbed inputs designed to fool machine learning models. Adversarial training injects such examples into training data to increase robustness. To scale this technique to large datasets, perturbations are crafted using fast single-step methods that maximize a linear approximation of the model's loss. We show that this form of adversarial training converges to a degenerate global minimum, wherein small curvature artifacts near the data points obfuscate a linear approximation of the loss. The model thus learns to generate weak perturbations, rather than defend against strong ones. As a result, we find that adversarial training remains vulnerable to black-box attacks, where we transfer perturbations computed on undefended models, as well as to a powerful novel single-step attack that escapes the non-smooth vicinity of the input data via a small random step. We further introduce Ensemble Adversarial Training, a technique that augments training data with perturbations transferred from other models. On ImageNet, Ensemble Adversarial Training yields models with stronger robustness to blackbox attacks. In particular, our most robust model won the first round of the NIPS 2017 competition on Defenses against Adversarial Attacks (Kurakin et al., 2017c). However, subsequent work found that more elaborate black-box attacks could significantly enhance transferability and reduce the accuracy of our models.

添加到输入的最小侵犯扰动已被证明在愚弄深度神经网络方面有效。在本文中，我们介绍了几种创新，使白盒子目标攻击遵循攻击者的目标：欺骗模型将更高的目标类概率分配比任何其他更高的概率，同时停留在距原始距离的指定距离内输入。首先，我们提出了一种新的损失函数，明确地捕获了目标攻击的目标，特别是通过使用所有类的Logits而不是仅仅是一个子集。我们表明，具有这种损失功能的自动PGD比与其他常用损耗功能相比发现更多的对抗示例。其次，我们提出了一种新的攻击方法，它使用进一步发发版本的我们的损失函数捕获错误分类目标和$ l _ {\ infty} $距离限制$ \ epsilon $。这种新的攻击方法在CIFAR10 DataSet上比较成功了1.5--4.2％，而在ImageNet DataSet上比下一个最先进的攻击更成功。我们使用统计测试确认，我们的攻击优于最先进的攻击不同数据集和$ \ epsilon $和不同防御的价值。

虽然深度神经网络在各种任务中表现出前所未有的性能，但对对抗性示例的脆弱性阻碍了他们在安全关键系统中的部署。许多研究表明，即使在黑盒设置中也可能攻击，其中攻击者无法访问目标模型的内部信息。大多数黑匣子攻击基于查询，每个都可以获得目标模型的输入输出，并且许多研究侧重于减少所需查询的数量。在本文中，我们注意了目标模型的输出完全对应于查询输入的隐含假设。如果将某些随机性引入模型中，它可以打破假设，因此，基于查询的攻击可能在梯度估计和本地搜索中具有巨大的困难，这是其攻击过程的核心。从这种动机来看，我们甚至观察到一个小的添加剂输入噪声可以中和大多数基于查询的攻击和名称这个简单但有效的方法小噪声防御（SND）。我们分析了SND如何防御基于查询的黑匣子攻击，并展示其与CIFAR-10和ImageNet数据集的八种最先进的攻击有效性。即使具有强大的防御能力，SND几乎保持了原始的分类准确性和计算速度。通过在推断下仅添加一行代码，SND很容易适用于预先训练的模型。

评估防御模型的稳健性是对抗对抗鲁棒性研究的具有挑战性的任务。僵化的渐变，先前已经发现了一种梯度掩蔽，以许多防御方法存在并导致鲁棒性的错误信号。在本文中，我们确定了一种更细微的情况，称为不平衡梯度，也可能导致过高的对抗性鲁棒性。当边缘损耗的一个术语的梯度主导并将攻击朝向次优化方向推动时，发生不平衡梯度的现象。为了利用不平衡的梯度，我们制定了分解利润率损失的边缘分解（MD）攻击，并通过两阶段过程分别探讨了这些术语的攻击性。我们还提出了一个Multared和Ensemble版本的MD攻击。通过调查自2018年以来提出的17个防御模型，我们发现6种型号易受不平衡梯度的影响，我们的MD攻击可以减少由最佳基线独立攻击评估的鲁棒性另外2％。我们还提供了对不平衡梯度的可能原因和有效对策的深入分析。

随着现实世界图像的大小不同，机器学习模型是包括上游图像缩放算法的较大系统的一部分。在本文中，我们研究了基于决策的黑框设置中图像缩放过程的漏洞与机器学习模型之间的相互作用。我们提出了一种新颖的采样策略，以端到端的方式使黑框攻击利用漏洞在缩放算法，缩放防御和最终的机器学习模型中。基于这种缩放感知的攻击，我们揭示了大多数现有的缩放防御能力在下游模型的威胁下无效。此外，我们从经验上观察到，标准的黑盒攻击可以通过利用脆弱的缩放程序来显着提高其性能。我们进一步在具有基于决策的黑盒攻击的商业图像分析API上证明了这个问题。