深度图像对小的Adver-Sarial投入扰动的脆性已被广泛研究了持续数年。然而，前面扰动的主要目标主要是限于通过不正确的一个更改了Cor-Rective预测的上1个，这不打算改变顶级k预测。在许多Digi-Tal真实世界的情景中，顶级Kpriction更加相关。在这项工作中，我们提出了一种快速准确的方法，使Top-Kadversarial示例的快速准确的方法是简单的多目标优化。我们通过将其与其他对抗性示例分泌技术进行比较来展示其功效安培。此外，基于这种方法，Wepropose Top-kuniversal对抗扰动，图像无关的微小扰动，导致真正的类别在大多数NAT-Ulal图像中的顶级Kpriction。我们通过实验表明我们的接近基线方法，甚至改善了寻找普遍对抗扰动的现有技术。

The authors thank Nicholas Carlini (UC Berkeley) and Dimitris Tsipras (MIT) for feedback to improve the survey quality. We also acknowledge X. Huang (Uni. Liverpool), K. R. Reddy (IISC), E. Valle (UNICAMP), Y. Yoo (CLAIR) and others for providing pointers to make the survey more comprehensive.

State-of-the-art deep neural networks have achieved impressive results on many image classification tasks. However, these same architectures have been shown to be unstable to small, well sought, perturbations of the images. Despite the importance of this phenomenon, no effective methods have been proposed to accurately compute the robustness of state-of-the-art deep classifiers to such perturbations on large-scale datasets. In this paper, we fill this gap and propose the DeepFool algorithm to efficiently compute perturbations that fool deep networks, and thus reliably quantify the robustness of these classifiers. Extensive experimental results show that our approach outperforms recent methods in the task of computing adversarial perturbations and making classifiers more robust. 1

Recent work has demonstrated that deep neural networks are vulnerable to adversarial examples-inputs that are almost indistinguishable from natural data and yet classified incorrectly by the network. In fact, some of the latest findings suggest that the existence of adversarial attacks may be an inherent weakness of deep learning models. To address this problem, we study the adversarial robustness of neural networks through the lens of robust optimization. This approach provides us with a broad and unifying view on much of the prior work on this topic. Its principled nature also enables us to identify methods for both training and attacking neural networks that are reliable and, in a certain sense, universal. In particular, they specify a concrete security guarantee that would protect against any adversary. These methods let us train networks with significantly improved resistance to a wide range of adversarial attacks. They also suggest the notion of security against a first-order adversary as a natural and broad security guarantee. We believe that robustness against such well-defined classes of adversaries is an important stepping stone towards fully resistant deep learning models. 1

对抗性实例的有趣现象引起了机器学习中的显着关注，对社区可能更令人惊讶的是存在普遍对抗扰动（UAPS），即欺骗目标DNN的单一扰动。随着对深层分类器的关注，本调查总结了最近普遍对抗攻击的进展，讨论了攻击和防御方的挑战，以及uap存在的原因。我们的目标是将此工作扩展为动态调查，该调查将定期更新其内容，以遵循关于在广泛的域中的UAP或通用攻击的新作品，例如图像，音频，视频，文本等。将讨论相关更新：https://bit.ly/2sbqlgg。我们欢迎未来的作者在该领域的作品，联系我们，包括您的新发现。

Given a state-of-the-art deep neural network classifier, we show the existence of a universal (image-agnostic) and very small perturbation vector that causes natural images to be misclassified with high probability. We propose a systematic algorithm for computing universal perturbations, and show that state-of-the-art deep neural networks are highly vulnerable to such perturbations, albeit being quasiimperceptible to the human eye. We further empirically analyze these universal perturbations and show, in particular, that they generalize very well across neural networks. The surprising existence of universal perturbations reveals important geometric correlations among the high-dimensional decision boundary of classifiers. It further outlines potential security breaches with the existence of single directions in the input space that adversaries can possibly exploit to break a classifier on most natural images. 1

许多最先进的ML模型在各种任务中具有优于图像分类的人类。具有如此出色的性能，ML模型今天被广泛使用。然而，存在对抗性攻击和数据中毒攻击的真正符合ML模型的稳健性。例如，Engstrom等人。证明了最先进的图像分类器可以容易地被任意图像上的小旋转欺骗。由于ML系统越来越纳入安全性和安全敏感的应用，对抗攻击和数据中毒攻击构成了相当大的威胁。本章侧重于ML安全的两个广泛和重要的领域：对抗攻击和数据中毒攻击。

深度学习（DL）系统的安全性是一个极为重要的研究领域，因为它们正在部署在多个应用程序中，因为它们不断改善，以解决具有挑战性的任务。尽管有压倒性的承诺，但深度学习系统容易受到制作的对抗性例子的影响，这可能是人眼无法察觉的，但可能会导致模型错误分类。对基于整体技术的对抗性扰动的保护已被证明很容易受到更强大的对手的影响，或者证明缺乏端到端评估。在本文中，我们试图开发一种新的基于整体的解决方案，该解决方案构建具有不同决策边界的防御者模型相对于原始模型。通过（1）通过一种称为拆分和剃须的方法转换输入的分类器的合奏，以及（2）通过一种称为对比度功能的方法限制重要特征，显示出相对于相对于不同的梯度对抗性攻击，这减少了将对抗性示例从原始示例转移到针对同一类的防御者模型的机会。我们使用标准图像分类数据集（即MNIST，CIFAR-10和CIFAR-100）进行了广泛的实验，以实现最新的对抗攻击，以证明基于合奏的防御的鲁棒性。我们还在存在更强大的对手的情况下评估稳健性，该对手同时靶向合奏中的所有模型。已经提供了整体假阳性和误报的结果，以估计提出的方法的总体性能。

The evaluation of robustness against adversarial manipulation of neural networks-based classifiers is mainly tested with empirical attacks as methods for the exact computation, even when available, do not scale to large networks. We propose in this paper a new white-box adversarial attack wrt the l p -norms for p ∈ {1, 2, ∞} aiming at finding the minimal perturbation necessary to change the class of a given input. It has an intuitive geometric meaning, yields quickly high quality results, minimizes the size of the perturbation (so that it returns the robust accuracy at every threshold with a single run). It performs better or similar to stateof-the-art attacks which are partially specialized to one l p -norm, and is robust to the phenomenon of gradient masking.

已知深度神经网络（DNN）容易受到用不可察觉的扰动制作的对抗性示例的影响，即，输入图像的微小变化会引起错误的分类，从而威胁着基于深度学习的部署系统的可靠性。经常采用对抗训练（AT）来通过训练损坏和干净的数据的混合物来提高DNN的鲁棒性。但是，大多数基于AT的方法在处理\ textit {转移的对抗示例}方面是无效的，这些方法是生成以欺骗各种防御模型的生成的，因此无法满足现实情况下提出的概括要求。此外，对抗性训练一般的国防模型不能对具有扰动的输入产生可解释的预测，而不同的领域专家则需要一个高度可解释的强大模型才能了解DNN的行为。在这项工作中，我们提出了一种基于Jacobian规范和选择性输入梯度正则化（J-SIGR）的方法，该方法通过Jacobian归一化提出了线性化的鲁棒性，还将基于扰动的显着性图正规化，以模仿模型的可解释预测。因此，我们既可以提高DNN的防御能力和高解释性。最后，我们评估了跨不同体系结构的方法，以针对强大的对抗性攻击。实验表明，提出的J-Sigr赋予了针对转移的对抗攻击的鲁棒性，我们还表明，来自神经网络的预测易于解释。

尽管深层神经网络在各种任务中取得了巨大的成功，但它们对不可察觉的对抗性扰动的脆弱性阻碍了他们在现实世界中的部署。最近，与随机合奏的作品相对于经过最小的计算开销的标准对手训练（AT）模型，对对抗性训练（AT）模型的对抗性鲁棒性有了显着改善，这使它们成为安全临界资源限制应用程序的有前途解决方案。但是，这种令人印象深刻的表现提出了一个问题：这些稳健性是由随机合奏提供的吗？在这项工作中，我们从理论和经验上都解决了这个问题。从理论上讲，我们首先确定通常采用的鲁棒性评估方法（例如自适应PGD）在这种情况下提供了错误的安全感。随后，我们提出了一种理论上有效的对抗攻击算法（ARC），即使在自适应PGD无法做到这一点的情况下，也能妥协随机合奏。我们在各种网络体系结构，培训方案，数据集和规范上进行全面的实验，以支持我们的主张，并经验证明，随机合奏实际上比在模型上更容易受到$ \ ell_p $结合的对抗性扰动的影响。我们的代码可以在https://github.com/hsndbk4/arc上找到。

Although deep learning has made remarkable progress in processing various types of data such as images, text and speech, they are known to be susceptible to adversarial perturbations: perturbations specifically designed and added to the input to make the target model produce erroneous output. Most of the existing studies on generating adversarial perturbations attempt to perturb the entire input indiscriminately. In this paper, we propose ExploreADV, a general and flexible adversarial attack system that is capable of modeling regional and imperceptible attacks, allowing users to explore various kinds of adversarial examples as needed. We adapt and combine two existing boundary attack methods, DeepFool and Brendel\&Bethge Attack, and propose a mask-constrained adversarial attack system, which generates minimal adversarial perturbations under the pixel-level constraints, namely ``mask-constraints''. We study different ways of generating such mask-constraints considering the variance and importance of the input features, and show that our adversarial attack system offers users good flexibility to focus on sub-regions of inputs, explore imperceptible perturbations and understand the vulnerability of pixels/regions to adversarial attacks. We demonstrate our system to be effective based on extensive experiments and user study.

评估对抗性鲁棒性的量，以找到有输入样品被错误分类所需的最小扰动。底层优化的固有复杂性需要仔细调整基于梯度的攻击，初始化，并且可能为许多计算苛刻的迭代而被执行，即使专门用于给定的扰动模型也是如此。在这项工作中，我们通过提出使用不同$ \ ell_p $ -norm扰动模型（$ p = 0，1,2，\ idty $）的快速最小规范（FMN）攻击来克服这些限制（FMN）攻击选择，不需要对抗性起点，并在很少的轻量级步骤中收敛。它通过迭代地发现在$ \ ell_p $ -norm的最大信心被错误分类的样本进行了尺寸的尺寸$ \ epsilon $的限制，同时适应$ \ epsilon $，以最小化当前样本到决策边界的距离。广泛的实验表明，FMN在收敛速度和计算时间方面显着优于现有的攻击，同时报告可比或甚至更小的扰动尺寸。

深度学习的进步使得广泛的有希望的应用程序。然而，这些系统容易受到对抗机器学习（AML）攻击的影响;对他们的意见的离前事实制作的扰动可能导致他们错误分类。若干最先进的对抗性攻击已经证明他们可以可靠地欺骗分类器，使这些攻击成为一个重大威胁。对抗性攻击生成算法主要侧重于创建成功的例子，同时控制噪声幅度和分布，使检测更加困难。这些攻击的潜在假设是脱机产生的对抗噪声，使其执行时间是次要考虑因素。然而，最近，攻击者机会自由地产生对抗性示例的立即对抗攻击已经可能。本文介绍了一个新问题：我们如何在实时约束下产生对抗性噪音，以支持这种实时对抗攻击？了解这一问题提高了我们对这些攻击对实时系统构成的威胁的理解，并为未来防御提供安全评估基准。因此，我们首先进行对抗生成算法的运行时间分析。普遍攻击脱机产生一般攻击，没有在线开销，并且可以应用于任何输入;然而，由于其一般性，他们的成功率是有限的。相比之下，在特定输入上工作的在线算法是计算昂贵的，使它们不适合在时间约束下的操作。因此，我们提出房间，一种新型实时在线脱机攻击施工模型，其中离线组件用于预热在线算法，使得可以在时间限制下产生高度成功的攻击。

Adaptive attacks have (rightfully) become the de facto standard for evaluating defenses to adversarial examples. We find, however, that typical adaptive evaluations are incomplete. We demonstrate that thirteen defenses recently published at ICLR, ICML and NeurIPS-and which illustrate a diverse set of defense strategies-can be circumvented despite attempting to perform evaluations using adaptive attacks. While prior evaluation papers focused mainly on the end result-showing that a defense was ineffective-this paper focuses on laying out the methodology and the approach necessary to perform an adaptive attack. Some of our attack strategies are generalizable, but no single strategy would have been sufficient for all defenses. This underlines our key message that adaptive attacks cannot be automated and always require careful and appropriate tuning to a given defense. We hope that these analyses will serve as guidance on how to properly perform adaptive attacks against defenses to adversarial examples, and thus will allow the community to make further progress in building more robust models.

在本讨论文件中，我们调查了有关机器学习模型鲁棒性的最新研究。随着学习算法在数据驱动的控制系统中越来越流行，必须确保它们对数据不确定性的稳健性，以维持可靠的安全至关重要的操作。我们首先回顾了这种鲁棒性的共同形式主义，然后继续讨论训练健壮的机器学习模型的流行和最新技术，以及可证明这种鲁棒性的方法。从强大的机器学习的这种统一中，我们识别并讨论了该地区未来研究的迫切方向。

基于深度神经网络（DNN）的智能信息（IOT）系统已被广泛部署在现实世界中。然而，发现DNNS易受对抗性示例的影响，这提高了人们对智能物联网系统的可靠性和安全性的担忧。测试和评估IOT系统的稳健性成为必要和必要。最近已经提出了各种攻击和策略，但效率问题仍未纠正。现有方法是计算地广泛或耗时，这在实践中不适用。在本文中，我们提出了一种称为攻击启发GaN（AI-GaN）的新框架，在有条件地产生对抗性实例。曾经接受过培训，可以有效地给予对抗扰动的输入图像和目标类。我们在白盒设置的不同数据集中应用AI-GaN，黑匣子设置和由最先进的防御保护的目标模型。通过广泛的实验，AI-GaN实现了高攻击成功率，优于现有方法，并显着降低了生成时间。此外，首次，AI-GaN成功地缩放到复杂的数据集。 Cifar-100和Imagenet，所有课程中的成功率约为90美元。

State-of-the-art classifiers have been shown to be largely vulnerable to adversarial perturbations. One of the most effective strategies to improve robustness is adversarial training. In this paper, we investigate the effect of adversarial training on the geometry of the classification landscape and decision boundaries. We show in particular that adversarial training leads to a significant decrease in the curvature of the loss surface with respect to inputs, leading to a drastically more "linear" behaviour of the network. Using a locally quadratic approximation, we provide theoretical evidence on the existence of a strong relation between large robustness and small curvature. To further show the importance of reduced curvature for improving the robustness, we propose a new regularizer that directly minimizes curvature of the loss surface, and leads to adversarial robustness that is on par with adversarial training. Besides being a more efficient and principled alternative to adversarial training, the proposed regularizer confirms our claims on the importance of exhibiting quasi-linear behavior in the vicinity of data points in order to achieve robustness.

Deep neural networks are vulnerable to adversarial examples, which poses security concerns on these algorithms due to the potentially severe consequences. Adversarial attacks serve as an important surrogate to evaluate the robustness of deep learning models before they are deployed. However, most of existing adversarial attacks can only fool a black-box model with a low success rate. To address this issue, we propose a broad class of momentum-based iterative algorithms to boost adversarial attacks. By integrating the momentum term into the iterative process for attacks, our methods can stabilize update directions and escape from poor local maxima during the iterations, resulting in more transferable adversarial examples. To further improve the success rates for black-box attacks, we apply momentum iterative algorithms to an ensemble of models, and show that the adversarially trained models with a strong defense ability are also vulnerable to our black-box attacks. We hope that the proposed methods will serve as a benchmark for evaluating the robustness of various deep models and defense methods. With this method, we won the first places in NIPS 2017 Non-targeted Adversarial Attack and Targeted Adversarial Attack competitions.

Although deep neural networks (DNNs) have achieved great success in many tasks, they can often be fooled by adversarial examples that are generated by adding small but purposeful distortions to natural examples. Previous studies to defend against adversarial examples mostly focused on refining the DNN models, but have either shown limited success or required expensive computation. We propose a new strategy, feature squeezing, that can be used to harden DNN models by detecting adversarial examples. Feature squeezing reduces the search space available to an adversary by coalescing samples that correspond to many different feature vectors in the original space into a single sample. By comparing a DNN model's prediction on the original input with that on squeezed inputs, feature squeezing detects adversarial examples with high accuracy and few false positives.This paper explores two feature squeezing methods: reducing the color bit depth of each pixel and spatial smoothing. These simple strategies are inexpensive and complementary to other defenses, and can be combined in a joint detection framework to achieve high detection rates against state-of-the-art attacks.