神经网络在检测嘈杂数据中的模式方面非常成功，并且已成为许多领域的首选技术。但是，他们对对抗攻击的敏感性阻碍了它们的有用性。最近，已经提出了许多用于衡量和改善网络对对抗性扰动的鲁棒性的方法，并且这项不断增长的研究体现了许多明确或隐性的鲁棒性观念。这些概念之间的联系通常是微妙的，文献中缺少它们之间的系统比较。在本文中，我们开始解决这一差距，通过在网络的培训阶段，其验证和部署之后设置对网络鲁棒性作为数学属性的经验分析和评估的一般原则。然后，我们应用这些原则并进行案例研究，以展示我们一般方法的实际好处。

Explainability has been widely stated as a cornerstone of the responsible and trustworthy use of machine learning models. With the ubiquitous use of Deep Neural Network (DNN) models expanding to risk-sensitive and safety-critical domains, many methods have been proposed to explain the decisions of these models. Recent years have also seen concerted efforts that have shown how such explanations can be distorted (attacked) by minor input perturbations. While there have been many surveys that review explainability methods themselves, there has been no effort hitherto to assimilate the different methods and metrics proposed to study the robustness of explanations of DNN models. In this work, we present a comprehensive survey of methods that study, understand, attack, and defend explanations of DNN models. We also present a detailed review of different metrics used to evaluate explanation methods, as well as describe attributional attack and defense methods. We conclude with lessons and take-aways for the community towards ensuring robust explanations of DNN model predictions.

深度神经网络（DNN）已成为实现各种复杂任务的首选技术。但是，正如许多最近的研究所强调的那样，即使是对正确分类的输入的不可察觉的扰动也可能导致DNN错误分类。这使DNNS容易受到攻击者的战略输入操作，并且对环境噪声过敏。为了减轻这种现象，从业人员通过DNNS的“合奏”进行联合分类。通过汇总不同单个DNN的分类输出对相同的输入，基于合奏的分类可以减少因任何单个DNN的随机训练过程的特定实现而导致错误分类的风险。但是，DNN集合的有效性高度依赖于其成员 *在许多不同的输入上没有同时错误 *。在本案例研究中，我们利用DNN验证的最新进展，设计一种方法来识别一种合奏组成，即使输入对对抗性进行了扰动，也不太容易出现同时误差 - 从而导致基于更坚固的集合分类。我们提出的框架使用DNN验证器作为后端，并包括启发式方法，有助于降低直接验证合奏的高复杂性。从更广泛的角度来看，我们的工作提出了一个新颖的普遍目标，以实现正式验证，该目标可能可以改善各种应用领域的现实世界中基于深度学习的系统的鲁棒性。

Recent work has demonstrated that deep neural networks are vulnerable to adversarial examples-inputs that are almost indistinguishable from natural data and yet classified incorrectly by the network. In fact, some of the latest findings suggest that the existence of adversarial attacks may be an inherent weakness of deep learning models. To address this problem, we study the adversarial robustness of neural networks through the lens of robust optimization. This approach provides us with a broad and unifying view on much of the prior work on this topic. Its principled nature also enables us to identify methods for both training and attacking neural networks that are reliable and, in a certain sense, universal. In particular, they specify a concrete security guarantee that would protect against any adversary. These methods let us train networks with significantly improved resistance to a wide range of adversarial attacks. They also suggest the notion of security against a first-order adversary as a natural and broad security guarantee. We believe that robustness against such well-defined classes of adversaries is an important stepping stone towards fully resistant deep learning models. 1

背景信息：在过去几年中，机器学习（ML）一直是许多创新的核心。然而，包括在所谓的“安全关键”系统中，例如汽车或航空的系统已经被证明是非常具有挑战性的，因为ML的范式转变为ML带来完全改变传统认证方法。目的：本文旨在阐明与ML为基础的安全关键系统认证有关的挑战，以及文献中提出的解决方案，以解决它们，回答问题的问题如何证明基于机器学习的安全关键系统？'方法：我们开展2015年至2020年至2020年之间发布的研究论文的系统文献综述（SLR），涵盖了与ML系统认证有关的主题。总共确定了217篇论文涵盖了主题，被认为是ML认证的主要支柱：鲁棒性，不确定性，解释性，验证，安全强化学习和直接认证。我们分析了每个子场的主要趋势和问题，并提取了提取的论文的总结。结果：单反结果突出了社区对该主题的热情，以及在数据集和模型类型方面缺乏多样性。它还强调需要进一步发展学术界和行业之间的联系，以加深域名研究。最后，它还说明了必须在上面提到的主要支柱之间建立连接的必要性，这些主要柱主要主要研究。结论：我们强调了目前部署的努力，以实现ML基于ML的软件系统，并讨论了一些未来的研究方向。

机器学习算法和深度神经网络在几种感知和控制任务中的卓越性能正在推动该行业在安全关键应用中采用这种技术，作为自治机器人和自动驾驶车辆。然而，目前，需要解决几个问题，以使深入学习方法更可靠，可预测，安全，防止对抗性攻击。虽然已经提出了几种方法来提高深度神经网络的可信度，但大多数都是针对特定类的对抗示例量身定制的，因此未能检测到其他角落案件或不安全的输入，这些输入大量偏离训练样本。本文介绍了基于覆盖范式的轻量级监控架构，以增强针对不同不安全输入的模型鲁棒性。特别是，在用于评估多种检测逻辑的架构中提出并测试了四种覆盖分析方法。实验结果表明，该方法有效地检测强大的对抗性示例和分销外输入，引入有限的执行时间和内存要求。

The authors thank Nicholas Carlini (UC Berkeley) and Dimitris Tsipras (MIT) for feedback to improve the survey quality. We also acknowledge X. Huang (Uni. Liverpool), K. R. Reddy (IISC), E. Valle (UNICAMP), Y. Yoo (CLAIR) and others for providing pointers to make the survey more comprehensive.

在本讨论文件中，我们调查了有关机器学习模型鲁棒性的最新研究。随着学习算法在数据驱动的控制系统中越来越流行，必须确保它们对数据不确定性的稳健性，以维持可靠的安全至关重要的操作。我们首先回顾了这种鲁棒性的共同形式主义，然后继续讨论训练健壮的机器学习模型的流行和最新技术，以及可证明这种鲁棒性的方法。从强大的机器学习的这种统一中，我们识别并讨论了该地区未来研究的迫切方向。

尽管机器学习系统的效率和可扩展性，但最近的研究表明，许多分类方法，尤其是深神经网络（DNN），易受对抗的例子;即，仔细制作欺骗训练有素的分类模型的例子，同时无法区分从自然数据到人类。这使得在安全关键区域中应用DNN或相关方法可能不安全。由于这个问题是由Biggio等人确定的。 （2013）和Szegedy等人。（2014年），在这一领域已经完成了很多工作，包括开发攻击方法，以产生对抗的例子和防御技术的构建防范这些例子。本文旨在向统计界介绍这一主题及其最新发展，主要关注对抗性示例的产生和保护。在数值实验中使用的计算代码（在Python和R）公开可用于读者探讨调查的方法。本文希望提交人们将鼓励更多统计学人员在这种重要的令人兴奋的领域的产生和捍卫对抗的例子。

神经网络已广泛应用于垃圾邮件和网络钓鱼检测，入侵预防和恶意软件检测等安全应用程序。但是，这种黑盒方法通常在应用中具有不确定性和不良的解释性。此外，神经网络本身通常容易受到对抗攻击的影响。由于这些原因，人们对可信赖和严格的方法有很高的需求来验证神经网络模型的鲁棒性。对抗性的鲁棒性在处理恶意操纵输入时涉及神经网络的可靠性，是安全和机器学习中最热门的主题之一。在这项工作中，我们在神经网络的对抗性鲁棒性验证中调查了现有文献，并在机器学习，安全和软件工程领域收集了39项多元化研究工作。我们系统地分析了它们的方法，包括如何制定鲁棒性，使用哪种验证技术以及每种技术的优势和局限性。我们从正式验证的角度提供分类学，以全面理解该主题。我们根据财产规范，减少问题和推理策略对现有技术进行分类。我们还展示了使用样本模型在现有研究中应用的代表性技术。最后，我们讨论了未来研究的开放问题。

Adaptive attacks have (rightfully) become the de facto standard for evaluating defenses to adversarial examples. We find, however, that typical adaptive evaluations are incomplete. We demonstrate that thirteen defenses recently published at ICLR, ICML and NeurIPS-and which illustrate a diverse set of defense strategies-can be circumvented despite attempting to perform evaluations using adaptive attacks. While prior evaluation papers focused mainly on the end result-showing that a defense was ineffective-this paper focuses on laying out the methodology and the approach necessary to perform an adaptive attack. Some of our attack strategies are generalizable, but no single strategy would have been sufficient for all defenses. This underlines our key message that adaptive attacks cannot be automated and always require careful and appropriate tuning to a given defense. We hope that these analyses will serve as guidance on how to properly perform adaptive attacks against defenses to adversarial examples, and thus will allow the community to make further progress in building more robust models.

已知深度神经网络（DNN）容易受到用不可察觉的扰动制作的对抗性示例的影响，即，输入图像的微小变化会引起错误的分类，从而威胁着基于深度学习的部署系统的可靠性。经常采用对抗训练（AT）来通过训练损坏和干净的数据的混合物来提高DNN的鲁棒性。但是，大多数基于AT的方法在处理\ textit {转移的对抗示例}方面是无效的，这些方法是生成以欺骗各种防御模型的生成的，因此无法满足现实情况下提出的概括要求。此外，对抗性训练一般的国防模型不能对具有扰动的输入产生可解释的预测，而不同的领域专家则需要一个高度可解释的强大模型才能了解DNN的行为。在这项工作中，我们提出了一种基于Jacobian规范和选择性输入梯度正则化（J-SIGR）的方法，该方法通过Jacobian归一化提出了线性化的鲁棒性，还将基于扰动的显着性图正规化，以模仿模型的可解释预测。因此，我们既可以提高DNN的防御能力和高解释性。最后，我们评估了跨不同体系结构的方法，以针对强大的对抗性攻击。实验表明，提出的J-Sigr赋予了针对转移的对抗攻击的鲁棒性，我们还表明，来自神经网络的预测易于解释。

Deep learning algorithms have been shown to perform extremely well on many classical machine learning problems. However, recent studies have shown that deep learning, like other machine learning techniques, is vulnerable to adversarial samples: inputs crafted to force a deep neural network (DNN) to provide adversary-selected outputs. Such attacks can seriously undermine the security of the system supported by the DNN, sometimes with devastating consequences. For example, autonomous vehicles can be crashed, illicit or illegal content can bypass content filters, or biometric authentication systems can be manipulated to allow improper access. In this work, we introduce a defensive mechanism called defensive distillation to reduce the effectiveness of adversarial samples on DNNs. We analytically investigate the generalizability and robustness properties granted by the use of defensive distillation when training DNNs. We also empirically study the effectiveness of our defense mechanisms on two DNNs placed in adversarial settings. The study shows that defensive distillation can reduce effectiveness of sample creation from 95% to less than 0.5% on a studied DNN. Such dramatic gains can be explained by the fact that distillation leads gradients used in adversarial sample creation to be reduced by a factor of 10 30 . We also find that distillation increases the average minimum number of features that need to be modified to create adversarial samples by about 800% on one of the DNNs we tested.

对抗性的鲁棒性已经成为深度学习的核心目标，无论是在理论和实践中。然而，成功的方法来改善对抗的鲁棒性（如逆势训练）在不受干扰的数据上大大伤害了泛化性能。这可能会对对抗性鲁棒性如何影响现实世界系统的影响（即，如果它可以提高未受干扰的数据的准确性），许多人可能选择放弃鲁棒性）。我们提出内插对抗培训，该培训最近雇用了在对抗培训框架内基于插值的基于插值的培训方法。在CiFar -10上，对抗性训练增加了标准测试错误（当没有对手时）从4.43％到12.32％，而我们的内插对抗培训我们保留了对抗性的鲁棒性，同时实现了仅6.45％的标准测试误差。通过我们的技术，强大模型标准误差的相对增加从178.1％降至仅为45.5％。此外，我们提供内插对抗性培训的数学分析，以确认其效率，并在鲁棒性和泛化方面展示其优势。

While neural networks have achieved high accuracy on standard image classification benchmarks, their accuracy drops to nearly zero in the presence of small adversarial perturbations to test inputs. Defenses based on regularization and adversarial training have been proposed, but often followed by new, stronger attacks that defeat these defenses. Can we somehow end this arms race? In this work, we study this problem for neural networks with one hidden layer. We first propose a method based on a semidefinite relaxation that outputs a certificate that for a given network and test input, no attack can force the error to exceed a certain value. Second, as this certificate is differentiable, we jointly optimize it with the network parameters, providing an adaptive regularizer that encourages robustness against all attacks. On MNIST, our approach produces a network and a certificate that no attack that perturbs each pixel by at most = 0.1 can cause more than 35% test error.

This paper investigates recently proposed approaches for defending against adversarial examples and evaluating adversarial robustness. We motivate adversarial risk as an objective for achieving models robust to worst-case inputs. We then frame commonly used attacks and evaluation metrics as defining a tractable surrogate objective to the true adversarial risk. This suggests that models may optimize this surrogate rather than the true adversarial risk. We formalize this notion as obscurity to an adversary, and develop tools and heuristics for identifying obscured models and designing transparent models. We demonstrate that this is a significant problem in practice by repurposing gradient-free optimization techniques into adversarial attacks, which we use to decrease the accuracy of several recently proposed defenses to near zero. Our hope is that our formulations and results will help researchers to develop more powerful defenses.

随着深度机器学习对现实生活应用的扩散，该技术的一种特殊属性引起了人们的注意：稳健性神经网络臭名昭著地表现出低的鲁棒性，并且对小输入扰动非常敏感。最近，已经提出了许多用于验证网络鲁棒性的一般特性的方法，但是它们主要用于计算机视觉。在本文中，我们提出了基于较大感兴趣区域的自然语言理解分类的验证规范，我们讨论了此类任务的挑战。我们观察到，尽管数据几乎是线性可分离的，但验证者努力输出积极的结果，我们解释了问题和含义。

对对抗攻击的脆弱性是在安全至关重要应用中采用深度学习的主要障碍之一。尽管做出了巨大的努力，但无论是实用还是理论上的，培训深度学习模型对对抗性攻击仍然是一个悬而未决的问题。在本文中，我们分析了大数据，贝叶斯神经网络（BNNS）中的对抗性攻击的几何形状。我们表明，在限制下，由于数据分布的堕落而产生了基于梯度的攻击的脆弱性，即当数据位于环境空间的较低维度的亚策略上时。直接结果，我们证明，在此限制下，BNN后代对基于梯度的对抗性攻击是强大的。至关重要的是，我们证明，即使从后部采样的每个神经网络都很容易受到基于梯度的攻击，因此相对于BNN后验分布的预期损失梯度正在消失。 MNIST，时尚MNIST和半卫星数据集的实验结果，代表有限的数据制度，并接受了汉密尔顿蒙特卡洛和变异推理的BNN，支持这一论点，表明BNN可以在清洁数据和稳健性上表现出很高的精度对基于梯度和无梯度的对抗性攻击。

We develop and study new adversarial perturbations that enable an attacker to gain control over decisions in generic Artificial Intelligence (AI) systems including deep learning neural networks. In contrast to adversarial data modification, the attack mechanism we consider here involves alterations to the AI system itself. Such a stealth attack could be conducted by a mischievous, corrupt or disgruntled member of a software development team. It could also be made by those wishing to exploit a ``democratization of AI'' agenda, where network architectures and trained parameter sets are shared publicly. We develop a range of new implementable attack strategies with accompanying analysis, showing that with high probability a stealth attack can be made transparent, in the sense that system performance is unchanged on a fixed validation set which is unknown to the attacker, while evoking any desired output on a trigger input of interest. The attacker only needs to have estimates of the size of the validation set and the spread of the AI's relevant latent space. In the case of deep learning neural networks, we show that a one neuron attack is possible - a modification to the weights and bias associated with a single neuron - revealing a vulnerability arising from over-parameterization. We illustrate these concepts using state of the art architectures on two standard image data sets. Guided by the theory and computational results, we also propose strategies to guard against stealth attacks.

深度神经网络很容易被称为对抗攻击的小扰动都愚弄。对抗性培训（AT）是一种近似解决了稳健的优化问题，以最大限度地减少最坏情况损失，并且被广泛认为是对这种攻击的最有效的防御。由于产生了强大的对抗性示例的高计算时间，已经提出了单步方法来减少培训时间。然而，这些方法遭受灾难性的过度装备，在训练期间侵犯准确度下降。虽然提出了改进，但它们增加了培训时间和稳健性远非多步骤。我们为FW优化（FW-AT）开发了对抗的对抗培训的理论框架，揭示了损失景观与$ \ ell_2 $失真之间的几何连接。我们分析地表明FW攻击的高变形相当于沿攻击路径的小梯度变化。然后在各种深度神经网络架构上进行实验证明，$ \ ell \ infty $攻击对抗强大的模型实现近乎最大的$ \ ell_2 $失真，而标准网络具有较低的失真。此外，实验表明，灾难性的过度拟合与FW攻击的低变形强烈相关。为了展示我们理论框架的效用，我们开发FW-AT-Adap，这是一种新的逆势训练算法，它使用简单的失真度量来调整攻击步骤的数量，以提高效率而不会影响鲁棒性。 FW-AT-Adapt提供培训时间以单步快速分配方法，并改善了在白色盒子和黑匣子设置中的普发内精度的最小损失和多步PGD之间的差距。