Data poisoning is an attack on machine learning models wherein the attacker adds examples to the training set to manipulate the behavior of the model at test time. This paper explores poisoning attacks on neural nets. The proposed attacks use "clean-labels"; they don't require the attacker to have any control over the labeling of training data. They are also targeted; they control the behavior of the classifier on a specific test instance without degrading overall classifier performance. For example, an attacker could add a seemingly innocuous image (that is properly labeled) to a training set for a face recognition engine, and control the identity of a chosen person at test time. Because the attacker does not need to control the labeling function, poisons could be entered into the training set simply by leaving them on the web and waiting for them to be scraped by a data collection bot. We present an optimization-based method for crafting poisons, and show that just one single poison image can control classifier behavior when transfer learning is used. For full end-to-end training, we present a "watermarking" strategy that makes poisoning reliable using multiple (≈ 50) poisoned training instances. We demonstrate our method by generating poisoned frog images from the CIFAR dataset and using them to manipulate image classifiers.

With the success of deep learning algorithms in various domains, studying adversarial attacks to secure deep models in real world applications has become an important research topic. Backdoor attacks are a form of adversarial attacks on deep networks where the attacker provides poisoned data to the victim to train the model with, and then activates the attack by showing a specific small trigger pattern at the test time. Most state-of-the-art backdoor attacks either provide mislabeled poisoning data that is possible to identify by visual inspection, reveal the trigger in the poisoned data, or use noise to hide the trigger. We propose a novel form of backdoor attack where poisoned data look natural with correct labels and also more importantly, the attacker hides the trigger in the poisoned data and keeps the trigger secret until the test time.We perform an extensive study on various image classification settings and show that our attack can fool the model by pasting the trigger at random locations on unseen images although the model performs well on clean data. We also show that our proposed attack cannot be easily defended using a state-of-the-art defense algorithm for backdoor attacks.

许多最先进的ML模型在各种任务中具有优于图像分类的人类。具有如此出色的性能，ML模型今天被广泛使用。然而，存在对抗性攻击和数据中毒攻击的真正符合ML模型的稳健性。例如，Engstrom等人。证明了最先进的图像分类器可以容易地被任意图像上的小旋转欺骗。由于ML系统越来越纳入安全性和安全敏感的应用，对抗攻击和数据中毒攻击构成了相当大的威胁。本章侧重于ML安全的两个广泛和重要的领域：对抗攻击和数据中毒攻击。

We introduce camouflaged data poisoning attacks, a new attack vector that arises in the context of machine unlearning and other settings when model retraining may be induced. An adversary first adds a few carefully crafted points to the training dataset such that the impact on the model's predictions is minimal. The adversary subsequently triggers a request to remove a subset of the introduced points at which point the attack is unleashed and the model's predictions are negatively affected. In particular, we consider clean-label targeted attacks (in which the goal is to cause the model to misclassify a specific test point) on datasets including CIFAR-10, Imagenette, and Imagewoof. This attack is realized by constructing camouflage datapoints that mask the effect of a poisoned dataset.

与令人印象深刻的进步触动了我们社会的各个方面，基于深度神经网络（DNN）的AI技术正在带来越来越多的安全问题。虽然在考试时间运行的攻击垄断了研究人员的初始关注，但是通过干扰培训过程来利用破坏DNN模型的可能性，代表了破坏训练过程的可能性，这是破坏AI技术的可靠性的进一步严重威胁。在后门攻击中，攻击者损坏了培训数据，以便在测试时间诱导错误的行为。然而，测试时间误差仅在存在与正确制作的输入样本对应的触发事件的情况下被激活。通过这种方式，损坏的网络继续正常输入的预期工作，并且只有当攻击者决定激活网络内隐藏的后门时，才会发生恶意行为。在过去几年中，后门攻击一直是强烈的研究活动的主题，重点是新的攻击阶段的发展，以及可能对策的提议。此概述文件的目标是审查发表的作品，直到现在，分类到目前为止提出的不同类型的攻击和防御。指导分析的分类基于攻击者对培训过程的控制量，以及防御者验证用于培训的数据的完整性，并监控DNN在培训和测试中的操作时间。因此，拟议的分析特别适合于参考他们在运营的应用方案的攻击和防御的强度和弱点。

随着机器学习数据的策展变得越来越自动化，数据集篡改是一种安装威胁。后门攻击者通过培训数据篡改，以嵌入在该数据上培训的模型中的漏洞。然后通过将“触发”放入模型的输入中的推理时间以推理时间激活此漏洞。典型的后门攻击将触发器直接插入训练数据，尽管在检查时可能会看到这种攻击。相比之下，隐藏的触发后托攻击攻击达到中毒，而无需将触发器放入训练数据即可。然而，这种隐藏的触发攻击在从头开始培训的中毒神经网络时无效。我们开发了一个新的隐藏触发攻击，睡眠代理，在制备过程中使用梯度匹配，数据选择和目标模型重新培训。睡眠者代理是第一个隐藏的触发后门攻击，以对从头开始培训的神经网络有效。我们展示了Imagenet和黑盒设置的有效性。我们的实现代码可以在https://github.com/hsouri/sleeper-agent找到。

计算能力和大型培训数据集的可用性增加，机器学习的成功助长了。假设它充分代表了在测试时遇到的数据，则使用培训数据来学习新模型或更新现有模型。这种假设受到中毒威胁的挑战，这种攻击会操纵训练数据，以损害模型在测试时的表现。尽管中毒已被认为是行业应用中的相关威胁，到目前为止，已经提出了各种不同的攻击和防御措施，但对该领域的完整系统化和批判性审查仍然缺失。在这项调查中，我们在机器学习中提供了中毒攻击和防御措施的全面系统化，审查了过去15年中该领域发表的100多篇论文。我们首先对当前的威胁模型和攻击进行分类，然后相应地组织现有防御。虽然我们主要关注计算机视觉应用程序，但我们认为我们的系统化还包括其他数据模式的最新攻击和防御。最后，我们讨论了中毒研究的现有资源，并阐明了当前的局限性和该研究领域的开放研究问题。

Learning-based pattern classifiers, including deep networks, have shown impressive performance in several application domains, ranging from computer vision to cybersecurity. However, it has also been shown that adversarial input perturbations carefully crafted either at training or at test time can easily subvert their predictions. The vulnerability of machine learning to such wild patterns (also referred to as adversarial examples), along with the design of suitable countermeasures, have been investigated in the research field of adversarial machine learning. In this work, we provide a thorough overview of the evolution of this research area over the last ten years and beyond, starting from pioneering, earlier work on the security of non-deep learning algorithms up to more recent work aimed to understand the security properties of deep learning algorithms, in the context of computer vision and cybersecurity tasks. We report interesting connections between these apparently-different lines of work, highlighting common misconceptions related to the security evaluation of machine-learning algorithms. We review the main threat models and attacks defined to this end, and discuss the main limitations of current work, along with the corresponding future challenges towards the design of more secure learning algorithms.

在对抗机器学习中，防止对深度学习系统的攻击的新防御能力在释放更强大的攻击后不久就会破坏。在这种情况下，法医工具可以通过追溯成功的根本原因来为现有防御措施提供宝贵的补充，并为缓解措施提供前进的途径，以防止将来采取类似的攻击。在本文中，我们描述了我们为开发用于深度神经网络毒物攻击的法医追溯工具的努力。我们提出了一种新型的迭代聚类和修剪解决方案，该解决方案修剪了“无辜”训练样本，直到所有剩余的是一组造成攻击的中毒数据。我们的方法群群训练样本基于它们对模型参数的影响，然后使用有效的数据解读方法来修剪无辜簇。我们从经验上证明了系统对三种类型的肮脏标签（后门）毒物攻击和三种类型的清洁标签毒药攻击的功效，这些毒物跨越了计算机视觉和恶意软件分类。我们的系统在所有攻击中都达到了98.4％的精度和96.8％的召回。我们还表明，我们的系统与专门攻击它的四种抗纤维法措施相对强大。

有针对性的训练集攻击将恶意实例注入训练集中，以导致训练有素的模型错误地标记一个或多个特定的测试实例。这项工作提出了目标识别的任务，该任务决定了特定的测试实例是否是训练集攻击的目标。目标识别可以与对抗性识别相结合，以查找（并删除）攻击实例，从而减轻对其他预测的影响，从而减轻攻击。我们没有专注于单个攻击方法或数据模式，而是基于影响力估计，这量化了每个培训实例对模型预测的贡献。我们表明，现有的影响估计量的不良实际表现通常来自于他们对训练实例和迭代次数的过度依赖。我们重新归一化的影响估计器解决了这一弱点。他们的表现远远超过了原始估计量，可以在对抗和非对抗环境中识别有影响力的训练示例群体，甚至发现多达100％的对抗训练实例，没有清洁数据误报。然后，目标识别简化以检测具有异常影响值的测试实例。我们证明了我们的方法对各种数据域的后门和中毒攻击的有效性，包括文本，视觉和语音，以及针对灰色盒子的自适应攻击者，该攻击者专门优化了逃避我们方法的对抗性实例。我们的源代码可在https://github.com/zaydh/target_indistification中找到。

从外界培训的机器学习模型可能会被数据中毒攻击损坏，将恶意指向到模型的培训集中。对这些攻击的常见防御是数据消毒：在培训模型之前首先过滤出异常培训点。在本文中，我们开发了三次攻击，可以绕过广泛的常见数据消毒防御，包括基于最近邻居，训练损失和奇异值分解的异常探测器。通过增加3％的中毒数据，我们的攻击成功地将Enron垃圾邮件检测数据集的测试错误从3％增加到24％，并且IMDB情绪分类数据集从12％到29％。相比之下，没有明确占据这些数据消毒防御的现有攻击被他们击败。我们的攻击基于两个想法：（i）我们协调我们的攻击将中毒点彼此放置在彼此附近，（ii）我们将每个攻击制定为受限制的优化问题，限制旨在确保中毒点逃避检测。随着这种优化涉及解决昂贵的Bilevel问题，我们的三个攻击对应于基于影响功能的近似近似这个问题的方式; minimax二元性;和karush-kuhn-tucker（kkt）条件。我们的结果强调了对数据中毒攻击产生更强大的防御的必要性。

Neural networks provide state-of-the-art results for most machine learning tasks. Unfortunately, neural networks are vulnerable to adversarial examples: given an input x and any target classification t, it is possible to find a new input x that is similar to x but classified as t. This makes it difficult to apply neural networks in security-critical areas. Defensive distillation is a recently proposed approach that can take an arbitrary neural network, and increase its robustness, reducing the success rate of current attacks' ability to find adversarial examples from 95% to 0.5%.In this paper, we demonstrate that defensive distillation does not significantly increase the robustness of neural networks by introducing three new attack algorithms that are successful on both distilled and undistilled neural networks with 100% probability. Our attacks are tailored to three distance metrics used previously in the literature, and when compared to previous adversarial example generation algorithms, our attacks are often much more effective (and never worse). Furthermore, we propose using high-confidence adversarial examples in a simple transferability test we show can also be used to break defensive distillation. We hope our attacks will be used as a benchmark in future defense attempts to create neural networks that resist adversarial examples.

A recent line of work has uncovered a new form of data poisoning: so-called backdoor attacks. These attacks are particularly dangerous because they do not affect a network's behavior on typical, benign data. Rather, the network only deviates from its expected output when triggered by a perturbation planted by an adversary. In this paper, we identify a new property of all known backdoor attacks, which we call spectral signatures. This property allows us to utilize tools from robust statistics to thwart the attacks. We demonstrate the efficacy of these signatures in detecting and removing poisoned examples on real image sets and state of the art neural network architectures. We believe that understanding spectral signatures is a crucial first step towards designing ML systems secure against such backdoor attacks.

Although deep neural networks (DNNs) have achieved great success in many tasks, they can often be fooled by adversarial examples that are generated by adding small but purposeful distortions to natural examples. Previous studies to defend against adversarial examples mostly focused on refining the DNN models, but have either shown limited success or required expensive computation. We propose a new strategy, feature squeezing, that can be used to harden DNN models by detecting adversarial examples. Feature squeezing reduces the search space available to an adversary by coalescing samples that correspond to many different feature vectors in the original space into a single sample. By comparing a DNN model's prediction on the original input with that on squeezed inputs, feature squeezing detects adversarial examples with high accuracy and few false positives.This paper explores two feature squeezing methods: reducing the color bit depth of each pixel and spatial smoothing. These simple strategies are inexpensive and complementary to other defenses, and can be combined in a joint detection framework to achieve high detection rates against state-of-the-art attacks.

数据中毒是对机器学习和数据驱动技术的最相关的安全威胁之一。由于许多应用程序依赖于不受信任的培训数据，因此攻击者可以轻松地将恶意样本轻松地将其注入训练数据集，以降低机器学习模型的性能。正如最近的工作所示，这种拒绝服务（DOS）数据中毒攻击非常有效。为了减轻这种威胁，我们提出了一种检测DOS中毒实例的新方法。与相关工作相比，我们偏离基于聚类和异常检测的方法，这通常遭受维度的诅咒和任意异常阈值选择。相反，我们的防御是基于以这种广义的方式从训练数据中提取信息，使得我们可以基于存在于数据的未被占部分中存在的信息来识别中毒样本。我们评估我们对两个DOS中毒攻击和七个数据集的防御，并发现它可靠地识别中毒实例。与相关的工作相比，我们的防范将误报/假负率提高至少50％，通常更多。

Deep neural networks (DNNs) provide excellent performance across a wide range of classification tasks, but their training requires high computational resources and is often outsourced to third parties. Recent work has shown that outsourced training introduces the risk that a malicious trainer will return a backdoored DNN that behaves normally on most inputs but causes targeted misclassifications or degrades the accuracy of the network when a trigger known only to the attacker is present. In this paper, we provide the first effective defenses against backdoor attacks on DNNs. We implement three backdoor attacks from prior work and use them to investigate two promising defenses, pruning and fine-tuning. We show that neither, by itself, is sufficient to defend against sophisticated attackers. We then evaluate fine-pruning, a combination of pruning and fine-tuning, and show that it successfully weakens or even eliminates the backdoors, i.e., in some cases reducing the attack success rate to 0% with only a 0.4% drop in accuracy for clean (non-triggering) inputs. Our work provides the first step toward defenses against backdoor attacks in deep neural networks.

深度神经网络容易受到来自对抗性投入的攻击，并且最近，特洛伊木马误解或劫持模型的决定。我们通过探索有界抗逆性示例空间和生成的对抗网络内的自然输入空间来揭示有界面的对抗性实例 - 通用自然主义侵害贴片的兴趣类 - 我们呼叫TNT。现在，一个对手可以用一个自然主义的补丁来手臂自己，不太恶意，身体上可实现，高效 - 实现高攻击成功率和普遍性。 TNT是普遍的，因为在场景中的TNT中捕获的任何输入图像都将：i）误导网络（未确定的攻击）;或ii）迫使网络进行恶意决定（有针对性的攻击）。现在，有趣的是，一个对抗性补丁攻击者有可能发挥更大的控制水平 - 选择一个独立，自然的贴片的能力，与被限制为嘈杂的扰动的触发器 - 到目前为止只有可能与特洛伊木马攻击方法有可能干扰模型建设过程，以嵌入风险发现的后门;但是，仍然意识到在物理世界中部署的补丁。通过对大型视觉分类任务的广泛实验，想象成在其整个验证集50,000张图像中进行评估，我们展示了TNT的现实威胁和攻击的稳健性。我们展示了攻击的概括，以创建比现有最先进的方法实现更高攻击成功率的补丁。我们的结果表明，攻击对不同的视觉分类任务（CIFAR-10，GTSRB，PUBFIG）和多个最先进的深神经网络，如WieredEnet50，Inception-V3和VGG-16。

随着深度神经网络（DNNS）的进步在许多关键应用中表现出前所未有的性能水平，它们的攻击脆弱性仍然是一个悬而未决的问题。我们考虑在测试时间进行逃避攻击，以防止在受约束的环境中进行深入学习，其中需要满足特征之间的依赖性。这些情况可能自然出现在表格数据中，也可能是特定应用程序域中功能工程的结果，例如网络安全中的威胁检测。我们提出了一个普通的基于迭代梯度的框架，称为围栏，用于制定逃避攻击，考虑到约束域和应用要求的细节。我们将其应用于针对两个网络安全应用培训的前馈神经网络：网络流量僵尸网络分类和恶意域分类，以生成可行的对抗性示例。我们广泛评估了攻击的成功率和绩效，比较它们对几个基线的改进，并分析影响攻击成功率的因素，包括优化目标和数据失衡。我们表明，通过最少的努力（例如，生成12个其他网络连接），攻击者可以将模型的预测从恶意类更改为良性并逃避分类器。我们表明，在具有更高失衡的数据集上训练的模型更容易受到我们的围栏攻击。最后，我们证明了在受限领域进行对抗训练的潜力，以提高针对这些逃避攻击的模型弹性。

With rapid progress and significant successes in a wide spectrum of applications, deep learning is being applied in many safety-critical environments. However, deep neural networks have been recently found vulnerable to well-designed input samples, called adversarial examples. Adversarial perturbations are imperceptible to human but can easily fool deep neural networks in the testing/deploying stage. The vulnerability to adversarial examples becomes one of the major risks for applying deep neural networks in safety-critical environments. Therefore, attacks and defenses on adversarial examples draw great attention. In this paper, we review recent findings on adversarial examples for deep neural networks, summarize the methods for generating adversarial examples, and propose a taxonomy of these methods. Under the taxonomy, applications for adversarial examples are investigated. We further elaborate on countermeasures for adversarial examples. In addition, three major challenges in adversarial examples and the potential solutions are discussed.

The authors thank Nicholas Carlini (UC Berkeley) and Dimitris Tsipras (MIT) for feedback to improve the survey quality. We also acknowledge X. Huang (Uni. Liverpool), K. R. Reddy (IISC), E. Valle (UNICAMP), Y. Yoo (CLAIR) and others for providing pointers to make the survey more comprehensive.