Adversarial training is one of the most powerful methods to improve the robustness of pre-trained language models (PLMs). However, this approach is typically more expensive than traditional fine-tuning because of the necessity to generate adversarial examples via gradient descent. Delving into the optimization process of adversarial training, we find that robust connectivity patterns emerge in the early training phase (typically $0.15\sim0.3$ epochs), far before parameters converge. Inspired by this finding, we dig out robust early-bird tickets (i.e., subnetworks) to develop an efficient adversarial training method: (1) searching for robust tickets with structured sparsity in the early stage; (2) fine-tuning robust tickets in the remaining time. To extract the robust tickets as early as possible, we design a ticket convergence metric to automatically terminate the searching process. Experiments show that the proposed efficient adversarial training method can achieve up to $7\times \sim 13 \times$ training speedups while maintaining comparable or even better robustness compared to the most competitive state-of-the-art adversarial training methods.

Recent works on Lottery Ticket Hypothesis have shown that pre-trained language models (PLMs) contain smaller matching subnetworks(winning tickets) which are capable of reaching accuracy comparable to the original models. However, these tickets are proved to be notrobust to adversarial examples, and even worse than their PLM counterparts. To address this problem, we propose a novel method based on learning binary weight masks to identify robust tickets hidden in the original PLMs. Since the loss is not differentiable for the binary mask, we assign the hard concrete distribution to the masks and encourage their sparsity using a smoothing approximation of L0 regularization.Furthermore, we design an adversarial loss objective to guide the search for robust tickets and ensure that the tickets perform well bothin accuracy and robustness. Experimental results show the significant improvement of the proposed method over previous work on adversarial robustness evaluation.

深度神经网络近似高度复杂功能的能力是其成功的关键。但是，这种好处是以巨大的模型大小为代价的，这挑战了其在资源受限环境中的部署。修剪是一种用于限制此问题的有效技术，但通常以降低准确性和对抗性鲁棒性为代价。本文解决了这些缺点，并引入了Deadwooding，这是一种新型的全球修剪技术，它利用了Lagrangian双重方法来鼓励模型稀疏性，同时保持准确性并确保鲁棒性。所得模型显示出在鲁棒性和准确性度量方面的最先进研究大大优于最先进的模型。

已知深神经网络（DNN）容易受到对抗性攻击的影响，即对输入的不可察觉的扰动可以误导DNN在清洁图像上培训，以制造错误的预测。为了解决这一目标，对抗性训练是目前最有效的防御方法，通过增强速度设定的训练，在飞行中产生的对抗样本。有趣的是，我们首次发现，在随机初始化的网络中，在没有任何模型训练的随机初始化网络中，第一次发现具有天生稳健性，匹配或超越对抗训练网络的强大准确性的鲁棒准确性，表明对模型权重的对抗训练不是对抗性鲁棒性不可或缺。我们命名为强大的临时票故障票（RST），也是自然效率的那种。不同于流行的彩票假设，既不需要培训原始密集的网络也不需要训练。为了验证和理解这种迷人的发现，我们进一步开展了广泛的实验，以研究不同模型，数据集，稀疏模式和攻击下RST的存在性和性质，绘制关于DNNS鲁棒性与其初始化/过度分辨率之间的关系的洞察。此外，我们确定从同一随机初始化的密集网络绘制的不同稀疏比率的RST之间的差的对抗性转移性，并提出了一种随机切换不同RST之间的随机切换的随机性，作为基于顶部的新型防御方法第一次。我们相信我们对RST的调查结果已经开辟了一个新的视角，以研究模型稳健性并扩大彩票假设。

Robustness evaluation against adversarial examples has become increasingly important to unveil the trustworthiness of the prevailing deep models in natural language processing (NLP). However, in contrast to the computer vision domain where the first-order projected gradient descent (PGD) is used as the benchmark approach to generate adversarial examples for robustness evaluation, there lacks a principled first-order gradient-based robustness evaluation framework in NLP. The emerging optimization challenges lie in 1) the discrete nature of textual inputs together with the strong coupling between the perturbation location and the actual content, and 2) the additional constraint that the perturbed text should be fluent and achieve a low perplexity under a language model. These challenges make the development of PGD-like NLP attacks difficult. To bridge the gap, we propose TextGrad, a new attack generator using gradient-driven optimization, supporting high-accuracy and high-quality assessment of adversarial robustness in NLP. Specifically, we address the aforementioned challenges in a unified optimization framework. And we develop an effective convex relaxation method to co-optimize the continuously-relaxed site selection and perturbation variables and leverage an effective sampling method to establish an accurate mapping from the continuous optimization variables to the discrete textual perturbations. Moreover, as a first-order attack generation method, TextGrad can be baked into adversarial training to further improve the robustness of NLP models. Extensive experiments are provided to demonstrate the effectiveness of TextGrad not only in attack generation for robustness evaluation but also in adversarial defense.

Pre-trained language models achieve superior performance, but they are computationally expensive due to their large size. Techniques such as pruning and knowledge distillation (KD) have been developed to reduce their size and latency. In most structural pruning methods, the pruning units, such as attention heads and feed-forward hidden dimensions, only span a small model structure space and limit the structures that the pruning algorithm can explore. In this work, we propose Gradient-based Intra-attention pruning (GRAIN), which inspects fine intra-attention structures, and allows different heads to have different sizes. Intra-attention pruning greatly expands the searching space of model structures and yields highly heterogeneous structures. We further propose structure regularization to encourage generating more regular structures, which achieves higher speedups than heterogeneous ones. We also integrate KD into the pruning process with a gradient separation strategy to reduce the interference of KD with the pruning process. GRAIN is evaluated on a variety of tasks. Results show that it notably outperforms other methods at the same or similar model size. Even under extreme compression where only $3\%$ weights in transformers remain, the pruned model is still competitive.

预训练是在各种下游任务上转移学习的广泛采用的起点。对彩票假说（LTH）的最新研究表明，这种巨大的预训练模型可以用极稀疏的子网（又称匹配子网络）代替，而无需牺牲可传递性。但是，实际的安全 - 重要应用程序通常在标准转移之外提出了更具挑战性的要求，这也要求这些子网克服对抗性脆弱性。在本文中，我们制定了一个更严格的概念，双赢彩票，其中预训练模型的位置可以在各种下游任务上独立传输，以在两个标准下达到相同的标准和可靠的概括正如完整的预培训模型可以做到的那样，对抗性训练制度。我们全面检查了各种训练机制，发现强大的预训练倾向于制作出更少的双赢彩票，其性能优于标准对应物。例如，在下游CIFAR-10/100数据集上，我们识别出具有标准的，快速的对抗性和对抗性预训练的双赢匹配子网，以89.26％/73.79％，89.26％/79.03％和91.41％的匹配培训。 /83.22%稀疏。此外，我们观察到获得的双赢彩票票可以在实用数据限制（例如1％和10％）下游方案下传输的数据效率更高。我们的结果表明，彩票票务方案以及数据限制的转移设置可以扩大稳健的预训练的好处。代码可在https://github.com/vita-group/double-win-lth上找到。

修剪是一种众所周知的机制，用于降低深度卷积网络的计算成本。然而，研究表明，作为正规化形式修剪的可能性，这减少了过度拟合并改善了泛化。我们证明，这种战略系列提供了额外的益处，超出了计算绩效和泛化。我们的分析表明，来自卷积网络的修剪结构（滤波器和/或层）不仅增加了泛化，而且增加了对抗性图像的鲁棒性（具有内容修改的自然图像）。由于修剪降低了网络容量并提供了正规化，因此可以获得对抗对抗图像的有效工具。与需要对对抗性图像和仔细正规化的培训需要培训的有希望的防御机制，我们表明修剪仅考虑自然图像（例如，标准和低成本训练）。我们在几种对抗攻击和架构上确认这些结果;因此，暗示了作为对抗对抗性图像的新型防御机制修剪的潜力。

到目前为止对抗训练是抵御对抗例子的最有效的策略。然而，由于每个训练步骤中的迭代对抗性攻击，它遭受了高的计算成本。最近的研究表明，通过随机初始化执行单步攻击，可以实现快速的对抗训练。然而，这种方法仍然落后于稳定性和模型稳健性的最先进的对手训练算法。在这项工作中，我们通过观察随机平滑的随机初始化来更好地优化内部最大化问题，对快速对抗培训进行新的理解。在这种新的视角之后，我们还提出了一种新的初始化策略，向后平滑，进一步提高单步强大培训方法的稳定性和模型稳健性。多个基准测试的实验表明，我们的方法在使用更少的训练时间（使用相同的培训计划时，使用更少的培训时间（$ \ sim $ 3x改进）时，我们的方法达到了类似的模型稳健性。

We present VILLA, the first known effort on large-scale adversarial training for vision-and-language (V+L) representation learning. VILLA consists of two training stages: (i) task-agnostic adversarial pre-training; followed by (ii) task-specific adversarial finetuning. Instead of adding adversarial perturbations on image pixels and textual tokens, we propose to perform adversarial training in the embedding space of each modality. To enable large-scale training, we adopt the "free" adversarial training strategy, and combine it with KL-divergence-based regularization to promote higher invariance in the embedding space. We apply VILLA to current best-performing V+L models, and achieve new state of the art on a wide range of tasks, including Visual Question Answering, Visual Commonsense Reasoning, Image-Text Retrieval, Referring Expression Comprehension, Visual Entailment, and NLVR 2 . 1

AD相关建模在包括Microsoft Bing在内的在线广告系统中起着至关重要的作用。为了利用强大的变压器在这种低延迟设置中，许多现有方法脱机执行广告端计算。虽然有效，但这些方法无法提供冷启动广告，从而导致对此类广告的相关性预测不佳。这项工作旨在通过结构化修剪设计一种新的低延迟BERT，以在CPU平台上授权实时在线推断对Cold Start Ads相关性。我们的挑战是，以前的方法通常将变压器的所有层都缩减为高，均匀的稀疏性，从而产生无法以可接受的精度实现令人满意的推理速度的模型。在本文中，我们提出了SwiftPruner - 一个有效的框架，利用基于进化的搜索自动在所需的延迟约束下自动找到表现最佳的稀疏BERT模型。与进行随机突变的现有进化算法不同，我们提出了一个具有潜伏意见的多目标奖励的增强突变器，以进行更好的突变，以有效地搜索层稀疏模型的大空间。广泛的实验表明，与均匀的稀疏基线和最先进的搜索方法相比，我们的方法始终达到更高的ROC AUC和更低的潜伏度。值得注意的是，根据我们在1900年的延迟需求，SwiftPruner的AUC比Bert-Mini在大型现实世界数据集中的最先进的稀疏基线高0.86％。在线A/B测试表明，我们的模型还达到了有缺陷的冷启动广告的比例，并获得了令人满意的实时服务延迟。

巨大的预训练模型已成为自然语言处理（NLP）的核心，它是针对一系列下游任务进行微调的起点。然而，此范式的两个疼痛点持续：（a）随着预训练的模型的增长越大（例如，GPT-3的175b参数），即使是微调过程也可能是耗时的，并且计算昂贵； （b）默认情况下，微调模型的大小与起点相同，由于其更专业的功能，这既不明智，也不是实际的，因为许多微调模型将部署在资源受限的环境中。为了解决这些疼痛点，我们通过在重量更新和最终模型权重中利用稀疏性来提出一个用于资源和参数有效的微调的框架。我们提出的框架被称为双重稀疏性的有效调整（DSEE），旨在实现两个关键目标：（i）参数有效的微调 - 通过在预训练的权重的顶部强制实施稀疏性的低级更新； （ii）资源有效的推论 - 通过鼓励对最终微调模型的稀疏重量结构。我们通过统一的方法在预训练的语言模型中利用非结构化和结构化的稀疏模式来利用这两个方向的稀疏性。广泛的实验和深入研究，对数十个数据集进行了不同的网络骨干（即Bert，Roberta和GPT-2），始终显示出令人印象深刻的参数 - /推理效率，同时保持竞争性下游性能。例如，DSEE在达到可比性能的同时节省了约25％的推理拖失lo，在BERT上具有0.5％的可训练参数。代码可在https://github.com/vita-group/dsee中找到。

自然语言视频本地化（NLVL）是视觉语言理解区域的重要任务，该方面还要求深入了解单独的计算机视觉和自然语言侧，但更重要的是两侧之间的相互作用。对抗性脆弱性得到了很好的认可，作为深度神经网络模型的关键安全问题，需要谨慎调查。尽管在视频和语言任务中进行了广泛但分开的研究，但目前对NLVL等愿景联合任务的对抗鲁棒性的理解较少。因此，本文旨在通过检查攻击和防御方面的三个脆弱性，全面调查NLVL模型的对抗性鲁棒性。为了实现攻击目标，我们提出了一种新的对抗攻击范式，称为同义句子感知对抗对抗攻击对逆向（潜行），这捕获了视觉和语言侧面之间的跨模式相互作用。

Adversarial training is widely acknowledged as the most effective defense against adversarial attacks. However, it is also well established that achieving both robustness and generalization in adversarially trained models involves a trade-off. The goal of this work is to provide an in depth comparison of different approaches for adversarial training in language models. Specifically, we study the effect of pre-training data augmentation as well as training time input perturbations vs. embedding space perturbations on the robustness and generalization of BERT-like language models. Our findings suggest that better robustness can be achieved by pre-training data augmentation or by training with input space perturbation. However, training with embedding space perturbation significantly improves generalization. A linguistic correlation analysis of neurons of the learned models reveal that the improved generalization is due to `more specialized' neurons. To the best of our knowledge, this is the first work to carry out a deep qualitative analysis of different methods of generating adversarial examples in adversarial training of language models.

我们将自然语言处理模型的脆弱性归因于以下事实：类似的输入转换为嵌入空间中不同的表示形式，导致输出不一致，我们提出了一种新颖的强大训练方法，称为快速三胞胎度量度量学习（FTML）。具体而言，我们认为原始样本应具有相似的表示及其对手对应物，并将其代表与其他样品区分开，以提高鲁棒性。为此，我们将三胞胎度量学习采用标准培训中，以将单词更接近其正样本（即同义词），并在嵌入空间中推出其负面样本（即非综合样品）。广泛的实验表明，FTML可以显着促进模型的鲁棒性，以针对各种高级对抗攻击，同时保持对原始样品的竞争性分类精度。此外，我们的方法是有效的，因为它只需要调整嵌入方式，并且在标准培训上引入了很少的开销。我们的工作显示出通过稳健的单词嵌入来改善文本鲁棒性的巨大潜力。

在本文中，我们询问视觉变形金刚（VIT）是否可以作为改善机器学习模型对抗逃避攻击的对抗性鲁棒性的基础结构。尽管较早的作品集中在改善卷积神经网络上，但我们表明VIT也非常适合对抗训练以实现竞争性能。我们使用自定义的对抗训练配方实现了这一目标，该配方是在Imagenet数据集的一部分上使用严格的消融研究发现的。与卷积相比，VIT的规范培训配方建议强大的数据增强，部分是为了补偿注意力模块的视力归纳偏置。我们表明，该食谱在用于对抗训练时可实现次优性能。相比之下，我们发现省略所有重型数据增强，并添加一些额外的零件（$ \ varepsilon $ -Warmup和更大的重量衰减），从而大大提高了健壮的Vits的性能。我们表明，我们的配方在完整的Imagenet-1k上概括了不同类别的VIT体系结构和大规模模型。此外，调查了模型鲁棒性的原因，我们表明，在使用我们的食谱时，在训练过程中产生强烈的攻击更加容易，这会在测试时提高鲁棒性。最后，我们通过提出一种量化对抗性扰动的语义性质并强调其与模型的鲁棒性的相关性来进一步研究对抗训练的结果。总体而言，我们建议社区应避免将VIT的规范培训食谱转换为在对抗培训的背景下进行强大的培训和重新思考常见的培训选择。

近年来，深神经网络（DNN）应用的流行和成功促使对DNN压缩的研究，例如修剪和量化。这些技术加速了模型推断，减少功耗，并降低运行DNN所需的硬件的大小和复杂性，而准确性几乎没有损失。但是，由于DNN容易受到对抗输入的影响，因此重要的是要考虑压缩和对抗性鲁棒性之间的关系。在这项工作中，我们研究了几种不规则修剪方案和8位量化产生的模型的对抗性鲁棒性。此外，尽管常规修剪消除了DNN中最不重要的参数，但我们研究了一种非常规修剪方法的效果：根据对抗输入的梯度去除最重要的模型参数。我们称这种方法称贪婪的对抗修剪（GAP），我们发现这种修剪方法会导致模型可抵抗从其未压缩的对应物转移攻击的模型。

Model compression and model defense for deep neural networks (DNNs) have been extensively and individually studied. Considering the co-importance of model compactness and robustness in practical applications, several prior works have explored to improve the adversarial robustness of the sparse neural networks. However, the structured sparse models obtained by the exiting works suffer severe performance degradation for both benign and robust accuracy, thereby causing a challenging dilemma between robustness and structuredness of the compact DNNs. To address this problem, in this paper, we propose CSTAR, an efficient solution that can simultaneously impose the low-rankness-based Compactness, high STructuredness and high Adversarial Robustness on the target DNN models. By formulating the low-rankness and robustness requirement within the same framework and globally determining the ranks, the compressed DNNs can simultaneously achieve high compression performance and strong adversarial robustness. Evaluations for various DNN models on different datasets demonstrate the effectiveness of CSTAR. Compared with the state-of-the-art robust structured pruning methods, CSTAR shows consistently better performance. For instance, when compressing ResNet-18 on CIFAR-10, CSTAR can achieve up to 20.07% and 11.91% improvement for benign accuracy and robust accuracy, respectively. For compressing ResNet-18 with 16x compression ratio on Imagenet, CSTAR can obtain 8.58% benign accuracy gain and 4.27% robust accuracy gain compared to the existing robust structured pruning method.

现有的研究表明，对抗性示例可以直接归因于具有高度预测性的非稳态特征的存在，但很容易被对手对愚弄NLP模型进行操纵。在这项研究中，我们探讨了捕获特定于任务的鲁棒特征的可行性，同时使用信息瓶颈理论消除了非舒适的特征。通过广泛的实验，我们表明，通过我们的信息基于瓶颈的方法训练的模型能够在稳健的精度上取得显着提高，超过了所有先前报道的防御方法的性能，而在SST-2上几乎没有遭受清洁准确性的表现下降，Agnews和IMDB数据集。

Despite achieving state-of-the-art performance on many NLP tasks, the high energy cost and long inference delay prevent Transformer-based pretrained language models (PLMs) from seeing broader adoption including for edge and mobile computing. Efficient NLP research aims to comprehensively consider computation, time and carbon emission for the entire life-cycle of NLP, including data preparation, model training and inference. In this survey, we focus on the inference stage and review the current state of model compression and acceleration for pretrained language models, including benchmarks, metrics and methodology.