Recent studies have shown the impressive efficacy of counterfactually augmented data (CAD) for reducing NLU models' reliance on spurious features and improving their generalizability. However, current methods still heavily rely on human efforts or task-specific designs to generate counterfactuals, thereby impeding CAD's applicability to a broad range of NLU tasks. In this paper, we present AutoCAD, a fully automatic and task-agnostic CAD generation framework. AutoCAD first leverages a classifier to unsupervisedly identify rationales as spans to be intervened, which disentangles spurious and causal features. Then, AutoCAD performs controllable generation enhanced by unlikelihood training to produce diverse counterfactuals. Extensive evaluations on multiple out-of-domain and challenge benchmarks demonstrate that AutoCAD consistently and significantly boosts the out-of-distribution performance of powerful pre-trained models across different NLU tasks, which is comparable or even better than previous state-of-the-art human-in-the-loop or task-specific CAD methods. The code is publicly available at https://github.com/thu-coai/AutoCAD.

几乎没有命名的实体识别（NER）对于在有限的资源领域中标记的实体标记至关重要，因此近年来受到了适当的关注。现有的几声方法主要在域内设置下进行评估。相比之下，对于这些固有的忠实模型如何使用一些标记的域内示例在跨域NER中执行的方式知之甚少。本文提出了一种两步以理性为中心的数据增强方法，以提高模型的泛化能力。几个数据集中的结果表明，与先前的最新方法相比，我们的模型无形方法可显着提高跨域NER任务的性能，包括反事实数据增强和及时调用方法。我们的代码可在\ url {https://github.com/lifan-yuan/factmix}上获得。

Recent methods demonstrate that data augmentation using counterfactual knowledge can teach models the causal structure of a task, leading to robust and generalizable models. However, such counterfactual data often has a limited scale and diversity if crowdsourced and is computationally expensive to extend to new perturbation types if generated using supervised methods. To address this, we introduce a new framework called DISCO for automatically generating high-quality counterfactual data at scale. DISCO engineers prompts to generate phrasal perturbations with a large general language model. Then, a task-specific teacher model filters the generation to distill high-quality counterfactual data. We show that learning with this counterfactual data yields a comparatively small student model that is 6% (absolute) more robust and generalizes 5% better across distributions than baselines on various challenging evaluations. This model is also 15% more sensitive in differentiating original and counterfactual examples, on three evaluation sets written by human workers and via human-AI collaboration.

我们提出了两种小型无监督方法，用于消除文本中的毒性。我们的第一个方法结合了最近的两个想法：（1）使用小型条件语言模型的生成过程的指导和（2）使用释义模型进行风格传输。我们使用良好的令人措辞的令人愉快的释放器，由风格培训的语言模型引导，以保持文本内容并消除毒性。我们的第二种方法使用BERT用他们的非攻击性同义词取代毒性单词。我们通过使BERT替换具有可变数量的单词的屏蔽令牌来使该方法更灵活。最后，我们介绍了毒性去除任务的风格转移模型的第一个大规模比较研究。我们将模型与许多用于样式传输的方法进行比较。使用无监督的样式传输指标的组合以可参考方式评估该模型。两种方法都建议产生新的SOTA结果。

生成反事实测试箱是测试NLP模型并使其像传统软件一样坚固且可靠的重要主体。在生成测试箱时，所需的特性是能够以灵活的方式控制测试案例生成以测试各种故障案例并以目标方式解释和修复它们。在这个方向上，通过手动编写生成受控反事实的规则，在先前的作品中取得了重大进展。但是，这种方法需要大量的手动监督，并且缺乏轻松引入新控件的灵活性。由PPLM的插件方法令人印象深刻的灵活性的激励，我们建议将插件的框架带入反事实测试案例生成任务。我们介绍了Casper，这是一种插件的反事实生成框架，以生成满足需求目标属性的测试用例。我们的插件模型可以在给定任何属性模型的情况下引导测试案例生成过程，而无需对模型的属性特定培训。在实验中，我们表明Casper有效地生成了反事实文本，该文本遵循属性模型提供的转向，同时流利，多样化并保留原始内容。我们还表明，CASPER的生成的反事实可用于增强训练数据，从而固定并使测试模型更加可靠。

Controllable Text Generation (CTG) is emerging area in the field of natural language generation (NLG). It is regarded as crucial for the development of advanced text generation technologies that are more natural and better meet the specific constraints in practical applications. In recent years, methods using large-scale pre-trained language models (PLMs), in particular the widely used transformer-based PLMs, have become a new paradigm of NLG, allowing generation of more diverse and fluent text. However, due to the lower level of interpretability of deep neural networks, the controllability of these methods need to be guaranteed. To this end, controllable text generation using transformer-based PLMs has become a rapidly growing yet challenging new research hotspot. A diverse range of approaches have emerged in the recent 3-4 years, targeting different CTG tasks which may require different types of controlled constraints. In this paper, we present a systematic critical review on the common tasks, main approaches and evaluation methods in this area. Finally, we discuss the challenges that the field is facing, and put forward various promising future directions. To the best of our knowledge, this is the first survey paper to summarize CTG techniques from the perspective of PLMs. We hope it can help researchers in related fields to quickly track the academic frontier, providing them with a landscape of the area and a roadmap for future research.

数据增强是通过转换为机器学习的人工创建数据的人工创建，是一个跨机器学习学科的研究领域。尽管它对于增加模型的概括功能很有用，但它还可以解决许多其他挑战和问题，从克服有限的培训数据到正规化目标到限制用于保护隐私的数据的数量。基于对数据扩展的目标和应用的精确描述以及现有作品的分类法，该调查涉及用于文本分类的数据增强方法，并旨在为研究人员和从业者提供简洁而全面的概述。我们将100多种方法划分为12种不同的分组，并提供最先进的参考文献来阐述哪种方法可以通过将它们相互关联，从而阐述了哪种方法。最后，提供可能构成未来工作的基础的研究观点。

基于方面的情感分析（ABSA）是一项精细的情感分析任务，它的重点是检测句子中的情感极性。但是，它始终对多方面的挑战敏感，在句子中，多个方面的特征将相互影响。为了减轻此问题，我们设计了一个新颖的培训框架，称为对比度跨通道数据增强（C3 DA），该框架利用了一个内域的发电机来构建更多的多种相应样本，然后通过对比度模型通过对比度学习的稳健性，从而通过对比度学习的稳健性这些生成的数据。实际上，鉴于生成预审预测的语言模型和一些有限的ABSA标记数据，我们首先采用一些参数效率的方法来执行内域微调。然后，所获得的内域发生器用于从两个通道（即方面增强通道和极性增强通道）生成合成句子，该句子分别在给定的方面和极性上生成句子条件。具体而言，我们的C3 DA以跨渠道的方式执行句子生成以获取更多句子，并提出了熵最小化过滤器以滤除低质量生成的样品。广泛的实验表明，我们的C3 DA可以在准确性和宏观上胜过约1％的基准，而不会增加1％。代码和数据在https://github.com/wangbing1416/c3da中发布。

用于预培训语言模型的自我监督学习的核心包括预训练任务设计以及适当的数据增强。语言模型中的大多数数据增强都是独立于上下文的。最近在电子中提出了一个开创性的增强，并通过引入辅助生成网络（发电机）来实现最先进的性能，以产生用于培训主要辨别网络（鉴别者）的上下文化数据增强。然而，这种设计引入了发电机的额外计算成本，并且需要调整发电机和鉴别器之间的相对能力。在本文中，我们提出了一种自增强策略（SAS），其中单个网络用于审视以后的时期的培训常规预训练和上下文化数据增强。基本上，该策略消除了单独的发电机，并使用单个网络共同执行具有MLM（屏蔽语言建模）和RTD（替换令牌检测）头的两个预训练任务。它避免了寻找适当大小的发电机的挑战，这对于在电子中证明的性能至关重要，以及其随后的变体模型至关重要。此外，SAS是一项常规策略，可以与最近或将来的许多新技术无缝地结合，例如杜伯塔省的解除关注机制。我们的实验表明，SAS能够在具有相似或更少的计算成本中优于胶水任务中的电磁和其他最先进的模型。

Existing reference-free metrics have obvious limitations for evaluating controlled text generation models. Unsupervised metrics can only provide a task-agnostic evaluation result which correlates weakly with human judgments, whereas supervised ones may overfit task-specific data with poor generalization ability to other datasets. In this paper, we propose an unsupervised reference-free metric called CTRLEval, which evaluates controlled text generation from different aspects by formulating each aspect into multiple text infilling tasks. On top of these tasks, the metric assembles the generation probabilities from a pre-trained language model without any model training. Experimental results show that our metric has higher correlations with human judgments than other baselines, while obtaining better generalization of evaluating generated texts from different models and with different qualities.

最近的作品表明了解释性和鲁棒性是值得信赖和可靠的文本分类的两个关键成分。然而，以前的作品通常是解决了两个方面的一个：i）如何提取准确的理由，以便在有利于预测的同时解释; ii）如何使预测模型对不同类型的对抗性攻击稳健。直观地，一种产生有用的解释的模型应该对对抗性攻击更加强大，因为我们无法信任输出解释的模型，而是在小扰动下改变其预测。为此，我们提出了一个名为-BMC的联合分类和理由提取模型。它包括两个关键机制：混合的对手训练（AT）旨在在离散和嵌入空间中使用各种扰动，以改善模型的鲁棒性，边界匹配约束（BMC）有助于利用边界信息的引导来定位理由。基准数据集的性能表明，所提出的AT-BMC优于分类和基本原子的基础，由大边距提取。鲁棒性分析表明，建议的AT-BMC将攻击成功率降低了高达69％。经验结果表明，强大的模型与更好的解释之间存在连接。

Natural Language Generation (NLG) has improved exponentially in recent years thanks to the development of sequence-to-sequence deep learning technologies such as Transformer-based language models. This advancement has led to more fluent and coherent NLG, leading to improved development in downstream tasks such as abstractive summarization, dialogue generation and data-to-text generation. However, it is also apparent that deep learning based generation is prone to hallucinate unintended text, which degrades the system performance and fails to meet user expectations in many real-world scenarios. To address this issue, many studies have been presented in measuring and mitigating hallucinated texts, but these have never been reviewed in a comprehensive manner before. In this survey, we thus provide a broad overview of the research progress and challenges in the hallucination problem in NLG. The survey is organized into two parts: (1) a general overview of metrics, mitigation methods, and future directions; and (2) an overview of task-specific research progress on hallucinations in the following downstream tasks, namely abstractive summarization, dialogue generation, generative question answering, data-to-text generation, machine translation, and visual-language generation. This survey serves to facilitate collaborative efforts among researchers in tackling the challenge of hallucinated texts in NLG.

Masked language modeling (MLM) pre-training methods such as BERT corrupt the input by replacing some tokens with [MASK] and then train a model to reconstruct the original tokens. While they produce good results when transferred to downstream NLP tasks, they generally require large amounts of compute to be effective. As an alternative, we propose a more sample-efficient pre-training task called replaced token detection. Instead of masking the input, our approach corrupts it by replacing some tokens with plausible alternatives sampled from a small generator network. Then, instead of training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that predicts whether each token in the corrupted input was replaced by a generator sample or not. Thorough experiments demonstrate this new pre-training task is more efficient than MLM because the task is defined over all input tokens rather than just the small subset that was masked out. As a result, the contextual representations learned by our approach substantially outperform the ones learned by BERT given the same model size, data, and compute. The gains are particularly strong for small models; for example, we train a model on one GPU for 4 days that outperforms GPT (trained using 30x more compute) on the GLUE natural language understanding benchmark. Our approach also works well at scale, where it performs comparably to RoBERTa and XLNet while using less than 1/4 of their compute and outperforms them when using the same amount of compute.

Recent studies have revealed the intriguing few-shot learning ability of pretrained language models (PLMs): They can quickly adapt to a new task when fine-tuned on a small amount of labeled data formulated as prompts, without requiring abundant task-specific annotations. Despite their promising performance, most existing few-shot approaches that only learn from the small training set still underperform fully supervised training by nontrivial margins. In this work, we study few-shot learning with PLMs from a different perspective: We first tune an autoregressive PLM on the few-shot samples and then use it as a generator to synthesize a large amount of novel training samples which augment the original training set. To encourage the generator to produce label-discriminative samples, we train it via weighted maximum likelihood where the weight of each token is automatically adjusted based on a discriminative meta-learning objective. A classification PLM can then be fine-tuned on both the few-shot and the synthetic samples with regularization for better generalization and stability. Our approach FewGen achieves an overall better result across seven classification tasks of the GLUE benchmark than existing few-shot learning methods, improving no-augmentation methods by 5+ average points, and outperforming augmentation methods by 3+ average points.

GPT-3等大型语言模型是优秀的几次学习者，允许他们通过自然文本提示来控制。最近的研究报告称，基于及时的直接分类消除了对微调的需求，但缺乏数据和推理可扩展性。本文提出了一种新的数据增强技术，利用大规模语言模型来生成来自真实样本的混合的现实文本样本。我们还建议利用语言模型预测的软标签，从大规模语言模型中有效地蒸馏知识并同时创建文本扰动。我们对各种分类任务进行数据增强实验，并显示我们的方法非常优于现有的文本增强方法。消融研究和定性分析为我们的方法提供了更多的见解。

由于NLP模型实现了基准测试的最先进的性能并获得了广泛的应用程序，因此确保在现实世界中的安全部署这些模型的安全部署，例如，确保模型对未经调用或具有挑战性的情景稳健。尽管具有越来越多的学习主题，但它在视觉和NLP等应用中分别探讨了，具有多种研究中的各种定义，评估和缓解策略。在本文中，我们的目标是提供对如何定义，测量和提高NLP鲁棒性的统一调查。我们首先连接多种稳健性的定义，然后统一各种各样的工作方面识别稳健性失败和评估模型的鲁棒性。相应地，我们呈现了数据驱动，模型驱动和基于归纳的缓解策略，具有如何有效地改善NLP模型中的鲁棒性的更系统的观点。最后，我们通过概述开放的挑战和未来方向来促进在这一领域的进一步研究。

The recent GPT-3 model (Brown et al., 2020) achieves remarkable few-shot performance solely by leveraging a natural-language prompt and a few task demonstrations as input context. Inspired by their findings, we study few-shot learning in a more practical scenario, where we use smaller language models for which fine-tuning is computationally efficient. We present LM-BFF-better few-shot fine-tuning of language models 1 -a suite of simple and complementary techniques for finetuning language models on a small number of annotated examples. Our approach includes (1) prompt-based fine-tuning together with a novel pipeline for automating prompt generation; and (2) a refined strategy for dynamically and selectively incorporating demonstrations into each context. Finally, we present a systematic evaluation for analyzing few-shot performance on a range of NLP tasks, including classification and regression. Our experiments demonstrate that our methods combine to dramatically outperform standard fine-tuning procedures in this low resource setting, achieving up to 30% absolute improvement, and 11% on average across all tasks. Our approach makes minimal assumptions on task resources and domain expertise, and hence constitutes a strong task-agnostic method for few-shot learning. 2 * The first two authors contributed equally. 1 Alternatively, language models' best friends forever. 2 Our implementation is publicly available at https:// github.com/princeton-nlp/LM-BFF.

文本样式传输是自然语言生成中的重要任务，旨在控制生成的文本中的某些属性，例如礼貌，情感，幽默和许多其他特性。它在自然语言处理领域拥有悠久的历史，最近由于深神经模型带来的有希望的性能而重大关注。在本文中，我们对神经文本转移的研究进行了系统调查，自2017年首次神经文本转移工作以来跨越100多个代表文章。我们讨论了任务制定，现有数据集和子任务，评估，以及丰富的方法在存在并行和非平行数据存在下。我们还提供关于这项任务未来发展的各种重要主题的讨论。我们的策据纸张列表在https://github.com/zhijing-jin/text_style_transfer_survey

Data Augmentation (DA) is frequently used to automatically provide additional training data without extra human annotation. However, data augmentation may introduce noisy data that impairs training. To guarantee the quality of augmented data, existing methods either assume no noise exists in the augmented data and adopt consistency training or use simple heuristics such as training loss and diversity constraints to filter out ``noisy'' data. However, those filtered examples may still contain useful information, and dropping them completely causes loss of supervision signals. In this paper, based on the assumption that the original dataset is cleaner than the augmented data, we propose an on-the-fly denoising technique for data augmentation that learns from soft augmented labels provided by an organic teacher model trained on the cleaner original data. A simple self-regularization module is applied to force the model prediction to be consistent across two distinct dropouts to further prevent overfitting on noisy labels. Our method can be applied to augmentation techniques in general and can consistently improve the performance on both text classification and question-answering tasks.

非平行文本样式转移是自然语言生成的重要任务。但是，先前的研究集中在令牌或句子级别上，例如句子情绪和形式转移，但在话语水平上忽略了长时间的转移。长文本通常涉及更复杂的作者语言偏好，例如话语结构，而不是句子。在本文中，我们制定了非并行故事作者风格转移的任务，该任务需要将输入故事传输到指定的作者样式的同时，同时维护源语义。为了解决这个问题，我们提出了一个名为StoryTrans的一代模型，该模型利用话语表示捕获源内容信息并将其传输到具有可学习样式嵌入的目标样式中。我们使用额外的培训目标将文学的文学特征与学习的话语表示，以防止模型退化为自动编码器。此外，为了增强内容保存，我们设计了一个面具和填充框架，以将源文本的特定于特定于样式的关键字定为生成。此外，我们分别用中文和英语构建了此任务的新数据集。广泛的实验表明，我们的模型在样式传输和内容保存的总体性能方面优于强大的基线。