常规作品通常采用两阶段模型，其中生成器选择最重要的部分，然后是根据所选零件进行预测的预测因子。但是，这样的两相模型可能会引起变性问题，其中预测变量过度适合尚未训练的发电机生成的噪声，然后导致发电机收敛到倾向于选择无意义的碎片的亚最佳模型。为了应对这一挑战，我们提出了折叠的合理化（FR），将理由模型的两个阶段折叠成一个文本语义提取的角度。FR的关键思想是在发电机和预测器之间采用统一的编码器，基于FR可以通过访问传统两相模型中发电机阻止的有价值的信息来促进更好的预测指标，从而带来更好的生成器。从经验上讲，我们表明，与最先进的方法相比，FR将F1得分提高了10.3％。

Recent studies have shown the impressive efficacy of counterfactually augmented data (CAD) for reducing NLU models' reliance on spurious features and improving their generalizability. However, current methods still heavily rely on human efforts or task-specific designs to generate counterfactuals, thereby impeding CAD's applicability to a broad range of NLU tasks. In this paper, we present AutoCAD, a fully automatic and task-agnostic CAD generation framework. AutoCAD first leverages a classifier to unsupervisedly identify rationales as spans to be intervened, which disentangles spurious and causal features. Then, AutoCAD performs controllable generation enhanced by unlikelihood training to produce diverse counterfactuals. Extensive evaluations on multiple out-of-domain and challenge benchmarks demonstrate that AutoCAD consistently and significantly boosts the out-of-distribution performance of powerful pre-trained models across different NLU tasks, which is comparable or even better than previous state-of-the-art human-in-the-loop or task-specific CAD methods. The code is publicly available at https://github.com/thu-coai/AutoCAD.

基于方面的情绪分析（ABSA）任务由三个典型的子特点组成：术语术语提取，意见术语提取和情感极性分类。这三个子组织通常是共同执行的，以节省资源并减少管道中的错误传播。但是，大多数现有联合模型只关注编码器共享的福利在子任务之间共享，但忽略差异。因此，我们提出了一个关节ABSA模型，它不仅享有编码器共享的好处，而且还专注于提高模型效率的差异。详细地，我们介绍了双编码器设计，其中一对编码器特别侧重于候选方识对分类，并且原始编码器对序列标记进行注意。经验结果表明，我们的拟议模型显示了鲁棒性，并显着优于前一个基准数据集的先前最先进。

Nowadays, time-stamped web documents related to a general news query floods spread throughout the Internet, and timeline summarization targets concisely summarizing the evolution trajectory of events along the timeline. Unlike traditional document summarization, timeline summarization needs to model the time series information of the input events and summarize important events in chronological order. To tackle this challenge, in this paper, we propose a Unified Timeline Summarizer (UTS) that can generate abstractive and extractive timeline summaries in time order. Concretely, in the encoder part, we propose a graph-based event encoder that relates multiple events according to their content dependency and learns a global representation of each event. In the decoder part, to ensure the chronological order of the abstractive summary, we propose to extract the feature of event-level attention in its generation process with sequential information remained and use it to simulate the evolutionary attention of the ground truth summary. The event-level attention can also be used to assist in extracting summary, where the extracted summary also comes in time sequence. We augment the previous Chinese large-scale timeline summarization dataset and collect a new English timeline dataset. Extensive experiments conducted on these datasets and on the out-of-domain Timeline 17 dataset show that UTS achieves state-of-the-art performance in terms of both automatic and human evaluations.

数据增强是通过转换为机器学习的人工创建数据的人工创建，是一个跨机器学习学科的研究领域。尽管它对于增加模型的概括功能很有用，但它还可以解决许多其他挑战和问题，从克服有限的培训数据到正规化目标到限制用于保护隐私的数据的数量。基于对数据扩展的目标和应用的精确描述以及现有作品的分类法，该调查涉及用于文本分类的数据增强方法，并旨在为研究人员和从业者提供简洁而全面的概述。我们将100多种方法划分为12种不同的分组，并提供最先进的参考文献来阐述哪种方法可以通过将它们相互关联，从而阐述了哪种方法。最后，提供可能构成未来工作的基础的研究观点。

近年来，人们对开发自然语言处理（NLP）中可解释模型的利益越来越多。大多数现有模型旨在识别输入功能，例如对于模型预测而言重要的单词或短语。然而，在NLP中开发的神经模型通常以层次结构的方式构成单词语义，文本分类需要层次建模来汇总本地信息，以便处理主题和标签更有效地转移。因此，单词或短语的解释不能忠实地解释文本分类中的模型决策。本文提出了一种新型的层次解释性神经文本分类器，称为提示，该分类器可以自动以层次结构方式以标记相关主题的形式生成模型预测的解释。模型解释不再处于单词级别，而是基于主题作为基本语义单元。评论数据集和新闻数据集的实验结果表明，我们所提出的方法与现有最新的文本分类器相当地达到文本分类结果，并比其他可解释的神经文本更忠实于模型的预测和更好地理解人类的解释分类器。

本文对过去二十年来对自然语言生成（NLG）的研究提供了全面的审查，特别是与数据到文本生成和文本到文本生成深度学习方法有关，以及NLG的新应用技术。该调查旨在（a）给出关于NLG核心任务的最新综合，以及该领域采用的建筑;（b）详细介绍各种NLG任务和数据集，并提请注意NLG评估中的挑战，专注于不同的评估方法及其关系;（c）强调一些未来的强调和相对近期的研究问题，因为NLG和其他人工智能领域的协同作用而增加，例如计算机视觉，文本和计算创造力。

多任务学习是一个框架，可执行多个学习任务以共享知识以提高其概括能力。虽然浅做多任务学习可以学习任务关系，但它只能处理预定义的功能。现代深度多任务学习可以共同学习潜在的功能和任务共享，但任务关系却很晦涩。同样，他们预先定义哪些层和神经元应该跨任务共享，并且不能适应地学习。为了应对这些挑战，本文提出了一个新的多任务学习框架，该框架通过补充现有浅层和深层多任务学习方案的强度，共同学习潜在特征和明确的任务关系。具体而言，我们建议将任务关系建模为任务输入梯度之间的相似性，并对它们的等效性进行理论分析。此外，我们创新地提出了一个多任务学习目标，该目标可以通过新的正规机明确学习任务关系。理论分析表明，由于提出的正常化程序，概括性误差已减少。在多个多任务学习和图像分类基准上进行的广泛实验证明了所提出的方法有效性，效率以及在学习任务关系模式中的合理性。

情绪分析中最突出的任务是为文本分配情绪，并了解情绪如何在语言中表现出来。自然语言处理的一个重要观察结果是，即使没有明确提及情感名称，也可以通过单独参考事件来隐式传达情绪。在心理学中，被称为评估理论的情感理论类别旨在解释事件与情感之间的联系。评估可以被形式化为变量，通过他们认为相关的事件的人们的认知评估来衡量认知评估。其中包括评估事件是否是新颖的，如果该人认为自己负责，是否与自己的目标以及许多其他人保持一致。这样的评估解释了哪些情绪是基于事件开发的，例如，新颖的情况会引起惊喜或不确定后果的人可能引起恐惧。我们在文本中分析了评估理论对情绪分析的适用性，目的是理解注释者是否可以可靠地重建评估概念，如果可以通过文本分类器预测，以及评估概念是否有助于识别情感类别。为了实现这一目标，我们通过要求人们发短信描述触发特定情绪并披露其评估的事件来编译语料库。然后，我们要求读者重建文本中的情感和评估。这种设置使我们能够衡量是否可以纯粹从文本中恢复情绪和评估，并为判断模型的绩效指标提供人体基准。我们将文本分类方法与人类注释者的比较表明，两者都可以可靠地检测出具有相似性能的情绪和评估。我们进一步表明，评估概念改善了文本中情绪的分类。

Recent work pre-training Transformers with self-supervised objectives on large text corpora has shown great success when fine-tuned on downstream NLP tasks including text summarization. However, pre-training objectives tailored for abstractive text summarization have not been explored. Furthermore there is a lack of systematic evaluation across diverse domains. In this work, we propose pre-training large Transformer-based encoder-decoder models on massive text corpora with a new selfsupervised objective. In PEGASUS, important sentences are removed/masked from an input document and are generated together as one output sequence from the remaining sentences, similar to an extractive summary. We evaluated our best PEGASUS model on 12 downstream summarization tasks spanning news, science, stories, instructions, emails, patents, and legislative bills. Experiments demonstrate it achieves state-of-the-art performance on all 12 downstream datasets measured by ROUGE scores. Our model also shows surprising performance on low-resource summarization, surpassing previous state-of-the-art results on 6 datasets with only 1000 examples. Finally we validated our results using human evaluation and show that our model summaries achieve human performance on multiple datasets.

As an important fine-grained sentiment analysis problem, aspect-based sentiment analysis (ABSA), aiming to analyze and understand people's opinions at the aspect level, has been attracting considerable interest in the last decade. To handle ABSA in different scenarios, various tasks are introduced for analyzing different sentiment elements and their relations, including the aspect term, aspect category, opinion term, and sentiment polarity. Unlike early ABSA works focusing on a single sentiment element, many compound ABSA tasks involving multiple elements have been studied in recent years for capturing more complete aspect-level sentiment information. However, a systematic review of various ABSA tasks and their corresponding solutions is still lacking, which we aim to fill in this survey. More specifically, we provide a new taxonomy for ABSA which organizes existing studies from the axes of concerned sentiment elements, with an emphasis on recent advances of compound ABSA tasks. From the perspective of solutions, we summarize the utilization of pre-trained language models for ABSA, which improved the performance of ABSA to a new stage. Besides, techniques for building more practical ABSA systems in cross-domain/lingual scenarios are discussed. Finally, we review some emerging topics and discuss some open challenges to outlook potential future directions of ABSA.

众所周知，端到端的神经NLP体系结构很难理解，这引起了近年来为解释性建模的许多努力。模型解释的基本原则是忠诚，即，解释应准确地代表模型预测背后的推理过程。这项调查首先讨论了忠诚的定义和评估及其对解释性的意义。然后，我们通过将方法分为五类来介绍忠实解释的最新进展：相似性方法，模型内部结构的分析，基于反向传播的方法，反事实干预和自我解释模型。每个类别将通过其代表性研究，优势和缺点来说明。最后，我们从它们的共同美德和局限性方面讨论了上述所有方法，并反思未来的工作方向忠实的解释性。对于有兴趣研究可解释性的研究人员，这项调查将为该领域提供可访问且全面的概述，为进一步探索提供基础。对于希望更好地了解自己的模型的用户，该调查将是一项介绍性手册，帮助选择最合适的解释方法。

在本文中，我们试图通过引入深度学习模型的句法归纳偏见来建立两所学校之间的联系。我们提出了两个归纳偏见的家族，一个家庭用于选区结构，另一个用于依赖性结构。选区归纳偏见鼓励深度学习模型使用不同的单位（或神经元）分别处理长期和短期信息。这种分离为深度学习模型提供了一种方法，可以从顺序输入中构建潜在的层次表示形式，即更高级别的表示由高级表示形式组成，并且可以分解为一系列低级表示。例如，在不了解地面实际结构的情况下，我们提出的模型学会通过根据其句法结构组成变量和运算符的表示来处理逻辑表达。另一方面，依赖归纳偏置鼓励模型在输入序列中找到实体之间的潜在关系。对于自然语言，潜在关系通常被建模为一个定向依赖图，其中一个单词恰好具有一个父节点和零或几个孩子的节点。将此约束应用于类似变压器的模型之后，我们发现该模型能够诱导接近人类专家注释的有向图，并且在不同任务上也优于标准变压器模型。我们认为，这些实验结果为深度学习模型的未来发展展示了一个有趣的选择。

最近的作品表明了解释性和鲁棒性是值得信赖和可靠的文本分类的两个关键成分。然而，以前的作品通常是解决了两个方面的一个：i）如何提取准确的理由，以便在有利于预测的同时解释; ii）如何使预测模型对不同类型的对抗性攻击稳健。直观地，一种产生有用的解释的模型应该对对抗性攻击更加强大，因为我们无法信任输出解释的模型，而是在小扰动下改变其预测。为此，我们提出了一个名为-BMC的联合分类和理由提取模型。它包括两个关键机制：混合的对手训练（AT）旨在在离散和嵌入空间中使用各种扰动，以改善模型的鲁棒性，边界匹配约束（BMC）有助于利用边界信息的引导来定位理由。基准数据集的性能表明，所提出的AT-BMC优于分类和基本原子的基础，由大边距提取。鲁棒性分析表明，建议的AT-BMC将攻击成功率降低了高达69％。经验结果表明，强大的模型与更好的解释之间存在连接。

Controllable Text Generation (CTG) is emerging area in the field of natural language generation (NLG). It is regarded as crucial for the development of advanced text generation technologies that are more natural and better meet the specific constraints in practical applications. In recent years, methods using large-scale pre-trained language models (PLMs), in particular the widely used transformer-based PLMs, have become a new paradigm of NLG, allowing generation of more diverse and fluent text. However, due to the lower level of interpretability of deep neural networks, the controllability of these methods need to be guaranteed. To this end, controllable text generation using transformer-based PLMs has become a rapidly growing yet challenging new research hotspot. A diverse range of approaches have emerged in the recent 3-4 years, targeting different CTG tasks which may require different types of controlled constraints. In this paper, we present a systematic critical review on the common tasks, main approaches and evaluation methods in this area. Finally, we discuss the challenges that the field is facing, and put forward various promising future directions. To the best of our knowledge, this is the first survey paper to summarize CTG techniques from the perspective of PLMs. We hope it can help researchers in related fields to quickly track the academic frontier, providing them with a landscape of the area and a roadmap for future research.

Masked language modeling (MLM) pre-training methods such as BERT corrupt the input by replacing some tokens with [MASK] and then train a model to reconstruct the original tokens. While they produce good results when transferred to downstream NLP tasks, they generally require large amounts of compute to be effective. As an alternative, we propose a more sample-efficient pre-training task called replaced token detection. Instead of masking the input, our approach corrupts it by replacing some tokens with plausible alternatives sampled from a small generator network. Then, instead of training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that predicts whether each token in the corrupted input was replaced by a generator sample or not. Thorough experiments demonstrate this new pre-training task is more efficient than MLM because the task is defined over all input tokens rather than just the small subset that was masked out. As a result, the contextual representations learned by our approach substantially outperform the ones learned by BERT given the same model size, data, and compute. The gains are particularly strong for small models; for example, we train a model on one GPU for 4 days that outperforms GPT (trained using 30x more compute) on the GLUE natural language understanding benchmark. Our approach also works well at scale, where it performs comparably to RoBERTa and XLNet while using less than 1/4 of their compute and outperforms them when using the same amount of compute.

广义文本表示是许多自然语言理解任务的基础。要充分利用不同的语料库，不可避免地需要了解它们之间的相关性。但是，许多方法忽略了相关性，并直接用于所有任务的单通道模型（粗糙的范式），这缺乏足够的理性和解释。此外，一些现有的作品通过针迹技能块（一个精细的范式）学习下游任务，这可能会导致其冗余和噪音，从而导致非理性。在这项工作中，我们首先通过三种不同的观点分析任务相关性，即数据属性，手动设计和基于模型的相关性，基于相似的任务被分组在一起。然后，我们提出了一个用粗到细范式的层次结构框架，其最底层共享了所有任务，中层级别分为不同的组，以及分配给每个任务的顶级级别。这使我们的模型可以从所有任务中学习基本的语言属性，提高相关任务的性能，并减少不相关任务的负面影响。我们在五个自然语言理解任务的13个基准数据集上进行的实验证明了我们方法的优势。

在本文中，我们设计和训练生成的图像到文本变压器Git，以统一视觉语言任务，例如图像/视频字幕和问题答案。尽管生成模型在预训练和微调之间提供了一致的网络体系结构，但现有工作通常包含复杂的结构（Uni/多模式编码器/解码器），并取决于外部模块，例如对象检测器/标记器和光学角色识别（OCR） ）。在git中，我们将体系结构简化为一个图像编码器，而在单语言建模任务下将架构简化为一个文本解码器。我们还扩展了预训练数据和模型大小，以提高模型性能。没有铃铛和哨子，我们的git在12个具有挑战性的基准下建立了新的艺术状态。例如，我们的模型在文本贴图上首次超过了人类的表现（138.2 vs. 125.5在苹果酒中）。此外，我们提出了一种新的基于一代的图像分类和场景文本识别的方案，在标准基准上实现了不错的表现。

Transfer learning, where a model is first pre-trained on a data-rich task before being finetuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new "Colossal Clean Crawled Corpus", we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.

Transfer learning aims at improving the performance of target learners on target domains by transferring the knowledge contained in different but related source domains. In this way, the dependence on a large number of target domain data can be reduced for constructing target learners. Due to the wide application prospects, transfer learning has become a popular and promising area in machine learning. Although there are already some valuable and impressive surveys on transfer learning, these surveys introduce approaches in a relatively isolated way and lack the recent advances in transfer learning. Due to the rapid expansion of the transfer learning area, it is both necessary and challenging to comprehensively review the relevant studies. This survey attempts to connect and systematize the existing transfer learning researches, as well as to summarize and interpret the mechanisms and the strategies of transfer learning in a comprehensive way, which may help readers have a better understanding of the current research status and ideas. Unlike previous surveys, this survey paper reviews more than forty representative transfer learning approaches, especially homogeneous transfer learning approaches, from the perspectives of data and model. The applications of transfer learning are also briefly introduced. In order to show the performance of different transfer learning models, over twenty representative transfer learning models are used for experiments. The models are performed on three different datasets, i.e., Amazon Reviews, Reuters-21578, and Office-31. And the experimental results demonstrate the importance of selecting appropriate transfer learning models for different applications in practice.