This work aims to employ natural language generation (NLG) to rapidly generate items for English language learning applications: this requires both language models capable of generating fluent, high-quality English, and to control the output of the generation to match the requirements of the relevant items. We experiment with deep pretrained models for this task, developing novel methods for controlling items for factors relevant in language learning: diverse sentences for different proficiency levels and argument structure to test grammar. Human evaluation demonstrates high grammatically scores for all models (3.4 and above out of 4), and higher length (24%) and complexity (9%) over the baseline for the advanced proficiency model. Our results show that we can achieve strong performance while adding additional control to ensure diverse, tailored content for individual users.

人类使用自然语言来撰写普通概念，将他们的环境归结为合理的日常场景描述。然而，这种生成的致辞推理（GCSR）技能缺乏最先进的文本生成方法。关于由神经文本生成模型（例如，预先接受的文本到文本变压器）生成的任意概念的描述性句子通常是语法流畅的，但可能与人类常识不相符，这主要是由于它们缺乏捕获概念关系的机制识别隐式概念，并对看不见的概念组成来执行概括的推理。在本文中，我们提出了一种想象的 - 言语（I＆V）方法，其学会在输入概念之间的关系中想象一个关系场景知识图（SKG），并在生成合理的场景描述时利用SKG作为约束。我们收集和协调来自不同领域和方式的一套知识资源，为I＆v提供丰富的辅助监督信号。该实验展示了I＆V在提高概念到句子和概念到故事的生成任务上的语言模型的有效性，同时使模型能够从更少的任务示例中学习并生成对人类注入者常识的SKG。

Grammatical Error Correction (GEC) is the task of automatically detecting and correcting errors in text. The task not only includes the correction of grammatical errors, such as missing prepositions and mismatched subject-verb agreement, but also orthographic and semantic errors, such as misspellings and word choice errors respectively. The field has seen significant progress in the last decade, motivated in part by a series of five shared tasks, which drove the development of rule-based methods, statistical classifiers, statistical machine translation, and finally neural machine translation systems which represent the current dominant state of the art. In this survey paper, we condense the field into a single article and first outline some of the linguistic challenges of the task, introduce the most popular datasets that are available to researchers (for both English and other languages), and summarise the various methods and techniques that have been developed with a particular focus on artificial error generation. We next describe the many different approaches to evaluation as well as concerns surrounding metric reliability, especially in relation to subjective human judgements, before concluding with an overview of recent progress and suggestions for future work and remaining challenges. We hope that this survey will serve as comprehensive resource for researchers who are new to the field or who want to be kept apprised of recent developments.

机器循环撰写旨在使人类能够与模型进行协作，以更有效地完成他们的写作任务。事先工作发现，为人类书面的草案或句子级持续存在有限，因为生成的文本往往偏离人类的意图。为了允许用户保留对内容的控制，我们训练重写模型，在提示时，修改用户原始草稿中的指定跨度的文本，以在文本中本地介绍本地的描述性和比喻元素。我们评估了在创造性图像标题任务上与人类合作的能力。通过亚马逊机械土耳其人的用户学习，我们的模型被评为比基线缺陷语言模型更有用。此外，第三方评估表明，与单独完成任务相比，用户在与模型合作时，用户编写更多描述性和比喻标题。

我们调查使用图像中包含的多模式信息作为增强文本生成的变压器模型的勤义的有效方法。我们在概念到文本生成中使用BART和T5进行实验，特别是生成致辞推理或蒙的任务。我们称之为Visctg：视觉地基础的概念到文本生成。VisctG涉及代表适当日常方案的标题图像，并使用这些标题来丰富和转向生成过程。综合评估和分析表明，VisctG显着提高了模型性能，同时成功地解决了基线几代的几个问题，包括差的致辞，流畅性和特异性。

Text detoxification has the potential to mitigate the harms of toxicity by rephrasing text to remove offensive meaning, but subtle toxicity remains challenging to tackle. We introduce MaRCo, a detoxification algorithm that combines controllable generation and text rewriting methods using a Product of Experts with autoencoder language models (LMs). MaRCo uses likelihoods under a non-toxic LM (expert) and a toxic LM (anti-expert) to find candidate words to mask and potentially replace. We evaluate our method on several subtle toxicity and microaggressions datasets, and show that it not only outperforms baselines on automatic metrics, but MaRCo's rewrites are preferred 2.1 $\times$ more in human evaluation. Its applicability to instances of subtle toxicity is especially promising, demonstrating a path forward for addressing increasingly elusive online hate.

数据增强是自然语言处理（NLP）模型的鲁棒性评估的重要组成部分，以及增强他们培训的数据的多样性。在本文中，我们呈现NL-Cogmenter，这是一种新的参与式Python的自然语言增强框架，它支持创建两个转换（对数据的修改）和过滤器（根据特定功能的数据拆分）。我们描述了框架和初始的117个变换和23个过滤器，用于各种自然语言任务。我们通过使用其几个转换来分析流行自然语言模型的鲁棒性来证明NL-Upmenter的功效。基础架构，Datacards和稳健性分析结果在NL-Augmenter存储库上公开可用（\ url {https://github.com/gem-benchmark/nl-augmenter}）。

Large "instruction-tuned" language models (finetuned to respond to instructions) have demonstrated a remarkable ability to generalize zero-shot to new tasks. Nevertheless, they depend heavily on human-written instruction data that is limited in quantity, diversity, and creativity, therefore hindering the generality of the tuned model. We introduce Self-Instruct, a framework for improving the instruction-following capabilities of pretrained language models by bootstrapping off its own generations. Our pipeline generates instruction, input, and output samples from a language model, then prunes them before using them to finetune the original model. Applying our method to vanilla GPT3, we demonstrate a 33% absolute improvement over the original model on Super-NaturalInstructions, on par with the performance of InstructGPT_001, which is trained with private user data and human annotations. For further evaluation, we curate a set of expert-written instructions for novel tasks, and show through human evaluation that tuning GPT3 with Self-Instruct outperforms using existing public instruction datasets by a large margin, leaving only a 5% absolute gap behind InstructGPT_001. Self-Instruct provides an almost annotation-free method for aligning pre-trained language models with instructions, and we release our large synthetic dataset to facilitate future studies on instruction tuning.

基于变压器的语言模型最近在许多自然语言任务中取得了显着的结果。但是，通常通过利用大量培训数据来实现排行榜的性能，并且很少通过将明确的语言知识编码为神经模型。这使许多人质疑语言学对现代自然语言处理的相关性。在本文中，我介绍了几个案例研究，以说明理论语言学和神经语言模型仍然相互关联。首先，语言模型通过提供一个客观的工具来测量语义距离，这对语言学家很有用，语义距离很难使用传统方法。另一方面，语言理论通过提供框架和数据源来探究我们的语言模型，以了解语言理解的特定方面，从而有助于语言建模研究。本论文贡献了三项研究，探讨了语言模型中语法 - 听觉界面的不同方面。在论文的第一部分中，我将语言模型应用于单词类灵活性的问题。我将Mbert作为语义距离测量的来源，我提供了有利于将单词类灵活性分析为方向过程的证据。在论文的第二部分中，我提出了一种方法来测量语言模型中间层的惊奇方法。我的实验表明，包含形态句法异常的句子触发了语言模型早期的惊喜，而不是语义和常识异常。最后，在论文的第三部分中，我适应了一些心理语言学研究，以表明语言模型包含了论证结构结构的知识。总而言之，我的论文在自然语言处理，语言理论和心理语言学之间建立了新的联系，以为语言模型的解释提供新的观点。

Knowledge about outcomes is critical for complex event understanding but is hard to acquire. We show that by pre-identifying a participant in a complex event, crowd workers are able to (1) infer the collective impact of salient events that make up the situation, (2) annotate the volitional engagement of participants in causing the situation, and (3) ground the outcome of the situation in state changes of the participants. By creating a multi-step interface and a careful quality control strategy, we collect a high quality annotated dataset of 8K short newswire narratives and ROCStories with high inter-annotator agreement (0.74-0.96 weighted Fleiss Kappa). Our dataset, POQue (Participant Outcome Questions), enables the exploration and development of models that address multiple aspects of semantic understanding. Experimentally, we show that current language models lag behind human performance in subtle ways through our task formulations that target abstract and specific comprehension of a complex event, its outcome, and a participant's influence over the event culmination.

临床票据是记录患者信息的有效方法，但难以破译非专家的难以破译。自动简化医学文本可以使患者提供有关其健康的有价值的信息，同时节省临床医生。我们提出了一种基于词频率和语言建模的医学文本自动简化的新方法，基于富裕的外行术语的医疗本体。我们发布了一对公开可用的医疗句子的新数据集，并由临床医生简化了它们的版本。此外，我们定义了一种新颖的文本简化公制和评估框架，我们用于对我们对现有技术的方法进行大规模人类评估。我们基于在医学论坛数据上培训的语言模型的方法在保留语法和原始含义时产生更简单的句子，超越现有技术。

拟人化是一种语音人物，它赋予无生命实体具有属性和行动，通常被视为需要动画。在本文中，我们探讨了人格化生成的任务。为此，我们提出了菠萝：通过获取平行的人格化数据来学习增强的产生，来拟人化无生命的实体。我们策划了一个名为PersonifCorp的拟人化语料库，并自动生成了这些拟人化的文字化。我们通过训练SEQ2SEQ模型来拟人化给定的文字输入，从而证明了该平行语料库的有用性。自动评估和人类评估都表明，通过人格科目进行微调会带来与人格化相关的素质（例如动画和兴趣）的显着提高。详细的定性分析还强调了菠萝在基准上的关键优势和瑕疵，表明具有强大的能力产生多样化和创造性的拟人化，从而增强了句子的整体吸引力。

舌头是有意义的句子，难以发音。自动产生舌头扭曲的过程具有挑战性，因为产生的话语必须立即满足两个条件：语音难度和语义含义。此外，语音难度本身很难表征，并且通过异质的现象（例如垂涎和谐音）的异质组合以自然的扭曲词来表达。在本文中，我们提出了Pancetta：音素意识到的神经完成，以自动引起舌头扭曲。我们利用音素表示来捕获语音难度的概念，并训练语言模型以在两个提出的任务设置上生成原始的舌头扭曲。为此，我们策划了一个名为Pancetta的数据集，该数据集由现有的英语舌头组成。通过自动和人类评估以及定性分析，我们表明pancetta产生了新颖，语音上的困难，流利和语义上有意义的舌头扭曲。

This paper addresses the quality issues in existing Twitter-based paraphrase datasets, and discusses the necessity of using two separate definitions of paraphrase for identification and generation tasks. We present a new Multi-Topic Paraphrase in Twitter (MultiPIT) corpus that consists of a total of 130k sentence pairs with crowdsoursing (MultiPIT_crowd) and expert (MultiPIT_expert) annotations using two different paraphrase definitions for paraphrase identification, in addition to a multi-reference test set (MultiPIT_NMR) and a large automatically constructed training set (MultiPIT_Auto) for paraphrase generation. With improved data annotation quality and task-specific paraphrase definition, the best pre-trained language model fine-tuned on our dataset achieves the state-of-the-art performance of 84.2 F1 for automatic paraphrase identification. Furthermore, our empirical results also demonstrate that the paraphrase generation models trained on MultiPIT_Auto generate more diverse and high-quality paraphrases compared to their counterparts fine-tuned on other corpora such as Quora, MSCOCO, and ParaNMT.

定义生成任务旨在自动在特定上下文中生成一个单词的定义。但是，由于缺乏针对不同复杂性的数据集，模型产生的定义往往会保持相同的复杂度。本文提出了为具有可控复杂性级别的单词生成定义的新任务。相应地，我们介绍了编译，一个数据集给出了有关中国定义的详细信息，并且每个定义都标有其复杂性级别。编译数据集包括74,303个单词和106,882个定义。据我们所知，它是中国定义生成任务的最大数据集。我们选择各种代表性生成方法作为此任务的基准和进行评估，这说明我们的数据集在协助模型生成不同的复杂性级别定义方面发挥了出色的作用。我们认为，编译数据集将使复杂性可控定义生成的进一步研究受益。

当前的语言模型可以产生高质量的文本。他们只是复制他们之前看到的文本，或者他们学习了普遍的语言抽象吗？要取笑这些可能性，我们介绍了乌鸦，这是一套评估生成文本的新颖性，专注于顺序结构（n-gram）和句法结构。我们将这些分析应用于四种神经语言模型（LSTM，变压器，变换器-XL和GPT-2）。对于本地结构 - 例如，单个依赖性 - 模型生成的文本比来自每个模型的测试集的人类生成文本的基线显着不那么新颖。对于大规模结构 - 例如，总句结构 - 模型生成的文本与人生成的基线一样新颖甚至更新颖，但模型仍然有时复制，在某些情况下，在训练集中重复超过1000字超过1,000字的通道。我们还表现了广泛的手动分析，表明GPT-2的新文本通常在形态学和语法中形成良好，但具有合理的语义问题（例如，是自相矛盾）。

生成反事实测试箱是测试NLP模型并使其像传统软件一样坚固且可靠的重要主体。在生成测试箱时，所需的特性是能够以灵活的方式控制测试案例生成以测试各种故障案例并以目标方式解释和修复它们。在这个方向上，通过手动编写生成受控反事实的规则，在先前的作品中取得了重大进展。但是，这种方法需要大量的手动监督，并且缺乏轻松引入新控件的灵活性。由PPLM的插件方法令人印象深刻的灵活性的激励，我们建议将插件的框架带入反事实测试案例生成任务。我们介绍了Casper，这是一种插件的反事实生成框架，以生成满足需求目标属性的测试用例。我们的插件模型可以在给定任何属性模型的情况下引导测试案例生成过程，而无需对模型的属性特定培训。在实验中，我们表明Casper有效地生成了反事实文本，该文本遵循属性模型提供的转向，同时流利，多样化并保留原始内容。我们还表明，CASPER的生成的反事实可用于增强训练数据，从而固定并使测试模型更加可靠。

Controllable Text Generation (CTG) is emerging area in the field of natural language generation (NLG). It is regarded as crucial for the development of advanced text generation technologies that are more natural and better meet the specific constraints in practical applications. In recent years, methods using large-scale pre-trained language models (PLMs), in particular the widely used transformer-based PLMs, have become a new paradigm of NLG, allowing generation of more diverse and fluent text. However, due to the lower level of interpretability of deep neural networks, the controllability of these methods need to be guaranteed. To this end, controllable text generation using transformer-based PLMs has become a rapidly growing yet challenging new research hotspot. A diverse range of approaches have emerged in the recent 3-4 years, targeting different CTG tasks which may require different types of controlled constraints. In this paper, we present a systematic critical review on the common tasks, main approaches and evaluation methods in this area. Finally, we discuss the challenges that the field is facing, and put forward various promising future directions. To the best of our knowledge, this is the first survey paper to summarize CTG techniques from the perspective of PLMs. We hope it can help researchers in related fields to quickly track the academic frontier, providing them with a landscape of the area and a roadmap for future research.

Large pretrained language models generate fluent text but are notoriously hard to controllably sample from. In this work, we study constrained sampling from such language models: generating text that satisfies user-defined constraints, while maintaining fluency and the model's performance in a downstream task. We propose MuCoLa -- a sampling procedure that combines the log-likelihood of the language model with arbitrary (differentiable) constraints in a single energy function, and then generates samples in a non-autoregressive manner. Specifically, it initializes the entire output sequence with noise and follows a Markov chain defined by Langevin Dynamics using the gradients of the energy function. We evaluate MuCoLa on text generation with soft and hard constraints as well as their combinations obtaining significant improvements over competitive baselines for toxicity avoidance, sentiment control, and keyword-guided generation.

An optimal delivery of arguments is key to persuasion in any debate, both for humans and for AI systems. This requires the use of clear and fluent claims relevant to the given debate. Prior work has studied the automatic assessment of argument quality extensively. Yet, no approach actually improves the quality so far. Our work is the first step towards filling this gap. We propose the task of claim optimization: to rewrite argumentative claims to optimize their delivery. As an initial approach, we first generate a candidate set of optimized claims using a sequence-to-sequence model, such as BART, while taking into account contextual information. Our key idea is then to rerank generated candidates with respect to different quality metrics to find the best optimization. In automatic and human evaluation, we outperform different reranking baselines on an English corpus, improving 60% of all claims (worsening 16% only). Follow-up analyses reveal that, beyond copy editing, our approach often specifies claims with details, whereas it adds less evidence than humans do. Moreover, its capabilities generalize well to other domains, such as instructional texts.