数据到文本生成系统旨在基于输入数据生成文本描述（通常以表格形式表示）。典型系统使用巨大的训练样本来学习表和文本之间的对应关系。然而，大型训练套装昂贵，可以获得这些方法在现实世界方案中的适用性。在这项工作中，我们专注于几次数据到文本生成。我们观察到，虽然微调预训练的语言模型可能会产生合理的句子，但它们在几次拍摄设置中遭受了低语义覆盖问题。换句话说，生成的文本中的重要输入时隙往往丢失。为此，我们提出了一种搜索和学习方法，可以利用预训练的语言模型，而是插入丢失的插槽以提高语义覆盖。我们根据搜索结果进一步微调我们的系统，以平滑搜索噪声，在很大程度上产生更好的质量文本并提高推理效率。实验表明，我们的模型在E2E和Wikibio数据集上实现了高性能。特别是，我们在E2E上覆盖了98.35％的输入槽，很大程度上减轻了低覆盖问题。

这项研究讨论了半监督学习的影响与验证的语言模型，以生成数据到文本。当还补充大规模语言模型时，尚不清楚半监督学习是否仍然有用。这项研究的目的是通过将仅补充语言模型的数据到文本系统与两个数据到文本系统进行比较，这些系统通过数据增强或伪标记的半固定学习方法而富含数据。结果表明，半监督学习会导致多样性指标的得分更高。在输出质量方面，使用伪标记方法扩展数据到文本系统的训练集确实提高了文本质量分数，但是数据增强方法在没有训练设置扩展的情况下得出了与系统相似的分数。这些结果表明，即使也存在语言模型，半监督的学习方法也可以增强产出质量和多样性。

Natural Language Generation (NLG) has improved exponentially in recent years thanks to the development of sequence-to-sequence deep learning technologies such as Transformer-based language models. This advancement has led to more fluent and coherent NLG, leading to improved development in downstream tasks such as abstractive summarization, dialogue generation and data-to-text generation. However, it is also apparent that deep learning based generation is prone to hallucinate unintended text, which degrades the system performance and fails to meet user expectations in many real-world scenarios. To address this issue, many studies have been presented in measuring and mitigating hallucinated texts, but these have never been reviewed in a comprehensive manner before. In this survey, we thus provide a broad overview of the research progress and challenges in the hallucination problem in NLG. The survey is organized into two parts: (1) a general overview of metrics, mitigation methods, and future directions; and (2) an overview of task-specific research progress on hallucinations in the following downstream tasks, namely abstractive summarization, dialogue generation, generative question answering, data-to-text generation, machine translation, and visual-language generation. This survey serves to facilitate collaborative efforts among researchers in tackling the challenge of hallucinated texts in NLG.

神经桌面到文本的生成方法是渴望数据的，限制了它们对低资源现实世界应用的适应性。先前的工作主要诉诸于训练的语言模型（PLM），以生成表格的表格摘要。但是，由于PLM的性质不受控制，它们通常包含幻觉内容。此外，很少研究表和序列之间的拓扑差异。最后但并非最不重要的一点是，在PLM上进行少量实例进行微调可能会导致过度贴合和灾难性的遗忘。为了减轻这些问题，我们提出了一种基于及时的方法，前缀控制的发电机（即PCG），用于几乎没有表格到文本的生成。我们为PLM的特定于任务的前缀预备，以使表结构更适合预训练的输入。此外，我们生成一个特定于输入的前缀，以控制生成的文本的事实内容和单词顺序。对Wikibio数据集的不同领域（人类，书籍和歌曲）的自动评估和人类评估都显示出对基线方法的实质性改进。

Powerful generative models have led to recent progress in question generation (QG). However, it is difficult to measure advances in QG research since there are no standardized resources that allow a uniform comparison among approaches. In this paper, we introduce QG-Bench, a multilingual and multidomain benchmark for QG that unifies existing question answering datasets by converting them to a standard QG setting. It includes general-purpose datasets such as SQuAD for English, datasets from ten domains and two styles, as well as datasets in eight different languages. Using QG-Bench as a reference, we perform an extensive analysis of the capabilities of language models for the task. First, we propose robust QG baselines based on fine-tuning generative language models. Then, we complement automatic evaluation based on standard metrics with an extensive manual evaluation, which in turn sheds light on the difficulty of evaluating QG models. Finally, we analyse both the domain adaptability of these models as well as the effectiveness of multilingual models in languages other than English. QG-Bench is released along with the fine-tuned models presented in the paper https://github.com/asahi417/lm-question-generation, which are also available as a demo https://autoqg.net/.

Controllable Text Generation (CTG) is emerging area in the field of natural language generation (NLG). It is regarded as crucial for the development of advanced text generation technologies that are more natural and better meet the specific constraints in practical applications. In recent years, methods using large-scale pre-trained language models (PLMs), in particular the widely used transformer-based PLMs, have become a new paradigm of NLG, allowing generation of more diverse and fluent text. However, due to the lower level of interpretability of deep neural networks, the controllability of these methods need to be guaranteed. To this end, controllable text generation using transformer-based PLMs has become a rapidly growing yet challenging new research hotspot. A diverse range of approaches have emerged in the recent 3-4 years, targeting different CTG tasks which may require different types of controlled constraints. In this paper, we present a systematic critical review on the common tasks, main approaches and evaluation methods in this area. Finally, we discuss the challenges that the field is facing, and put forward various promising future directions. To the best of our knowledge, this is the first survey paper to summarize CTG techniques from the perspective of PLMs. We hope it can help researchers in related fields to quickly track the academic frontier, providing them with a landscape of the area and a roadmap for future research.

查询聚焦的文本摘要（QFTS）任务旨在构建基于给定查询的文本文档摘要的构建系统。解决此任务的关键挑战是缺乏培训摘要模型的大量标记数据。在本文中，我们通过探索一系列域适应技术来解决这一挑战。鉴于最近在广泛的自然语言处理任务中进行预先接受的变压器模型的成功，我们利用此类模型为单文档和多文件方案的QFTS任务产生抽象摘要。对于域适应，我们使用预先训练的变压器的摘要模型应用了各种技术，包括转移学习，弱监督学习和远程监督。六个数据集的广泛实验表明，我们所提出的方法非常有效地为QFTS任务产生抽象摘要，同时在一组自动和人类评估指标上设置新的最先进的结果。

This paper addresses the quality issues in existing Twitter-based paraphrase datasets, and discusses the necessity of using two separate definitions of paraphrase for identification and generation tasks. We present a new Multi-Topic Paraphrase in Twitter (MultiPIT) corpus that consists of a total of 130k sentence pairs with crowdsoursing (MultiPIT_crowd) and expert (MultiPIT_expert) annotations using two different paraphrase definitions for paraphrase identification, in addition to a multi-reference test set (MultiPIT_NMR) and a large automatically constructed training set (MultiPIT_Auto) for paraphrase generation. With improved data annotation quality and task-specific paraphrase definition, the best pre-trained language model fine-tuned on our dataset achieves the state-of-the-art performance of 84.2 F1 for automatic paraphrase identification. Furthermore, our empirical results also demonstrate that the paraphrase generation models trained on MultiPIT_Auto generate more diverse and high-quality paraphrases compared to their counterparts fine-tuned on other corpora such as Quora, MSCOCO, and ParaNMT.

Dialogue State Tracking (DST), a key component of task-oriented conversation systems, represents user intentions by determining the values of pre-defined slots in an ongoing dialogue. Existing approaches use hand-crafted templates and additional slot information to fine-tune and prompt large pre-trained language models and elicit slot values from the dialogue context. Significant manual effort and domain knowledge is required to design effective prompts, limiting the generalizability of these approaches to new domains and tasks. In this work, we propose DiSTRICT, a generalizable in-context tuning approach for DST that retrieves highly relevant training examples for a given dialogue to fine-tune the model without any hand-crafted templates. Experiments with the MultiWOZ benchmark datasets show that DiSTRICT outperforms existing approaches in various zero-shot and few-shot settings using a much smaller model, thereby providing an important advantage for real-world deployments that often have limited resource availability.

我们介绍了Godel（接地开放对话语言模型），这是对话框的大型预训练的语言模型。与诸如Dialogpt之类的早期模型相比，Godel利用了一个新的扎根预训练阶段，旨在更好地支持将Godel适应广泛的下游对话框任务，这些任务需要当前对话外部的信息（例如，数据库或文档）到产生良好的回应。针对一系列基准测试的实验，这些基准涵盖了面向任务的对话框，对话质量质量检查和接地的开放式对话框，表明Godel在几次以上的微调设置中优于最先进的预训练的对话模型，就人类和自动评估。我们评估方法的一个新颖特征是引入了一个效用概念，该概念除了其交流特征（内在评估）外，还评估了响应的有用性（外部评估）。我们表明，外部评估提供了改进的通道间一致性和与自动指标的相关性。代码和数据处理脚本公开可用。

学术研究是解决以前从未解决过的问题的探索活动。通过这种性质，每个学术研究工作都需要进行文献审查，以区分其Novelties尚未通过事先作品解决。在自然语言处理中，该文献综述通常在“相关工作”部分下进行。鉴于研究文件的其余部分和引用的论文列表，自动相关工作生成的任务旨在自动生成“相关工作”部分。虽然这项任务是在10年前提出的，但直到最近，它被认为是作为科学多文件摘要问题的变种。然而，即使在今天，尚未标准化了自动相关工作和引用文本生成的问题。在这项调查中，我们进行了一个元研究，从问题制定，数据集收集，方法方法，绩效评估和未来前景的角度来比较相关工作的现有文献，以便为读者洞察到国家的进步 - 最内容的研究，以及如何进行未来的研究。我们还调查了我们建议未来工作要考虑整合的相关研究领域。

差异化（DP）学习在建立大型文本模型方面的成功有限，并尝试直接将差异化私有随机梯度下降（DP-SGD）应用于NLP任务，从而导致了大量的性能下降和高度计算的开销。我们表明，通过（1）使用大型验证模型可以缓解这种性能下降； （2）适合DP优化的超参数； （3）与训练过程对齐的微调目标。通过正确设定这些因素，我们将获得私人NLP模型，以优于最先进的私人培训方法和强大的非私人基准 - 通过直接对中等大小的Corpora进行DP优化的预审计模型。为了解决使用大型变压器运行DP-SGD的计算挑战，我们提出了一种存储器保存技术，该技术允许DP-SGD中的剪辑在不实例化模型中任何层的每个示例梯度的情况下运行。该技术使私人训练变压器的内存成本几乎与非私人培训相同，并以适度的运行时间开销。与传统的观点相反，即DP优化在学习高维模型（由于尺寸缩放的噪声）方面失败的经验结果表明，使用预审预周化模型的私人学习往往不会遭受维度依赖性性能降低的障碍。

We report on novel investigations into training models that make sentences concise. We define the task and show that it is different from related tasks such as summarization and simplification. For evaluation, we release two test sets, consisting of 2000 sentences each, that were annotated by two and five human annotators, respectively. We demonstrate that conciseness is a difficult task for which zero-shot setups with large neural language models often do not perform well. Given the limitations of these approaches, we propose a synthetic data generation method based on round-trip translations. Using this data to either train Transformers from scratch or fine-tune T5 models yields our strongest baselines that can be further improved by fine-tuning on an artificial conciseness dataset that we derived from multi-annotator machine translation test sets.

Training learnable metrics using modern language models has recently emerged as a promising method for the automatic evaluation of machine translation. However, existing human evaluation datasets in text simplification are limited by a lack of annotations, unitary simplification types, and outdated models, making them unsuitable for this approach. To address these issues, we introduce the SIMPEVAL corpus that contains: SIMPEVAL_ASSET, comprising 12K human ratings on 2.4K simplifications of 24 systems, and SIMPEVAL_2022, a challenging simplification benchmark consisting of over 1K human ratings of 360 simplifications including generations from GPT-3.5. Training on SIMPEVAL_ASSET, we present LENS, a Learnable Evaluation Metric for Text Simplification. Extensive empirical results show that LENS correlates better with human judgment than existing metrics, paving the way for future progress in the evaluation of text simplification. To create the SIMPEVAL datasets, we introduce RANK & RATE, a human evaluation framework that rates simplifications from several models in a list-wise manner by leveraging an interactive interface, which ensures both consistency and accuracy in the evaluation process. Our metric, dataset, and annotation toolkit are available at https://github.com/Yao-Dou/LENS.

Transfer learning, where a model is first pre-trained on a data-rich task before being finetuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new "Colossal Clean Crawled Corpus", we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.

语言模型（LMS）被证明具有对物理世界的常识知识，这对于在日常情况下完成任务至关重要。但是，LMS是否有能力为具体任务生成扎根的可执行计划，这仍然是一个悬而未决的问题。这是非常具有挑战性的，因为LMS没有“眼睛”或“手”来感知现实的环境。在这项工作中，我们展示了有关这个重要研究问题的第一个研究。我们首先提出了一个名为G-Planet的新型问题公式，它将其作为输入一个高级目标和在特定环境中的对象表。预期输出是一个计划，该计划包括逐步指令供代理执行。为了实现此问题的研究，我们建立了一个评估协议，并设计了一个专门的指标来评估计划的质量。在我们的广泛实验中，我们表明，为编码环境添加扁平表并使用迭代解码策略都可以提高LMS的基础计划能力。我们对结果的分析也导致有趣的非平凡发现。

多文件摘要（MDS）是信息聚合的有效工具，它从与主题相关文档集群生成信息和简洁的摘要。我们的调查是，首先，系统地概述了最近的基于深度学习的MDS模型。我们提出了一种新的分类学，总结神经网络的设计策略，并进行全面的最先进的概要。我们突出了在现有文献中很少讨论的各种客观函数之间的差异。最后，我们提出了与这个新的和令人兴奋的领域有关的几个方向。

临床前和临床领域中的结构化（表格）数据包含有关个人的有价值信息，有效的表格到文本摘要系统可以大大减少手动努力，以将该数据凝结到报告中。但是，实际上，该问题受到最先进的自然语言生成模型（包括T5，Pegasus和GPT-NEO）的数据稀疏性和无法产生准确可靠的输出的严重阻碍。在本文中，我们提出了一种新颖的桌面到文本方法，并通过新颖的两步结构解决这些问题，通过自动校正，复制机制和合成数据增强来增强这些问题。研究表明，所提出的方法从结构化数据中选择了显着的生物医学实体和值，以提高精度（最高0.13个绝对增加），以复制表格值，以生成相干和准确的文本以进行测定验证报告和毒理学报告。此外，我们还通过微调示例进行微调来展示提出的系统对新数据集的轻量重量改编。我们模型的输出在人类的场景中得到了人类专家的验证。

The recent GPT-3 model (Brown et al., 2020) achieves remarkable few-shot performance solely by leveraging a natural-language prompt and a few task demonstrations as input context. Inspired by their findings, we study few-shot learning in a more practical scenario, where we use smaller language models for which fine-tuning is computationally efficient. We present LM-BFF-better few-shot fine-tuning of language models 1 -a suite of simple and complementary techniques for finetuning language models on a small number of annotated examples. Our approach includes (1) prompt-based fine-tuning together with a novel pipeline for automating prompt generation; and (2) a refined strategy for dynamically and selectively incorporating demonstrations into each context. Finally, we present a systematic evaluation for analyzing few-shot performance on a range of NLP tasks, including classification and regression. Our experiments demonstrate that our methods combine to dramatically outperform standard fine-tuning procedures in this low resource setting, achieving up to 30% absolute improvement, and 11% on average across all tasks. Our approach makes minimal assumptions on task resources and domain expertise, and hence constitutes a strong task-agnostic method for few-shot learning. 2 * The first two authors contributed equally. 1 Alternatively, language models' best friends forever. 2 Our implementation is publicly available at https:// github.com/princeton-nlp/LM-BFF.

我们提出了两种小型无监督方法，用于消除文本中的毒性。我们的第一个方法结合了最近的两个想法：（1）使用小型条件语言模型的生成过程的指导和（2）使用释义模型进行风格传输。我们使用良好的令人措辞的令人愉快的释放器，由风格培训的语言模型引导，以保持文本内容并消除毒性。我们的第二种方法使用BERT用他们的非攻击性同义词取代毒性单词。我们通过使BERT替换具有可变数量的单词的屏蔽令牌来使该方法更灵活。最后，我们介绍了毒性去除任务的风格转移模型的第一个大规模比较研究。我们将模型与许多用于样式传输的方法进行比较。使用无监督的样式传输指标的组合以可参考方式评估该模型。两种方法都建议产生新的SOTA结果。