产生连贯的长文本的困难在于，现有模型压倒性地集中在预测本地单词上，并且无法制定高级计划，以生成或捕获文本块之间的高级话语依赖性。受到人类写作过程的启发，首先概述了子弹点或目录的列表，然后扩展了每个子弹点以构成整个文章，我们建议{\ it soe}，这是一个涉及总结，概述和概述和概述和概述和的系统的系统详细说明长文本生成：模型首先概述了长文本不同段的摘要，然后在每个子弹点上详细阐述以生成相应的段。为了避免劳动密集型的摘要征集过程，我们提出了{\ it重建}策略，该策略通过选择其最有用的部分来重建该细分市场，以无监督的方式提取细分市场摘要。拟议的生成系统具有以下优点：（1）摘要为文本生成提供了高级指导，并避免了单个单词预测的局部最小值； （2）高级话语依赖性是在摘要之间的条件依赖性中捕获的，并在摘要扩展过程中保留了，并且（3）此外，我们能够通过将上下文表示为简洁的摘要来考虑更多的上下文。广泛的实验表明，SOE产生的长文本具有明显更好的质量，以及更快的收敛速度。

与单案摘要相比，抽象性多文件摘要（MDS）对其冗长和链接的来源的表示和覆盖范围提出了挑战。这项研究开发了一个平行的层次变压器（PHT），具有MDS的注意对齐。通过合并单词和段落级的多头注意，PHT的层次结构可以更好地处理令牌和文档级别的依赖项。为了指导解码到更好的源文档覆盖范围，然后将注意力调整机制引入以校准光束搜索，并预测的最佳注意力分布。根据Wikisum数据，进行了全面的评估，以测试拟议的体系结构对MD的改进。通过更好地处理内部和跨文档的信息，结果胭脂和人类评估都表明，我们的分层模型以相对较低的计算成本生成较高质量的摘要。

诸如学术文章和商业报告之类的长期文件一直是详细说明重要问题和需要额外关注的复杂主题的标准格式。自动汇总系统可以有效地将长文档置于简短而简洁的文本中，以封装最重要的信息，从而在帮助读者的理解中很重要。最近，随着神经体系结构的出现，已经做出了重大的研究工作，以推动自动文本摘要系统，以及有关将这些系统扩展到长期文档领域的挑战的大量研究。在这项调查中，我们提供了有关长期文档摘要的研究的全面概述，以及其研究环境的三个主要组成部分的系统评估：基准数据集，汇总模型和评估指标。对于每个组成部分，我们在长期汇总的背景下组织文献，并进行经验分析，以扩大有关当前研究进度的观点。实证分析包括一项研究基准数据集的内在特征，摘要模型的多维分析以及摘要评估指标的综述。根据总体发现，我们通过提出可能在这个快速增长的领域中提出未来探索的方向来得出结论。

Text summarization is a user-preference based task, i.e., for one document, users often have different priorities for summary. As a key aspect of customization in summarization, granularity is used to measure the semantic coverage between the summary and source document. However, developing systems that can generate summaries with customizable semantic coverage is still an under-explored topic. In this paper, we propose the first unsupervised multi-granularity summarization framework, GranuSum. We take events as the basic semantic units of the source documents and propose to rank these events by their salience. We also develop a model to summarize input documents with given events as anchors and hints. By inputting different numbers of events, GranuSum is capable of producing multi-granular summaries in an unsupervised manner. Meanwhile, we annotate a new benchmark GranuDUC that contains multiple summaries at different granularities for each document cluster. Experimental results confirm the substantial superiority of GranuSum on multi-granularity summarization over strong baselines. Further, by exploiting the event information, GranuSum also exhibits state-of-the-art performance under the conventional unsupervised abstractive setting. Dataset for this paper can be found at: https://github.com/maszhongming/GranuDUC

Sentence summarization shortens given texts while maintaining core contents of the texts. Unsupervised approaches have been studied to summarize texts without human-written summaries. However, recent unsupervised models are extractive, which remove words from texts and thus they are less flexible than abstractive summarization. In this work, we devise an abstractive model based on reinforcement learning without ground-truth summaries. We formulate the unsupervised summarization based on the Markov decision process with rewards representing the summary quality. To further enhance the summary quality, we develop a multi-summary learning mechanism that generates multiple summaries with varying lengths for a given text, while making the summaries mutually enhance each other. Experimental results show that the proposed model substantially outperforms both abstractive and extractive models, yet frequently generating new words not contained in input texts.

具有释义生成的长期问题是如何获得可靠的监督信号。在本文中，我们基于假设产生与鉴定相同的上下文相同的含义的两个句子的概率应该是相同的，提出了一种无监督的范例。灵感来自这一基本因的主意，我们提出了一种流水线系统，该系统由基于上下文语言模型的候选候选生成组成，使用评分函数的候选滤波，以及基于所选候选者的释放模型训练。提议的范例提供了现有的释义生成方法的优点：（1）使用上下文规范器在含义上，该模型能够产生大量的高质量释义对; （2）使用人为可解释的评分功能来选择来自候选者的释义对，所提出的框架为开发人员提供了一种与数据生成过程进行干预的通道，导致更可控的模型。不同任务和数据集的实验结果表明，拟议模型在监督和无人监督的设置中的有效性。

多文件摘要（MDS）是信息聚合的有效工具，它从与主题相关文档集群生成信息和简洁的摘要。我们的调查是，首先，系统地概述了最近的基于深度学习的MDS模型。我们提出了一种新的分类学，总结神经网络的设计策略，并进行全面的最先进的概要。我们突出了在现有文献中很少讨论的各种客观函数之间的差异。最后，我们提出了与这个新的和令人兴奋的领域有关的几个方向。

Although lyrics generation has achieved significant progress in recent years, it has limited practical applications because the generated lyrics cannot be performed without composing compatible melodies. In this work, we bridge this practical gap by proposing a song rewriting system which rewrites the lyrics of an existing song such that the generated lyrics are compatible with the rhythm of the existing melody and thus singable. In particular, we propose SongRewriter, a controllable Chinese lyric generation and editing system which assists users without prior knowledge of melody composition. The system is trained by a randomized multi-level masking strategy which produces a unified model for generating entirely new lyrics or editing a few fragments. To improve the controllabiliy of the generation process, we further incorporate a keyword prompt to control the lexical choices of the content and propose novel decoding constraints and a vowel modeling task to enable flexible end and internal rhyme schemes. While prior rhyming metrics are mainly for rap lyrics, we propose three novel rhyming evaluation metrics for song lyrics. Both automatic and human evaluations show that the proposed model performs better than the state-of-the-art models in both contents and rhyming quality. Our code and models implemented in MindSpore Lite tool will be available.

Prompts with different control signals (e.g., length, keywords, etc.) can be used to control text summarization. When control signals are available, they can control the properties of generated summaries and potentially improve summarization quality (since more information are given). Unfortunately, control signals are not already available during inference time. In this paper, we propose Lotus (shorthand for Latent Prompt Tuning for Summarization), which is a single model that can be applied in both controlled and uncontrolled (without control signals) modes. During training, Lotus learns latent prompt representations from prompts with gold control signals using a contrastive learning objective. Experiments show Lotus in uncontrolled mode consistently improves upon strong (uncontrollable) summarization models across four different summarization datasets. We also demonstrate generated summaries can be controlled using prompts with user specified control tokens.

本文对过去二十年来对自然语言生成（NLG）的研究提供了全面的审查，特别是与数据到文本生成和文本到文本生成深度学习方法有关，以及NLG的新应用技术。该调查旨在（a）给出关于NLG核心任务的最新综合，以及该领域采用的建筑;（b）详细介绍各种NLG任务和数据集，并提请注意NLG评估中的挑战，专注于不同的评估方法及其关系;（c）强调一些未来的强调和相对近期的研究问题，因为NLG和其他人工智能领域的协同作用而增加，例如计算机视觉，文本和计算创造力。

尽管在产生流利的文本方面取得了进步，但现有的预训练模型倾向于在产生诸如故事和新闻之类的叙述时将不连贯的事件序列附加到相关实体上。我们猜想，这些问题是由将实体表示为浅表词的静态嵌入而导致的，同时忽略了对其不断变化的状态建模，即随着文本的展开，即它们所携带的信息。因此，我们将变压器模型扩展到动态执行实体状态更新和叙事生成的句子实现。我们提出了一个对比框架，以在离散空间中学习状态表示，并将其他注意层插入解码器中以更好地利用这些状态。两个叙述数据集的实验表明，与有意义的实体状态的指导相比，我们的模型可以产生更多的连贯和多样化的叙事。

Recent work has improved language models (LMs) remarkably by equipping them with a non-parametric memory component. However, most existing approaches only introduce mem-ories at testing time or represent them using a separately trained encoder, resulting in suboptimal training of the language model. In this work, we present TRIME, a novel yet simple training approach designed for training LMs with memory augmentation. Our approach uses a training objective that directly takes in-batch examples as accessible memory. We also present new methods for memory construction and data batching, which are used for adapting to different sets of memories--local, long-term, and external memory--at testing time. We evaluate TRIME on multiple language modeling and machine translation benchmarks and show that it is able to achieve significant improvements across all the settings. Concretely, TRIME reduces the perplexity from 18.70 to 15.37 on WIKITEXT-103, by effectively leveraging a large memory set from the training corpus. Compared to standard LM training, TRIME adds negligible computational overhead and is compatible with different neural architectures, making it a versatile solution for training memory-augmented LMs.

深度神经语言模型的最新进展与大规模数据集的能力相结合，加速了自然语言生成系统的发展，这些系统在多种任务和应用程序上下文中产生流利和连贯的文本（在各种成功程度上）。但是，为所需的用户控制这些模型的输出仍然是一个开放的挑战。这不仅对于自定义生成语言的内容和样式至关重要，而且对于他们在现实世界中的安全可靠部署至关重要。我们提出了一项关于受约束神经语言生成的新兴主题的广泛调查，在该主题中，我们通过区分条件和约束（后者是在输出文本上而不是输入的可检验条件），正式定义和分类自然语言生成问题，目前是可检验的）约束文本生成任务，并查看受限文本生成的现有方法和评估指标。我们的目的是强调这个新兴领域的最新进展和趋势，以告知最有希望的方向和局限性，以推动受约束神经语言生成研究的最新作品。

预训练的语言模型（PLM）无法生成长形式的叙事文本，因为它们不考虑全局结构。结果，生成的文本通常是不巧妙的，重复的或缺乏内容的。故事发电的最新工作以提示，关键字或语义框架的形式重新引入了明确的内容计划。经过大型平行语料库的培训，这些模型可以生成更合乎逻辑的事件序列，从而产生更满足的故事。但是，这些中间表示通常不使用自然语言，并且不需要微调就无法使用。我们建议使用现成的PLM生成故事情节，同时保持内容计划的好处，以产生凝聚力和满足的故事。我们提出的方法ScratchPlot首先提示PLM构成内容计划。然后，我们生成故事的身体并以内容计划结束。此外，我们通过使用其他PLM来对生成的（故事，结尾）对进行排名。我们用各种基线基准测试我们的方法，并在人类和自动评估中取得了卓越的结果。

Controllable Text Generation (CTG) is emerging area in the field of natural language generation (NLG). It is regarded as crucial for the development of advanced text generation technologies that are more natural and better meet the specific constraints in practical applications. In recent years, methods using large-scale pre-trained language models (PLMs), in particular the widely used transformer-based PLMs, have become a new paradigm of NLG, allowing generation of more diverse and fluent text. However, due to the lower level of interpretability of deep neural networks, the controllability of these methods need to be guaranteed. To this end, controllable text generation using transformer-based PLMs has become a rapidly growing yet challenging new research hotspot. A diverse range of approaches have emerged in the recent 3-4 years, targeting different CTG tasks which may require different types of controlled constraints. In this paper, we present a systematic critical review on the common tasks, main approaches and evaluation methods in this area. Finally, we discuss the challenges that the field is facing, and put forward various promising future directions. To the best of our knowledge, this is the first survey paper to summarize CTG techniques from the perspective of PLMs. We hope it can help researchers in related fields to quickly track the academic frontier, providing them with a landscape of the area and a roadmap for future research.

Nowadays, time-stamped web documents related to a general news query floods spread throughout the Internet, and timeline summarization targets concisely summarizing the evolution trajectory of events along the timeline. Unlike traditional document summarization, timeline summarization needs to model the time series information of the input events and summarize important events in chronological order. To tackle this challenge, in this paper, we propose a Unified Timeline Summarizer (UTS) that can generate abstractive and extractive timeline summaries in time order. Concretely, in the encoder part, we propose a graph-based event encoder that relates multiple events according to their content dependency and learns a global representation of each event. In the decoder part, to ensure the chronological order of the abstractive summary, we propose to extract the feature of event-level attention in its generation process with sequential information remained and use it to simulate the evolutionary attention of the ground truth summary. The event-level attention can also be used to assist in extracting summary, where the extracted summary also comes in time sequence. We augment the previous Chinese large-scale timeline summarization dataset and collect a new English timeline dataset. Extensive experiments conducted on these datasets and on the out-of-domain Timeline 17 dataset show that UTS achieves state-of-the-art performance in terms of both automatic and human evaluations.

复制机制允许序列到序列模型从输入中选择单词并将它们直接放入输出中，这在抽象总结中发现越来越多的使用。但是，由于汉语句子中没有明确的分隔符，所以最现有的中国抽象摘要模型只能执行字符副本，从而导致效率低下。为了解决这个问题，我们提出了一个词典约束的复制网络，在编码器和解码器中模拟多粒度。在源端，单词和字符使用变换器基编码器聚合到相同的输入存储器中。在目标方面，解码器可以在每个时间步骤复制字符或多字符字，并且解码过程由一个词增强的搜索算法引导，其促进并行计算并鼓励模型复制更多单词。此外，我们采用单词选择器来集成关键字信息。实验结果在中国社交媒体数据集显示我们的模型可以独立或使用单词选择器。这两种形式都可以胜过以前的基于角色的模型并实现竞争性表现。

Sequence-to-sequence neural network models for generation of conversational responses tend to generate safe, commonplace responses (e.g., I don't know) regardless of the input. We suggest that the traditional objective function, i.e., the likelihood of output (response) given input (message) is unsuited to response generation tasks. Instead we propose using Maximum Mutual Information (MMI) as the objective function in neural models. Experimental results demonstrate that the proposed MMI models produce more diverse, interesting, and appropriate responses, yielding substantive gains in BLEU scores on two conversational datasets and in human evaluations. Input: What are you doing? −0.86 I don't know. −1.09 Get out of here. −1.03 I don't know! −1.09 I'm going home. −1.06 Nothing. −1.09 Oh my god! −1.09 Get out of the way. −1.10 I'm talking to you. Input: what is your name? −0.91 I don't know. ... −0.92 I don't know! −1.55 My name is Robert. −0.92 I don't know, sir. −1.58 My name is John. −0.97 Oh, my god! −1.59 My name's John. Input: How old are you? −0.79 I don't know. ... −1.06 I'm fine.−1.64 Twenty-five. −1.17 I'm all right.−1.66 Five. −1.17 I'm not sure.−1.71 Eight.

Bidirectional Encoder Representations from Transformers (BERT; Devlin et al. 2019) represents the latest incarnation of pretrained language models which have recently advanced a wide range of natural language processing tasks. In this paper, we showcase how BERT can be usefully applied in text summarization and propose a general framework for both extractive and abstractive models. We introduce a novel document-level encoder based on BERT which is able to express the semantics of a document and obtain representations for its sentences. Our extractive model is built on top of this encoder by stacking several intersentence Transformer layers. For abstractive summarization, we propose a new fine-tuning schedule which adopts different optimizers for the encoder and the decoder as a means of alleviating the mismatch between the two (the former is pretrained while the latter is not). We also demonstrate that a two-staged fine-tuning approach can further boost the quality of the generated summaries. Experiments on three datasets show that our model achieves stateof-the-art results across the board in both extractive and abstractive settings. 1

Information overloading requires the need for summarizers to extract salient information from the text. Currently, there is an overload of dialogue data due to the rise of virtual communication platforms. The rise of Covid-19 has led people to rely on online communication platforms like Zoom, Slack, Microsoft Teams, Discord, etc. to conduct their company meetings. Instead of going through the entire meeting transcripts, people can use meeting summarizers to select useful data. Nevertheless, there is a lack of comprehensive surveys in the field of meeting summarizers. In this survey, we aim to cover recent meeting summarization techniques. Our survey offers a general overview of text summarization along with datasets and evaluation metrics for meeting summarization. We also provide the performance of each summarizer on a leaderboard. We conclude our survey with different challenges in this domain and potential research opportunities for future researchers.