Narrative summarization aims to produce a distilled version of a narrative to describe its most salient events and characters. Summarizing a narrative is challenging as it requires an understanding of event causality and character behaviors. To encourage research in this direction, we propose NarraSum, a large-scale narrative summarization dataset. It contains 122K narrative documents, which are collected from plot descriptions of movies and TV episodes with diverse genres, and their corresponding abstractive summaries. Experiments show that there is a large performance gap between humans and the state-of-the-art summarization models on NarraSum. We hope that this dataset will promote future research in summarization, as well as broader studies of natural language understanding and generation. The dataset is available at https://github.com/zhaochaocs/narrasum.

This paper introduces the shared task of summarizing documents in several creative domains, namely literary texts, movie scripts, and television scripts. Summarizing these creative documents requires making complex literary interpretations, as well as understanding non-trivial temporal dependencies in texts containing varied styles of plot development and narrative structure. This poses unique challenges and is yet underexplored for text summarization systems. In this shared task, we introduce four sub-tasks and their corresponding datasets, focusing on summarizing books, movie scripts, primetime television scripts, and daytime soap opera scripts. We detail the process of curating these datasets for the task, as well as the metrics used for the evaluation of the submissions. As part of the CREATIVESUMM workshop at COLING 2022, the shared task attracted 18 submissions in total. We discuss the submissions and the baselines for each sub-task in this paper, along with directions for facilitating future work in the field.

诸如学术文章和商业报告之类的长期文件一直是详细说明重要问题和需要额外关注的复杂主题的标准格式。自动汇总系统可以有效地将长文档置于简短而简洁的文本中，以封装最重要的信息，从而在帮助读者的理解中很重要。最近，随着神经体系结构的出现，已经做出了重大的研究工作，以推动自动文本摘要系统，以及有关将这些系统扩展到长期文档领域的挑战的大量研究。在这项调查中，我们提供了有关长期文档摘要的研究的全面概述，以及其研究环境的三个主要组成部分的系统评估：基准数据集，汇总模型和评估指标。对于每个组成部分，我们在长期汇总的背景下组织文献，并进行经验分析，以扩大有关当前研究进度的观点。实证分析包括一项研究基准数据集的内在特征，摘要模型的多维分析以及摘要评估指标的综述。根据总体发现，我们通过提出可能在这个快速增长的领域中提出未来探索的方向来得出结论。

Aspect or query-based summarization has recently caught more attention, as it can generate differentiated summaries based on users' interests. However, the current dataset for aspect or query-based summarization either focuses on specific domains, contains relatively small-scale instances, or includes only a few aspect types. Such limitations hinder further explorations in this direction. In this work, we take advantage of crowd-sourcing knowledge on Wikipedia.org and automatically create a high-quality, large-scale open-domain aspect-based summarization dataset named OASum, which contains more than 3.7 million instances with around 1 million different aspects on 2 million Wikipedia pages. We provide benchmark results on OAsum and demonstrate its ability for diverse aspect-based summarization generation. To overcome the data scarcity problem on specific domains, we also perform zero-shot, few-shot, and fine-tuning on seven downstream datasets. Specifically, zero/few-shot and fine-tuning results show that the model pre-trained on our corpus demonstrates a strong aspect or query-focused generation ability compared with the backbone model. Our dataset and pre-trained checkpoints are publicly available.

随着大型语言模型的出现，抽象性摘要的方法取得了长足的进步，从而在应用程序中使用了帮助知识工人处理笨拙的文档收集的潜力。一个这样的环境是民权诉讼交换所（CRLC）（https://clearinghouse.net），其中发布了有关大规模民权诉讼，服务律师，学者和公众的信息。如今，CRLC中的摘要需要对律师和法律专业的学生进行广泛的培训，这些律师和法律专业的学生花费数小时了解多个相关文件，以便产生重要事件和结果的高质量摘要。在这种持续的现实世界摘要工作的激励下，我们引入了Multi-iplesum，这是由正在进行的CRLC写作中绘制的9,280个专家作者的摘要集。鉴于源文档的长度，多文章介绍了一个具有挑战性的多文档摘要任务，通常每个情况超过200页。此外，多胎sum与其多个目标摘要中的其他数据集不同，每个数据集都处于不同的粒度（从一句“极端”摘要到超过五百个单词的多段落叙述）。我们提供了广泛的分析，表明，尽管培训数据（遵守严格的内容和样式准则）中的摘要很高，但最新的摘要模型在此任务上的表现较差。我们发布了多体式的摘要方法，以及促进应用程序的开发，以协助CRLC的任务https://multilexsum.github.io。

The majority of available text summarization datasets include short-form source documents that lack long-range causal and temporal dependencies, and often contain strong layout and stylistic biases. While relevant, such datasets will offer limited challenges for future generations of text summarization systems. We address these issues by introducing BookSum, a collection of datasets for long-form narrative summarization. Our dataset covers source documents from the literature domain, such as novels, plays and stories, and includes highly abstractive, human written summaries on three levels of granularity of increasing difficulty: paragraph-, chapter-, and book-level. The domain and structure of our dataset poses a unique set of challenges for summarization systems, which include: processing very long documents, non-trivial causal and temporal dependencies, and rich discourse structures. To facilitate future work, we trained and evaluated multiple extractive and abstractive summarization models as baselines for our dataset.

由于免费的在线百科全书具有大量内容，因此Wikipedia和Wikidata是许多自然语言处理（NLP）任务的关键，例如信息检索，知识基础构建，机器翻译，文本分类和文本摘要。在本文中，我们介绍了Wikides，这是一个新颖的数据集，用于为文本摘要问题提供Wikipedia文章的简短描述。该数据集由6987个主题上的80K英语样本组成。我们设置了一种两阶段的摘要方法 - 描述生成（I阶段）和候选排名（II阶段）作为一种依赖于转移和对比学习的强大方法。对于描述生成，与其他小规模的预训练模型相比，T5和BART表现出了优越性。通过将对比度学习与Beam Search的不同输入一起应用，基于度量的排名模型优于直接描述生成模型，在主题独立拆分和独立于主题的独立拆分中，最高可达22个胭脂。此外，第II期中的结果描述得到了人类评估的支持，其中45.33％以上，而I阶段的23.66％则支持针对黄金描述。在情感分析方面，生成的描述无法有效地从段落中捕获所有情感极性，同时从黄金描述中更好地完成此任务。自动产生的新描述减少了人类为创建它们的努力，并丰富了基于Wikidata的知识图。我们的论文对Wikipedia和Wikidata产生了实际影响，因为有成千上万的描述。最后，我们预计Wikides将成为从短段落中捕获显着信息的相关作品的有用数据集。策划的数据集可公开可用：https：//github.com/declare-lab/wikides。

Text summarization is a user-preference based task, i.e., for one document, users often have different priorities for summary. As a key aspect of customization in summarization, granularity is used to measure the semantic coverage between the summary and source document. However, developing systems that can generate summaries with customizable semantic coverage is still an under-explored topic. In this paper, we propose the first unsupervised multi-granularity summarization framework, GranuSum. We take events as the basic semantic units of the source documents and propose to rank these events by their salience. We also develop a model to summarize input documents with given events as anchors and hints. By inputting different numbers of events, GranuSum is capable of producing multi-granular summaries in an unsupervised manner. Meanwhile, we annotate a new benchmark GranuDUC that contains multiple summaries at different granularities for each document cluster. Experimental results confirm the substantial superiority of GranuSum on multi-granularity summarization over strong baselines. Further, by exploiting the event information, GranuSum also exhibits state-of-the-art performance under the conventional unsupervised abstractive setting. Dataset for this paper can be found at: https://github.com/maszhongming/GranuDUC

NLP基准在很大程度上主要集中在短篇文本上，例如句子和段落，即使长文本在野外占相当数量的自然语言。我们介绍卷轴，这是一套需要在长文本上推理的任务套件。我们检查现有的长文本数据集，文本自然是长期的，同时优先考虑涉及在输入上扫描信息的任务。滚动包含概述，问题应答和自然语言推理任务，包括多个域，包括文学，科学，业务和娱乐。初始基线（包括啰覆编码器），表明滚动有充足的改进空间。我们以统一的文本到文本格式提供所有数据集，并托管Live Refordboard，以促进模型架构和预用方法的研究。

Dialogue summarization has recently garnered significant attention due to its wide range of applications. However, existing methods for summarizing dialogues are suboptimal because they do not take into account the inherent structure of dialogue and rely heavily on labeled data, which can lead to poor performance in new domains. In this work, we propose DIONYSUS (dynamic input optimization in pre-training for dialogue summarization), a pre-trained encoder-decoder model for summarizing dialogues in any new domain. To pre-train DIONYSUS, we create two pseudo summaries for each dialogue example: one is produced by a fine-tuned summarization model, and the other is a collection of dialogue turns that convey important information. We then choose one of these pseudo summaries based on the difference in information distribution across different types of dialogues. This selected pseudo summary serves as the objective for pre-training DIONYSUS using a self-supervised approach on a large dialogue corpus. Our experiments show that DIONYSUS outperforms existing methods on six datasets, as demonstrated by its ROUGE scores in zero-shot and few-shot settings.

Nowadays, time-stamped web documents related to a general news query floods spread throughout the Internet, and timeline summarization targets concisely summarizing the evolution trajectory of events along the timeline. Unlike traditional document summarization, timeline summarization needs to model the time series information of the input events and summarize important events in chronological order. To tackle this challenge, in this paper, we propose a Unified Timeline Summarizer (UTS) that can generate abstractive and extractive timeline summaries in time order. Concretely, in the encoder part, we propose a graph-based event encoder that relates multiple events according to their content dependency and learns a global representation of each event. In the decoder part, to ensure the chronological order of the abstractive summary, we propose to extract the feature of event-level attention in its generation process with sequential information remained and use it to simulate the evolutionary attention of the ground truth summary. The event-level attention can also be used to assist in extracting summary, where the extracted summary also comes in time sequence. We augment the previous Chinese large-scale timeline summarization dataset and collect a new English timeline dataset. Extensive experiments conducted on these datasets and on the out-of-domain Timeline 17 dataset show that UTS achieves state-of-the-art performance in terms of both automatic and human evaluations.

学术研究是解决以前从未解决过的问题的探索活动。通过这种性质，每个学术研究工作都需要进行文献审查，以区分其Novelties尚未通过事先作品解决。在自然语言处理中，该文献综述通常在“相关工作”部分下进行。鉴于研究文件的其余部分和引用的论文列表，自动相关工作生成的任务旨在自动生成“相关工作”部分。虽然这项任务是在10年前提出的，但直到最近，它被认为是作为科学多文件摘要问题的变种。然而，即使在今天，尚未标准化了自动相关工作和引用文本生成的问题。在这项调查中，我们进行了一个元研究，从问题制定，数据集收集，方法方法，绩效评估和未来前景的角度来比较相关工作的现有文献，以便为读者洞察到国家的进步 - 最内容的研究，以及如何进行未来的研究。我们还调查了我们建议未来工作要考虑整合的相关研究领域。

Human evaluation is the foundation upon which the evaluation of both summarization systems and automatic metrics rests. However, existing human evaluation protocols and benchmarks for summarization either exhibit low inter-annotator agreement or lack the scale needed to draw statistically significant conclusions, and an in-depth analysis of human evaluation is lacking. In this work, we address the shortcomings of existing summarization evaluation along the following axes: 1) We propose a modified summarization salience protocol, Atomic Content Units (ACUs), which relies on fine-grained semantic units and allows for high inter-annotator agreement. 2) We curate the Robust Summarization Evaluation (RoSE) benchmark, a large human evaluation dataset consisting of over 22k summary-level annotations over state-of-the-art systems on three datasets. 3) We compare our ACU protocol with three other human evaluation protocols, underscoring potential confounding factors in evaluation setups. 4) We evaluate existing automatic metrics using the collected human annotations across evaluation protocols and demonstrate how our benchmark leads to more statistically stable and significant results. Furthermore, our findings have important implications for evaluating large language models (LLMs), as we show that LLMs adjusted by human feedback (e.g., GPT-3.5) may overfit unconstrained human evaluation, which is affected by the annotators' prior, input-agnostic preferences, calling for more robust, targeted evaluation methods.

多文件摘要（MDS）是信息聚合的有效工具，它从与主题相关文档集群生成信息和简洁的摘要。我们的调查是，首先，系统地概述了最近的基于深度学习的MDS模型。我们提出了一种新的分类学，总结神经网络的设计策略，并进行全面的最先进的概要。我们突出了在现有文献中很少讨论的各种客观函数之间的差异。最后，我们提出了与这个新的和令人兴奋的领域有关的几个方向。

Natural Language Generation (NLG) has improved exponentially in recent years thanks to the development of sequence-to-sequence deep learning technologies such as Transformer-based language models. This advancement has led to more fluent and coherent NLG, leading to improved development in downstream tasks such as abstractive summarization, dialogue generation and data-to-text generation. However, it is also apparent that deep learning based generation is prone to hallucinate unintended text, which degrades the system performance and fails to meet user expectations in many real-world scenarios. To address this issue, many studies have been presented in measuring and mitigating hallucinated texts, but these have never been reviewed in a comprehensive manner before. In this survey, we thus provide a broad overview of the research progress and challenges in the hallucination problem in NLG. The survey is organized into two parts: (1) a general overview of metrics, mitigation methods, and future directions; and (2) an overview of task-specific research progress on hallucinations in the following downstream tasks, namely abstractive summarization, dialogue generation, generative question answering, data-to-text generation, machine translation, and visual-language generation. This survey serves to facilitate collaborative efforts among researchers in tackling the challenge of hallucinated texts in NLG.

Though many algorithms can be used to automatically summarize legal case decisions, most fail to incorporate domain knowledge about how important sentences in a legal decision relate to a representation of its document structure. For example, analysis of a legal case summarization dataset demonstrates that sentences serving different types of argumentative roles in the decision appear in different sections of the document. In this work, we propose an unsupervised graph-based ranking model that uses a reweighting algorithm to exploit properties of the document structure of legal case decisions. We also explore the impact of using different methods to compute the document structure. Results on the Canadian Legal Case Law dataset show that our proposed method outperforms several strong baselines.

传达相关和忠实信息的能力对于有条件生成的许多任务至关重要，但对于神经SEQ-seq seq模型仍然难以捉摸，这些模型的输出通常显示出幻觉，并且无法正确涵盖重要细节。在这项工作中，我们主张规划作为有用的中间表示，以使有条件的一代减少不透明和扎根。我们的作品提出了将文本计划作为一系列提问（QA）对的新概念化。我们用QA蓝图作为内容选择（即〜说什么）和计划（即〜按什么顺序）来增强现有数据集（例如，用于摘要）。我们通过利用最先进的问题生成技术并将输入输出对自动获取蓝图，并将其转换为输入 - 蓝图输出输出元组。我们开发了基于变压器的模型，每个模型都在它们如何将蓝图合并到生成的输出中（例如，作为全局计划或迭代）。跨指标和数据集的评估表明，蓝图模型比不采取计划并允许对生成输出进行更严格控制的替代方案更为事实。

我们介绍了微博观点摘要（MOS）的任务，并共享3100个金标准意见摘要的数据集，以促进该领域的研究。该数据集包含跨越2年期的推文的摘要，并且涵盖了比任何其他公共Twitter摘要数据集更多的主题。摘要本质上是抽象的，是由熟练的记者创建的，这些记者在将事实信息（主要故事）与作者观点分开的模板之后，总结了新闻文章。我们的方法不同于以前在社交媒体中生成金标准摘要的工作，这些摘要通常涉及选择代表性帖子，从而有利于提取性摘要模型。为了展示数据集的实用性和挑战，我们基准了一系列抽象性和提取性的最先进的摘要模型，并实现良好的性能，前者的表现优于后者。我们还表明，微调对于提高性能和研究使用不同样本量的好处是必要的。

社区问题应答（CQA）FORA，如堆栈溢出和雅虎！答案包含丰富的资源，对广泛的基于社区的问题答案。每个问题线程都可以通过不同的角度接收大量答案。答案摘要的一个目标是产生反映答案视角范围的摘要。抽象答案概述的主要障碍是没有数据集，可以提供监督制作这些摘要。最近的作品提出了创建此类数据的启发式，但这些是嘈杂的，并且不会涵盖答案中存在的所有观点。这项工作介绍了4,631个CQA线程的新型数据集，用于答案摘要，由专业语言学家策划。我们的管道收集了答案概述所涉及的所有子特设的注释，包括选择与问题相关的答案句子，根据透视图对这些句子进行分组，总结每个视角，并生成整体摘要。我们在这些子组织上分析和基准最先进的模型，并为多视角数据增强引入了一种新的无监督方法，这进一步提高了根据自动评估的整体摘要性能。最后，我们提出了加强学习奖励，以改善事实一致性和答案覆盖范围和分析改进领域。

Knowledge about outcomes is critical for complex event understanding but is hard to acquire. We show that by pre-identifying a participant in a complex event, crowd workers are able to (1) infer the collective impact of salient events that make up the situation, (2) annotate the volitional engagement of participants in causing the situation, and (3) ground the outcome of the situation in state changes of the participants. By creating a multi-step interface and a careful quality control strategy, we collect a high quality annotated dataset of 8K short newswire narratives and ROCStories with high inter-annotator agreement (0.74-0.96 weighted Fleiss Kappa). Our dataset, POQue (Participant Outcome Questions), enables the exploration and development of models that address multiple aspects of semantic understanding. Experimentally, we show that current language models lag behind human performance in subtle ways through our task formulations that target abstract and specific comprehension of a complex event, its outcome, and a participant's influence over the event culmination.