Controllable summarization allows users to generate customized summaries with specified attributes. However, due to the lack of designated annotations of controlled summaries, existing works have to craft pseudo datasets by adapting generic summarization benchmarks. Furthermore, most research focuses on controlling single attributes individually (e.g., a short summary or a highly abstractive summary) rather than controlling a mix of attributes together (e.g., a short and highly abstractive summary). In this paper, we propose MACSum, the first human-annotated summarization dataset for controlling mixed attributes. It contains source texts from two domains, news articles and dialogues, with human-annotated summaries controlled by five designed attributes (Length, Extractiveness, Specificity, Topic, and Speaker). We propose two simple and effective parameter-efficient approaches for the new task of mixed controllable summarization based on hard prompt tuning and soft prefix tuning. Results and analysis demonstrate that hard prompt models yield the best performance on all metrics and human evaluations. However, mixed-attribute control is still challenging for summarization tasks. Our dataset and code are available at https://github.com/psunlpgroup/MACSum.

Text summarization is a user-preference based task, i.e., for one document, users often have different priorities for summary. As a key aspect of customization in summarization, granularity is used to measure the semantic coverage between the summary and source document. However, developing systems that can generate summaries with customizable semantic coverage is still an under-explored topic. In this paper, we propose the first unsupervised multi-granularity summarization framework, GranuSum. We take events as the basic semantic units of the source documents and propose to rank these events by their salience. We also develop a model to summarize input documents with given events as anchors and hints. By inputting different numbers of events, GranuSum is capable of producing multi-granular summaries in an unsupervised manner. Meanwhile, we annotate a new benchmark GranuDUC that contains multiple summaries at different granularities for each document cluster. Experimental results confirm the substantial superiority of GranuSum on multi-granularity summarization over strong baselines. Further, by exploiting the event information, GranuSum also exhibits state-of-the-art performance under the conventional unsupervised abstractive setting. Dataset for this paper can be found at: https://github.com/maszhongming/GranuDUC

我们报告了Dialogsum挑战的结果，即在INLG 2022上汇总现实生活中的对话的共同任务。四个团队参与了这项共享任务，并提交了他们的系统报告，探索了不同的方法来提高对话摘要的性能。尽管对于自动评估指标（例如Rouge分数），基线模型有很大的改进，但我们发现模型生成的输出与通过多个方面的人类评估之间的人类评估之间存在显着差距。这些发现表明了对话摘要的困难，并表明需要更细粒度的评估指标。

The diverse demands of different summarization tasks and their high annotation costs are driving a need for few-shot summarization. However, despite the emergence of many summarization tasks and datasets, the current training paradigm for few-shot summarization systems ignores potentially shareable knowledge in heterogeneous datasets. To this end, we propose \textsc{UniSumm}, a unified few-shot summarization model pre-trained with multiple summarization tasks and can be prefix-tuned to excel at any few-shot summarization datasets. Meanwhile, to better evaluate few-shot summarization systems, under the principles of diversity and robustness, we assemble and publicize a new benchmark \textsc{SummZoo}. It consists of $8$ diverse summarization tasks with multiple sets of few-shot samples for each task, covering both monologue and dialogue domains. Experimental results and ablation studies show that \textsc{UniSumm} outperforms strong baseline systems by a large margin across all tasks in \textsc{SummZoo} under both automatic and human evaluations. We release our code and benchmark at \url{https://github.com/microsoft/UniSumm}.

Content-Controllable Summarization generates summaries focused on the given controlling signals. Due to the lack of large-scale training corpora for the task, we propose a plug-and-play module RelAttn to adapt any general summarizers to the content-controllable summarization task. RelAttn first identifies the relevant content in the source documents, and then makes the model attend to the right context by directly steering the attention weight. We further apply an unsupervised online adaptive parameter searching algorithm to determine the degree of control in the zero-shot setting, while such parameters are learned in the few-shot setting. By applying the module to three backbone summarization models, experiments show that our method effectively improves all the summarizers, and outperforms the prefix-based method and a widely used plug-and-play model in both zero- and few-shot settings. Tellingly, more benefit is observed in the scenarios when more control is needed.

Prompts with different control signals (e.g., length, keywords, etc.) can be used to control text summarization. When control signals are available, they can control the properties of generated summaries and potentially improve summarization quality (since more information are given). Unfortunately, control signals are not already available during inference time. In this paper, we propose Lotus (shorthand for Latent Prompt Tuning for Summarization), which is a single model that can be applied in both controlled and uncontrolled (without control signals) modes. During training, Lotus learns latent prompt representations from prompts with gold control signals using a contrastive learning objective. Experiments show Lotus in uncontrolled mode consistently improves upon strong (uncontrollable) summarization models across four different summarization datasets. We also demonstrate generated summaries can be controlled using prompts with user specified control tokens.

Human evaluation is the foundation upon which the evaluation of both summarization systems and automatic metrics rests. However, existing human evaluation protocols and benchmarks for summarization either exhibit low inter-annotator agreement or lack the scale needed to draw statistically significant conclusions, and an in-depth analysis of human evaluation is lacking. In this work, we address the shortcomings of existing summarization evaluation along the following axes: 1) We propose a modified summarization salience protocol, Atomic Content Units (ACUs), which relies on fine-grained semantic units and allows for high inter-annotator agreement. 2) We curate the Robust Summarization Evaluation (RoSE) benchmark, a large human evaluation dataset consisting of over 22k summary-level annotations over state-of-the-art systems on three datasets. 3) We compare our ACU protocol with three other human evaluation protocols, underscoring potential confounding factors in evaluation setups. 4) We evaluate existing automatic metrics using the collected human annotations across evaluation protocols and demonstrate how our benchmark leads to more statistically stable and significant results. Furthermore, our findings have important implications for evaluating large language models (LLMs), as we show that LLMs adjusted by human feedback (e.g., GPT-3.5) may overfit unconstrained human evaluation, which is affected by the annotators' prior, input-agnostic preferences, calling for more robust, targeted evaluation methods.

生成摘要中的事实不一致严重限制了抽象对话摘要的实际应用。尽管通过使用预先训练的模型实现了显着进展，但在人类评估期间发现了大量的幻觉含量。预先接受的模型最常见的是微调文本摘要的跨熵损失，这可能不是最佳策略。在这项工作中，我们为带注释数据提供了事实错误的类型，以突出显示错误的类型并远离对事实的二进制了解。我们进一步提出了一种培训策略，通过新颖的对比微调，改善了摘要的事实一致性和整体素质。基于我们的语言信息的错误类型，我们设计了各个目标的不同模块化目标。具体而言，我们利用硬阴性样本具有误差，以减少事实不一致的产生。为了捕获扬声器之间的关键信息，我们还设计了特定于对话的损失。使用人类评估和自动忠实度量指标，我们表明我们的模型在对话摘要，Samsum语料库中大大降低了各种事实错误。此外，我们的模型可以推广到会议概述，AMI语料库，它产生的分数明显高于两个数据集关于单词 - 重叠度量标准的基线。

A primary objective of news articles is to establish the factual record for an event, frequently achieved by conveying both the details of the specified event (i.e., the 5 Ws; Who, What, Where, When and Why regarding the event) and how people reacted to it (i.e., reported statements). However, existing work on news summarization almost exclusively focuses on the event details. In this work, we propose the novel task of summarizing the reactions of different speakers, as expressed by their reported statements, to a given event. To this end, we create a new multi-document summarization benchmark, SUMREN, comprising 745 summaries of reported statements from various public figures obtained from 633 news articles discussing 132 events. We propose an automatic silver training data generation approach for our task, which helps smaller models like BART achieve GPT-3 level performance on this task. Finally, we introduce a pipeline-based framework for summarizing reported speech, which we empirically show to generate summaries that are more abstractive and factual than baseline query-focused summarization approaches.

诸如学术文章和商业报告之类的长期文件一直是详细说明重要问题和需要额外关注的复杂主题的标准格式。自动汇总系统可以有效地将长文档置于简短而简洁的文本中，以封装最重要的信息，从而在帮助读者的理解中很重要。最近，随着神经体系结构的出现，已经做出了重大的研究工作，以推动自动文本摘要系统，以及有关将这些系统扩展到长期文档领域的挑战的大量研究。在这项调查中，我们提供了有关长期文档摘要的研究的全面概述，以及其研究环境的三个主要组成部分的系统评估：基准数据集，汇总模型和评估指标。对于每个组成部分，我们在长期汇总的背景下组织文献，并进行经验分析，以扩大有关当前研究进度的观点。实证分析包括一项研究基准数据集的内在特征，摘要模型的多维分析以及摘要评估指标的综述。根据总体发现，我们通过提出可能在这个快速增长的领域中提出未来探索的方向来得出结论。

输出长度对于对话摘要系统至关重要。对话摘要长度由多个因素决定，包括对话复杂性，摘要目标和个人偏好。在这项工作中，我们从三个角度来对话摘要长度。首先，我们分析了现有模型的输出与相应的人类参考之间的长度差异，并发现摘要模型由于其预训练的目标而倾向于产生更多的详细摘要。其次，我们通过比较不同的模型设置来确定摘要长度预测的显着特征。第三，我们尝试使用长度意识的摘要，并在现有模型上显示出显着改进，如果汇总长度可以很好地整合。分析和实验是在流行的对话和Samsum数据集中进行的，以验证我们的发现。

Information overloading requires the need for summarizers to extract salient information from the text. Currently, there is an overload of dialogue data due to the rise of virtual communication platforms. The rise of Covid-19 has led people to rely on online communication platforms like Zoom, Slack, Microsoft Teams, Discord, etc. to conduct their company meetings. Instead of going through the entire meeting transcripts, people can use meeting summarizers to select useful data. Nevertheless, there is a lack of comprehensive surveys in the field of meeting summarizers. In this survey, we aim to cover recent meeting summarization techniques. Our survey offers a general overview of text summarization along with datasets and evaluation metrics for meeting summarization. We also provide the performance of each summarizer on a leaderboard. We conclude our survey with different challenges in this domain and potential research opportunities for future researchers.

当前有效的微调方法（例如，适配器，前缀调整等）通过培训一小组神经语言模型的额外参数进行优化的条件文本生成，同时冻结其余效率。虽然在某些一代任务中显示出强大表现，但它们不会概括所有一代任务。在这项工作中，我们表明可以提高基于迅速的条件文本生成，简单而有效的方法模拟了人类书面文本的话语结构建模。我们介绍了两个关键设计选择：首先，我们表明人写文本的更高级别的话语结构可以用前缀参数上的\ Textit {分层阻塞}建模，使得能够跨越输入和输出文本的不同部分，并产生更长度的输出几代人。其次，我们通过在网络上的不同层的前缀参数上引入\ texit {注意稀疏性}来提出稀疏的前缀调整，并分别学习SoftMax函数上的稀疏变换。我们发现稀疏的注意力使前缀调整能够更好地控制输入内容（突出事实），从而更有效地调整前缀参数。在各种文本生成任务上的实验表明，前缀参数的结构化设计可以实现可比的结果，以微调所有参数，同时即使在低资源设置中也表现出所有生成任务的标准前缀调整。

Abstractive dialogue summarization has received increasing attention recently. Despite the fact that most of the current dialogue summarization systems are trained to maximize the likelihood of human-written summaries and have achieved significant results, there is still a huge gap in generating high-quality summaries as determined by humans, such as coherence and faithfulness, partly due to the misalignment in maximizing a single human-written summary. To this end, we propose to incorporate different levels of human feedback into the training process. This will enable us to guide the models to capture the behaviors humans care about for summaries. Specifically, we ask humans to highlight the salient information to be included in summaries to provide the local feedback , and to make overall comparisons among summaries in terms of coherence, accuracy, coverage, concise and overall quality, as the global feedback. We then combine both local and global feedback to fine-tune the dialog summarization policy with Reinforcement Learning. Experiments conducted on multiple datasets demonstrate the effectiveness and generalization of our methods over the state-of-the-art supervised baselines, especially in terms of human judgments.

GPT-3等模型的零和少量提示的最新成功导致了NLP研究的范式转移。在本文中，我们研究了其对文本摘要的影响，重点是新闻摘要的经典基准领域。首先，我们研究了零击GPT-3与在大型摘要数据集中训练的微调模型的比较。我们表明，不仅人类压倒性地更喜欢GPT-3摘要，而且这些摘要也不遭受普通数据集特异性问题（例如事实差的问题）。接下来，我们研究这对评估意味着什么，尤其是黄金标准测试集的作用。我们的实验表明，基于参考和无参考的自动指标，例如最近提出的基于质量检查或基于质量的事实方法无法可靠地评估零击摘要。最后，我们讨论了未来的研究挑战，除了通用摘要之外，特别是基于关键字和方面的摘要，表明了优势微调方法与零拍的提示相比如何。为了支持进一步的研究，我们发布：（a）在4个标准摘要基准中，从微调和零摄像模型中产生的10K生成的摘要，（b）1K人类偏好判断和比较不同系统的普通系统，以进行通用和关键字的不同系统。基于摘要。

In long document controllable summarization, where labeled data is scarce, pretrained models struggle to adapt to the task and effectively respond to user queries. In this paper, we introduce Socratic pretraining, a question-driven, unsupervised pretraining objective specifically designed to improve controllability in summarization tasks. By training a model to generate and answer relevant questions in a given context, Socratic pretraining enables the model to more effectively adhere to user-provided queries and identify relevant content to be summarized. We demonstrate the effectiveness of this approach through extensive experimentation on two summarization domains, short stories and dialogue, and multiple control strategies: keywords, questions, and factoid QA pairs. Our pretraining method relies only on unlabeled documents and a question generation system and outperforms pre-finetuning approaches that use additional supervised data. Furthermore, our results show that Socratic pretraining cuts task-specific labeled data requirements in half, is more faithful to user-provided queries, and achieves state-of-the-art performance on QMSum and SQuALITY.

主题控制的摘要是一个具有广泛潜在应用的新兴研究领域。但是，现有方法受到重大局限性。首先，目前尚无针对此任务的确定评估指标。此外，现有的方法基于经常性架构，与最新的基于变压器的架构相比，这可能会大大限制其性能，同时它们还需要对模型的架构进行修改以控制主题。在这项工作中，我们提出了一种新的面向主题的评估措施，以根据生成的摘要与所需主题之间的主题亲和力自动评估生成的摘要。我们还进行了一项用户研究，以验证该措施的可靠性。最后，我们提出了简单而有力的方法，用于将主题控制的摘要要么将主题嵌入到模型的体系结构中，要么采用控制令牌来指导摘要生成。实验结果表明，与更复杂的基于嵌入的方法相比，对照令牌可以实现更好的性能，同时更快。

Narrative summarization aims to produce a distilled version of a narrative to describe its most salient events and characters. Summarizing a narrative is challenging as it requires an understanding of event causality and character behaviors. To encourage research in this direction, we propose NarraSum, a large-scale narrative summarization dataset. It contains 122K narrative documents, which are collected from plot descriptions of movies and TV episodes with diverse genres, and their corresponding abstractive summaries. Experiments show that there is a large performance gap between humans and the state-of-the-art summarization models on NarraSum. We hope that this dataset will promote future research in summarization, as well as broader studies of natural language understanding and generation. The dataset is available at https://github.com/zhaochaocs/narrasum.

Aspect or query-based summarization has recently caught more attention, as it can generate differentiated summaries based on users' interests. However, the current dataset for aspect or query-based summarization either focuses on specific domains, contains relatively small-scale instances, or includes only a few aspect types. Such limitations hinder further explorations in this direction. In this work, we take advantage of crowd-sourcing knowledge on Wikipedia.org and automatically create a high-quality, large-scale open-domain aspect-based summarization dataset named OASum, which contains more than 3.7 million instances with around 1 million different aspects on 2 million Wikipedia pages. We provide benchmark results on OAsum and demonstrate its ability for diverse aspect-based summarization generation. To overcome the data scarcity problem on specific domains, we also perform zero-shot, few-shot, and fine-tuning on seven downstream datasets. Specifically, zero/few-shot and fine-tuning results show that the model pre-trained on our corpus demonstrates a strong aspect or query-focused generation ability compared with the backbone model. Our dataset and pre-trained checkpoints are publicly available.

文本生成的广泛使用的评估指标要么与更长的文本效果不错，要么无法评估文本质量的所有方面。在本文中，我们引入了一个名为SMART的新指标，以减轻此类限制。具体而言，我们将句子视为匹配的基本单位，而不是代币，并使用句子匹配函数来匹配匹配候选和参考句子。还将候选句子与源文件中的句子进行了比较，以允许接地（例如，事实）评估。我们的结果表明，我们提出的指标与基于模型的匹配函数的系统级相关性优于萨姆瓦尔摘要元评估数据集上的所有竞争指标指标。后者不使用任何神经模型，这在模型开发阶段很有用，在这些阶段，资源可以受到限制且需要快速评估。最后，我们还进行了广泛的分析，表明我们提出的指标与较长的摘要很好地运行，并且对特定模型的偏见较小。