自动摘要方法是有效的，但可能患有低质量。相比之下，手动摘要很昂贵，但质量更高。人类和人工智能可以协作以提高总结性能吗？在类似的文本生成任务（例如机器翻译）中，人类AI合作的形式是“后编辑” AI生成的文本，可减少人类的工作量并提高AI输出的质量。因此，我们探讨了邮政编辑是否提供文本摘要中的优势。具体来说，我们对72名参与者进行了实验，将提供的后编辑摘要与手动摘要进行了摘要，以摘要质量，人为效率和用户在正式新闻（XSUM新闻）和非正式（REDDIT帖子）文本方面进行了比较。这项研究对何时编辑的文本摘要提供了宝贵的见解：在某些情况下（例如，何时参与者缺乏领域知识），但在其他情况下却没有帮助（例如，何时提供的摘要包括不准确的信息）。参与者的不同编辑策略和援助需求为未来的人类摘要系统提供了影响。

Many real-world applications of language models (LMs), such as code autocomplete and writing assistance, involve human-LM interaction, but the main LM benchmarks are non-interactive, where a system produces output without human intervention. To evaluate human-LM interaction, we develop a framework, Human-AI Language-based Interaction Evaluation (H-LINE), that expands non-interactive evaluation along three dimensions, capturing (i) the interactive process, not only the final output; (ii) the first-person subjective experience, not just a third-party assessment; and (iii) notions of preference beyond quality. We then design five tasks ranging from goal-oriented to open-ended to capture different forms of interaction. On four state-of-the-art LMs (three variants of OpenAI's GPT-3 and AI21's J1-Jumbo), we find that non-interactive performance does not always result in better human-LM interaction and that first-person and third-party metrics can diverge, suggesting the importance of examining the nuances of human-LM interaction.

随着近期自然语言生成（NLG）模型的各种应用程序的改进，它变得必须具有识别和评估NLG输出是否仅共享关于外部世界的可验证信息的手段。在这项工作中，我们提出了一个归属于识别的来源（AIS）的新评估框架，用于评估自然语言生成模型的输出，当这种输出涉及外部世界时。我们首先定义AIS，并引入两级注释管道，用于允许注释器根据AIS指南适当地评估模型输出。通过人为评估研究，我们在三个代数据集（会话QA域中的两个中和总结一下，概括地验证了这种方法，表明AIS可以作为测量模型生成的语句是否支持基础来源的常见框架。我们释放人类评估研究指南。

随着大型语言模型的出现，抽象性摘要的方法取得了长足的进步，从而在应用程序中使用了帮助知识工人处理笨拙的文档收集的潜力。一个这样的环境是民权诉讼交换所（CRLC）（https://clearinghouse.net），其中发布了有关大规模民权诉讼，服务律师，学者和公众的信息。如今，CRLC中的摘要需要对律师和法律专业的学生进行广泛的培训，这些律师和法律专业的学生花费数小时了解多个相关文件，以便产生重要事件和结果的高质量摘要。在这种持续的现实世界摘要工作的激励下，我们引入了Multi-iplesum，这是由正在进行的CRLC写作中绘制的9,280个专家作者的摘要集。鉴于源文档的长度，多文章介绍了一个具有挑战性的多文档摘要任务，通常每个情况超过200页。此外，多胎sum与其多个目标摘要中的其他数据集不同，每个数据集都处于不同的粒度（从一句“极端”摘要到超过五百个单词的多段落叙述）。我们提供了广泛的分析，表明，尽管培训数据（遵守严格的内容和样式准则）中的摘要很高，但最新的摘要模型在此任务上的表现较差。我们发布了多体式的摘要方法，以及促进应用程序的开发，以协助CRLC的任务https://multilexsum.github.io。

GPT-3等模型的零和少量提示的最新成功导致了NLP研究的范式转移。在本文中，我们研究了其对文本摘要的影响，重点是新闻摘要的经典基准领域。首先，我们研究了零击GPT-3与在大型摘要数据集中训练的微调模型的比较。我们表明，不仅人类压倒性地更喜欢GPT-3摘要，而且这些摘要也不遭受普通数据集特异性问题（例如事实差的问题）。接下来，我们研究这对评估意味着什么，尤其是黄金标准测试集的作用。我们的实验表明，基于参考和无参考的自动指标，例如最近提出的基于质量检查或基于质量的事实方法无法可靠地评估零击摘要。最后，我们讨论了未来的研究挑战，除了通用摘要之外，特别是基于关键字和方面的摘要，表明了优势微调方法与零拍的提示相比如何。为了支持进一步的研究，我们发布：（a）在4个标准摘要基准中，从微调和零摄像模型中产生的10K生成的摘要，（b）1K人类偏好判断和比较不同系统的普通系统，以进行通用和关键字的不同系统。基于摘要。

我们提出了一个文本编辑器，以帮助用户计划，结构并反思其写作过程。它使用自动文本摘要提供了不断更新的段落摘要作为边缘注释。摘要级别范围从全文到选定的（中央）句子，一直到关键字的集合。为了了解用户在写作过程中如何与该系统进行交互，我们进行了两项用户研究（n = 4和n = 8），人们在其中写了有关给定主题和文章的分析文章。作为关键发现，这些摘要使用户对他们的写作有了外部视角，并帮助他们修改了草稿段落的内容和范围。人们进一步使用该工具快速获得文本概述，并制定了整合自动摘要中见解的策略。从更广泛的角度来看，这项工作探索并突出了为作家设计AI工具的价值，其自然语言处理（NLP）功能超出了直接文本生成和更正。

神经语言模型有可能支持人类写作。但是，关于其整合和对写作和产出的影响仍然存在问题。为了解决这个问题，我们设计并比较了两个用于写作的用户界面与移动设备上的AI，这些用户界面操纵主动性和控制级别：1）使用连续生成的文本编写，AI添加了逐字文字和用户转向。 2）编写建议，AI建议短语和用户从列表中选择。在监督的在线研究（n = 18）中，参与者使用了这些原型和无AI的基线。我们收集了触摸互动，关于灵感和作者的评分以及访谈数据。有了AI的建议，人们的写作不那么积极，但觉得他们是作者。连续生成的文本减少了这种感知的作者身份，但编辑行为增加了。在这两种设计中，AI都会增加文本长度，并被认为会影响措辞。我们的发现为UI设计决策对用户体验和共同创造系统的产出的影响增加了新的经验证据。

创意支持工具中的反馈可以帮助人群推动他们的意思。但是，目前的反馈方法需要从促进者或同行中进行人力评估。这不可扩展到大人群。我们提出可解释的定向多样性来自动预测观点的质量和多样性分数，并提供AI解释 - 归因，对比归因和反事实建议 - 反馈意见（低），以及如何获得更高的分数。由于用户迭代地提高其想象，这些解释提供了多面反馈。我们进行了形成性和控制的用户研究，以了解解释的使用和有用性，以提高观念多样性和质量。用户感谢解释反馈帮助重点努力，并提供了改进的方向。这导致解释与没有反馈或反馈仅具有预测的反馈和反馈相比提高了多样性。因此，我们的方法为解释和丰富的反馈开辟了可解释的AI的机会，以获得迭代人群思想和创造力支​​持工具。

Language models (LMs) now excel at many tasks such as few-shot learning, question answering, reasoning, and dialog. However, they sometimes generate unsupported or misleading content. A user cannot easily determine whether their outputs are trustworthy or not, because most LMs do not have any built-in mechanism for attribution to external evidence. To enable attribution while still preserving all the powerful advantages of recent generation models, we propose RARR (Retrofit Attribution using Research and Revision), a system that 1) automatically finds attribution for the output of any text generation model and 2) post-edits the output to fix unsupported content while preserving the original output as much as possible. When applied to the output of several state-of-the-art LMs on a diverse set of generation tasks, we find that RARR significantly improves attribution while otherwise preserving the original input to a much greater degree than previously explored edit models. Furthermore, the implementation of RARR requires only a handful of training examples, a large language model, and standard web search.

The proliferation of automatic faithfulness metrics for summarization has produced a need for benchmarks to evaluate them. While existing benchmarks measure the correlation with human judgements of faithfulness on model-generated summaries, they are insufficient for diagnosing whether metrics are: 1) consistent, i.e., decrease as errors are introduced into a summary, 2) effective on human-written texts, and 3) sensitive to different error types (as summaries can contain multiple errors). To address these needs, we present a benchmark of unfaithful minimal pairs (BUMP), a dataset of 889 human-written, minimally different summary pairs, where a single error (from an ontology of 7 types) is introduced to a summary from the CNN/DailyMail dataset to produce an unfaithful summary. We find BUMP complements existing benchmarks in a number of ways: 1) the summaries in BUMP are harder to discriminate and less probable under SOTA summarization models, 2) BUMP enables measuring the consistency of metrics, and reveals that the most discriminative metrics tend not to be the most consistent, 3) BUMP enables the measurement of metrics' performance on individual error types and highlights areas of weakness for future work.

在线众包平台使对算法输出进行评估变得容易，并提出诸如“哪个图像更好，A或B？”之类的问题的调查，在视觉和图形研究论文中的这些“用户研究”的扩散导致了增加匆忙进行的研究充其量是草率且无知的，并且可能有害和误导。我们认为，在计算机视觉和图形论文中的用户研究的设计和报告需要更多关注。为了提高从业者的知识并提高用户研究的可信度和可复制性，我们提供了用户体验研究（UXR），人类计算机互动（HCI）和相关领域的方法论的概述。我们讨论了目前在计算机视觉和图形研究中未利用的基础用户研究方法（例如，需要调查），但可以为研究项目提供宝贵的指导。我们为有兴趣探索其他UXR方法的读者提供了进一步的指导。最后，我们描述了研究界的更广泛的开放问题和建议。我们鼓励作者和审稿人都认识到，并非每项研究贡献都需要用户研究，而且根本没有研究比不小心进行的研究更好。

Recent developments in natural language generation (NLG) using neural language models have brought us closer than ever to the goal of building AI-powered creative writing tools. However, most prior work on human-AI collaboration in the creative writing domain has evaluated new systems with amateur writers, typically in contrived user studies of limited scope. In this work, we commissioned 13 professional, published writers from a diverse set of creative writing backgrounds to craft stories using Wordcraft, a text editor with built-in AI-powered writing assistance tools. Using interviews and participant journals, we discuss the potential of NLG to have significant impact in the creative writing domain--especially with respect to brainstorming, generation of story details, world-building, and research assistance. Experienced writers, more so than amateurs, typically have well-developed systems and methodologies for writing, as well as distinctive voices and target audiences. Our work highlights the challenges in building for these writers; NLG technologies struggle to preserve style and authorial voice, and they lack deep understanding of story contents. In order for AI-powered writing assistants to realize their full potential, it is essential that they take into account the diverse goals and expertise of human writers.

情绪分析中最突出的任务是为文本分配情绪，并了解情绪如何在语言中表现出来。自然语言处理的一个重要观察结果是，即使没有明确提及情感名称，也可以通过单独参考事件来隐式传达情绪。在心理学中，被称为评估理论的情感理论类别旨在解释事件与情感之间的联系。评估可以被形式化为变量，通过他们认为相关的事件的人们的认知评估来衡量认知评估。其中包括评估事件是否是新颖的，如果该人认为自己负责，是否与自己的目标以及许多其他人保持一致。这样的评估解释了哪些情绪是基于事件开发的，例如，新颖的情况会引起惊喜或不确定后果的人可能引起恐惧。我们在文本中分析了评估理论对情绪分析的适用性，目的是理解注释者是否可以可靠地重建评估概念，如果可以通过文本分类器预测，以及评估概念是否有助于识别情感类别。为了实现这一目标，我们通过要求人们发短信描述触发特定情绪并披露其评估的事件来编译语料库。然后，我们要求读者重建文本中的情感和评估。这种设置使我们能够衡量是否可以纯粹从文本中恢复情绪和评估，并为判断模型的绩效指标提供人体基准。我们将文本分类方法与人类注释者的比较表明，两者都可以可靠地检测出具有相似性能的情绪和评估。我们进一步表明，评估概念改善了文本中情绪的分类。

Incivility remains a major challenge for online discussion platforms, to such an extent that even conversations between well-intentioned users can often derail into uncivil behavior. Traditionally, platforms have relied on moderators to -- with or without algorithmic assistance -- take corrective actions such as removing comments or banning users. In this work we propose a complementary paradigm that directly empowers users by proactively enhancing their awareness about existing tension in the conversation they are engaging in and actively guides them as they are drafting their replies to avoid further escalation. As a proof of concept for this paradigm, we design an algorithmic tool that provides such proactive information directly to users, and conduct a user study in a popular discussion platform. Through a mixed methods approach combining surveys with a randomized controlled experiment, we uncover qualitative and quantitative insights regarding how the participants utilize and react to this information. Most participants report finding this proactive paradigm valuable, noting that it helps them to identify tension that they may have otherwise missed and prompts them to further reflect on their own replies and to revise them. These effects are corroborated by a comparison of how the participants draft their reply when our tool warns them that their conversation is at risk of derailing into uncivil behavior versus in a control condition where the tool is disabled. These preliminary findings highlight the potential of this user-centered paradigm and point to concrete directions for future implementations.

我们提出了一项探索性定性研究，以了解作家如何与下一页建议相互作用。尽管对建议系统对写作的影响进行了一些定量研究，但几乎没有定性的工作来理解作家如何与建议系统互动及其如何影响他们的写作过程 - 特别是针对非本地但英国作家的。我们进行了一项研究，要求业余作家分别写两部电影评论，一本没有建议。我们发现作家以各种复杂的方式与下一页建议互动 - 作家能够抽象建议的多个部分并将其纳入他们的写作中 - 即使他们不同意整个建议。建议系统对写作过程也有各种影响 - 以独特的方式为写作过程的不同方面做出了影响。我们提出了一种用于与GPT-2写作的作家 - 探索互动模型，用于电影评论写作任务，然后是该模型可用于未来研究的方式，并概述了研究和设计的机会。

The acquisition of high-quality human annotations through crowdsourcing platforms like Amazon Mechanical Turk (MTurk) is more challenging than expected. The annotation quality might be affected by various aspects like annotation instructions, Human Intelligence Task (HIT) design, and wages paid to annotators, etc. To avoid potentially low-quality annotations which could mislead the evaluation of automatic summarization system outputs, we investigate the recruitment of high-quality MTurk workers via a three-step qualification pipeline. We show that we can successfully filter out bad workers before they carry out the evaluations and obtain high-quality annotations while optimizing the use of resources. This paper can serve as basis for the recruitment of qualified annotators in other challenging annotation tasks.

为了实现长文档理解的构建和测试模型，我们引入质量，具有中文段的多项选择QA DataSet，具有约5,000个令牌的平均长度，比典型的当前模型更长。与经过段落的事先工作不同，我们的问题是由阅读整个段落的贡献者编写和验证的，而不是依赖摘要或摘录。此外，只有一半的问题是通过在紧缩时间限制下工作的注释器来应答，表明略读和简单的搜索不足以一直表现良好。目前的模型在此任务上表现不佳（55.4％），并且落后于人类性能（93.5％）。

Despite the recent progress in language generation models, their outputs may not always meet user expectations. In this work, we study whether informational feedback in natural language can be leveraged to improve generation quality and user preference alignment. To this end, we consider factual consistency in summarization, the quality that the summary should only contain information supported by the input documents, for user preference alignment. We collect a high-quality dataset, DeFacto, containing human demonstrations and informational feedback in natural language consisting of corrective instructions, edited summaries, and explanations with respect to the factual consistency of the summary. Using our dataset, we study two natural language generation tasks: 1) editing a summary using the human feedback, and 2) generating human feedback from the original summary. Using the two tasks, we further evaluate if models can automatically correct factual inconsistencies in generated summaries. We show that the human-edited summaries we collected are more factually consistent, and pre-trained language models can leverage our dataset to improve the factual consistency of original system-generated summaries in our proposed generation tasks. We make the DeFacto dataset publicly available at https://github.com/microsoft/DeFacto.

自动源代码摘要是一种任务，它生成有关用于对这些代码实体的理解的方法和类别的用于方法和类的总结信息。已经提出了多种方法和技术在规范摘要中进行监督和无监督学习，但是，它们主要集中在为一段代码生成摘要。此外，很少有效利用非官方文件。本文提出了一种自动和新的方法，总结了堆栈溢出中讨论的Android API方法，以便我们认为这项研究中的非官方文档。我们的方法将API方法的名称作为输入，并基于该API方法的堆栈溢出讨论生成自然语言摘要。我们已经进行了一项调查，涉及16个Android开发人员，以评估我们自动生成的摘要的质量，并将它们与官方Android文档进行比较。我们的结果表明，虽然开发人员在普通方面找到官方文件更有用的虽然，所产生的摘要也具有竞争力，特别是用于提供实施细节，并且可以用作指导软件开发和维护任务中开发人员的补充来源。

Human evaluation is the foundation upon which the evaluation of both summarization systems and automatic metrics rests. However, existing human evaluation protocols and benchmarks for summarization either exhibit low inter-annotator agreement or lack the scale needed to draw statistically significant conclusions, and an in-depth analysis of human evaluation is lacking. In this work, we address the shortcomings of existing summarization evaluation along the following axes: 1) We propose a modified summarization salience protocol, Atomic Content Units (ACUs), which relies on fine-grained semantic units and allows for high inter-annotator agreement. 2) We curate the Robust Summarization Evaluation (RoSE) benchmark, a large human evaluation dataset consisting of over 22k summary-level annotations over state-of-the-art systems on three datasets. 3) We compare our ACU protocol with three other human evaluation protocols, underscoring potential confounding factors in evaluation setups. 4) We evaluate existing automatic metrics using the collected human annotations across evaluation protocols and demonstrate how our benchmark leads to more statistically stable and significant results. Furthermore, our findings have important implications for evaluating large language models (LLMs), as we show that LLMs adjusted by human feedback (e.g., GPT-3.5) may overfit unconstrained human evaluation, which is affected by the annotators' prior, input-agnostic preferences, calling for more robust, targeted evaluation methods.