Temporal reasoning is the task of predicting temporal relations of event pairs with corresponding contexts. While some temporal reasoning models perform reasonably well on in-domain benchmarks, we have little idea of the systems' generalizability due to existing datasets' limitations. In this work, we introduce a novel task named TODAY that bridges this gap with temporal differential analysis, which as the name suggests, evaluates if systems can correctly understand the effect of incremental changes. Specifically, TODAY makes slight context changes for given event pairs, and systems need to tell how this subtle contextual change will affect temporal relation distributions. To facilitate learning, TODAY also annotates human explanations. We show that existing models, including GPT-3, drop to random guessing on TODAY, suggesting that they heavily rely on spurious information rather than proper reasoning for temporal predictions. On the other hand, we show that TODAY's supervision style and explanation annotations can be used in joint learning and encourage models to use more appropriate signals during training and outperform across several benchmarks. TODAY can also be used to train models to solicit incidental supervision from noisy sources such as GPT-3 and moves farther towards generic temporal reasoning systems.

大型语言模型越来越能够通过相对较少的特定任务的监督产生流畅的出现文本。但这些模型可以准确解释分类决策吗？我们考虑使用少量人写的例子（即，以几滴方式）生成自由文本解释的任务。我们发现（1）创作更高质量的例子，以提示导致更高质量的世代; （2）令人惊讶的是，在头到头比较中，人群公司通常更喜欢GPT-3生成的解释，以众包中包含的人性写入的解释。然而，Crowdworker评级也表明，虽然模型产生了事实，语法和充分的解释，但它们具有改进的空间，例如沿着提供新颖信息和支持标签的轴。我们创建了一种管道，该管道将GPT-3与监督过滤器结合起来，该过滤器通过二进制可接受性判断来包含人类循环。尽管具有重要的主观性内在的判断可接受性，但我们的方法能够始终如一地过滤人类可接受的GPT-3生成的解释。

众包NLP数据集的反复挑战是，在制作示例时，人类作家通常会依靠重复的模式，从而导致缺乏语言多样性。我们介绍了一种基于工人和AI协作的数据集创建的新方法，该方法汇集了语言模型的生成力量和人类的评估力量。从现有的数据集，自然语言推理（NLI）的Multinli开始，我们的方法使用数据集制图自动识别示例来证明具有挑战性的推理模式，并指示GPT-3撰写具有相似模式的新示例。然后，机器生成的示例会自动过滤，并最终由人类人群工人修订和标记。最终的数据集Wanli由107,885个NLI示例组成，并在现有的NLI数据集上呈现出独特的经验优势。值得注意的是，培训有关Wanli的模型，而不是Multinli（$ 4 $ $倍）可改善我们考虑的七个外域测试集的性能，包括汉斯（Hans）的11％和对抗性NLI的9％。此外，将Multinli与Wanli结合起来比将其与其他NLI增强集相结合更有效。我们的结果表明，自然语言生成技术的潜力是策划增强质量和多样性的NLP数据集。

预测任务标签和为其预测生成自由文本阐述的自律化模型可以实现与NLP系统更直观的交互。然而，这些模型目前正在接受大量人为的自由文本解释，每个任务都会阻碍更广泛的使用。我们建议使用少数培训例子研究更现实的自律化建立。我们出示2月 - 一个标准化的四个现有英语数据集和相关指标。我们通过2月份广泛探索自然语言提示来确定正确的提示方法。然后，通过使用此提示并缩放模型大小，我们证明了几次拍摄自合合理化的进展。我们展示了这项任务的完善房间仍然有充足的改进空间：人类注册人评估的生成解释的平均合理性最多为51％，而人类解释的合理性是76％。我们希望2月份与我们的拟议方法一起促使社区承担几次拍摄的自我合理化挑战。

Recent methods demonstrate that data augmentation using counterfactual knowledge can teach models the causal structure of a task, leading to robust and generalizable models. However, such counterfactual data often has a limited scale and diversity if crowdsourced and is computationally expensive to extend to new perturbation types if generated using supervised methods. To address this, we introduce a new framework called DISCO for automatically generating high-quality counterfactual data at scale. DISCO engineers prompts to generate phrasal perturbations with a large general language model. Then, a task-specific teacher model filters the generation to distill high-quality counterfactual data. We show that learning with this counterfactual data yields a comparatively small student model that is 6% (absolute) more robust and generalizes 5% better across distributions than baselines on various challenging evaluations. This model is also 15% more sensitive in differentiating original and counterfactual examples, on three evaluation sets written by human workers and via human-AI collaboration.

在有问题的回答需要常识的问题上，语言模型（例如，GPT-3）已用于生成表达有助于提高性能的背景知识的文本。然而，使用此类模型的成本很高。在这项工作中，我们对较小的语言模型产生有用的中间上下文，此处称为阐述。我们的框架在更新两个语言模型之间交替使用 - 阐述生成器和一个答案预测变量 - 允许每个语言都影响彼此。我们的模型使用少于GPT-3的参数的0.5％优于具有相似尺寸的替代方案，并在四个常识性问题上回答基准测试的GPT-3上的差距缩小。人类评估表明，生成的阐述的质量很高。

对事件序列的预测对于信息检索和自然语言处理中的许多现实世界应用至关重要。在事件序列预测中，未来的活动生成（FEG）是一项具有挑战性的任务，因为它不仅需要流利的文本生成，而且需要常识性推理才能保持整个事件故事的逻辑连贯性。在本文中，我们提出了一个新颖的可解释的FEG框架COEP。它突出并整合了两种类型的事件知识，对直接事件事件关系的顺序知识以及推论知识，这些知识反映了事件之间的中间角色心理学（例如意图，原因，反应），这些心理本质地将故事推向了故事。为了减轻知识遗忘问题，我们为每种类型的知识设计了两个模块，即IM和GM，它们是通过及时调整组合的。首先，IM专注于理解推论知识，以产生常识性解释并为通用汽车提供软提示向量。我们还设计了一种对比歧视器，以提高概括能力。其次，GM通过用IM的指导对直接顺序知识进行建模来生成未来事件。自动和人类评估表明，我们的方法可以产生更连贯，具体和逻辑的未来事件。

This paper investigates models of event implications. Specifically, how well models predict entity state-changes, by targeting their understanding of physical attributes. Nominally, Large Language models (LLM) have been exposed to procedural knowledge about how objects interact, yet our benchmarking shows they fail to reason about the world. Conversely, we also demonstrate that existing approaches often misrepresent the surprising abilities of LLMs via improper task encodings and that proper model prompting can dramatically improve performance of reported baseline results across multiple tasks. In particular, our results indicate that our prompting technique is especially useful for unseen attributes (out-of-domain) or when only limited data is available.

Instruction tuning enables pretrained language models to perform new tasks from inference-time natural language descriptions. These approaches rely on vast amounts of human supervision in the form of crowdsourced datasets or user interactions. In this work, we introduce Unnatural Instructions: a large dataset of creative and diverse instructions, collected with virtually no human labor. We collect 64,000 examples by prompting a language model with three seed examples of instructions and eliciting a fourth. This set is then expanded by prompting the model to rephrase each instruction, creating a total of approximately 240,000 examples of instructions, inputs, and outputs. Experiments show that despite containing a fair amount of noise, training on Unnatural Instructions rivals the effectiveness of training on open-source manually-curated datasets, surpassing the performance of models such as T0++ and Tk-Instruct across various benchmarks. These results demonstrate the potential of model-generated data as a cost-effective alternative to crowdsourcing for dataset expansion and diversification.

我们提出了一种可解释的关系提取方法，通过共同训练这两个目标来减轻概括和解释性之间的张力。我们的方法使用多任务学习体系结构，该体系结构共同训练分类器以进行关系提取，并在解释关系分类器的决策的关系中标记单词的序列模型。我们还将模型输出转换为规则，以将全局解释带入这种方法。使用混合策略对此序列模型进行训练：有监督，当可获得预先存在的模式的监督时，另外还要半监督。在后一种情况下，我们将序列模型的标签视为潜在变量，并学习最大化关系分类器性能的最佳分配。我们评估了两个数据集中的提议方法，并表明序列模型提供了标签，可作为关系分类器决策的准确解释，并且重要的是，联合培训通常可以改善关系分类器的性能。我们还评估了生成的规则的性能，并表明新规则是手动规则的重要附加功能，并使基于规则的系统更接近神经模型。

相同上下文的可能后果可能会因我们所指的情况而异。但是，当前在自然语言处理中的研究并不集中于多种可能情况下的常识性推理。本研究通过短篇小说文字提出与候选人答案相同的结尾的多个问题来构成这项任务。我们由此产生的数据集，可能的故事，包括超过1.3k的故事文本超过4.5k的问题。我们发现，即使是目前的强训练性语言模型也很难始终如一地回答问题，这强调了无监督环境中最高的准确性（60.2％）远远落后于人类准确性（92.5％）。通过与现有数据集进行比较，我们观察到数据集中的问题包含答案选项中的最小注释伪像。此外，我们的数据集还包括需要反事实推理的示例，以及需要读者的反应和虚构信息的示例，这表明我们的数据集可以作为对未来常识性推理的未来研究的挑战性测试。

The common practice for training commonsense models has gone from-human-to-corpus-to-machine: humans author commonsense knowledge graphs in order to train commonsense models. In this work, we investigate an alternative, from-machine-to-corpus-to-machine: general language models author these commonsense knowledge graphs to train commonsense models. Our study leads to a new framework, Symbolic Knowledge Distillation. As with prior art in Knowledge Distillation (Hinton et al., 2015), our approach uses larger models to teach smaller models. A key difference is that we distill knowledge symbolically-as text-in addition to the neural model. We also distill only one aspect-the commonsense of a general language model teacher, allowing the student to be a different type, a commonsense model. Altogether, we show that careful prompt engineering and a separately trained critic model allow us to selectively distill high-quality causal commonsense from GPT-3, a general language model. Empirical results demonstrate that, for the first time, a human-authored commonsense knowledge graph is surpassed by our automatically distilled variant in all three criteria: quantity, quality, and diversity. In addition, it results in a neural commonsense model that surpasses the teacher model's commonsense capabilities despite its 100x smaller size. We apply this to the ATOMIC resource, and share our new symbolic knowledge graph and commonsense models.

Natural Language Generation (NLG) has improved exponentially in recent years thanks to the development of sequence-to-sequence deep learning technologies such as Transformer-based language models. This advancement has led to more fluent and coherent NLG, leading to improved development in downstream tasks such as abstractive summarization, dialogue generation and data-to-text generation. However, it is also apparent that deep learning based generation is prone to hallucinate unintended text, which degrades the system performance and fails to meet user expectations in many real-world scenarios. To address this issue, many studies have been presented in measuring and mitigating hallucinated texts, but these have never been reviewed in a comprehensive manner before. In this survey, we thus provide a broad overview of the research progress and challenges in the hallucination problem in NLG. The survey is organized into two parts: (1) a general overview of metrics, mitigation methods, and future directions; and (2) an overview of task-specific research progress on hallucinations in the following downstream tasks, namely abstractive summarization, dialogue generation, generative question answering, data-to-text generation, machine translation, and visual-language generation. This survey serves to facilitate collaborative efforts among researchers in tackling the challenge of hallucinated texts in NLG.

我们研究了检查问题的事实，旨在识别给定索赔的真实性。具体而言，我们专注于事实提取和验证（发烧）及其伴随数据集的任务。该任务包括从维基百科检索相关文件（和句子）并验证文件中的信息是否支持或驳斥所索赔的索赔。此任务至关重要，可以是假新闻检测和医疗索赔验证等应用程序块。在本文中，我们以通过以结构化和全面的方式呈现文献来更好地了解任务的挑战。我们通过分析不同方法的技术视角并讨论发热数据集的性能结果，描述了所提出的方法，这是最熟悉的和正式结构化的数据集，就是事实提取和验证任务。我们还迄今为止迄今为止确定句子检索组件的有益损失函数的最大实验研究。我们的分析表明，采样负句对于提高性能并降低计算复杂性很重要。最后，我们描述了开放的问题和未来的挑战，我们激励了未来的任务研究。

可解释的NLP（EXNLP）越来越关注收集人类注释的文本解释。这些解释在三种方面使用下游：作为数据增强，以提高预测任务的性能，因为对培训模型的监督，为他们的预测产生解释，以及评估模型生成的解释的理论。在本次审查中，我们识别65个具有三个主要类别的文本解释的数据集（突出显示，自由文本和结构），组织关于注释每种类型的文献，识别现有收集方法的优势和缺点，并为收集EXNLP数据集提供建议在将来。

众所周知，端到端的神经NLP体系结构很难理解，这引起了近年来为解释性建模的许多努力。模型解释的基本原则是忠诚，即，解释应准确地代表模型预测背后的推理过程。这项调查首先讨论了忠诚的定义和评估及其对解释性的意义。然后，我们通过将方法分为五类来介绍忠实解释的最新进展：相似性方法，模型内部结构的分析，基于反向传播的方法，反事实干预和自我解释模型。每个类别将通过其代表性研究，优势和缺点来说明。最后，我们从它们的共同美德和局限性方面讨论了上述所有方法，并反思未来的工作方向忠实的解释性。对于有兴趣研究可解释性的研究人员，这项调查将为该领域提供可访问且全面的概述，为进一步探索提供基础。对于希望更好地了解自己的模型的用户，该调查将是一项介绍性手册，帮助选择最合适的解释方法。

In many task settings, text classification models are likely to encounter examples from novel classes on which they cannot predict correctly. Selective prediction, in which models abstain on low-confidence examples, provides a possible solution, but existing models are often overly confident on OOD examples. To remedy this overconfidence, we introduce Contrastive Novelty-Augmented Learning (CoNAL), a two-step method that generates OOD examples representative of novel classes, then trains to decrease confidence on them. First, we generate OOD examples by prompting a large language model twice: we prompt it to enumerate relevant novel labels, then generate examples from each novel class matching the task format. Second, we train our classifier with a novel contrastive objective that encourages lower confidence on generated OOD examples than training examples. When trained with CoNAL, classifiers improve in their ability to detect and abstain on OOD examples over prior methods by an average of 2.3% AUAC and 5.5% AUROC across 4 NLP datasets, with no cost to in-distribution accuracy.

Two key obstacles in biomedical relation extraction (RE) are the scarcity of annotations and the prevalence of instances without explicitly pre-defined labels due to low annotation coverage. Existing approaches, which treat biomedical RE as a multi-class classification task, often result in poor generalization in low-resource settings and do not have the ability to make selective prediction on unknown cases but give a guess from seen relations, hindering the applicability of those approaches. We present NBR, which converts biomedical RE as natural language inference formulation through indirect supervision. By converting relations to natural language hypotheses, NBR is capable of exploiting semantic cues to alleviate annotation scarcity. By incorporating a ranking-based loss that implicitly calibrates abstinent instances, NBR learns a clearer decision boundary and is instructed to abstain on uncertain instances. Extensive experiments on three widely-used biomedical RE benchmarks, namely ChemProt, DDI and GAD, verify the effectiveness of NBR in both full-set and low-resource regimes. Our analysis demonstrates that indirect supervision benefits biomedical RE even when a domain gap exists, and combining NLI knowledge with biomedical knowledge leads to the best performance gains.

叙事中的事件可以通过其参与者的基本状态理解为一致的整体。通常，这些参与者在叙述中没有明确提及，而是通过常识性或推论填写。理解叙述的模型应该能够推断出这些隐性参与者状态，以及有关这些状态对叙事的影响的原因。为了促进这一目标，我们介绍了一个新的众包参与者指出的数据集意大利面。该数据集包含有效的，可推断的参与者状态；对国家的反事实扰动；如果反事实是真实的，那么故事的变化将是必要的。我们介绍了三项基于州的推理任务，这些任务测试了一个故事何时由故事启用，修改一个反事实状态的故事，并解释给定经过修订的故事的最有可能的状态变化。我们的基准测试实验表明，尽管当今的LLM能够在某种程度上推理有关州的推理，但仍有很大的改进空间，这表明了未来研究的潜在途径。

As large language models (LLMs) grow larger and more sophisticated, assessing their "reasoning" capabilities in natural language grows more challenging. Recent question answering (QA) benchmarks that attempt to assess reasoning are often limited by a narrow scope of covered situations and subject matters. We introduce WikiWhy, a QA dataset built around a novel auxiliary task: explaining why an answer is true in natural language. WikiWhy contains over 9,000 "why" question-answer-rationale triples, grounded on Wikipedia facts across a diverse set of topics. Each rationale is a set of supporting statements connecting the question to the answer. WikiWhy serves as a benchmark for the reasoning capabilities of LLMs because it demands rigorous explicit rationales for each answer to demonstrate the acquisition of implicit commonsense knowledge, which is unlikely to be easily memorized. GPT-3 baselines achieve only 38.7% human-evaluated correctness in the end-to-end answer & explain condition, leaving significant room for future improvements.