Question-answering datasets require a broad set of reasoning skills. We show how to use question decompositions to teach language models these broad reasoning skills in a robust fashion. Specifically, we use widely available QDMR representations to programmatically create hard-to-cheat synthetic contexts for real questions in six multi-step reasoning datasets. These contexts are carefully designed to avoid reasoning shortcuts prevalent in real contexts that prevent models from learning the right skills. This results in a pretraining dataset, named TeaBReaC, containing 525K multi-step questions (with associated formal programs) covering about 900 reasoning patterns. We show that pretraining standard language models (LMs) on TeaBReaC before fine-tuning them on target datasets improves their performance by up to 13 F1 points across 4 multi-step QA datasets, with up to 21 point gain on more complex questions. The resulting models also demonstrate higher robustness, with a 5-8 F1 point improvement on two contrast sets. Furthermore, TeaBReaC pretraining substantially improves model performance and robustness even when starting with numerate LMs pretrained using recent methods (e.g., PReasM, POET). Our work thus shows how to effectively use decomposition-guided contexts to robustly teach multi-step reasoning.

Answering complex questions that require making latent decisions is a challenging task, especially when limited supervision is available. Recent works leverage the capabilities of large language models (LMs) to perform complex question answering in a few-shot setting by demonstrating how to output intermediate rationalizations while solving the complex question in a single pass. We introduce ``Successive Prompting'', where we iteratively break down a complex task into a simple task, solve it, and then repeat the process until we get the final solution. Successive prompting decouples the supervision for decomposing complex questions from the supervision for answering simple questions, allowing us to (1) have multiple opportunities to query in-context examples at each reasoning step (2) learn question decomposition separately from question answering, including using synthetic data, and (3) use bespoke (fine-tuned) components for reasoning steps where a large LM does not perform well. The intermediate supervision is typically manually written, which can be expensive to collect. We introduce a way to generate a synthetic dataset which can be used to bootstrap a model's ability to decompose and answer intermediate questions. Our best model (with successive prompting) achieves an improvement of ~5% absolute F1 on a few-shot version of the DROP dataset when compared with a state-of-the-art model with the same supervision.

Multi-hop reading comprehension requires not only the ability to reason over raw text but also the ability to combine multiple evidence. We propose a novel learning approach that helps language models better understand difficult multi-hop questions and perform "complex, compositional" reasoning. Our model first learns to decompose each multi-hop question into several sub-questions by a trainable question decomposer. Instead of answering these sub-questions, we directly concatenate them with the original question and context, and leverage a reading comprehension model to predict the answer in a sequence-to-sequence manner. By using the same language model for these two components, our best seperate/unified t5-base variants outperform the baseline by 7.2/6.1 absolute F1 points on a hard subset of DROP dataset.

Transformers have been shown to be able to perform deductive reasoning on a logical rulebase containing rules and statements written in English natural language. While the progress is promising, it is currently unclear if these models indeed perform logical reasoning by understanding the underlying logical semantics in the language. To this end, we propose RobustLR, a suite of evaluation datasets that evaluate the robustness of these models to minimal logical edits in rulebases and some standard logical equivalence conditions. In our experiments with RoBERTa and T5, we find that the models trained in prior works do not perform consistently on the different perturbations in RobustLR, thus showing that the models are not robust to the proposed logical perturbations. Further, we find that the models find it especially hard to learn logical negation and disjunction operators. Overall, using our evaluation sets, we demonstrate some shortcomings of the deductive reasoning-based language models, which can eventually help towards designing better models for logical reasoning over natural language. All the datasets and code base have been made publicly available.

Recent work has shown that large language models are capable of generating natural language reasoning steps or Chains-of-Thoughts (CoT) to answer a multi-step question when prompted to do so. This is insufficient, however, when the necessary knowledge is not available or up-to-date within a model's parameters. A straightforward approach to address this is to retrieve text from an external knowledge source using the question as a query and prepend it as context to the model's input. This, however, is also insufficient for multi-step QA where \textit{what to retrieve} depends on \textit{what has already been derived}. To address this issue we propose IRCoT, a new approach that interleaves retrieval with CoT for multi-step QA, guiding the retrieval with CoT and in turn using retrieved results to improve CoT. Our experiments with GPT3 show substantial improvements in retrieval (up to 22 points) and downstream QA (up to 16 points) over the baselines on four datasets: HotpotQA, 2WikiMultihopQA, MuSiQue, and IIRC. Notably, our method also works well for much smaller models such as T5-Flan-large (0.7B) without any additional training.

Structured tabular data exist across nearly all fields. Reasoning task over these data aims to answer questions or determine the truthiness of hypothesis sentences by understanding the semantic meaning of a table. While previous works have devoted significant efforts to the tabular reasoning task, they always assume there are sufficient labeled data. However, constructing reasoning samples over tables (and related text) is labor-intensive, especially when the reasoning process is complex. When labeled data is insufficient, the performance of models will suffer an unendurable decline. In this paper, we propose a unified framework for unsupervised complex tabular reasoning (UCTR), which generates sufficient and diverse synthetic data with complex logic for tabular reasoning tasks, assuming no human-annotated data at all. We first utilize a random sampling strategy to collect diverse programs of different types and execute them on tables based on a "Program-Executor" module. To bridge the gap between the programs and natural language sentences, we design a powerful "NL-Generator" module to generate natural language sentences with complex logic from these programs. Since a table often occurs with its surrounding texts, we further propose novel "Table-to-Text" and "Text-to-Table" operators to handle joint table-text reasoning scenarios. This way, we can adequately exploit the unlabeled table resources to obtain a well-performed reasoning model under an unsupervised setting. Our experiments cover different tasks (question answering and fact verification) and different domains (general and specific), showing that our unsupervised methods can achieve at most 93% performance compared to supervised models. We also find that it can substantially boost the supervised performance in low-resourced domains as a data augmentation technique. Our code is available at https://github.com/leezythu/UCTR.

有效的多跳问答（QA）需要在多个分散的段落上进行推理，并提供答案的解释。大多数现有方法无法提供可解释的推理过程，以说明这些模型如何得出答案。在本文中，我们提出了一种基于多跳QA的抽象含义表示形式（QDAMR）的问题分解方法，该方法通过将多跳问题分解为更简单的子问题并按顺序回答它们来实现可解释的推理。由于注释分解很昂贵，因此我们首先将理解多跳问题的复杂性委托给AMR解析器。然后，我们通过基于所需的推理类型对相应的AMR图进行分割实现多跳问题的分解。最后，我们使用AMR到文本生成模型生成子问题，并使用现成的QA模型回答它们。 HOTPOTQA的实验结果表明，我们的方法在可解释的推理方面具有竞争力，并且QDAMR产生的子问题是良好的，表现优于现有的基于问题分解的多跳质量质量检查方法。

最近的开放式域问题回答表明，新颖的测试问题之间的模型性能和那些在很大程度上与培训问题重叠的模型性能存在很大差异。然而，目前尚不清楚新颖的问题的哪些方面使他们成为挑战。在进行系统泛化的研究时，我们根据三个类别介绍和注释问题，这些类别测量了不同的水平和概括的种类：培训设定重叠，组成泛化（Comp-Gen）和新颖的实体概括（新实体）。在评估六个流行的参数和非参数模型时，我们发现，对于既定的自然问题和TriviaQA数据集，即使是Comp-Gen /新颖实体的最强的模型性能也是13.1 / 5.4％和9.6 / 1.5％，而与此相比降低对于完整的测试集 - 表示这些类型的问题所带来的挑战。此外，我们表明，虽然非参数模型可以相对良好地处理含有新颖实体的问题，但它们与那些需要组成泛化的问题斗争。最后，我们发现关键问题是：来自检索组件的级联错误，问题模式的频率和实体的频率。

Pre-trained language models (LMs) have shown remarkable reasoning performance using explanations (or ``chain-of-thought'' (CoT)) for in-context learning. On the other hand, these reasoning tasks are usually presumed to be more approachable for symbolic programming. To make progress towards understanding in-context learning, we curate synthetic datasets containing equivalent (natural, symbolic) data pairs, where symbolic examples contain first-order logic rules and predicates from knowledge bases (KBs). Then we revisit neuro-symbolic approaches and use Language Models as Logic Programmer (LMLP) that learns from demonstrations containing logic rules and corresponding examples to iteratively reason over KBs, recovering Prolog's backward chaining algorithm. Comprehensive experiments are included to systematically compare LMLP with CoT in deductive reasoning settings, showing that LMLP enjoys more than 25% higher accuracy than CoT on length generalization benchmarks even with fewer parameters.

我们提出了一种系统推理的方法，该方法生产了基于事实基础的人类可解释的证明树。我们的解决方案类似于经典的基于序言的推理引擎的风格，在该引擎中，我们通过神经语言建模，指导生成和半磁头密集检索的结合来代替手工制作的规则。这款新颖的推理引擎Nellie动态实例化了可解释的推理规则，这些规则捕获和分数构成（DE）在自然语言陈述上。内莉（Nellie）在科学质量检查数据集上提供竞争性能，需要对多个事实进行结构化解释。

许多现实世界问题需要综合应用采用合适的抽象，致辞认识和创造性的解决问题策略的多种推理能力。为了帮助推进AI系统实现这种能力，我们提出了一个新的推理挑战，即费银问题（FPS），这是答案只能估计的问题，因为它们的精确计算是不切实际或不可能的。例如，“如果世界上所有的冰融化，那么海平面会增加多少海平面？” FPS通常用于测验和访谈，以发出和评估人类的创造性推理能力。为AI系统做同样的事情，我们展示了两个数据集：1）来自测验和奥林匹克的1K现实世界FPS的集合; 2）一个10K的中间复杂合成FPS的银行，作为较难的真实挑战的沙箱。除问题答案对之外，数据集还包含可执行计划形式的详细解决方案，并提供支持事实，帮助监督和评估中间步骤。我们展示了甚至广泛的微调大规模语言模型在这些数据集上表现不佳，平均估计是由两个数量级的估计值。因此，我们的贡献是几个未解决的AI问题的结晶，以至于我们希望将促进可以推理的建筑系统进一步前进。

Answering complex questions often requires multi-step reasoning in order to obtain the final answer. Most research into decompositions of complex questions involves open-domain systems, which have shown success in using these decompositions for improved retrieval. In the machine reading setting, however, work to understand when decompositions are helpful is understudied. We conduct experiments on decompositions in machine reading to unify recent work in this space, using a range of models and datasets. We find that decompositions can be helpful in the few-shot case, giving several points of improvement in exact match scores. However, we also show that when models are given access to datasets with around a few hundred or more examples, decompositions are not helpful (and can actually be detrimental). Thus, our analysis implies that models can learn decompositions implicitly even with limited data.

知识基础问题回答（KBQA）旨在通过知识库（KB）回答问题。早期研究主要集中于回答有关KB的简单问题，并取得了巨大的成功。但是，他们在复杂问题上的表现远非令人满意。因此，近年来，研究人员提出了许多新颖的方法，研究了回答复杂问题的挑战。在这项调查中，我们回顾了KBQA的最新进展，重点是解决复杂问题，这些问题通常包含多个主题，表达复合关系或涉及数值操作。详细说明，我们从介绍复杂的KBQA任务和相关背景开始。然后，我们描述用于复杂KBQA任务的基准数据集，并介绍这些数据集的构建过程。接下来，我们提出两个复杂KBQA方法的主流类别，即基于语义解析的方法（基于SP）的方法和基于信息检索的方法（基于IR）。具体而言，我们通过流程设计说明了他们的程序，并讨论了它们的主要差异和相似性。之后，我们总结了这两类方法在回答复杂问题时会遇到的挑战，并解释了现有工作中使用的高级解决方案和技术。最后，我们结论并讨论了与复杂的KBQA有关的几个有希望的方向，以进行未来的研究。

Parsing natural language questions into executable logical forms is a useful and interpretable way to perform question answering on structured data such as knowledge bases (KB) or databases (DB). However, existing approaches on semantic parsing cannot adapt to both modalities, as they suffer from the exponential growth of the logical form candidates and can hardly generalize to unseen data. In this work, we propose Uni-Parser, a unified semantic parser for question answering (QA) on both KB and DB. We introduce the primitive (relation and entity in KB, and table name, column name and cell value in DB) as an essential element in our framework. The number of primitives grows linearly with the number of retrieved relations in KB and DB, preventing us from dealing with exponential logic form candidates. We leverage the generator to predict final logical forms by altering and composing topranked primitives with different operations (e.g. select, where, count). With sufficiently pruned search space by a contrastive primitive ranker, the generator is empowered to capture the composition of primitives enhancing its generalization ability. We achieve competitive results on multiple KB and DB QA benchmarks more efficiently, especially in the compositional and zero-shot settings.

Powerful generative models have led to recent progress in question generation (QG). However, it is difficult to measure advances in QG research since there are no standardized resources that allow a uniform comparison among approaches. In this paper, we introduce QG-Bench, a multilingual and multidomain benchmark for QG that unifies existing question answering datasets by converting them to a standard QG setting. It includes general-purpose datasets such as SQuAD for English, datasets from ten domains and two styles, as well as datasets in eight different languages. Using QG-Bench as a reference, we perform an extensive analysis of the capabilities of language models for the task. First, we propose robust QG baselines based on fine-tuning generative language models. Then, we complement automatic evaluation based on standard metrics with an extensive manual evaluation, which in turn sheds light on the difficulty of evaluating QG models. Finally, we analyse both the domain adaptability of these models as well as the effectiveness of multilingual models in languages other than English. QG-Bench is released along with the fine-tuned models presented in the paper https://github.com/asahi417/lm-question-generation, which are also available as a demo https://autoqg.net/.

Existing question answering (QA) datasets fail to train QA systems to perform complex reasoning and provide explanations for answers. We introduce HOTPOTQA, a new dataset with 113k Wikipedia-based question-answer pairs with four key features: (1) the questions require finding and reasoning over multiple supporting documents to answer; (2) the questions are diverse and not constrained to any pre-existing knowledge bases or knowledge schemas; (3) we provide sentence-level supporting facts required for reasoning, allowing QA systems to reason with strong supervision and explain the predictions; (4) we offer a new type of factoid comparison questions to test QA systems' ability to extract relevant facts and perform necessary comparison. We show that HOTPOTQA is challenging for the latest QA systems, and the supporting facts enable models to improve performance and make explainable predictions.

Question answering models commonly have access to two sources of "knowledge" during inference time: (1) parametric knowledge - the factual knowledge encoded in the model weights, and (2) contextual knowledge - external knowledge (e.g., a Wikipedia passage) given to the model to generate a grounded answer. Having these two sources of knowledge entangled together is a core issue for generative QA models as it is unclear whether the answer stems from the given non-parametric knowledge or not. This unclarity has implications on issues of trust, interpretability and factuality. In this work, we propose a new paradigm in which QA models are trained to disentangle the two sources of knowledge. Using counterfactual data augmentation, we introduce a model that predicts two answers for a given question: one based on given contextual knowledge and one based on parametric knowledge. Our experiments on the Natural Questions dataset show that this approach improves the performance of QA models by making them more robust to knowledge conflicts between the two knowledge sources, while generating useful disentangled answers.

Large language models show improved downstream task performance when prompted to generate step-by-step reasoning to justify their final answers. These reasoning steps greatly improve model interpretability and verification, but objectively studying their correctness (independent of the final answer) is difficult without reliable methods for automatic evaluation. We simply do not know how often the stated reasoning steps actually support the final end task predictions. In this work, we present ROSCOE, a suite of interpretable, unsupervised automatic scores that improve and extend previous text generation evaluation metrics. To evaluate ROSCOE against baseline metrics, we design a typology of reasoning errors and collect synthetic and human evaluation scores on commonly used reasoning datasets. In contrast with existing metrics, ROSCOE can measure semantic consistency, logicality, informativeness, fluency, and factuality - among other traits - by leveraging properties of step-by-step rationales. We empirically verify the strength of our metrics on five human annotated and six programmatically perturbed diagnostics datasets - covering a diverse set of tasks that require reasoning skills and show that ROSCOE can consistently outperform baseline metrics.

当一个神经语言模型（LM）适于执行新任务时，任务的哪些方面预测了模型的最终性能？在NLP中，LM概括到个别示例的系统特征很好，但LM对新任务的系统的系统性方面并不理解。我们使用500个程序生成的序列建模任务构建的新基准测试，展示了LM适应性的特性和限制的大规模实证研究。这些任务组合了语言处理的核心方面，包括词汇语义，序列处理，记忆，逻辑推理和世界知识。使用TaskBench500，我们评估了三个适应性的方面，发现：（1）适应程序在他们记忆小型数据集的能力中急剧差异; （2）在任务类型的子集中，适应程序表现出对复杂任务的组成适应性; （3）未能匹配培训标签分布，在预测个别标签的内在难度中解释了不匹配。我们的实验表明，可以系统地描述和理解新的任务，如新示例的泛化，以及讨论可以使用新基准研究的适应性的其他方面的结论。

从头开始解决复杂问题通常是有挑战性的，但如果我们可以访问其解决方案的其他类似问题，则更容易 - 一种称为基于案例的推理（CBR）的范式。我们提出了一种神经象征性的CBR方法（CBR-KBQA），用于在大知识库上应答。 CBR-KBQA由非参数内存组成，该内存存储案例（问题和逻辑表单）和参数模型，该参数模型可以通过检索与其相关的案例来为新问题生成逻辑表单。在包含复杂问题的几个KBQA数据集上，CBR-KBQA实现了竞争性能。例如，在ComplexWebQuestions数据集上，CBR-KBQA以11 \％的准确度优于当前最新状态。此外，我们表明CBR-KBQA能够使用新案例\ EMPH {没有}任何进一步的培训：通过在案例存储器中纳入一些人类标记的示例，CBR-KBQA能够成功地生成包含未经看线KB实体的逻辑表格以及关系。