临床问题应答(QA)旨在根据临床文本自动回答医疗专业人员的问题。研究表明,在一个语料库上培训的神经QA模型可能对来自不同研究所或不同患者组的新临床文本概括,其中大规模的QA对不容易获得模型再培训。为了解决这一挑战,我们提出了一个简单但有效的框架CliniQG4QA,它利用问题生成(QG)在新的临床环境中综合QA对,并在不需要手动注释的情况下提升QA模型。为了生成对训练QA模型至关重要的不同类型的问题,我们进一步引入了基于SEQ2SEQ的问题短语预测(QPP)模块,可以与大多数现有的QG模型一起使用以使生成多样化。我们的综合实验结果表明,我们的框架产生的QA​​语料库可以改善新上下文的QA模型(在完全匹配方面最高8%的绝对增益),QPP模块在实现增益方面发挥着至关重要的作用。
translated by 谷歌翻译
预审前的语言模型通过提供高质量的上下文化单词嵌入来显着改善了下游语言理解任务(包括提取性问题)的性能。但是,培训问答模型仍然需要大量特定域的注释数据。在这项工作中,我们提出了一个合作的自我训练框架RGX,用于自动生成更非平凡的问题 - 解答对以提高模型性能。 RGX建立在带有答案实体识别器,问题生成器和答案提取器的交互式学习环境的蒙版答案提取任务上。给定带有蒙版实体的段落,生成器会在实体周围生成一个问题,并培训了提取器,以提取蒙面实体,并使用生成的问题和原始文本。该框架允许对任何文本语料库的问题产生和回答模型进行培训,而无需注释。实验结果表明,RGX优于最先进的语言模型(SOTA)的语言模型,并在标准提问基准的基准上采用转移学习方法,并在给定的模型大小和传输学习设置下产生新的SOTA性能。
translated by 谷歌翻译
虽然通过简单的因素问题回答,文本理解的大量进展,但更加全面理解话语仍然存在重大挑战。批判性地反映出文本的人将造成好奇心驱动,通常是开放的问题,这反映了对内容的深刻理解,并要求复杂的推理来回答。建立和评估这种类型的话语理解模型的关键挑战是缺乏注释数据,特别是因为找到了这些问题的答案(可能根本不回答),需要高度的注释载荷的高认知负荷。本文提出了一种新的范式,使可扩展的数据收集能够针对新闻文件的理解,通过话语镜头查看这些问题。由此产生的语料库DCQA(疑问回答的话语理解)包括在607名英语文件中的22,430个问题答案对组成。 DCQA以自由形式,开放式问题的形式捕获句子之间的话语和语义链接。在评估集中,我们向问题上的问题提交了来自好奇数据集的问题,我们表明DCQA提供了有价值的监督,以回答开放式问题。我们还在使用现有的问答资源设计预训练方法,并使用合成数据来适应不可批售的问题。
translated by 谷歌翻译
传达相关和忠实信息的能力对于有条件生成的许多任务至关重要,但对于神经SEQ-seq seq模型仍然难以捉摸,这些模型的输出通常显示出幻觉,并且无法正确涵盖重要细节。在这项工作中,我们主张规划作为有用的中间表示,以使有条件的一代减少不透明和扎根。我们的作品提出了将文本计划作为一系列提问(QA)对的新概念化。我们用QA蓝图作为内容选择(即〜说什么)和计划(即〜按什么顺序)来增强现有数据集(例如,用于摘要)。我们通过利用最先进的问题生成技术并将输入输出对自动获取蓝图,并将其转换为输入 - 蓝图输出输出元组。我们开发了基于变压器的模型,每个模型都在它们如何将蓝图合并到生成的输出中(例如,作为全局计划或迭代)。跨指标和数据集的评估表明,蓝图模型比不采取计划并允许对生成输出进行更严格控制的替代方案更为事实。
translated by 谷歌翻译
Natural Language Generation (NLG) has improved exponentially in recent years thanks to the development of sequence-to-sequence deep learning technologies such as Transformer-based language models. This advancement has led to more fluent and coherent NLG, leading to improved development in downstream tasks such as abstractive summarization, dialogue generation and data-to-text generation. However, it is also apparent that deep learning based generation is prone to hallucinate unintended text, which degrades the system performance and fails to meet user expectations in many real-world scenarios. To address this issue, many studies have been presented in measuring and mitigating hallucinated texts, but these have never been reviewed in a comprehensive manner before. In this survey, we thus provide a broad overview of the research progress and challenges in the hallucination problem in NLG. The survey is organized into two parts: (1) a general overview of metrics, mitigation methods, and future directions; and (2) an overview of task-specific research progress on hallucinations in the following downstream tasks, namely abstractive summarization, dialogue generation, generative question answering, data-to-text generation, machine translation, and visual-language generation. This survey serves to facilitate collaborative efforts among researchers in tackling the challenge of hallucinated texts in NLG.
translated by 谷歌翻译
自Bert(Devlin等,2018)以来,学习上下文化的单词嵌入一直是NLP中的事实上的标准。然而,学习上下文化短语嵌入的进展受到缺乏人类通知的语句基准基准的阻碍。为了填补这一空白,我们提出了PIC- 〜28K名词短语的数据集伴随着它们的上下文Wikipedia页面,以及一套三个任务,这些任务增加了评估短语嵌入质量的难度。我们发现,在我们的数据集中进行的培训提高了排名模型的准确性,并明显地将问题答案(QA)模型推向了近人类的准确性,而在语义搜索上,鉴于询问短语和段落,在语义搜索上是95%的精确匹配(EM)。有趣的是,我们发现这种令人印象深刻的性能的证据是因为质量检查模型学会了更好地捕获短语的共同含义,而不管其实际背景如何。也就是说,在我们的短语中歧义歧义(PSD)任务上,SOTA模型的精度大大下降(60%EM),在两个不同情况下未能区分相同短语的两种不同感觉。在我们的3任任务基准测试中的进一步结果表明,学习上下文化的短语嵌入仍然是一个有趣的开放挑战。
translated by 谷歌翻译
Question Answering (QA) is a growing area of research, often used to facilitate the extraction of information from within documents. State-of-the-art QA models are usually pre-trained on domain-general corpora like Wikipedia and thus tend to struggle on out-of-domain documents without fine-tuning. We demonstrate that synthetic domain-specific datasets can be generated easily using domain-general models, while still providing significant improvements to QA performance. We present two new tools for this task: A flexible pipeline for validating the synthetic QA data and training downstream models on it, and an online interface to facilitate human annotation of this generated data. Using this interface, crowdworkers labelled 1117 synthetic QA pairs, which we then used to fine-tune downstream models and improve domain-specific QA performance by 8.75 F1.
translated by 谷歌翻译
问答(QA)在回答定制域中的问题方面表现出了令人印象深刻的进展。然而,域的适应性仍然是质量检查系统最难以捉摸的挑战之一,尤其是当质量检查系统在源域中训练但部署在不同的目标域中时。在这项工作中,我们调查了问题分类对质量检查域适应的潜在好处。我们提出了一个新颖的框架:问题回答的问题分类(QC4QA)。具体而言,采用问题分类器将问题类分配给源数据和目标数据。然后,我们通过伪标记以自我监督的方式进行联合培训。为了优化,源和目标域之间的域间差异通过最大平均差异(MMD)距离降低。我们还最大程度地减少了同一问题类别的质量质量适应性表现的QA样本中的类内部差异。据我们所知,这是质量检查域适应中的第一部作品,以通过自我监督的适应来利用问题分类。我们证明了拟议的QC4QA的有效性,并在多个数据集上针对最先进的基线进行了一致的改进。
translated by 谷歌翻译
学术研究是解决以前从未解决过的问题的探索活动。通过这种性质,每个学术研究工作都需要进行文献审查,以区分其Novelties尚未通过事先作品解决。在自然语言处理中,该文献综述通常在“相关工作”部分下进行。鉴于研究文件的其余部分和引用的论文列表,自动相关工作生成的任务旨在自动生成“相关工作”部分。虽然这项任务是在10年前提出的,但直到最近,它被认为是作为科学多文件摘要问题的变种。然而,即使在今天,尚未标准化了自动相关工作和引用文本生成的问题。在这项调查中,我们进行了一个元研究,从问题制定,数据集收集,方法方法,绩效评估和未来前景的角度来比较相关工作的现有文献,以便为读者洞察到国家的进步 - 最内容的研究,以及如何进行未来的研究。我们还调查了我们建议未来工作要考虑整合的相关研究领域。
translated by 谷歌翻译
表中的信息可能是文本的重要补充,使基于表的问题答案(QA)具有巨大的价值。处理表的内在复杂性通常会增加模型设计和数据注释的额外负担。在本文中,我们旨在以最少的注释工作开发一个简单的基于表的质量检查模型。由于基于表的质量检查需要问题和表之间的对齐方式以及在多个表元素上执行复杂推理的能力,因此我们提出了一种杂食性的预读方法,该方法既可以消耗自然数据,又提出了合成数据,以使模型具有这些各自的能力。具体而言,鉴于可免费获得的表,我们利用检索将它们与相关的自然句子配对,以进行掩盖预处理,并通过将SQL从表中进行转换为QA损失进行预处理而合成NL问题。我们在几次和完整的设置中都进行了广泛的实验,结果清楚地证明了模型omnitab的优势,最好的多任务方法分别实现了16.2%和2.7%的绝对增益,在128次和完整的设置中也获得了2.7%建立有关Wickitable Questions的最新最新。详细的消融和分析揭示了自然和合成数据的不同特征,从而阐明了杂食性预处理的未来方向。可以在https://github.com/jzbjyb/omnitab上获得代码,预读数据和预算模型。
translated by 谷歌翻译
生物医学机器阅读理解(生物医学MRC)旨在理解复杂的生物医学叙事,并协助医疗保健专业人员从中检索信息。现代神经网络的MRC系统的高性能取决于高质量的大规模,人为宣传的培训数据集。在生物医学领域中,创建此类数据集的一个至关重要的挑战是域知识的要求,引起了标记数据的稀缺性以及从标记的通用(源)域转移学习到生物医学(目标)域的需求。然而,由于主题方差,通用和生物医学领域之间的边际分布存在差异。因此,从在通用域上训练的模型到生物医学领域的模型直接转移学会的表示可能会损害模型的性能。我们为生物医学机器阅读理解任务(BioAdapt-MRC)提供了基于对抗性学习的域适应框架,这是一种基于神经网络的方法,可解决一般和生物医学域数据之间边际分布中的差异。 Bioadapt-MRC松弛了生成伪标签的需求,以训练表现出色的生物医学MRC模型。我们通过将生物ADAPT-MRC与三种广泛使用的基准生物医学MRC数据集进行比较,从而广泛评估了生物ADAPT-MRC的性能-Bioasq-7B,BioASQ-8B和BioASQ-9B。我们的结果表明,如果不使用来自生物医学领域的任何合成或人类通知的数据,Bioadapt-MRC可以在这些数据集中实现最先进的性能。可用性:bioadapt-MRC可作为开放源项目免费获得,\ url {https://github.com/mmahbub/bioadapt-mrc}。
translated by 谷歌翻译
通过将搜索任务框架作为解释跨度检测来绘制语义搜索问题,即给定文本作为查询短语的段,任务是在给定文档中识别其释放,与通常相同的建模设置用于提取问题的回答。在Turku释放语料库中的100,000手动提取芬兰语释义对,包括其原始文档上下文,我们发现我们的扫描跨度检测模型分别优于31.9pp和22.4pp的两个强烈的检索基线(词汇相似性和BERT句子嵌入)。匹配,达到22.3pp和12.9pp的令牌级F分数。这展示了在跨度检索而不是句子相似性方面建模任务的强大优点。此外,我们介绍了一种通过背部翻译创建人工释义数据的方法,适用于手动注释用于训练的跨度检测模型的剖析资源。
translated by 谷歌翻译
Supervised Question Answering systems (QA systems) rely on domain-specific human-labeled data for training. Unsupervised QA systems generate their own question-answer training pairs, typically using secondary knowledge sources to achieve this outcome. Our approach (called PIE-QG) uses Open Information Extraction (OpenIE) to generate synthetic training questions from paraphrased passages and uses the question-answer pairs as training data for a language model for a state-of-the-art QA system based on BERT. Triples in the form of <subject, predicate, object> are extracted from each passage, and questions are formed with subjects (or objects) and predicates while objects (or subjects) are considered as answers. Experimenting on five extractive QA datasets demonstrates that our technique achieves on-par performance with existing state-of-the-art QA systems with the benefit of being trained on an order of magnitude fewer documents and without any recourse to external reference data sources.
translated by 谷歌翻译
在过去的几年中,临床笔记中的问题回答(QA)引起了很多关注。临床领域中现有的机器阅读理解方法只能处理有关单个临床文本的问题,并且无法检索有关多个患者及其临床笔记的信息。为了处理更复杂的问题,我们旨在从临床注释中创建知识库,以将不同的患者和临床笔记联系起来,并进行知识基础问题答案(KBQA)。根据N2C2数据集中可用的专家注释,我们首先创建了ClinicalKBQA数据集,其中包括大约9K QA对,并使用300多个问题模板涵盖了有关七个医学主题的问题。然后,我们研究了KBQA的一种基于注意力的方面推理(AAR)方法,并分析了答案的不同方面(例如,实体,类型,路径和上下文)对预测的影响。由于设计精良的编码器和注意力机制,AAR方法可实现更好的性能。从我们的实验中,我们发现这两个方面,类型和路径都使模型能够识别满足一般条件的答案,并产生较低的精度和更高的回忆。另一方面,各个方面,实体和上下文通过特定于节点的信息限制答案,并导致更高的精度和较低的回忆。
translated by 谷歌翻译
The meaningful use of electronic health records (EHR) continues to progress in the digital era with clinical decision support systems augmented by artificial intelligence. A priority in improving provider experience is to overcome information overload and reduce the cognitive burden so fewer medical errors and cognitive biases are introduced during patient care. One major type of medical error is diagnostic error due to systematic or predictable errors in judgment that rely on heuristics. The potential for clinical natural language processing (cNLP) to model diagnostic reasoning in humans with forward reasoning from data to diagnosis and potentially reduce the cognitive burden and medical error has not been investigated. Existing tasks to advance the science in cNLP have largely focused on information extraction and named entity recognition through classification tasks. We introduce a novel suite of tasks coined as Diagnostic Reasoning Benchmarks, DR.BENCH, as a new benchmark for developing and evaluating cNLP models with clinical diagnostic reasoning ability. The suite includes six tasks from ten publicly available datasets addressing clinical text understanding, medical knowledge reasoning, and diagnosis generation. DR.BENCH is the first clinical suite of tasks designed to be a natural language generation framework to evaluate pre-trained language models. Experiments with state-of-the-art pre-trained generative language models using large general domain models and models that were continually trained on a medical corpus demonstrate opportunities for improvement when evaluated in DR. BENCH. We share DR. BENCH as a publicly available GitLab repository with a systematic approach to load and evaluate models for the cNLP community.
translated by 谷歌翻译
We introduce a large scale MAchine Reading COmprehension dataset, which we name MS MARCO. The dataset comprises of 1,010,916 anonymized questionssampled from Bing's search query logs-each with a human generated answer and 182,669 completely human rewritten generated answers. In addition, the dataset contains 8,841,823 passages-extracted from 3,563,535 web documents retrieved by Bing-that provide the information necessary for curating the natural language answers. A question in the MS MARCO dataset may have multiple answers or no answers at all. Using this dataset, we propose three different tasks with varying levels of difficulty: (i) predict if a question is answerable given a set of context passages, and extract and synthesize the answer as a human would (ii) generate a well-formed answer (if possible) based on the context passages that can be understood with the question and passage context, and finally (iii) rank a set of retrieved passages given a question. The size of the dataset and the fact that the questions are derived from real user search queries distinguishes MS MARCO from other well-known publicly available datasets for machine reading comprehension and question-answering. We believe that the scale and the real-world nature of this dataset makes it attractive for benchmarking machine reading comprehension and question-answering models.
translated by 谷歌翻译
Existing metrics for evaluating the quality of automatically generated questions such as BLEU, ROUGE, BERTScore, and BLEURT compare the reference and predicted questions, providing a high score when there is a considerable lexical overlap or semantic similarity between the candidate and the reference questions. This approach has two major shortcomings. First, we need expensive human-provided reference questions. Second, it penalises valid questions that may not have high lexical or semantic similarity to the reference questions. In this paper, we propose a new metric, RQUGE, based on the answerability of the candidate question given the context. The metric consists of a question-answering and a span scorer module, in which we use pre-trained models from the existing literature, and therefore, our metric can be used without further training. We show that RQUGE has a higher correlation with human judgment without relying on the reference question. RQUGE is shown to be significantly more robust to several adversarial corruptions. Additionally, we illustrate that we can significantly improve the performance of QA models on out-of-domain datasets by fine-tuning on the synthetic data generated by a question generation model and re-ranked by RQUGE.
translated by 谷歌翻译
在宣传,新闻和社交媒体中的虚假,不准确和误导信息中,现实世界的问题应答(QA)系统面临综合和推理相互矛盾的挑战,以获得正确答案的挑战。这种紧迫性导致需要使QA系统对错误信息的强大,这是一个先前未开发的主题。我们通过调查与实际和虚假信息混合的矛盾的情况下,通过调查QA模型的行为来研究对QA模型的错误信息的风险。我们为此问题创建了第一个大规模数据集,即对QA,其中包含超过10K的人写和模型生成的矛盾的上下文。实验表明,QA模型易受误导的背景下的攻击。为了防御这种威胁,我们建立一个错误信息感知的QA系统作为一个反措施,可以以联合方式整合问题应答和错误信息检测。
translated by 谷歌翻译
自动问题应答(QA)系统的目的是以时间有效的方式向用户查询提供答案。通常在数据库(或知识库)或通常被称为语料库的文件集合中找到答案。在过去的几十年里,收购知识的扩散,因此生物医学领域的新科学文章一直是指数增长。因此,即使对于领域专家,也难以跟踪域中的所有信息。随着商业搜索引擎的改进,用户可以在某些情况下键入其查询并获得最相关的一小组文档,以及在某些情况下从文档中的相关片段。但是,手动查找所需信息或答案可能仍然令人疑惑和耗时。这需要开发高效的QA系统,该系统旨在为用户提供精确和精确的答案提供了生物医学领域的自然语言问题。在本文中,我们介绍了用于开发普通域QA系统的基本方法,然后彻底调查生物医学QA系统的不同方面,包括使用结构化数据库和文本集合的基准数据集和几种提出的方​​法。我们还探讨了当前系统的局限性,并探索潜在的途径以获得进一步的进步。
translated by 谷歌翻译
We present TriviaQA, a challenging reading comprehension dataset containing over 650K question-answer-evidence triples. TriviaQA includes 95K questionanswer pairs authored by trivia enthusiasts and independently gathered evidence documents, six per question on average, that provide high quality distant supervision for answering the questions. We show that, in comparison to other recently introduced large-scale datasets, TriviaQA (1) has relatively complex, compositional questions, (2) has considerable syntactic and lexical variability between questions and corresponding answer-evidence sentences, and (3) requires more cross sentence reasoning to find answers. We also present two baseline algorithms: a featurebased classifier and a state-of-the-art neural network, that performs well on SQuAD reading comprehension. Neither approach comes close to human performance (23% and 40% vs. 80%), suggesting that Trivi-aQA is a challenging testbed that is worth significant future study. 1
translated by 谷歌翻译