通常观察到的最先进的自然语言技术问题,例如亚马逊alexa和苹果公司,是他们的服务不会因语言障碍而扩展到大多数发展中国家的公民。这种种群因其语言缺乏可用资源来构建NLP产品。本文介绍了allwoz,一个多语言多域面向任务的客户服务对话框数据集覆盖八种语言:英语,普通话,韩语,越南语,印地语,法国,葡萄牙语和泰国。此外,我们通过使用mt5与元学习来创建多语言数据集的基准。
translated by 谷歌翻译
Task-oriented dialogue (TOD) systems have been applied in a range of domains to support human users to achieve specific goals. Systems are typically constructed for a single domain or language and do not generalise well beyond this. Their extension to other languages in particular is restricted by the lack of available training data for many of the world's languages. To support work on Natural Language Understanding (NLU) in TOD across multiple languages and domains simultaneously, we constructed MULTI3NLU++, a multilingual, multi-intent, multi-domain dataset. MULTI3NLU++ extends the English-only NLU++ dataset to include manual translations into a range of high, medium and low resource languages (Spanish, Marathi, Turkish and Amharic), in two domains (banking and hotels). MULTI3NLU++ inherits the multi-intent property of NLU++, where an utterance may be labelled with multiple intents, providing a more realistic representation of a user's goals and aligning with the more complex tasks that commercial systems aim to model. We use MULTI3NLU++ to benchmark state-of-the-art multilingual language models as well as Machine Translation and Question Answering systems for the NLU task of intent detection for TOD systems in the multilingual setting. The results demonstrate the challenging nature of the dataset, particularly in the low-resource language setting.
translated by 谷歌翻译
我们介绍了用于插槽,意图分类和虚拟助手评估的大规模数据集 - 数字亚马逊SLU资源包(SLURP)。大规模包含1M现实,平行,标记为虚拟助手的话语,涵盖51种语言,18个域,60个意图和55个插槽。通过任务专业翻译人员将仅英文slurp数据集定位为29属的50种类型多样性的语言来创建大规模。我们还介绍了XLM-R和MT5上的建模结果,包括精确的匹配精度,意图分类精度和插槽填充F1分数。我们已经公开发布了数据集,建模代码和模型。
translated by 谷歌翻译
发展任务导向的对话助理的实用需求需要了解许多语言。多语言自然语言理解(NLU)的新型基准包括多种语言中的单声道句,用意图和插槽注释。在这种设置模型中,用于交叉传输在联合意图识别和槽填充方面表现出显着性能。然而,现有的基准缺乏代码切换话语,这难以收集和标签由于语法结构的复杂性。对于NLU模型的评估似乎偏见和有限,因为代码切换被遗漏了范围。我们的工作采用认可的方法来生成合理的和自然探测的代码切换话语,并使用它们来创建合成代码交换测试集。基于实验,我们报告说,最先进的NLU模型无法处理代码切换。在最糟糕的是,性能,通过语义精度评估,从横跨80 \%的8 \%的低至15 \%。此外,我们展示了,对合成码混合数据进行预训练有助于在具有单晶体数据的可比水平上保持所提出的测试中的性能。最后,我们分析了不同的语言对并表明语言越近,NLU模型越好地处理了交替。这符合对多语种模型在语言之间进行转移的共同理解
translated by 谷歌翻译
在过去的十年中,对对话系统的兴趣已经大大增长。从扩展过程中,也有兴趣开发和改进意图分类和插槽填充模型,这是两个组件,这些组件通常在以任务为导向的对话框系统中使用。此外,良好的评估基准对于帮助比较和分析结合此类模型的系统很重要。不幸的是,该领域的许多文献仅限于对相对较少的基准数据集的分析。为了促进针对任务的对话系统的更强大的分析,我们对意图分类和插槽填充任务进行了公开可用数据集的调查。我们分类每个数据集的重要特征,并就每个数据集的适用性,优势和劣势进行讨论。我们的目标是,这项调查有助于提高这些数据集的可访问性,我们希望它们能够在未来评估意图分类和填充插槽模型中用于以任务为导向的对话框系统。
translated by 谷歌翻译
针对任务导向的对话系统的强大状态跟踪目前仍然限于一些流行语言。本文显示,给定以一种语言设置的大规模对话数据,我们可以使用机器翻译自动为其他语言生成有效的语义解析器。我们提出了对话数据集的自动翻译,并进行对齐,以确保插槽值的忠实翻译,并消除以前的基准中使用的昂贵人类监督。我们还提出了一种新的上下文语义解析模型,它编码正式的插槽和值,只有最后一个代理和用户话语。我们表明,简洁的表示降低了翻译误差的复合效果,而不会损害实践中的准确性。我们评估我们对几个对话状态跟踪基准的方法。在Risawoz,Crosswoz,Crosswoz-Zh和Multiwoz-Zh Datasets,我们将最先进的技术提高11%,17%,20%和0.3%,以共同的目标准确度。我们为所有三个数据集提供了全面的错误分析,显示错误注释可以模糊模型质量的判断。最后,我们使用推荐方法创建了Risawoz英语和德语数据集。在这些数据集中,准确性在原始的11%以内,表示可能的高精度多语言对话数据集,而无需依赖昂贵的人类注释。
translated by 谷歌翻译
Token free approaches have been successfully applied to a series of word and span level tasks. In this work, we compare a byte-level (ByT5) and a wordpiece based (mT5) sequence to sequence model on the 51 languages of the MASSIVE multilingual semantic parsing dataset. We examine multiple experimental settings: (i) zero-shot, (ii) full gold data and (iii) zero-shot with synthetic data. By leveraging a state-of-the-art label projection method for machine translated examples, we are able to reduce the gap in exact match accuracy to only 5 points with respect to a model trained on gold data from all the languages. We additionally provide insights on the cross-lingual transfer of ByT5 and show how the model compares with respect to mT5 across all parameter sizes.
translated by 谷歌翻译
Despite recent progress in Natural Language Understanding (NLU), the creation of multilingual NLU systems remains a challenge. It is common to have NLU systems limited to a subset of languages due to lack of available data. They also often vary widely in performance. We launch a three-phase approach to address the limitations in NLU and help propel NLU technology to new heights. We release a 52 language dataset called the Multilingual Amazon SLU resource package (SLURP) for Slot-filling, Intent classification, and Virtual assistant Evaluation, or MASSIVE, in an effort to address parallel data availability for voice assistants. We organize the Massively Multilingual NLU 2022 Challenge to provide a competitive environment and push the state-of-the art in the transferability of models into other languages. Finally, we host the first Massively Multilingual NLU workshop which brings these components together. The MMNLU workshop seeks to advance the science behind multilingual NLU by providing a platform for the presentation of new research in the field and connecting teams working on this research direction. This paper summarizes the dataset, workshop and the competition and the findings of each phase.
translated by 谷歌翻译
Much recent progress in applications of machine learning models to NLP has been driven by benchmarks that evaluate models across a wide variety of tasks. However, these broad-coverage benchmarks have been mostly limited to English, and despite an increasing interest in multilingual models, a benchmark that enables the comprehensive evaluation of such methods on a diverse range of languages and tasks is still missing. To this end, we introduce the Cross-lingual TRansfer Evaluation of Multilingual Encoders (XTREME) benchmark, a multi-task benchmark for evaluating the cross-lingual generalization capabilities of multilingual representations across 40 languages and 9 tasks. We demonstrate that while models tested on English reach human performance on many tasks, there is still a sizable gap in the performance of cross-lingually transferred models, particularly on syntactic and sentence retrieval tasks. There is also a wide spread of results across languages. We release the benchmark 1 to encourage research on cross-lingual learning methods that transfer linguistic knowledge across a diverse and representative set of languages and tasks.
translated by 谷歌翻译
由于缺乏低资源语言的语料库,当前的对话生成作品主要集中在英语上。在本文中,我们介绍了MDIA,这是第一个大规模的多语言基准,用于跨低资源语言进行对话生成。它涵盖了19个语言家庭中46种语言的现实生活对话。我们介绍通过微调多语言,非拨号的预训练的模型MT5以及以英语为中心的,以对话为中心的预训练的预训练的聊天机器人对话,获得了基线结果。结果表明,基于MT5的模型在Sacrebleu和Bertscore上的表现更好,但在多样性方面的性能较差。即使在几乎没有射击和零拍的场景中发现了有希望的结果,但英语和其他语言的一代质量之间存在很大的差距。我们希望MDIA的发布可以鼓励更多关于多语言对话生成的作品,以促进语言多样性。
translated by 谷歌翻译
In this work, we introduce IndicXTREME, a benchmark consisting of nine diverse tasks covering 18 languages from the Indic sub-continent belonging to four different families. Across languages and tasks, IndicXTREME contains a total of 103 evaluation sets, of which 51 are new contributions to the literature. To maintain high quality, we only use human annotators to curate or translate\footnote{for IndicXParaphrase, where an automatic translation system is used, a second human verification and correction step is done.} our datasets. To the best of our knowledge, this is the first effort toward creating a standard benchmark for Indic languages that aims to test the zero-shot capabilities of pretrained language models. We also release IndicCorp v2, an updated and much larger version of IndicCorp that contains 20.9 billion tokens in 24 languages. We pretrain IndicBERT v2 on IndicCorp v2 and evaluate it on IndicXTREME to show that it outperforms existing multilingual language models such as XLM-R and MuRIL.
translated by 谷歌翻译
我们提出语言学家,这是一种通过微调Alexatm 5B生成带注释数据的方法,用于生成意图分类和插槽标记(IC+ST),这是一种5亿参数的多语言序列到序列(SEQ2SEQ)模型,在灵活的指令上迅速的。在SNIP数据集的10次新颖意图设置中,语言学家超过了最新的方法(反向翻译和示例外推),可以通过宽阔的边距,显示出IC回忆中+1.9点的目标意图的绝对改善ST F1分数和+2.5分。在MATIS ++数据集的零击跨语言设置中,语言学家表现出强大的机器翻译基线,插槽对齐的基线是+4.14的+4.14点在6个语言上绝对在ST F1分数上,同时在IC上匹配IC的性能。最后,我们在用于对话代理IC+ST的内部大规模多语言数据集上验证了我们的结果,并显示了使用背面翻译,释义和插槽目录重新采样采样的基线的显着改进。据我们所知,我们是第一个展示大规模SEQ2SEQ模型的指导微调的人,以控制多语言意图和插槽标记的数据生成的输出。
translated by 谷歌翻译
有监督的基于深度学习的方法已应用于以任务为导向的对话框,并在有足够数量的培训示例可用时对有限的域和语言应用有效。在实践中,这些方法遭受了域驱动设计和资源不足的语言的缺点。域和语言模型应该随着问题空间的发展而增长和变化。一方面,对转移学习的研究证明了基于多语言变压器模型学习语义丰富的表示的跨语性能力。另一方面,除了上述方法之外,元学习还能够开发任务和语言学习算法,能够实现泛滥。在这种情况下,本文提出了使用典型的神经网络和基于多语言变压器的模型来研究使用协同进行几次学习的跨语性可传递性。自然语言的实验理解多亚提斯++语料库的任务表明,我们的方法基本上改善了低资源和高资源语言之间观察到的转移学习表现。更普遍地说,我们的方法证实,可以将具有特定语言的有意义的潜在空间推广到使用元学习的情况下看不见和资源不足的潜在空间。
translated by 谷歌翻译
State-of-the-art natural language processing systems rely on supervision in the form of annotated data to learn competent models. These models are generally trained on data in a single language (usually English), and cannot be directly used beyond that language. Since collecting data in every language is not realistic, there has been a growing interest in crosslingual language understanding (XLU) and low-resource cross-language transfer. In this work, we construct an evaluation set for XLU by extending the development and test sets of the Multi-Genre Natural Language Inference Corpus (MultiNLI) to 15 languages, including low-resource languages such as Swahili and Urdu. We hope that our dataset, dubbed XNLI, will catalyze research in cross-lingual sentence understanding by providing an informative standard evaluation task. In addition, we provide several baselines for multilingual sentence understanding, including two based on machine translation systems, and two that use parallel data to train aligned multilingual bag-of-words and LSTM encoders. We find that XNLI represents a practical and challenging evaluation suite, and that directly translating the test data yields the best performance among available baselines.
translated by 谷歌翻译
Translating training data into many languages has emerged as a practical solution for improving cross-lingual transfer. For tasks that involve span-level annotations, such as information extraction or question answering, an additional label projection step is required to map annotated spans onto the translated texts. Recently, a few efforts have utilized a simple mark-then-translate method to jointly perform translation and projection by inserting special markers around the labeled spans in the original sentence. However, as far as we are aware, no empirical analysis has been conducted on how this approach compares to traditional annotation projection based on word alignment. In this paper, we present an extensive empirical study across 42 languages and three tasks (QA, NER, and Event Extraction) to evaluate the effectiveness and limitations of both methods, filling an important gap in the literature. Experimental results show that our optimized version of mark-then-translate, which we call EasyProject, is easily applied to many languages and works surprisingly well, outperforming the more complex word alignment-based methods. We analyze several key factors that affect end-task performance, and show EasyProject works well because it can accurately preserve label span boundaries after translation. We will publicly release all our code and data.
translated by 谷歌翻译
在零拍摄的情况下建立对话的生成系统仍然是一个巨大的挑战,因为对话生成中典型的零击方法很大程度上取决于大规模的预训练的语言生成模型,例如GPT-3和T5。由于缺乏相应的平行对话COLIDA,对无繁琐语言模型的零摄像对话生成的研究受到限制。在本文中,我们提出了一个简单但有效的多语言学习框架,用于零拍对对话(称为mulzdg),该框架可以有效地将知识从带有大规模培训样本的英语语料库转移到具有零样本的非英语语料库。此外,MulzDG可以被视为一种多语言数据增强方法,以提高资源丰富的语言的性能。首先,我们通过从单语英文数据集随机选择的翻译说法来构建多语言代码转换对话数据集。然后,我们使用MulzDG来培训基于代码转换数据集的统一的多语言对话模型。 mulzdg可以在不同语言之间进行隐性的语义一致性。关于DailyDialog和DSTC7数据集的实验表明,与有足够示例的培训相比,MulzDG不仅在零击中的情况下实现竞争性能,而且还可以大大提高源语言的性能。
translated by 谷歌翻译
最近,培训预培训方法在以任务为导向的对话框(TOD)系统中表现出了很大的成功。但是,大多数现有的预培训模型用于TOD专注于对话的理解或对话生成,但并非两者兼而有之。在本文中,我们提出了Space-3,这是一种新型的统一的半监督预培训的预训练的对话模型,从大规模对话CORPORA中学习有限的注释,可以有效地对广泛的下游对话任务进行微调。具体而言,Space-3由单个变压器中的四个连续组件组成,以维护TOD系统中的任务流:(i)对话框编码模块编码对话框历史记录,(ii)对话框理解模块以从任一用户中提取语义向量查询或系统响应,(iii)一个对话框策略模块,以生成包含响应高级语义的策略向量,以及(iv)对话框生成模块以产生适当的响应。我们为每个组件设计一个专门的预训练目标。具体而言,我们预先培训对话框编码模块,使用跨度掩码语言建模,以学习上下文化对话框信息。为了捕获“结构化对话框”语义,我们通过额外的对话注释通过新颖的树诱导的半监视对比度学习目标来预先培训对话框理解模块。此外,我们通过将其输出策略向量与响应响应的语义向量之间的L2距离最小化以进行策略优化,从而预先培训对话策略模块。最后,对话框生成模型由语言建模预先训练。结果表明,Space-3在八个下游对话框基准中实现最新性能,包括意图预测,对话框状态跟踪和端到端对话框建模。我们还表明,在低资源设置下,Space-3比现有模型具有更强的射击能力。
translated by 谷歌翻译
最近,通过“向导”模拟游戏收集了一类以任务为导向的对话(TOD)数据集。但是,《巫师》数据实际上是模拟的数据,因此与现实生活中的对话根本不同,这些对话更加嘈杂和随意。最近,Seretod挑战赛是组织的,并发布了Mobilecs数据集,该数据集由来自中国移动的真实用户和客户服务人员之间的真实世界对话框组成。基于Mobilecs数据集,Seretod挑战具有两个任务,不仅评估了对话系统本身的构建,而且还检查了对话框成绩单中的信息提取,这对于建立TOD的知识库至关重要。本文主要介绍了Mobilecs数据集对这两项任务的基线研究。我们介绍了如何构建两个基线,遇到的问题以及结果。我们预计基线可以促进令人兴奋的未来研究,以建立针对现实生活任务的人类机器人对话系统。
translated by 谷歌翻译
由于面向任务导向的对话系统在我们的生活中越来越受欢迎,提出并探索了更现实的任务。然而,出现了新的实际挑战。例如,由于在现有公共数据集中缺少这种情况,当前对话系统无法在查询数据库时有效处理多个搜索结果。在本文中,我们提出了数据库搜索结果(DSR)歧义,这是一个专注于消除数据库搜索结果的新任务,这通过允许它们从多个选项中选择了多个选项而不是只有一个来增强用户体验。为研究这项任务,我们增强了受到流行的面向任务的对话数据集(Multimoz和SGD),转弯,由(a)通过预定义的语法和(b)为子集收集人类释义的(b)来解析歧义。我们发现,我们的增强对话数据的培训提高了模型处理模糊方案的能力,而不会牺牲未修改的转弯。此外,即使在没有域名数据的情况下,也有助于我们的模型帮助我们的模型提高DSR消歧的性能,表明它可以被学习为普遍对话技能。我们的数据和代码将公开可用。
translated by 谷歌翻译
One of the biggest challenges of natural language generation (NLG) is the proper handling of named entities. Named entities are a common source of grammar mistakes such as wrong prepositions, wrong article handling, or incorrect entity inflection. Without factoring linguistic representation, such errors are often underrepresented when evaluating on a small set of arbitrarily picked argument values, or when translating a dataset from a linguistically simpler language, like English, to a linguistically complex language, like Russian. However, for some applications, broadly precise grammatical correctness is critical -- native speakers may find entity-related grammar errors silly, jarring, or even offensive. To enable the creation of more linguistically diverse NLG datasets, we release a Corpus of Linguistically Significant Entities (CLSE) annotated by linguist experts. The corpus includes 34 languages and covers 74 different semantic types to support various applications from airline ticketing to video games. To demonstrate one possible use of CLSE, we produce an augmented version of the Schema-Guided Dialog Dataset, SGD-CLSE. Using the CLSE's entities and a small number of human translations, we create a linguistically representative NLG evaluation benchmark in three languages: French (high-resource), Marathi (low-resource), and Russian (highly inflected language). We establish quality baselines for neural, template-based, and hybrid NLG systems and discuss the strengths and weaknesses of each approach.
translated by 谷歌翻译