Modern virtual assistants use internal semantic parsing engines to convert user utterances to actionable commands. However, prior work has demonstrated that semantic parsing is a difficult multilingual transfer task with low transfer efficiency compared to other tasks. In global markets such as India and Latin America, this is a critical issue as switching between languages is prevalent for bilingual users. In this work we dramatically improve the zero-shot performance of a multilingual and codeswitched semantic parsing system using two stages of multilingual alignment. First, we show that constrastive alignment pretraining improves both English performance and transfer efficiency. We then introduce a constrained optimization approach for hyperparameter-free adversarial alignment during finetuning. Our Doubly Aligned Multilingual Parser (DAMP) improves mBERT transfer performance by 3x, 6x, and 81x on the Spanglish, Hinglish and Multilingual Task Oriented Parsing benchmarks respectively and outperforms XLM-R and mT5-Large using 3.2x fewer parameters.

Token free approaches have been successfully applied to a series of word and span level tasks. In this work, we compare a byte-level (ByT5) and a wordpiece based (mT5) sequence to sequence model on the 51 languages of the MASSIVE multilingual semantic parsing dataset. We examine multiple experimental settings: (i) zero-shot, (ii) full gold data and (iii) zero-shot with synthetic data. By leveraging a state-of-the-art label projection method for machine translated examples, we are able to reduce the gap in exact match accuracy to only 5 points with respect to a model trained on gold data from all the languages. We additionally provide insights on the cross-lingual transfer of ByT5 and show how the model compares with respect to mT5 across all parameter sizes.

多语言语言模型（\ mllms），如mbert，xlm，xlm-r，\ textit {etc。}已成为一种可行的选择，使预先估计到大量语言的力量。鉴于他们的成功在零射击转移学习中，在（i）建立更大的\ mllms〜覆盖了大量语言（ii）创建覆盖更广泛的任务和语言来评估的详尽工作基准mllms〜（iii）分析单音零点，零拍摄交叉和双语任务（iv）对Monolingual的性能，了解\ mllms〜（v）增强（通常）学习的通用语言模式（如果有的话）有限的容量\ mllms〜以提高他们在已见甚至看不见语言的表现。在这项调查中，我们审查了现有的文学，涵盖了上述与\ MLLMS有关的广泛研究领域。根据我们的调查，我们建议您有一些未来的研究方向。

将语义解析器定位以支持新语言需要有效的跨语性概括。最近的工作发现了机器翻译或零击方法的成功，尽管这些方法可能难以模拟母语人士如何提出问题。我们考虑如何有效利用新语言的最小注释示例来进行几次跨语性语义解析。我们引入了一阶元学习算法，以在跨语性转移过程中训练具有最大样品效率的语义解析器。我们的算法使用高资源语言来训练解析器，并同时优化低资源语言的跨语性概括。 ATIS上六种语言的结果表明，我们的泛化步骤的组合产生了准确的语义解析器，以每种新语言中的源培训数据$ 10％的$ 10％。我们的方法还使用英语对蜘蛛的竞争模型进行训练，并将其推广到中文，同样对$ 10％的培训数据进行了采样。

我们提出语言学家，这是一种通过微调Alexatm 5B生成带注释数据的方法，用于生成意图分类和插槽标记（IC+ST），这是一种5亿参数的多语言序列到序列（SEQ2SEQ）模型，在灵活的指令上迅速的。在SNIP数据集的10次新颖意图设置中，语言学家超过了最新的方法（反向翻译和示例外推），可以通过宽阔的边距，显示出IC回忆中+1.9点的目标意图的绝对改善ST F1分数和+2.5分。在MATIS ++数据集的零击跨语言设置中，语言学家表现出强大的机器翻译基线，插槽对齐的基线是+4.14的+4.14点在6个语言上绝对在ST F1分数上，同时在IC上匹配IC的性能。最后，我们在用于对话代理IC+ST的内部大规模多语言数据集上验证了我们的结果，并显示了使用背面翻译，释义和插槽目录重新采样采样的基线的显着改进。据我们所知，我们是第一个展示大规模SEQ2SEQ模型的指导微调的人，以控制多语言意图和插槽标记的数据生成的输出。

Universal cross-lingual sentence embeddings map semantically similar cross-lingual sentences into a shared embedding space. Aligning cross-lingual sentence embeddings usually requires supervised cross-lingual parallel sentences. In this work, we propose mSimCSE, which extends SimCSE to multilingual settings and reveal that contrastive learning on English data can surprisingly learn high-quality universal cross-lingual sentence embeddings without any parallel data. In unsupervised and weakly supervised settings, mSimCSE significantly improves previous sentence embedding methods on cross-lingual retrieval and multilingual STS tasks. The performance of unsupervised mSimCSE is comparable to fully supervised methods in retrieving low-resource languages and multilingual STS. The performance can be further enhanced when cross-lingual NLI data is available. Our code is publicly available at https://github.com/yaushian/mSimCSE.

跨语言嵌入（CLWE）已被证明在许多跨语性任务中有用。但是，大多数现有的学习Clwe的方法，包括具有上下文嵌入的方法是无知的。在这项工作中，我们提出了一个新颖的框架，以通过仅利用双语词典的跨语性信号来使上下文嵌入在感觉层面上。我们通过首先提出一种新颖的感知感知的跨熵损失来明确地提出一种新颖的感知跨熵损失来实现我们的框架。通过感知感知的跨熵损失预算的单语Elmo和BERT模型显示出对单词感官歧义任务的显着改善。然后，我们提出了一个感官对齐目标，除了跨语义模型预训练的感知感知跨熵损失以及几种语言对的跨语义模型（英语对德语/西班牙语/日本/中文）。与最佳的基线结果相比，我们的跨语言模型分别在零摄影，情感分类和XNLI任务上达到0.52％，2.09％和1.29％的平均绩效提高。

有监督的基于深度学习的方法已应用于以任务为导向的对话框，并在有足够数量的培训示例可用时对有限的域和语言应用有效。在实践中，这些方法遭受了域驱动设计和资源不足的语言的缺点。域和语言模型应该随着问题空间的发展而增长和变化。一方面，对转移学习的研究证明了基于多语言变压器模型学习语义丰富的表示的跨语性能力。另一方面，除了上述方法之外，元学习还能够开发任务和语言学习算法，能够实现泛滥。在这种情况下，本文提出了使用典型的神经网络和基于多语言变压器的模型来研究使用协同进行几次学习的跨语性可传递性。自然语言的实验理解多亚提斯++语料库的任务表明，我们的方法基本上改善了低资源和高资源语言之间观察到的转移学习表现。更普遍地说，我们的方法证实，可以将具有特定语言的有意义的潜在空间推广到使用元学习的情况下看不见和资源不足的潜在空间。

在这项工作中，我们提出了一个系统的实证研究，专注于最先进的多语言编码器在跨越多种不同语言对的交叉语言文档和句子检索任务的适用性。我们首先将这些模型视为多语言文本编码器，并在无监督的ad-hoc句子和文档级CLIR中基准性能。与监督语言理解相比，我们的结果表明，对于无监督的文档级CLIR - 一个没有针对IR特定的微调 - 预训练的多语言编码器的相关性判断，平均未能基于CLWE显着优于早期模型。对于句子级检索，我们确实获得了最先进的性能：然而，通过多语言编码器来满足高峰分数，这些编码器已经进一步专注于监督的时尚，以便句子理解任务，而不是使用他们的香草'现货'变体。在这些结果之后，我们介绍了文档级CLIR的本地化相关性匹配，在那里我们独立地对文件部分进行了查询。在第二部分中，我们评估了在一系列零拍语言和域转移CLIR实验中的英语相关数据中进行微调的微调编码器精细调整的微调我们的结果表明，监督重新排名很少提高多语言变压器作为无监督的基数。最后，只有在域名对比度微调（即，同一域名，只有语言转移），我们设法提高排名质量。我们在目标语言中单次检索的交叉定向检索结果和结果（零拍摄）交叉传输之间的显着实证差异，这指出了在单机数据上训练的检索模型的“单声道过度装备”。

对于多语言序列到序列预审预周序模型（多语言SEQ2SEQ PLM），例如姆巴特（Mbart），自制的预处理任务接受了多种单语言的培训，例如25种来自CommonCrawl的语言，而下游的跨语言任务通常在双语语言子集上进行，例如英语 - 德国人，存在数据差异，即领域的差异，以及跨语言学习客观差异，即在训练和填充阶段之间的任务差异。为了弥合上述跨语言域和任务差距，我们将使用额外的代码切换恢复任务扩展了香草预后管道。具体而言，第一阶段采用自我监督的代码转换还原任务作为借口任务，从而允许多语言SEQ2SEQ PLM获取一些域内对齐信息。在第二阶段，我们正常在下游数据上微调模型。 NLG评估（12个双语翻译任务，30个零射击任务和2项跨语言摘要任务）和NLU评估（7个跨语性自然语言推理任务）的实验表明，我们的模型超过了强大的基线MBART，具有标准的FINETUNNING，这表明了我们的模型策略，一致。分析表明，我们的方法可以缩小跨语性句子表示的欧几里得距离，并通过微不足道的计算成本改善模型概括。我们在：https：//github.com/zanchangtong/csr4mbart上发布代码。

State-of-the-art natural language processing systems rely on supervision in the form of annotated data to learn competent models. These models are generally trained on data in a single language (usually English), and cannot be directly used beyond that language. Since collecting data in every language is not realistic, there has been a growing interest in crosslingual language understanding (XLU) and low-resource cross-language transfer. In this work, we construct an evaluation set for XLU by extending the development and test sets of the Multi-Genre Natural Language Inference Corpus (MultiNLI) to 15 languages, including low-resource languages such as Swahili and Urdu. We hope that our dataset, dubbed XNLI, will catalyze research in cross-lingual sentence understanding by providing an informative standard evaluation task. In addition, we provide several baselines for multilingual sentence understanding, including two based on machine translation systems, and two that use parallel data to train aligned multilingual bag-of-words and LSTM encoders. We find that XNLI represents a practical and challenging evaluation suite, and that directly translating the test data yields the best performance among available baselines.

与辅助语言的元学习已经表明了对交叉语言自然语言处理的有希望的改进。然而，以前的研究采样使用相同语言的元培训和元测试数据，这限制了模型交叉传输的能力。在本文中，我们提出了XLA-MAML，在元学习阶段执行直接交叉调整。我们对自然语言推理和问题进行零射击和几次拍摄实验。实验结果表明了我们在不同语言，任务和预磨料模型中的方法的有效性。我们还对元学习的各种交叉特定设置进行了分析，包括采样策略和并行性。

Much recent progress in applications of machine learning models to NLP has been driven by benchmarks that evaluate models across a wide variety of tasks. However, these broad-coverage benchmarks have been mostly limited to English, and despite an increasing interest in multilingual models, a benchmark that enables the comprehensive evaluation of such methods on a diverse range of languages and tasks is still missing. To this end, we introduce the Cross-lingual TRansfer Evaluation of Multilingual Encoders (XTREME) benchmark, a multi-task benchmark for evaluating the cross-lingual generalization capabilities of multilingual representations across 40 languages and 9 tasks. We demonstrate that while models tested on English reach human performance on many tasks, there is still a sizable gap in the performance of cross-lingually transferred models, particularly on syntactic and sentence retrieval tasks. There is also a wide spread of results across languages. We release the benchmark 1 to encourage research on cross-lingual learning methods that transfer linguistic knowledge across a diverse and representative set of languages and tasks.

Task-oriented dialogue (TOD) systems have been applied in a range of domains to support human users to achieve specific goals. Systems are typically constructed for a single domain or language and do not generalise well beyond this. Their extension to other languages in particular is restricted by the lack of available training data for many of the world's languages. To support work on Natural Language Understanding (NLU) in TOD across multiple languages and domains simultaneously, we constructed MULTI3NLU++, a multilingual, multi-intent, multi-domain dataset. MULTI3NLU++ extends the English-only NLU++ dataset to include manual translations into a range of high, medium and low resource languages (Spanish, Marathi, Turkish and Amharic), in two domains (banking and hotels). MULTI3NLU++ inherits the multi-intent property of NLU++, where an utterance may be labelled with multiple intents, providing a more realistic representation of a user's goals and aligning with the more complex tasks that commercial systems aim to model. We use MULTI3NLU++ to benchmark state-of-the-art multilingual language models as well as Machine Translation and Question Answering systems for the NLU task of intent detection for TOD systems in the multilingual setting. The results demonstrate the challenging nature of the dataset, particularly in the low-resource language setting.

MARCO排名数据集已广泛用于培训IR任务的深度学习模型，在不同的零射击方案上实现了相当大的效果。但是，这种类型的资源是英语以外的语言的稀缺。在这项工作中，我们呈现MMARCO，MS Marco段落的多语言版本，该数据集包括使用机器翻译创建的13种语言。我们通过微调单语和多语言重新排名模型以及此数据集的密集多语言模型进行了评估。实验结果表明，在我们翻译的数据集上微调微调的多语言模型可以单独对原始英文版的模型进行微调的卓越效果。我们蒸馏的多语言RE-RANKER与非蒸馏模型具有竞争力，而参数较少的5.4倍。最后，我们展现了翻译质量和检索效果之间的正相关性，提供了证据，即翻译方法的改进可能导致多语言信息检索的改进。翻译的数据集和微调模型可在https://github.com/unicamp-dl/mmarco.git上获得。

Translating training data into many languages has emerged as a practical solution for improving cross-lingual transfer. For tasks that involve span-level annotations, such as information extraction or question answering, an additional label projection step is required to map annotated spans onto the translated texts. Recently, a few efforts have utilized a simple mark-then-translate method to jointly perform translation and projection by inserting special markers around the labeled spans in the original sentence. However, as far as we are aware, no empirical analysis has been conducted on how this approach compares to traditional annotation projection based on word alignment. In this paper, we present an extensive empirical study across 42 languages and three tasks (QA, NER, and Event Extraction) to evaluate the effectiveness and limitations of both methods, filling an important gap in the literature. Experimental results show that our optimized version of mark-then-translate, which we call EasyProject, is easily applied to many languages and works surprisingly well, outperforming the more complex word alignment-based methods. We analyze several key factors that affect end-task performance, and show EasyProject works well because it can accurately preserve label span boundaries after translation. We will publicly release all our code and data.

发展任务导向的对话助理的实用需求需要了解许多语言。多语言自然语言理解（NLU）的新型基准包括多种语言中的单声道句，用意图和插槽注释。在这种设置模型中，用于交叉传输在联合意图识别和槽填充方面表现出显着性能。然而，现有的基准缺乏代码切换话语，这难以收集和标签由于语法结构的复杂性。对于NLU模型的评估似乎偏见和有限，因为代码切换被遗漏了范围。我们的工作采用认可的方法来生成合理的和自然探测的代码切换话语，并使用它们来创建合成代码交换测试集。基于实验，我们报告说，最先进的NLU模型无法处理代码切换。在最糟糕的是，性能，通过语义精度评估，从横跨80 \％的8 \％的低至15 \％。此外，我们展示了，对合成码混合数据进行预训练有助于在具有单晶体数据的可比水平上保持所提出的测试中的性能。最后，我们分析了不同的语言对并表明语言越近，NLU模型越好地处理了交替。这符合对多语种模型在语言之间进行转移的共同理解

以前的工作主要侧重于改善NLU任务的交叉传输，具有多语言预用编码器（MPE），或提高与伯特的监督机器翻译的性能。然而，探索了，MPE是否可以有助于促进NMT模型的交叉传递性。在本文中，我们专注于NMT中的零射频转移任务。在此任务中，NMT模型培训，只有一个语言对的并行数据集和搁置架MPE，然后它直接测试在零拍语言对上。我们为此任务提出了Sixt，一个简单而有效的模型。 SIXT利用了两阶段培训计划利用MPE，并进一步改进了解离编码器和容量增强的解码器。使用此方法，SIMPT显着优于MBart，这是一个用于NMT的预磨削的多语言编码器解码器模型，平均改善了14个源语言的零拍摄的任何英语测试集上的7.1 BLEU。此外，培训计算成本和培训数据较少，我们的模型在15个任何英语测试组上实现了比Criss和M2M-100，两个强大的多语言NMT基线更好的性能。

Contrastive learning has been successfully used for retrieval of semantically aligned sentences, but it often requires large batch sizes or careful engineering to work well. In this paper, we instead propose a generative model for learning multilingual text embeddings which can be used to retrieve or score sentence pairs. Our model operates on parallel data in $N$ languages and, through an approximation we introduce, efficiently encourages source separation in this multilingual setting, separating semantic information that is shared between translations from stylistic or language-specific variation. We show careful large-scale comparisons between contrastive and generation-based approaches for learning multilingual text embeddings, a comparison that has not been done to the best of our knowledge despite the popularity of these approaches. We evaluate this method on a suite of tasks including semantic similarity, bitext mining, and cross-lingual question retrieval -- the last of which we introduce in this paper. Overall, our Variational Multilingual Source-Separation Transformer (VMSST) model outperforms both a strong contrastive and generative baseline on these tasks.

Software engineers working with the same programming language (PL) may speak different natural languages (NLs) and vice versa, erecting huge barriers to communication and working efficiency. Recent studies have demonstrated the effectiveness of generative pre-training in computer programs, yet they are always English-centric. In this work, we step towards bridging the gap between multilingual NLs and multilingual PLs for large language models (LLMs). We release ERNIE-Code, a unified pre-trained language model for 116 NLs and 6 PLs. We employ two methods for universal cross-lingual pre-training: span-corruption language modeling that learns patterns from monolingual NL or PL; and pivot-based translation language modeling that relies on parallel data of many NLs and PLs. Extensive results show that ERNIE-Code outperforms previous multilingual LLMs for PL or NL across a wide range of end tasks of code intelligence, including multilingual code-to-text, text-to-code, code-to-code, and text-to-text generation. We further show its advantage of zero-shot prompting on multilingual code summarization and text-to-text translation. We will make our code and pre-trained models publicly available.