预处理的多语言编码器可实现零拍的跨语性转移，但通常会产生不可靠的模型，这些模型在目标语言上表现出高性能差异。我们假设这种高差异是由零拍的跨语性转移解决了一个不明显的优化问题。我们表明，源语言单语言模型和源 +目标双语模型之间的任何线性交互模型都具有较低的源语言概括错误，但是当我们从单语模型移动到双语模型时，目标语言概括误差会顺利而线性地降低，这表明该模型努力仅使用源语言来识别源和目标语言的良好解决方案。此外，我们表明零击解决方案在于目标语言误差概括表面的非平板区域，从而导致较高的方差。

Much recent progress in applications of machine learning models to NLP has been driven by benchmarks that evaluate models across a wide variety of tasks. However, these broad-coverage benchmarks have been mostly limited to English, and despite an increasing interest in multilingual models, a benchmark that enables the comprehensive evaluation of such methods on a diverse range of languages and tasks is still missing. To this end, we introduce the Cross-lingual TRansfer Evaluation of Multilingual Encoders (XTREME) benchmark, a multi-task benchmark for evaluating the cross-lingual generalization capabilities of multilingual representations across 40 languages and 9 tasks. We demonstrate that while models tested on English reach human performance on many tasks, there is still a sizable gap in the performance of cross-lingually transferred models, particularly on syntactic and sentence retrieval tasks. There is also a wide spread of results across languages. We release the benchmark 1 to encourage research on cross-lingual learning methods that transfer linguistic knowledge across a diverse and representative set of languages and tasks.

In this paper, we show that Multilingual BERT (M-BERT), released by Devlin et al. (2019) as a single language model pre-trained from monolingual corpora in 104 languages, is surprisingly good at zero-shot cross-lingual model transfer, in which task-specific annotations in one language are used to fine-tune the model for evaluation in another language. To understand why, we present a large number of probing experiments, showing that transfer is possible even to languages in different scripts, that transfer works best between typologically similar languages, that monolingual corpora can train models for code-switching, and that the model can find translation pairs. From these results, we can conclude that M-BERT does create multilingual representations, but that these representations exhibit systematic deficiencies affecting certain language pairs.

Multilingual BERT (mBERT) has demonstrated considerable cross-lingual syntactic ability, whereby it enables effective zero-shot cross-lingual transfer of syntactic knowledge. The transfer is more successful between some languages, but it is not well understood what leads to this variation and whether it fairly reflects difference between languages. In this work, we investigate the distributions of grammatical relations induced from mBERT in the context of 24 typologically different languages. We demonstrate that the distance between the distributions of different languages is highly consistent with the syntactic difference in terms of linguistic formalisms. Such difference learnt via self-supervision plays a crucial role in the zero-shot transfer performance and can be predicted by variation in morphosyntactic properties between languages. These results suggest that mBERT properly encodes languages in a way consistent with linguistic diversity and provide insights into the mechanism of cross-lingual transfer.

我们考虑使用最新的MultieRlex数据集中考虑法律主题分类中的零射击跨语性转移。由于原始数据集包含并行文档，这对于零拍传输不现实是不现实的，因此我们开发了一个没有并行文档的数据集的新版本。我们使用它来表明，基于翻译的方法非常优于多绘制预训练的模型，这是多曲线的最佳先前的零弹性传输方法。我们还开发了一种双语的教师零摄像转移方法，该方法利用了目标语言的其他未标记文档，并且比直接在标记的目标语言文档上进行微调的模型更好。

Pre-trained multilingual language models show significant performance gains for zero-shot cross-lingual model transfer on a wide range of natural language understanding (NLU) tasks. Previously, for zero-shot cross-lingual evaluation, pre-trained models are only fine-tuned on English data and tested on a variety of target languages. In this paper, we do cross-lingual evaluation on various NLU tasks (sentence classification, sequence labeling, question answering) using prompt-tuning and compare it with fine-tuning. The results show that prompt tuning achieves much better cross-lingual transfer than fine-tuning across datasets, with only 0.1% to 0.3% tuned parameters. Additionally, we demonstrate through the analysis that prompt tuning can have better cross-lingual transferability of representations on downstream tasks with better aligned decision boundaries.

以前的工作主要侧重于改善NLU任务的交叉传输，具有多语言预用编码器（MPE），或提高与伯特的监督机器翻译的性能。然而，探索了，MPE是否可以有助于促进NMT模型的交叉传递性。在本文中，我们专注于NMT中的零射频转移任务。在此任务中，NMT模型培训，只有一个语言对的并行数据集和搁置架MPE，然后它直接测试在零拍语言对上。我们为此任务提出了Sixt，一个简单而有效的模型。 SIXT利用了两阶段培训计划利用MPE，并进一步改进了解离编码器和容量增强的解码器。使用此方法，SIMPT显着优于MBart，这是一个用于NMT的预磨削的多语言编码器解码器模型，平均改善了14个源语言的零拍摄的任何英语测试集上的7.1 BLEU。此外，培训计算成本和培训数据较少，我们的模型在15个任何英语测试组上实现了比Criss和M2M-100，两个强大的多语言NMT基线更好的性能。

最近的工作表明，通过多语种伯爵（MBENT）获得的知识有两个组件：特定于语言和语言中立的。本文分析了它们之间的关系，在两项任务的微调 - POS标记和自然语言推理的背景下 - 需要模型带来不同的语言特异性知识。可视化揭示MBERT失去了在微调后通过语言进行群集表示的能力，这是通过语言识别实验的证据支持的结果。然而，显示使用梯度逆转和迭代对抗性学习的“无学习”语言特定表示的进一步实验，不会在微调的效果之外增加对独立于语言无关的组件的进一步改进。此处提出的结果表明，微调的过程导致模型的重组有限的代表能力，以特定于语言特定的代表性的语言无关的表示。

将语义解析器定位以支持新语言需要有效的跨语性概括。最近的工作发现了机器翻译或零击方法的成功，尽管这些方法可能难以模拟母语人士如何提出问题。我们考虑如何有效利用新语言的最小注释示例来进行几次跨语性语义解析。我们引入了一阶元学习算法，以在跨语性转移过程中训练具有最大样品效率的语义解析器。我们的算法使用高资源语言来训练解析器，并同时优化低资源语言的跨语性概括。 ATIS上六种语言的结果表明，我们的泛化步骤的组合产生了准确的语义解析器，以每种新语言中的源培训数据$ 10％的$ 10％。我们的方法还使用英语对蜘蛛的竞争模型进行训练，并将其推广到中文，同样对$ 10％的培训数据进行了采样。

多语言语言模型（\ mllms），如mbert，xlm，xlm-r，\ textit {etc。}已成为一种可行的选择，使预先估计到大量语言的力量。鉴于他们的成功在零射击转移学习中，在（i）建立更大的\ mllms〜覆盖了大量语言（ii）创建覆盖更广泛的任务和语言来评估的详尽工作基准mllms〜（iii）分析单音零点，零拍摄交叉和双语任务（iv）对Monolingual的性能，了解\ mllms〜（v）增强（通常）学习的通用语言模式（如果有的话）有限的容量\ mllms〜以提高他们在已见甚至看不见语言的表现。在这项调查中，我们审查了现有的文学，涵盖了上述与\ MLLMS有关的广泛研究领域。根据我们的调查，我们建议您有一些未来的研究方向。

虽然最近关于多语种语言模型的工作已经证明了他们对下游任务的交叉零射击传输的能力，但社区缺乏符合语言之间的共享属性，可以实现这种转移。涉及成对的自然语言的分析通常是不确定的，并且矛盾以来，许多语言方面同时不同。在本文中，我们进行大规模的实证研究，通过测量四种不同的自然语言和通过修改脚本，单词顺序和语法等方面构造的零拍摄传递来隔离各种语言特性的影响。在其他事情之外，我们的实验表明，当语言的单词顺序不同时，缺乏子字重叠显着影响零拍摄传输，并且在语言之间的传输性能和Word嵌入对准之间存在强烈相关性（例如，r = 0.94关于NLI的任务）。我们的结果呼吁专注于在明确改进语言之间的嵌入对齐而不是依赖于隐含的出现。

与辅助语言的元学习已经表明了对交叉语言自然语言处理的有希望的改进。然而，以前的研究采样使用相同语言的元培训和元测试数据，这限制了模型交叉传输的能力。在本文中，我们提出了XLA-MAML，在元学习阶段执行直接交叉调整。我们对自然语言推理和问题进行零射击和几次拍摄实验。实验结果表明了我们在不同语言，任务和预磨料模型中的方法的有效性。我们还对元学习的各种交叉特定设置进行了分析，包括采样策略和并行性。

Universal cross-lingual sentence embeddings map semantically similar cross-lingual sentences into a shared embedding space. Aligning cross-lingual sentence embeddings usually requires supervised cross-lingual parallel sentences. In this work, we propose mSimCSE, which extends SimCSE to multilingual settings and reveal that contrastive learning on English data can surprisingly learn high-quality universal cross-lingual sentence embeddings without any parallel data. In unsupervised and weakly supervised settings, mSimCSE significantly improves previous sentence embedding methods on cross-lingual retrieval and multilingual STS tasks. The performance of unsupervised mSimCSE is comparable to fully supervised methods in retrieving low-resource languages and multilingual STS. The performance can be further enhanced when cross-lingual NLI data is available. Our code is publicly available at https://github.com/yaushian/mSimCSE.

最近，大型预用语言模型（LMS）越来越受欢迎。培训这些模型需要更多的计算资源，并且大多数现有模型仅在英文文本上培训。以其他语言训练这些模型非常昂贵。为了缓解这个问题，我们介绍了一种叫做威施塞的方法 - 将英语模型传输到新语言。我们将英语模型的销量与目标语言中的销量交换，并初始化令牌嵌入式，以便通过利用覆盖英语和目标语言的多语言静态字嵌入来初始化令牌嵌入式。我们使用Wechsel将GPT-2和Roberta模型转移到4种其他语言（法语，德语，中文和斯瓦希里语）。 Wechsel通过以前提出的跨语言参数转移和优于比较大小的模型来改善从目标语言的划痕训练的相当大小的型号，距离培训速度较小。我们的方法使培训大型语言模型为新语言更容易访问，更少损害环境。我们宣传我们的代码和型号。

Multi-lingual language models (LM), such as mBERT, XLM-R, mT5, mBART, have been remarkably successful in enabling natural language tasks in low-resource languages through cross-lingual transfer from high-resource ones. In this work, we try to better understand how such models, specifically mT5, transfer *any* linguistic and semantic knowledge across languages, even though no explicit cross-lingual signals are provided during pre-training. Rather, only unannotated texts from each language are presented to the model separately and independently of one another, and the model appears to implicitly learn cross-lingual connections. This raises several questions that motivate our study, such as: Are the cross-lingual connections between every language pair equally strong? What properties of source and target language impact the strength of cross-lingual transfer? Can we quantify the impact of those properties on the cross-lingual transfer? In our investigation, we analyze a pre-trained mT5 to discover the attributes of cross-lingual connections learned by the model. Through a statistical interpretation framework over 90 language pairs across three tasks, we show that transfer performance can be modeled by a few linguistic and data-derived features. These observations enable us to interpret cross-lingual understanding of the mT5 model. Through these observations, one can favorably choose the best source language for a task, and can anticipate its training data demands. A key finding of this work is that similarity of syntax, morphology and phonology are good predictors of cross-lingual transfer, significantly more than just the lexical similarity of languages. For a given language, we are able to predict zero-shot performance, that increases on a logarithmic scale with the number of few-shot target language data points.

Misinformation spread over social media has become an undeniable infodemic. However, not all spreading claims are made equal. If propagated, some claims can be destructive, not only on the individual level, but to organizations and even countries. Detecting claims that should be prioritized for fact-checking is considered the first step to fight against spread of fake news. With training data limited to a handful of languages, developing supervised models to tackle the problem over lower-resource languages is currently infeasible. Therefore, our work aims to investigate whether we can use existing datasets to train models for predicting worthiness of verification of claims in tweets in other languages. We present a systematic comparative study of six approaches for cross-lingual check-worthiness estimation across pairs of five diverse languages with the help of Multilingual BERT (mBERT) model. We run our experiments using a state-of-the-art multilingual Twitter dataset. Our results show that for some language pairs, zero-shot cross-lingual transfer is possible and can perform as good as monolingual models that are trained on the target language. We also show that in some languages, this approach outperforms (or at least is comparable to) state-of-the-art models.

在多语言甚至单语言中鉴定的模型的零拍跨语言能力刺激了许多假设，以解释这一有趣的经验结果。但是，由于预处理的成本，大多数研究都使用公共模型的公共模型，其预处理方法（例如代币化，语料库规模和计算预算的选择）可能会大不相同。当研究人员对自己的模型预识时，他们通常会在预算有限的情况下这样做，并且与SOTA模型相比，最终的模型的表现可能明显不足。这些实验差异导致有关这些模型跨语性能力的性质的各种不一致的结论。为了帮助对该主题进行进一步研究，我们发布了10个单语字节级模型，并在相同的配置下进行了严格审慎的概述，并具有大型计算预算（相当于V100的420天）和Corpora，比原始BERT大4倍。由于它们不含令牌，因此消除了看不见的令牌嵌入的问题，从而使研究人员可以在具有不同脚本的语言中尝试更广泛的跨语言实验。此外，我们释放了在不自然语言文本上预测的两个模型，这些模型可用于理智检查实验。关于质量检查和NLI任务的实验表明，我们的单语模型实现了多语言的竞争性能，因此可以加强我们对语言模型中跨语性可传递性的理解。

虽然对多语言视觉语言预测的模型实现了一些好处，但是当将多句预训练的视力语言模型应用于非英语数据时，各种任务和语言的最新基准测试表明，跨语性概括不佳，并且在有监督之间存在很大的差距（ ）英语表现和（零射）跨语性转移。在这项工作中，我们探讨了这些模型在零拍的跨语性视觉响应（VQA）任务上的糟糕性能，其中模型在英语视觉问题数据上进行了微调，并对7种类型上多样的语言进行了评估。我们通过三种策略改善了跨语性转移：（1）我们引入了语言的先验目标，以增加基于相似性损失以指导模型在培训期间的跨渗透损失，（2）我们学习了一个特定于任务的子网络，改善跨语性概括并减少不修改模型的方差，（3）我们使用合成代码混合来扩大培训示例，以促进源和目标语言之间的嵌入。我们使用预审计的多语言多模式变压器UC2和M3P进行的XGQA实验证明了针对7种语言提出的微调策略的一致有效性，以稀疏模型优于现有的转移方法。复制我们发现的代码和数据已公开可用。

Translating training data into many languages has emerged as a practical solution for improving cross-lingual transfer. For tasks that involve span-level annotations, such as information extraction or question answering, an additional label projection step is required to map annotated spans onto the translated texts. Recently, a few efforts have utilized a simple mark-then-translate method to jointly perform translation and projection by inserting special markers around the labeled spans in the original sentence. However, as far as we are aware, no empirical analysis has been conducted on how this approach compares to traditional annotation projection based on word alignment. In this paper, we present an extensive empirical study across 42 languages and three tasks (QA, NER, and Event Extraction) to evaluate the effectiveness and limitations of both methods, filling an important gap in the literature. Experimental results show that our optimized version of mark-then-translate, which we call EasyProject, is easily applied to many languages and works surprisingly well, outperforming the more complex word alignment-based methods. We analyze several key factors that affect end-task performance, and show EasyProject works well because it can accurately preserve label span boundaries after translation. We will publicly release all our code and data.

State-of-the-art natural language processing systems rely on supervision in the form of annotated data to learn competent models. These models are generally trained on data in a single language (usually English), and cannot be directly used beyond that language. Since collecting data in every language is not realistic, there has been a growing interest in crosslingual language understanding (XLU) and low-resource cross-language transfer. In this work, we construct an evaluation set for XLU by extending the development and test sets of the Multi-Genre Natural Language Inference Corpus (MultiNLI) to 15 languages, including low-resource languages such as Swahili and Urdu. We hope that our dataset, dubbed XNLI, will catalyze research in cross-lingual sentence understanding by providing an informative standard evaluation task. In addition, we provide several baselines for multilingual sentence understanding, including two based on machine translation systems, and two that use parallel data to train aligned multilingual bag-of-words and LSTM encoders. We find that XNLI represents a practical and challenging evaluation suite, and that directly translating the test data yields the best performance among available baselines.