Using a single model across various tasks is beneficial for training and applying deep neural sequence models. We address the problem of developing generalist representations of text that can be used to perform a range of different tasks rather than being specialised to a single application. We focus on processing short questions and developing an embedding for these questions that is useful on a diverse set of problems, such as question topic classification, equivalent question recognition, and question answering. This paper introduces QBERT, a generalist model for processing questions. With QBERT, we demonstrate how we can train a multi-task network that performs all question-related tasks and has achieved similar performance compared to its corresponding single-task models.

问题答案（QA）是自然语言处理中最具挑战性的最具挑战性的问题之一（NLP）。问答（QA）系统试图为给定问题产生答案。这些答案可以从非结构化或结构化文本生成。因此，QA被认为是可以用于评估文本了解系统的重要研究区域。大量的QA研究致力于英语语言，调查最先进的技术和实现最先进的结果。然而，由于阿拉伯QA中的研究努力和缺乏大型基准数据集，在阿拉伯语问答进展中的研究努力得到了很大速度的速度。最近许多预先接受的语言模型在许多阿拉伯语NLP问题中提供了高性能。在这项工作中，我们使用四个阅读理解数据集来评估阿拉伯QA的最先进的接种变压器模型，它是阿拉伯语 - 队，ArcD，AQAD和TYDIQA-GoldP数据集。我们微调并比较了Arabertv2基础模型，ArabertV0.2大型型号和ARAElectra模型的性能。在最后，我们提供了一个分析，了解和解释某些型号获得的低绩效结果。

我们提供了从文本到文本变换器（T5）的第一次探索句子嵌入式。句子嵌入式广泛适用于语言处理任务。虽然T5在作为序列到序列映射问题的语言任务上实现令人印象深刻的性能，但目前尚不清楚如何从编码器解码器模型生成陈列嵌入的句子。我们调查三种方法提取T5句子嵌入方法：两个仅利用T5编码器，一个使用全T5编码器解码器模型。为了支持我们的调查，我们建立了一个新的句子代表转移基准，SentGlue，它将Senteval Toolkit扩展到粘合基准的九个任务。我们的编码器的型号优于Senteval和SentGlue传输任务的句子 - BERT和SIMCSE句子嵌入，包括语义文本相似性（STS）。发现从数百万到数十亿参数的缩放T5产生一致的进一步改进。最后，我们的编码器 - 解码器方法在使用句子嵌入时在STS上实现了新的最先进的。我们的模型在https://tfhub.dev/google/collections/sentence-t5/1发布。

We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models (Peters et al., 2018a;Radford et al., 2018), BERT is designed to pretrain deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be finetuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial taskspecific architecture modifications.BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

大型语言模型在各种任务上显示出令人印象深刻的几次结果。但是，当知识是此类结果的关键时，就像问题回答和事实检查之类的任务一样，似乎需要存储知识的大量参数计数。众所周知，检索增强模型可以在不需要多个参数的情况下在知识密集的任务上表现出色，但是目前尚不清楚它们是否在几个弹药设置中工作。在这项工作中，我们介绍了地图集，这是一个经过精心设计和预先训练的增强语言模型，能够通过很少的培训示例学习知识密集型任务。我们对包括MMLU，苏格兰短裙和归类等各种任务进行评估，并研究文档索引内容的影响，表明它可以很容易地进行更新。值得注意的是，在自然问题上仅使用64个示例在自然问题上达到超过42 \％的准确性，尽管参数少了50倍，但比540B参数模型的表现优于540b参数模型。

BERT (Devlin et al., 2018) and RoBERTa has set a new state-of-the-art performance on sentence-pair regression tasks like semantic textual similarity (STS). However, it requires that both sentences are fed into the network, which causes a massive computational overhead: Finding the most similar pair in a collection of 10,000 sentences requires about 50 million inference computations (~65 hours) with BERT. The construction of BERT makes it unsuitable for semantic similarity search as well as for unsupervised tasks like clustering.In this publication, we present Sentence-BERT (SBERT), a modification of the pretrained BERT network that use siamese and triplet network structures to derive semantically meaningful sentence embeddings that can be compared using cosine-similarity. This reduces the effort for finding the most similar pair from 65 hours with BERT / RoBERTa to about 5 seconds with SBERT, while maintaining the accuracy from BERT.We evaluate SBERT and SRoBERTa on common STS tasks and transfer learning tasks, where it outperforms other state-of-the-art sentence embeddings methods. 1

Open-domain question answering relies on efficient passage retrieval to select candidate contexts, where traditional sparse vector space models, such as TF-IDF or BM25, are the de facto method. In this work, we show that retrieval can be practically implemented using dense representations alone, where embeddings are learned from a small number of questions and passages by a simple dualencoder framework. When evaluated on a wide range of open-domain QA datasets, our dense retriever outperforms a strong Lucene-BM25 system greatly by 9%-19% absolute in terms of top-20 passage retrieval accuracy, and helps our end-to-end QA system establish new state-of-the-art on multiple open-domain QA benchmarks. 1 * Equal contribution 1 The code and trained models have been released at https://github.com/facebookresearch/DPR.

Existing pre-training methods for extractive Question Answering (QA) generate cloze-like queries different from natural questions in syntax structure, which could overfit pre-trained models to simple keyword matching. In order to address this problem, we propose a novel Momentum Contrastive pRe-training fOr queStion anSwering (MCROSS) method for extractive QA. Specifically, MCROSS introduces a momentum contrastive learning framework to align the answer probability between cloze-like and natural query-passage sample pairs. Hence, the pre-trained models can better transfer the knowledge learned in cloze-like samples to answering natural questions. Experimental results on three benchmarking QA datasets show that our method achieves noticeable improvement compared with all baselines in both supervised and zero-shot scenarios.

广义文本表示是许多自然语言理解任务的基础。要充分利用不同的语料库，不可避免地需要了解它们之间的相关性。但是，许多方法忽略了相关性，并直接用于所有任务的单通道模型（粗糙的范式），这缺乏足够的理性和解释。此外，一些现有的作品通过针迹技能块（一个精细的范式）学习下游任务，这可能会导致其冗余和噪音，从而导致非理性。在这项工作中，我们首先通过三种不同的观点分析任务相关性，即数据属性，手动设计和基于模型的相关性，基于相似的任务被分组在一起。然后，我们提出了一个用粗到细范式的层次结构框架，其最底层共享了所有任务，中层级别分为不同的组，以及分配给每个任务的顶级级别。这使我们的模型可以从所有任务中学习基本的语言属性，提高相关任务的性能，并减少不相关任务的负面影响。我们在五个自然语言理解任务的13个基准数据集上进行的实验证明了我们方法的优势。

This paper presents a new UNIfied pre-trained Language Model (UNILM) that can be fine-tuned for both natural language understanding and generation tasks. The model is pre-trained using three types of language modeling tasks: unidirectional, bidirectional, and sequence-to-sequence prediction. The unified modeling is achieved by employing a shared Transformer network and utilizing specific self-attention masks to control what context the prediction conditions on. UNILM compares favorably with BERT on the GLUE benchmark, and the SQuAD 2.0 and CoQA question answering tasks. Moreover, UNILM achieves new state-ofthe-art results on five natural language generation datasets, including improving the CNN/DailyMail abstractive summarization ROUGE-L to 40.51 (2.04 absolute improvement), the Gigaword abstractive summarization ROUGE-L to 35.75 (0.86 absolute improvement), the CoQA generative question answering F1 score to 82.5 (37.1 absolute improvement), the SQuAD question generation BLEU-4 to 22.12 (3.75 absolute improvement), and the DSTC7 document-grounded dialog response generation NIST-4 to 2.67 (human performance is 2.65). The code and pre-trained models are available at https://github.com/microsoft/unilm. * Equal contribution. † Contact person.

文本匹配是信息检索和自然语言处理的基本技术。文本匹配任务共享确定两个给定文本之间关系的相同范例。这些关系因任务而异，例如〜在文档检索中相关性，释义识别中的语义一致性和所回答的可回答判断。但是，文本匹配的基本信号保留在有限范围中，即〜精确匹配，语义匹配和推理匹配。理想情况下，良好的文本匹配模型可以学会捕获和汇总这些信号，以实现不同的匹配任务以实现竞争性能，而最近的最新文本匹配模型，例如〜预训练的语言模型（PLM）很难概括。这是因为在特定于任务的数据集上的端到端监督学习使模型过分强调了数据示例偏置和特定于任务的信号，而不是基本的匹配信号。为了克服这个问题，我们采用了专业化的将军培训策略，并将其称为比赛推出。在专业阶段，对不同匹配任务的描述映射到一些提示令牌。在概括阶段，匹配模型通过接受各种匹配任务的培训来探索基本匹配信号。高不同的匹配任务避免了模型拟合特定任务的数据偏差，因此该模型可以专注于学习基本匹配信号。同时，在第一步中获得的提示令牌有助于模型区分不同的特定任务匹配信号。公共数据集上的实验结果表明，匹配点可以提高PLM在文本匹配中的多任务概括能力，并产生更好的内域多任务，外域多任务和新任务适应性性能由以前的微调范式训练的特定于任务模型。

句子嵌入通常用于文本聚类和语义检索任务中。最先进的句子表示方法基于大量手动标记句子对集合的人工神经网络。高资源语言（例如英语或中文）可以使用足够数量的注释数据。在不太受欢迎的语言中，必须使用多语言模型，从而提供较低的性能。在本出版物中，我们通过提出一种培训有效的语言特定句子编码的方法来解决此问题，而无需手动标记数据。我们的方法是从句子对准双语文本语料库中自动构建释义对数据集。然后，我们使用收集的数据来微调具有附加复发池层的变压器语言模型。我们的句子编码器可以在不到一天的时间内在一张图形卡上进行培训，从而在各种句子级的任务上实现高性能。我们在波兰语中评估了八个语言任务的方法，并将其与最佳可用多语言句子编码器进行比较。

Language model pre-training has proven to be useful in learning universal language representations. As a state-of-the-art language model pre-training model, BERT (Bidirectional Encoder Representations from Transformers) has achieved amazing results in many language understanding tasks. In this paper, we conduct exhaustive experiments to investigate different fine-tuning methods of BERT on text classification task and provide a general solution for BERT fine-tuning. Finally, the proposed solution obtains new state-of-the-art results on eight widely-studied text classification datasets. 1

We present SpaceQA, to the best of our knowledge the first open-domain QA system in Space mission design. SpaceQA is part of an initiative by the European Space Agency (ESA) to facilitate the access, sharing and reuse of information about Space mission design within the agency and with the public. We adopt a state-of-the-art architecture consisting of a dense retriever and a neural reader and opt for an approach based on transfer learning rather than fine-tuning due to the lack of domain-specific annotated data. Our evaluation on a test set produced by ESA is largely consistent with the results originally reported by the evaluated retrievers and confirms the need of fine tuning for reading comprehension. As of writing this paper, ESA is piloting SpaceQA internally.

本文通过将深度递归编码器添加到具有深递归编码器（BERT-DRE）的伯爵，提供了一种深度神经阵列匹配（NLSM）。我们对模型行为的分析表明，BERT仍未捕获文本的全部复杂性，因此伯特顶部应用了一个深递归编码器。具有残留连接的三个Bi-LSTM层用于设计递归编码器，并在此编码器顶部使用注意模块。为了获得最终的载体，使用由平均值和最大池组成的池化层。我们在四个基准，SNLI，贝尔船，Multinli，Scitail和新的波斯宗教问题数据集上进行模型。本文侧重于改善NLSM任务中的BERT结果。在这方面，进行BERT-DRE和BERT之间的比较，并且显示在所有情况下，BERT-DRE优于伯特。宗教数据集的BERT算法实现了89.70％的精度，并且BERT-DRE架构使用相同的数据集提高了90.29％。

预先接受的语言模型实现了最先进的导致各种自然语言处理（NLP）任务。 GPT-3表明，缩放预先训练的语言模型可以进一步利用它们的巨大潜力。最近提出了一个名为Ernie 3.0的统一框架，以预先培训大型知识增强型号，并培训了具有10亿参数的模型。 Ernie 3.0在各种NLP任务上表现出最先进的模型。为了探讨缩放的表现，我们培养了百卢比的3.0泰坦参数型号，在PaddlePaddle平台上有高达260亿参数的泰坦。此外，我们设计了一种自我监督的对抗性损失和可控语言建模损失，以使ERNIE 3.0 TITAN产生可信和可控的文本。为了减少计算开销和碳排放，我们向Ernie 3.0泰坦提出了一个在线蒸馏框架，教师模型将同时教授学生和培训。埃塞尼3.0泰坦是迄今为止最大的中国密集预训练模型。经验结果表明，Ernie 3.0泰坦在68个NLP数据集中优于最先进的模型。

Current state-of-the-art approaches to text classification typically leverage BERT-style Transformer models with a softmax classifier, jointly fine-tuned to predict class labels of a target task. In this paper, we instead propose an alternative training objective in which we learn task-specific embeddings of text: our proposed objective learns embeddings such that all texts that share the same target class label should be close together in the embedding space, while all others should be far apart. This allows us to replace the softmax classifier with a more interpretable k-nearest-neighbor classification approach. In a series of experiments, we show that this yields a number of interesting benefits: (1) The resulting order induced by distances in the embedding space can be used to directly explain classification decisions. (2) This facilitates qualitative inspection of the training data, helping us to better understand the problem space and identify labelling quality issues. (3) The learned distances to some degree generalize to unseen classes, allowing us to incrementally add new classes without retraining the model. We present extensive experiments which show that the benefits of ante-hoc explainability and incremental learning come at no cost in overall classification accuracy, thus pointing to practical applicability of our proposed approach.

近年来，统一的视觉语言框架已经大大提高，其中大多数采用编码器架构将图像文本任务统一为序列到序列的生成。但是，现有的视频语言（VIDL）模型仍需要在每个任务的模型体系结构和培训目标中进行特定于任务的设计。在这项工作中，我们探索了一个统一的VIDL框架薰衣草，其中蒙版语言建模（MLM）用作所有前训练和下游任务的常见接口。这样的统一导致了简化的模型体系结构，在多模式编码器之上，只需要一个轻巧的MLM头，而不是具有更多参数的解码器。令人惊讶的是，实验结果表明，这个统一的框架在14个VIDL基准测试中实现了竞争性能，涵盖了视频问答，文本到视频检索和视频字幕。广泛的分析进一步证明了薰衣草比现有VIDL方法的优势：（i）在多任务列出时仅使用一组参数值支持所有下游任务； （ii）对各种下游任务的几乎没有概括； （iii）在视频问题回答任务上启用零射门评估。代码可从https://github.com/microsoft/lavender获得。

Recent progress in pretraining language models on large textual corpora led to a surge of improvements for downstream NLP tasks. Whilst learning linguistic knowledge, these models may also be storing relational knowledge present in the training data, and may be able to answer queries structured as "fillin-the-blank" cloze statements. Language models have many advantages over structured knowledge bases: they require no schema engineering, allow practitioners to query about an open class of relations, are easy to extend to more data, and require no human supervision to train. We present an in-depth analysis of the relational knowledge already present (without fine-tuning) in a wide range of state-of-theart pretrained language models. We find that (i) without fine-tuning, BERT contains relational knowledge competitive with traditional NLP methods that have some access to oracle knowledge, (ii) BERT also does remarkably well on open-domain question answering against a supervised baseline, and (iii) certain types of factual knowledge are learned much more readily than others by standard language model pretraining approaches. The surprisingly strong ability of these models to recall factual knowledge without any fine-tuning demonstrates their potential as unsupervised open-domain QA systems. The code to reproduce our analysis is available at https: //github.com/facebookresearch/LAMA.

This paper presents E5, a family of state-of-the-art text embeddings that transfer well to a wide range of tasks. The model is trained in a contrastive manner with weak supervision signals from our curated large-scale text pair dataset (called CCPairs). E5 can be readily used as a general-purpose embedding model for any tasks requiring a single-vector representation of texts such as retrieval, clustering, and classification, achieving strong performance in both zero-shot and fine-tuned settings. We conduct extensive evaluations on 56 datasets from the BEIR and MTEB benchmarks. For zero-shot settings, E5 is the first model that outperforms the strong BM25 baseline on the BEIR retrieval benchmark without using any labeled data. When fine-tuned, E5 obtains the best results on the MTEB benchmark, beating existing embedding models with 40x more parameters.