微调预训练的语言模型可以提高商业回复建议系统的质量，但以不可持续的培训时间的成本。流行的训练时间减少方法是资源密集型，因此我们探索了低成本的模型压缩技术，如层次掉线和冻结层。我们展示了这些技术在大数据场景中的功效，使商业电子邮件回复建议系统的培训时间减少了42％，而不会影响模型相关性或用户参与。我们进一步研究了这些技术的稳健性，以预先训练的模型和数据集大小消融，并为商业应用分享了几个见解和建议。

基于变压器的NLP模型是使用数亿甚至数十亿个参数训练的，从而限制了其在计算受限环境中的适用性。尽管参数的数量通常与性能相关，但尚不清楚下游任务是否需要整个网络。在最新的修剪和提炼预培训模型的工作中，我们探索了在预训练模型中放下层的策略，并观察修剪对下游胶水任务的影响。我们能够修剪Bert，Roberta和XLNet型号高达40％，同时保持其原始性能的98％。此外，我们证明，在大小和性能方面，您的修剪模型与使用知识蒸馏的型号相提并论。我们的实验产生有趣的观察结果，例如（i）下层对于维持下游任务性能最重要，（ii）某些任务（例如释义检测和句子相似性）对于降低层的降低和（iii）经过训练的模型更强大。使用不同的目标函数表现出不同的学习模式，并且层掉落。

已经表明，在一个域上训练的双编码器经常概括到其他域以获取检索任务。一种广泛的信念是，一个双编码器的瓶颈层，其中最终得分仅仅是查询向量和通道向量之间的点产品，它过于局限，使得双编码器是用于域外概括的有效检索模型。在本文中，我们通过缩放双编码器模型的大小{\ em同时保持固定的瓶颈嵌入尺寸固定的瓶颈的大小来挑战这一信念。令人惊讶的是，令人惊讶的是，缩放模型尺寸会对各种缩放提高检索任务，特别是对于域外泛化。实验结果表明，我们的双编码器，\ textbf {g} enovalizable \ textbf {t} eTrievers（gtr），优先级％colbert〜\ cite {khattab2020colbertt}和现有的稀疏和密集的索取Beir DataSet〜\ Cite {Thakur2021Beir}显着显着。最令人惊讶的是，我们的消融研究发现，GTR是非常数据的高效，因为它只需要10 \％MARCO监督数据，以实现最佳域的性能。所有GTR模型都在https://tfhub.dev/google/collections/gtr/1发布。

查询聚焦的文本摘要（QFTS）任务旨在构建基于给定查询的文本文档摘要的构建系统。解决此任务的关键挑战是缺乏培训摘要模型的大量标记数据。在本文中，我们通过探索一系列域适应技术来解决这一挑战。鉴于最近在广泛的自然语言处理任务中进行预先接受的变压器模型的成功，我们利用此类模型为单文档和多文件方案的QFTS任务产生抽象摘要。对于域适应，我们使用预先训练的变压器的摘要模型应用了各种技术，包括转移学习，弱监督学习和远程监督。六个数据集的广泛实验表明，我们所提出的方法非常有效地为QFTS任务产生抽象摘要，同时在一组自动和人类评估指标上设置新的最先进的结果。

大型的语言模型（PRELMS）正在彻底改变所有基准的自然语言处理。但是，它们的巨大尺寸对于小型实验室或移动设备上的部署而言是过分的。修剪和蒸馏等方法可减少模型尺寸，但通常保留相同的模型体系结构。相反，我们探索了蒸馏预告片中的更有效的架构，单词的持续乘法（CMOW），该构造将每个单词嵌入为矩阵，并使用矩阵乘法来编码序列。我们扩展了CMOW体系结构及其CMOW/CBOW-HYBRID变体，具有双向组件，以提供更具表现力的功能，在预绘制期间进行一般（任务无义的）蒸馏的单次表示，并提供了两种序列编码方案，可促进下游任务。句子对，例如句子相似性和自然语言推断。我们的基于矩阵的双向CMOW/CBOW-HYBRID模型在问题相似性和识别文本范围内的Distilbert具有竞争力，但仅使用参数数量的一半，并且在推理速度方面快三倍。除了情感分析任务SST-2和语言可接受性任务COLA外，我们匹配或超过ELMO的ELMO分数。但是，与以前的跨架结构蒸馏方法相比，我们证明了检测语言可接受性的分数增加了一倍。这表明基于基质的嵌入可用于将大型预赛提炼成竞争模型，并激励朝这个方向进行进一步的研究。

将最新的变压器模型蒸馏成轻量级的学生模型是降低推理时计算成本的有效方法。学生模型通常是紧凑的变压器，参数较少，而昂贵的操作（例如自我发项）持续存在。因此，对于实时或大量用例，提高的推理速度仍然不令人满意。在本文中，我们旨在通过将教师模型提炼成更大，更稀疏的学生模型来进一步推动推理速度的极限 - 更大的是它们扩展到数十亿个参数；稀疏，大多数模型参数是N-gram嵌入。我们对六个单词文本分类任务的实验表明，这些学生模型平均保留了罗伯塔大师教师表现的97％，同时推理时GPU和CPU的加速速度最高为600倍。进一步的调查表明，我们的管道也有助于句子对分类任务和域泛化设置。

对于许多任务，基于变压器的体系结构已经实现了最新的结果，从而导致实践从使用特定于任务的架构到预先训练的语言模型的微调。持续的趋势包括具有越来越多的数据和参数的培训模型，这需要大量资源。它导致了强有力的搜索，以提高基于仅针对英语评估的算法和硬件改进的算法和硬件改进。这引发了有关其可用性的疑问，当应用于小规模的学习问题时，对于资源不足的语言任务，有限的培训数据可用。缺乏适当尺寸的语料库是应用数据驱动和转移学习的方法的障碍。在本文中，我们建立了致力于基于变压器模型的可用性的最新努力，并建议评估这些改进的法语表现，而法语的效果很少。我们通过通过数据增强，超参数优化和跨语性转移来调查各种培训策略来解决与数据稀缺有关的不稳定。我们还为法国弗拉伯特（Fralbert）引入了一种新的紧凑型模型，该模型在低资源环境中被证明具有竞争力。

近年来，大型预训练的变压器网络已显示出许多自然语言理解任务的巨大改进。但是，由于延迟和成本限制，这些模型的巨大规模给他们的微调和在线部署带来了重大挑战。支持N：M半结构化的稀疏性和低精油整数计算的新硬件是提高DNN模型效率的有前途解决方案。但是，很少有研究系统地研究预先训练的变压器网络在多大程度上受益于这些技术的组合，以及如何最好地压缩变压器的每个组件。我们提出了一个灵活的压缩框架NXMiformer，该框架使用ADMM和基于Ste的QAT执行同时进行稀疏和量化。此外，我们介绍且廉价的启发式驱动搜索算法，该算法标识了满足压缩比约束的有希望的异质压缩配置。当通过NLU基准测试的胶水套件进行评估时，我们的方法可以达到BERT模型编码器的93％压缩，同时保留了98.2％的原始模型准确性并充分利用硬件功能。异质配置通过搜索启发式发现了基线准确性的99.5％，同时仍将模型压缩为87.5％。

Transfer learning, where a model is first pre-trained on a data-rich task before being finetuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new "Colossal Clean Crawled Corpus", we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.

语言模型是使用大量通用数据（如Book Copus，Common Crawl和Wikipedia）进行预训练的，这对于模型了解语言的语言特征至关重要。新的研究建议将域自适应预训练（DAPT）和任务自适应预训练（TAPT）作为最终填充任务之前的中间步骤。此步骤有助于涵盖目标域词汇，并改善下游任务的模型性能。在这项工作中，我们仅研究训练在TAPT和特定于任务的填充过程中嵌入层对模型性能的影响。基于我们的研究，我们提出了一种简单的方法，以通过对BERT层进行选择性预训练，使基于BERT的模型的中间步骤更有效。我们表明，在TAPT期间仅训练BERT嵌入层足以适应目标域的词汇并实现可比的性能。我们的方法在计算上是有效的，在TAPT期间训练了78％的参数。所提出的嵌入层列式方法也可以是一种有效的域适应技术。

Transformer-based models have pushed state of the art in many areas of NLP, but our understanding of what is behind their success is still limited. This paper is the first survey of over 150 studies of the popular BERT model. We review the current state of knowledge about how BERT works, what kind of information it learns and how it is represented, common modifications to its training objectives and architecture, the overparameterization issue and approaches to compression. We then outline directions for future research.

As Transfer Learning from large-scale pre-trained models becomes more prevalent in Natural Language Processing (NLP), operating these large models in on-theedge and/or under constrained computational training or inference budgets remains challenging. In this work, we propose a method to pre-train a smaller generalpurpose language representation model, called DistilBERT, which can then be finetuned with good performances on a wide range of tasks like its larger counterparts. While most prior work investigated the use of distillation for building task-specific models, we leverage knowledge distillation during the pre-training phase and show that it is possible to reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster. To leverage the inductive biases learned by larger models during pre-training, we introduce a triple loss combining language modeling, distillation and cosine-distance losses. Our smaller, faster and lighter model is cheaper to pre-train and we demonstrate its capabilities for on-device computations in a proof-of-concept experiment and a comparative on-device study.

我们介绍了BitFit，这是一种稀疏的重点方法，其中仅修改了模型的偏差（或其中一个子集）。我们表明，通过在预训练的BERT模型上应用BITFIT的小型至中等训练数据具有竞争力（有时比）对整个模型进行微调。对于较大的数据，该方法与其他稀疏微调方法具有竞争力。除了它们的实际实用性外，这些发现与理解常用的填补过程的问题有关：它们支持以下假设：填充主要是关于揭示通过语言模型培训引起的知识，而不是学习新的任务特定的语言知识。

在生物医学语料库中预先培训的语言模型，例如Biobert，最近在下游生物医学任务上显示出令人鼓舞的结果。另一方面，由于嵌入尺寸，隐藏尺寸和层数等因素，许多现有的预训练模型在资源密集型和计算上都是沉重的。自然语言处理（NLP）社区已经制定了许多策略来压缩这些模型，利用修剪，定量和知识蒸馏等技术，从而导致模型更快，更小，随后更易于使用。同样，在本文中，我们介绍了六种轻型模型，即Biodistilbert，Biotinybert，BioMobilebert，Distilbiobert，Tinybiobert和Cmpactactbiobert，并通过掩护的语言在PubMed DataSet上通过掩护数据进行了知识蒸馏而获得的知识蒸馏来获得。建模（MLM）目标。我们在三个生物医学任务上评估了所有模型，并将它们与Biobert-V1.1进行比较，以创建有效的轻量级模型，以与较大的对应物相同。所有模型将在我们的HuggingFace配置文件上公开可用，网址为https://huggingface.co/nlpie，用于运行实验的代码将在https://github.com/nlpie-research/compact-compact-biomedical-transformers上获得。

Leveraging shared learning through Massively Multilingual Models, state-of-the-art machine translation models are often able to adapt to the paucity of data for low-resource languages. However, this performance comes at the cost of significantly bloated models which are not practically deployable. Knowledge Distillation is one popular technique to develop competitive, lightweight models: In this work, we first evaluate its use to compress MT models focusing on languages with extremely limited training data. Through our analysis across 8 languages, we find that the variance in the performance of the distilled models due to their dependence on priors including the amount of synthetic data used for distillation, the student architecture, training hyperparameters and confidence of the teacher models, makes distillation a brittle compression mechanism. To mitigate this, we explore the use of post-training quantization for the compression of these models. Here, we find that while distillation provides gains across some low-resource languages, quantization provides more consistent performance trends for the entire range of languages, especially the lowest-resource languages in our target set.

从有限的资源中获得最大收益可以进步自然语言处理（NLP）研究和实践，同时保守资源。这些资源可能是数据，时间，存储或能源。NLP的最新工作从缩放率产生了有趣的结果。但是，仅使用比例来改善结果意味着资源消耗也会扩展。这种关系激发了对有效方法的研究，这些方法需要更少的资源才能获得相似的结果。这项调查涉及NLP效率的方法和发现，旨在指导该领域的新研究人员并激发新方法的发展。

我们为大规模训练的大规模训练语言模型提供了更简单，更稀疏，更快的算法，这些算法在许多标准的NLP任务上实现了最新的隐私与实用性权衡。我们为此问题提出了一个元框架，这是受高度参数效率方法进行微调成功的启发。我们的实验表明，这些方法的差异化适应能力在三个重要方面优于以前的私人算法：实用程序，隐私以及私人培训的计算和记忆成本。在许多经常研究的数据集中，私人模型的实用性接近了非私人模型的方法。例如，在MNLI数据集上，我们使用Roberta-large的准确度为87.8 \％$，使用Roberta-Base $ 83.5 \％$，其隐私预算为$ \ Epsilon = 6.7 $。相比之下，缺乏隐私限制，罗伯塔·莱格（Roberta-Large）的准确度为$ 90.2 \％$。我们的发现对于自然语言生成任务类似。与DART，GPT-2-SMALL，GPT-2中，GPT-2-MEDIUM，GPT-2-LARGE和GPT-2-XL的私人微调达到38.5、42.0、43.1和43.8（$ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ 43.8） epsilon = 6.8，\ delta = $ 1E-5），而非私人基线为$ 48.1 $。我们所有的实验都表明，较大的模型更适合私人微调：虽然众所周知，它们旨在非优先实现卓越的准确性，但我们发现当引入隐私时，它们也更好地保持其准确性。

In this work, we explore "prompt tuning," a simple yet effective mechanism for learning "soft prompts" to condition frozen language models to perform specific downstream tasks. Unlike the discrete text prompts used by GPT-3, soft prompts are learned through backpropagation and can be tuned to incorporate signals from any number of labeled examples. Our end-to-end learned approach outperforms GPT-3's few-shot learning by a large margin. More remarkably, through ablations on model size using T5, we show that prompt tuning becomes more competitive with scale: as models exceed billions of parameters, our method "closes the gap" and matches the strong performance of model tuning (where all model weights are tuned). This finding is especially relevant because large models are costly to share and serve and the ability to reuse one frozen model for multiple downstream tasks can ease this burden. Our method can be seen as a simplification of the recently proposed "prefix tuning" of Li and Liang (2021) and we provide a comparison to this and other similar approaches. Finally, we show that conditioning a frozen model with soft prompts confers benefits in robustness to domain transfer and enables efficient "prompt ensembling." * Work done as a Google AI Resident.

Text generation has made significant advances in the last few years. Yet, evaluation metrics have lagged behind, as the most popular choices (e.g., BLEU and ROUGE) may correlate poorly with human judgments. We propose BLEURT, a learned evaluation metric based on BERT that can model human judgments with a few thousand possibly biased training examples. A key aspect of our approach is a novel pre-training scheme that uses millions of synthetic examples to help the model generalize. BLEURT provides state-ofthe-art results on the last three years of the WMT Metrics shared task and the WebNLG Competition dataset. In contrast to a vanilla BERT-based approach, it yields superior results even when the training data is scarce and out-of-distribution.

Language model pre-training, such as BERT, has significantly improved the performances of many natural language processing tasks. However, pre-trained language models are usually computationally expensive, so it is difficult to efficiently execute them on resourcerestricted devices. To accelerate inference and reduce model size while maintaining accuracy, we first propose a novel Transformer distillation method that is specially designed for knowledge distillation (KD) of the Transformer-based models. By leveraging this new KD method, the plenty of knowledge encoded in a large "teacher" BERT can be effectively transferred to a small "student" Tiny-BERT. Then, we introduce a new two-stage learning framework for TinyBERT, which performs Transformer distillation at both the pretraining and task-specific learning stages. This framework ensures that TinyBERT can capture the general-domain as well as the task-specific knowledge in BERT. TinyBERT 41 with 4 layers is empirically effective and achieves more than 96.8% the performance of its teacher BERT BASE on GLUE benchmark, while being 7.5x smaller and 9.4x faster on inference. TinyBERT 4 is also significantly better than 4-layer state-of-the-art baselines on BERT distillation, with only ∼28% parameters and ∼31% inference time of them. Moreover, TinyBERT 6 with 6 layers performs on-par with its teacher BERT BASE .