Large language models have exhibited intriguing in-context learning capability, achieving promising zero- and few-shot performance without updating the parameters. However, conventional in-context learning is usually restricted by length constraints, rendering it ineffective to absorb supervision from a large number of examples. In order to go beyond few shots, we introduce structured prompting that breaks the length limit and scales in-context learning to thousands of examples. Specifically, demonstration examples are separately encoded with well-designed position embeddings, and then they are jointly attended by the test example using a rescaled attention mechanism. So we can scale the number of exemplars with linear complexity instead of quadratic complexity with respect to length. Experimental results on a diverse set of tasks show that our approach improves end-task performance and reduces evaluation variance over conventional in-context learning as the number of demonstration examples increases. Code has been released at https://aka.ms/structured-prompting.

基础模型由于在广泛的下游应用中的有效性而受到了很多关注。尽管在体系结构方面存在很大的融合，但大多数审慎的模型通常仍用于特定任务或模式。在这项工作中，我们建议将语言模型用作各种基础模型的通用接口。一系列预处理的编码者感知到了多种方式（例如视觉和语言），并与扮演通用任务层角色的语言模型对接。我们提出了一个半伴侣的语言建模目标，以共同确定界面和模块化编码器。我们从因果关系和非因果建模中涵盖了优势和能力，从而结合了两个世界的最佳状态。具体而言，所提出的方法不仅从因果语言建模中继承了内在学习和开放式生成的能力，而且由于双向编码器而有利于填补。更重要的是，我们的方法无缝地解锁了上述功能的组合，例如，通过填充编码器启用了文本学习或指导。各种仅语言和视觉语言基准的实验结果表明，我们的模型表现优于或与鉴定，零弹性概括和几乎没有的学习的专业模型竞争。

大型语言模型在各种任务上显示出令人印象深刻的几次结果。但是，当知识是此类结果的关键时，就像问题回答和事实检查之类的任务一样，似乎需要存储知识的大量参数计数。众所周知，检索增强模型可以在不需要多个参数的情况下在知识密集的任务上表现出色，但是目前尚不清楚它们是否在几个弹药设置中工作。在这项工作中，我们介绍了地图集，这是一个经过精心设计和预先训练的增强语言模型，能够通过很少的培训示例学习知识密集型任务。我们对包括MMLU，苏格兰短裙和归类等各种任务进行评估，并研究文档索引内容的影响，表明它可以很容易地进行更新。值得注意的是，在自然问题上仅使用64个示例在自然问题上达到超过42 \％的准确性，尽管参数少了50倍，但比540B参数模型的表现优于540b参数模型。

With the increasing ability of large language models (LLMs), in-context learning (ICL) has become a new paradigm for natural language processing (NLP), where LLMs make predictions only based on contexts augmented with a few training examples. It has been a new trend exploring ICL to evaluate and extrapolate the ability of LLMs. In this paper, we aim to survey and summarize the progress, challenges, and future work in ICL. We first present a formal definition of ICL and clarify its correlation to related studies. Then, we organize and discuss advanced techniques of ICL, including training strategies, prompting strategies, and so on. Finally, we present the challenges of ICL and provide potential directions for further research. We hope our work can encourage more research on uncovering how ICL works and improving ICL in future work.

The recent GPT-3 model (Brown et al., 2020) achieves remarkable few-shot performance solely by leveraging a natural-language prompt and a few task demonstrations as input context. Inspired by their findings, we study few-shot learning in a more practical scenario, where we use smaller language models for which fine-tuning is computationally efficient. We present LM-BFF-better few-shot fine-tuning of language models 1 -a suite of simple and complementary techniques for finetuning language models on a small number of annotated examples. Our approach includes (1) prompt-based fine-tuning together with a novel pipeline for automating prompt generation; and (2) a refined strategy for dynamically and selectively incorporating demonstrations into each context. Finally, we present a systematic evaluation for analyzing few-shot performance on a range of NLP tasks, including classification and regression. Our experiments demonstrate that our methods combine to dramatically outperform standard fine-tuning procedures in this low resource setting, achieving up to 30% absolute improvement, and 11% on average across all tasks. Our approach makes minimal assumptions on task resources and domain expertise, and hence constitutes a strong task-agnostic method for few-shot learning. 2 * The first two authors contributed equally. 1 Alternatively, language models' best friends forever. 2 Our implementation is publicly available at https:// github.com/princeton-nlp/LM-BFF.

Pre-trained language models (PLMs) have exhibited remarkable few-shot learning capabilities when provided a few examples in a natural language prompt as demonstrations of test instances, i.e., in-context learning. However, the performance of in-context learning is susceptible to the choice of prompt format, training examples and the ordering of the training examples. In this paper, we propose a novel nearest-neighbor calibration framework for in-context learning to ease this issue. It is inspired by a phenomenon that the in-context learning paradigm produces incorrect labels when inferring training instances, which provides a useful supervised signal to calibrate predictions. Thus, our method directly augments the predictions with a $k$-nearest-neighbor ($k$NN) classifier over a datastore of cached few-shot instance representations obtained by PLMs and their corresponding labels. Then adaptive neighbor selection and feature regularization modules are introduced to make full use of a few support instances to reduce the $k$NN retrieval noise. Experiments on various few-shot text classification tasks demonstrate that our method significantly improves in-context learning, while even achieving comparable performance with state-of-the-art tuning-based approaches in some sentiment analysis tasks.

预先接受的语言模型实现了最先进的导致各种自然语言处理（NLP）任务。 GPT-3表明，缩放预先训练的语言模型可以进一步利用它们的巨大潜力。最近提出了一个名为Ernie 3.0的统一框架，以预先培训大型知识增强型号，并培训了具有10亿参数的模型。 Ernie 3.0在各种NLP任务上表现出最先进的模型。为了探讨缩放的表现，我们培养了百卢比的3.0泰坦参数型号，在PaddlePaddle平台上有高达260亿参数的泰坦。此外，我们设计了一种自我监督的对抗性损失和可控语言建模损失，以使ERNIE 3.0 TITAN产生可信和可控的文本。为了减少计算开销和碳排放，我们向Ernie 3.0泰坦提出了一个在线蒸馏框架，教师模型将同时教授学生和培训。埃塞尼3.0泰坦是迄今为止最大的中国密集预训练模型。经验结果表明，Ernie 3.0泰坦在68个NLP数据集中优于最先进的模型。

We introduce TeSS (Text Similarity Comparison using Sentence Encoder), a framework for zero-shot classification where the assigned label is determined by the embedding similarity between the input text and each candidate label prompt. We leverage representations from sentence encoders optimized to locate semantically similar samples closer to each other in embedding space during pre-training. The label prompt embeddings serve as prototypes of their corresponding class clusters. Furthermore, to compensate for the potentially poorly descriptive labels in their original format, we retrieve semantically similar sentences from external corpora and additionally use them with the original label prompt (TeSS-R). TeSS outperforms strong baselines on various closed-set and open-set classification datasets under zero-shot setting, with further gains when combined with label prompt diversification through retrieval. These results are robustly attained to verbalizer variations, an ancillary benefit of using a bi-encoder. Altogether, our method serves as a reliable baseline for zero-shot classification and a simple interface to assess the quality of sentence encoders.

迅速的学习方法通​​过诱导更好的几次表现，在他们仍然遵循基于参数的学习范式的同时，引起了自然语言处理的波动。学习中的遗忘和死记硬背的记忆问题可能会遇到不稳定的概括问题。具体而言，香草及时的学习可能难以利用死记硬背的非典型实例，在完全监督的培训或过度贴身模式的情况下使用低射击数据。为了减轻此类局限性，我们以将知识从记忆中解耦的动机发展为有助于模型在概括和记忆之间取得平衡。与香草及时学习相反，重新启动构造了培训实例中的开放式知识店，并在输入，培训和推理过程中实现检索机制，从而使该模型能够从培训语料库中检索相关环境作为能力为提示增强。广泛的实验表明，Retroppt可以在几次射击和零拍设置中获得更好的性能。此外，我们进一步说明，我们提出的撤退可以通过新数据集获得更好的概括能力。对记忆的详细分析确实显示逆转可以减少语言模型对记忆的依赖；因此，改善下游任务的概括。

We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models (Peters et al., 2018a;Radford et al., 2018), BERT is designed to pretrain deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be finetuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial taskspecific architecture modifications.BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

及时调整是将预训练模型调整到下游任务的极其有效的工具。但是，基于标准及时的方法主要考虑下游任务的足够数据的情况。目前尚不清楚是否可以将优势传输到几杆式制度，在每个下游任务中只有有限的数据。尽管有些作品证明了在几次弹奏设置下及时调整的潜力，但通过搜索离散提示或使用有限数据调整软提示的主流方法仍然非常具有挑战性。通过广泛的实证研究，我们发现迅速调整和完全微调之间的学习差距仍然存在差距。为了弥合差距，我们提出了一个新的及时调整框架，称为软模板调整（STT）。 STT结合了手册和自动提示，并将下游分类任务视为掩盖语言建模任务。对不同设置的全面评估表明，STT可以在不引入其他参数的情况下缩小微调和基于及时的方法之间的差距。值得注意的是，它甚至可以胜过情感分类任务的时间和资源消耗的微调方法。

本文着重于几次NLP任务的文本数据增强。现有的数据增强算法要么使用一个小型培训集来生成新的合成数据，要么利用与任务无关的启发式规则（例如，同义词替代）或微调通用预训练的语言模型（例如GPT2）。因此，这些方法具有特定于任务的知识，并且仅限于在简单任务中为弱基线产生低质量的合成数据。为了解决这个问题，我们提出了知识混合数据增强模型（KNOWDA）：使用知识混合培训（KOMT）在不同的NLP任务的混合物上预测的编码器LM。 KOMT是一种培训程序，将各种异质NLP任务的输入示例重新定义为统一的文本到文本格式，并采用不同粒度的目标，以学习生成部分或完整的样本。在KOMT的帮助下，Knowda可以隐含地将所需的特定于任务的知识从任务的混合中隐含地结合在一起，并通过一些给定的实例迅速掌握目标任务的固有综合定律。据我们所知，我们是首次尝试将任务数量扩展到多任务共同培训以进行数据扩展。广泛的实验表明，i）Knowda成功地通过少量基准的基准成功地提高了Albert和Deberta的表现，表现优于先前的最新数据增强基线； ii）KNOWDA还可以改善少数弹药任务的模型性能，这是KOMT中未包含的固定任务类型。

本文探讨了提高语言模型的零次学习能力的简单方法。我们表明，指令调整 - 通过对说明书中所述的任务集合微调语言模型 - 大幅提升零射门上看不见任务中的表现。我们采取预训练的语言模型和指令调整它通过自然语言指令模板语言表达了60NLP任务137B参数。我们评估这种指令调整模型，我们称之为FLAN，在看不见的任务类型。FLAN显着改善其未修饰的对应的性能和超过25的20个任务，我们评估零射门175BGPT-3。FLAN甚至GPT-3通过在安利，RTE，BoolQ，AI2-ARC，OpenbookQA和StoryCloze大比分胜过几拍。消融研究显示任务和模型的规模，这个数字是指令调整取得成功的关键组成部分。

For applications that require processing large amounts of text at inference time, Large Language Models (LLMs) are handicapped by their limited context windows, which are typically 2048 tokens. In-context learning, an emergent phenomenon in LLMs in sizes above a certain parameter threshold, constitutes one significant example because it can only leverage training examples that fit into the context window. Existing efforts to address the context window limitation involve training specialized architectures, which tend to be smaller than the sizes in which in-context learning manifests due to the memory footprint of processing long texts. We present Parallel Context Windows (PCW), a method that alleviates the context window restriction for any off-the-shelf LLM without further training. The key to the approach is to carve a long context into chunks (``windows'') that fit within the architecture, restrict the attention mechanism to apply only within each window, and re-use the positional embeddings among the windows. We test the PCW approach on in-context learning with models that range in size between 750 million and 178 billion parameters, and show substantial improvements for tasks with diverse input and output spaces. Our results motivate further investigation of Parallel Context Windows as a method for applying off-the-shelf LLMs in other settings that require long text sequences.

In this work, we explore "prompt tuning," a simple yet effective mechanism for learning "soft prompts" to condition frozen language models to perform specific downstream tasks. Unlike the discrete text prompts used by GPT-3, soft prompts are learned through backpropagation and can be tuned to incorporate signals from any number of labeled examples. Our end-to-end learned approach outperforms GPT-3's few-shot learning by a large margin. More remarkably, through ablations on model size using T5, we show that prompt tuning becomes more competitive with scale: as models exceed billions of parameters, our method "closes the gap" and matches the strong performance of model tuning (where all model weights are tuned). This finding is especially relevant because large models are costly to share and serve and the ability to reuse one frozen model for multiple downstream tasks can ease this burden. Our method can be seen as a simplification of the recently proposed "prefix tuning" of Li and Liang (2021) and we provide a comparison to this and other similar approaches. Finally, we show that conditioning a frozen model with soft prompts confers benefits in robustness to domain transfer and enables efficient "prompt ensembling." * Work done as a Google AI Resident.

在维持预审预定序列模型的灵活性的同时，是否有利于常识性推理，这仍然是一个悬而未决的问题。为了调查这个问题，我们开发了生成的知识提示，该提示包括从语言模型中生成知识，然后在回答问题时提供知识作为附加输入。我们的方法不需要特定于任务的监督知识集成或访问结构化的知识库，但它可以提高四个常识性推理任务上的大规模，最先进的模型的性能，从而实现最先进-ART结果取决于数值常识（NumerSense），通用常识性（Commonsenseqa 2.0）和科学常识（QASC）基准。产生的知识促使大型语言模型是灵活的外部知识来源，以改善常识性推理。我们的代码可从https://github.com/liujch1998/gkp获得

大规模的预训练的语言模型（PLM）以能够仅通过在提示中调节一些被称为示范的示威演示的情况而不明确调整为所需的下游任务而被称为示威的示威来解决任务。但是，这种过程（即，在文章中的学习）自然会高度依赖通常从外部数据集中选择的演示。在本文中，我们提出了自我生成的文化学习（SG-ICL），该学习生成了从PLM本身中的文化学习演示，以最大程度地减少对外部演示的依赖。我们对四个不同的文本分类任务进行实验，并显示SG-ICL的表现明显优于零射击学习，并且通常价值约0.6个黄金训练样本。此外，与培训数据集的随机选择相比，我们的生成的演示表现出更一致的性能，方差较低。

几乎没有射击的内在学习（ICL）使预训练的语言模型能够通过为输入的一部分提供少量的培训示例来执行以前的任务，而无需任何基于梯度的培训。 ICL会产生大量的计算，内存和存储成本，因为它每次进行预测时都涉及处理所有培训示例。参数有效的微调（PEFT）（例如，适配器模块，提示调谐，稀疏更新方法等）提供了替代范式，其中训练了一组少量参数以启用模型来执行新任务。在本文中，我们严格地比较了几个ICL和PEFT，并证明后者提供了更好的准确性，并大大降低了计算成本。在此过程中，我们引入了一种称为（IA）$^3 $的新PEFT方法，该方法通过学习的向量来扩展激活，从而获得更强的性能，同时仅引入相对少量的新参数。我们还提出了一个基于称为T-FEW的T0模型的简单食谱，可以将其应用于新任务，而无需特定于任务的调整或修改。我们通过将T-FEW应用于木筏基准，首次实现超人性能，并以6％的绝对性能优于最先进的方法来验证T-FEW对完全看不见的任务的有效性。我们实验中使用的所有代码均可公开使用。

最近的自然语言理解进展（NLU）已经被驱动，部分是由胶水，超级格，小队等的基准。事实上，许多NLU模型现在在许多任务中匹配或超过“人类水平”性能这些基准。然而，大多数这些基准测试都提供模型访问相对大量的标记数据进行培训。因此，该模型提供了比人类所需的更多数据，以实现强大的性能。这有动机侧重于侧重于改善NLU模型的少量学习性能。然而，缺乏少量射门的标准化评估基准，导致不同纸张中的不同实验设置。为了帮助加速这一工作的工作，我们介绍了线索（受限制的语言理解评估标准），这是评估NLU模型的几次拍摄学习功能的基准。我们证明，虽然最近的模型在获得大量标记数据时达到人类性能，但对于大多数任务，少量拍摄设置中的性能存在巨大差距。我们还展示了几个拍摄设置中替代模型家族和适应技术之间的差异。最后，我们讨论了在设计实验设置时讨论了评估真实少量学习绩效的实验设置，并提出了统一的标准化方法，以获得少量学习评估。我们的目标是鼓励对NLU模型的研究，可以概括为具有少数示例的新任务。线索的代码和数据可以在https://github.com/microsoft/clues提供。

This paper presents a new UNIfied pre-trained Language Model (UNILM) that can be fine-tuned for both natural language understanding and generation tasks. The model is pre-trained using three types of language modeling tasks: unidirectional, bidirectional, and sequence-to-sequence prediction. The unified modeling is achieved by employing a shared Transformer network and utilizing specific self-attention masks to control what context the prediction conditions on. UNILM compares favorably with BERT on the GLUE benchmark, and the SQuAD 2.0 and CoQA question answering tasks. Moreover, UNILM achieves new state-ofthe-art results on five natural language generation datasets, including improving the CNN/DailyMail abstractive summarization ROUGE-L to 40.51 (2.04 absolute improvement), the Gigaword abstractive summarization ROUGE-L to 35.75 (0.86 absolute improvement), the CoQA generative question answering F1 score to 82.5 (37.1 absolute improvement), the SQuAD question generation BLEU-4 to 22.12 (3.75 absolute improvement), and the DSTC7 document-grounded dialog response generation NIST-4 to 2.67 (human performance is 2.65). The code and pre-trained models are available at https://github.com/microsoft/unilm. * Equal contribution. † Contact person.