The recent GPT-3 model (Brown et al., 2020) achieves remarkable few-shot performance solely by leveraging a natural-language prompt and a few task demonstrations as input context. Inspired by their findings, we study few-shot learning in a more practical scenario, where we use smaller language models for which fine-tuning is computationally efficient. We present LM-BFF-better few-shot fine-tuning of language models 1 -a suite of simple and complementary techniques for finetuning language models on a small number of annotated examples. Our approach includes (1) prompt-based fine-tuning together with a novel pipeline for automating prompt generation; and (2) a refined strategy for dynamically and selectively incorporating demonstrations into each context. Finally, we present a systematic evaluation for analyzing few-shot performance on a range of NLP tasks, including classification and regression. Our experiments demonstrate that our methods combine to dramatically outperform standard fine-tuning procedures in this low resource setting, achieving up to 30% absolute improvement, and 11% on average across all tasks. Our approach makes minimal assumptions on task resources and domain expertise, and hence constitutes a strong task-agnostic method for few-shot learning. 2 * The first two authors contributed equally. 1 Alternatively, language models' best friends forever. 2 Our implementation is publicly available at https:// github.com/princeton-nlp/LM-BFF.

提示将下游应用程序作为语言建模任务施放，与使用预训练的模型进行标准微调相比，已显示出样本有效的效率。但是，提示的一个陷阱是需要手动设计的模式，其结果可能是不直觉的，需要大量的验证集来调整。为了应对挑战，我们提出了一种全自动提示方法Autoseq：（1）我们在序列到序列模型上采用自然语言提示，从而实现自由形式生成和更大的标签搜索空间； （2）我们提出了标签序列 - 无限长度的短语以口头表达标签 - 这消除了手动模板的需求，并且比单个标签单词更具有表现力； （3）我们使用Beam Search自动生成大量的标签序列候选物，并提出对比度重新排列以获得最佳组合。 Autoseq显着胜过其他无手动设计方法，例如软提示调整，适配器调整和自动搜索单个标签单词；生成的标签序列比各种任务上的精选手动序列更好。我们的方法揭示了几次学习中序列模型的潜力，并阐明了通用通用和自动提示的途径。本文的源代码可以从https://github.com/thunlp/seq2seq-prompt获得。

最近，与“预训练，及时和预测”的新范式相比，与“预训练，微调”范式相比，新的范式“预训练，及时和预测”取得了显着的成就。在基于及时的GPT-3成功之后，一系列基于蒙版的语言模型（MLM）（例如Bert，Roberta）及时学习方法变得流行并广泛使用。但是，另一个有效的预训练的判别模型Electra可能被忽略了。在本文中，我们尝试使用拟议的替换代替令牌检测（RTD）基于基于的及时学习方法来完成零摄像的几个NLP任务。实验结果表明，基于RTD-Prompt学习的Electra模型可达到令人惊讶的最先进的零拍性能。在数字上，与MLM-Roberta-Large和MLM-Bert-Large相比，我们的RTD-Electra-Large在所有15个任务上平均提高了约8.4％和13.7％。特别是在SST-2任务上，我们的RTD-Electra-Large在没有任何培训数据的情况下达到了令人惊讶的90.1％精度。总体而言，与预先训练的蒙版语言模型相比，预先训练的代替令牌检测模型在零拍学习中的性能更好。因此，Electra是一位出色的零球学习者。源代码可在以下网址获得：https：//github.com/nishiwen1214/rtd-electra。

及时调整是将预训练模型调整到下游任务的极其有效的工具。但是，基于标准及时的方法主要考虑下游任务的足够数据的情况。目前尚不清楚是否可以将优势传输到几杆式制度，在每个下游任务中只有有限的数据。尽管有些作品证明了在几次弹奏设置下及时调整的潜力，但通过搜索离散提示或使用有限数据调整软提示的主流方法仍然非常具有挑战性。通过广泛的实证研究，我们发现迅速调整和完全微调之间的学习差距仍然存在差距。为了弥合差距，我们提出了一个新的及时调整框架，称为软模板调整（STT）。 STT结合了手册和自动提示，并将下游分类任务视为掩盖语言建模任务。对不同设置的全面评估表明，STT可以在不引入其他参数的情况下缩小微调和基于及时的方法之间的差距。值得注意的是，它甚至可以胜过情感分类任务的时间和资源消耗的微调方法。

Pre-trained language models (PLMs) have exhibited remarkable few-shot learning capabilities when provided a few examples in a natural language prompt as demonstrations of test instances, i.e., in-context learning. However, the performance of in-context learning is susceptible to the choice of prompt format, training examples and the ordering of the training examples. In this paper, we propose a novel nearest-neighbor calibration framework for in-context learning to ease this issue. It is inspired by a phenomenon that the in-context learning paradigm produces incorrect labels when inferring training instances, which provides a useful supervised signal to calibrate predictions. Thus, our method directly augments the predictions with a $k$-nearest-neighbor ($k$NN) classifier over a datastore of cached few-shot instance representations obtained by PLMs and their corresponding labels. Then adaptive neighbor selection and feature regularization modules are introduced to make full use of a few support instances to reduce the $k$NN retrieval noise. Experiments on various few-shot text classification tasks demonstrate that our method significantly improves in-context learning, while even achieving comparable performance with state-of-the-art tuning-based approaches in some sentiment analysis tasks.

大型预训练的语言模型（PLM）的最新进展导致了自然语言理解（NLU）任务的令人印象深刻的增长，并具有特定于任务的微调。但是，直接调整PLM在很大程度上依赖大量的标记实例，这些实例通常很难获得。迅速对PLM的调整已被证明对各种少数次任务很有价值。现有的作品研究基于迅速的NLU任务的基于及时的调整，主要集中于用语言器来得出正确的标签单词或生成及时的模板，以从PLM中启发语义。此外，还对常规数据增强方法进行了验证，可用于少量射击任务。但是，目前几乎没有针对基于及时的调整范式设计的数据增强方法。因此，我们研究了迅速的少数射击学习者的新数据增强问题。由于标签语义对于迅速的调整至关重要，因此我们提出了一种新颖的标签引导数据增强方法促进DA，该方法利用了丰富的标签语义信息以进行数据增强。很少的文本分类任务的广泛实验结果表明，我们提出的框架通过有效利用标签语义和数据扩展来实现自然语言理解来实现卓越的性能。

预先训练的蒙版语言模型通过将下游任务作为文本填充来成功执行几次学习。但是，作为全镜头环境中的强大替代方案，诸如Electra之类的判别预训练模型不适合范式。在这项工作中，我们调整了基于及时的几次学习来进行电信，并表明它在广泛的任务中优于蒙面的语言模型。Electra是预先训练的，以区分令牌是产生还是原始。我们自然地将其扩展到基于迅速的几次学习，通过培训来评分目标选项的原创性，而无需引入新参数。我们的方法很容易适应涉及多token预测的任务，而无需额外的计算开销。分析表明，Electra学习分布与下游任务更好。

迅速的学习方法通​​过诱导更好的几次表现，在他们仍然遵循基于参数的学习范式的同时，引起了自然语言处理的波动。学习中的遗忘和死记硬背的记忆问题可能会遇到不稳定的概括问题。具体而言，香草及时的学习可能难以利用死记硬背的非典型实例，在完全监督的培训或过度贴身模式的情况下使用低射击数据。为了减轻此类局限性，我们以将知识从记忆中解耦的动机发展为有助于模型在概括和记忆之间取得平衡。与香草及时学习相反，重新启动构造了培训实例中的开放式知识店，并在输入，培训和推理过程中实现检索机制，从而使该模型能够从培训语料库中检索相关环境作为能力为提示增强。广泛的实验表明，Retroppt可以在几次射击和零拍设置中获得更好的性能。此外，我们进一步说明，我们提出的撤退可以通过新数据集获得更好的概括能力。对记忆的详细分析确实显示逆转可以减少语言模型对记忆的依赖；因此，改善下游任务的概括。

How can we extend a pre-trained model to many language understanding tasks, without labeled or additional unlabeled data? Pre-trained language models (PLMs) have been effective for a wide range of NLP tasks. However, existing approaches either require fine-tuning on downstream labeled datasets or manually constructing proper prompts. In this paper, we propose nonparametric prompting PLM (NPPrompt) for fully zero-shot language understanding. Unlike previous methods, NPPrompt uses only pre-trained language models and does not require any labeled data or additional raw corpus for further fine-tuning, nor does it rely on humans to construct a comprehensive set of prompt label words. We evaluate NPPrompt against previous major few-shot and zero-shot learning methods on diverse NLP tasks: including text classification, text entailment, similar text retrieval, and paraphrasing. Experimental results demonstrate that our NPPrompt outperforms the previous best fully zero-shot method by big margins, with absolute gains of 12.8% in accuracy on text classification and 18.9% on the GLUE benchmark.

Prompt learning recently become an effective linguistic tool to motivate the PLMs' knowledge on few-shot-setting tasks. However, studies have shown the lack of robustness still exists in prompt learning, since suitable initialization of continuous prompt and expert-first manual prompt are essential in fine-tuning process. What is more, human also utilize their comparative ability to motivate their existing knowledge for distinguishing different examples. Motivated by this, we explore how to use contrastive samples to strengthen prompt learning. In detail, we first propose our model ConsPrompt combining with prompt encoding network, contrastive sampling module, and contrastive scoring module. Subsequently, two sampling strategies, similarity-based and label-based strategies, are introduced to realize differential contrastive learning. The effectiveness of proposed ConsPrompt is demonstrated in five different few-shot learning tasks and shown the similarity-based sampling strategy is more effective than label-based in combining contrastive learning. Our results also exhibits the state-of-the-art performance and robustness in different few-shot settings, which proves that the ConsPrompt could be assumed as a better knowledge probe to motivate PLMs.

在本文中，我们描述了我们参与Case-2022的子任务1，即与休闲新闻语料库的事件因果关系识别。我们通过在少数带注释的示例（即几次配置）上利用一组简单但互补的技术来解决因果关系识别（CRI）任务。我们遵循一种基于迅速的预测方法，用于微调LMS，其中CRI任务被视为掩盖语言建模问题（MLM）。这种方法允许LMS在MLM问题上进行本地预先训练，可以直接生成对CRI特异性提示的文本响应。我们将此方法的性能与在整个数据集中训练的集合技术进行比较。我们表现​​最佳的提交仅接受了每班256个实例，整个数据集的一小部分培训，但能够获得第二好的精度（0.82），第三好的精度（0.82）和F1得分。 （0.85）非常接近获胜者团队（0.86）的报道。

Recent studies have revealed the intriguing few-shot learning ability of pretrained language models (PLMs): They can quickly adapt to a new task when fine-tuned on a small amount of labeled data formulated as prompts, without requiring abundant task-specific annotations. Despite their promising performance, most existing few-shot approaches that only learn from the small training set still underperform fully supervised training by nontrivial margins. In this work, we study few-shot learning with PLMs from a different perspective: We first tune an autoregressive PLM on the few-shot samples and then use it as a generator to synthesize a large amount of novel training samples which augment the original training set. To encourage the generator to produce label-discriminative samples, we train it via weighted maximum likelihood where the weight of each token is automatically adjusted based on a discriminative meta-learning objective. A classification PLM can then be fine-tuned on both the few-shot and the synthetic samples with regularization for better generalization and stability. Our approach FewGen achieves an overall better result across seven classification tasks of the GLUE benchmark than existing few-shot learning methods, improving no-augmentation methods by 5+ average points, and outperforming augmentation methods by 3+ average points.

We describe PromptBoosting, a query-efficient procedure for building a text classifier from a neural language model (LM) without access to the LM's parameters, gradients, or hidden representations. This form of "black-box" classifier training has become increasingly important as the cost of training and inference in large-scale LMs grows. But existing black-box LM classifier learning approaches are themselves computationally inefficient, typically specializing LMs to the target task by searching in a large space of (discrete or continuous) prompts using zeroth-order optimization methods. Instead of directly optimizing in prompt space, PromptBoosting obtains a small pool of prompts via a gradient-free approach and then constructs a large pool of weak learners by pairing these prompts with different elements of the LM's output distribution. These weak learners are then ensembled using the AdaBoost algorithm. The entire learning process requires only a small number of forward passes and no backward pass. Experiments show that PromptBoosting achieves state-of-the-art performance in multiple black-box few-shot classification tasks, and matches or outperforms full fine-tuning in both few-shot and standard learning paradigms, while training 10x faster than existing black-box methods.

示范学习旨在通过在少数射击设置中提供回答的演示来指导及时的预测。尽管取得了令人鼓舞的结果，但现有工作仅将回答的示例与及时模板（包括原始上下文）相连，而无需任何其他操作，从而忽略了迅速示意的依赖性。此外，先前的研究发现，随机替换示威的标签极小地损害了性能，这表明该模型无法正确地了解示威活动所带来的知识。受到人类学习过程的启发，在本文中，我们引入了模仿演示学习（模仿），以通过明确模仿人类审查行为来加强演示学习，其中包括：（1）对比度学习机制，以专注于类似的演示。 （2）证明标签重新预测方法以合并已知知识。实验结果表明，我们提出的方法在14个分类中心中有11个实现了最先进的性能。进一步的研究还证明，模仿 - demo加强了迅速与示威之间的关联，这可以为探索示范学习的工作方式提供基础。

Pre-trained large language models can efficiently interpolate human-written prompts in a natural way. Multitask prompted learning can help generalization through a diverse set of tasks at once, thus enhancing the potential for more effective downstream fine-tuning. To perform efficient multitask-inference in the same batch, parameter-efficient fine-tuning methods such as prompt tuning have been proposed. However, the existing prompt tuning methods may lack generalization. We propose SPT, a semi-parametric prompt tuning method for multitask prompted learning. The novel component of SPT is a memory bank from where memory prompts are retrieved based on discrete prompts. Extensive experiments, such as (i) fine-tuning a full language model with SPT on 31 different tasks from 8 different domains and evaluating zero-shot generalization on 9 heldout datasets under 5 NLP task categories and (ii) pretraining SPT on the GLUE datasets and evaluating fine-tuning on the SuperGLUE datasets, demonstrate effectiveness of SPT.

最近的自然语言理解进展（NLU）已经被驱动，部分是由胶水，超级格，小队等的基准。事实上，许多NLU模型现在在许多任务中匹配或超过“人类水平”性能这些基准。然而，大多数这些基准测试都提供模型访问相对大量的标记数据进行培训。因此，该模型提供了比人类所需的更多数据，以实现强大的性能。这有动机侧重于侧重于改善NLU模型的少量学习性能。然而，缺乏少量射门的标准化评估基准，导致不同纸张中的不同实验设置。为了帮助加速这一工作的工作，我们介绍了线索（受限制的语言理解评估标准），这是评估NLU模型的几次拍摄学习功能的基准。我们证明，虽然最近的模型在获得大量标记数据时达到人类性能，但对于大多数任务，少量拍摄设置中的性能存在巨大差距。我们还展示了几个拍摄设置中替代模型家族和适应技术之间的差异。最后，我们讨论了在设计实验设置时讨论了评估真实少量学习绩效的实验设置，并提出了统一的标准化方法，以获得少量学习评估。我们的目标是鼓励对NLU模型的研究，可以概括为具有少数示例的新任务。线索的代码和数据可以在https://github.com/microsoft/clues提供。

少量学习时，基于及时的方法很强劲。然而，Perez等人。 （2021年）最近对他们的表现产生了疑问，因为它们难以在“真实”的几次拍摄设置中获得良好的结果，其中提示和超级参数无法在DEV集上调整。鉴于此，我们对PET进行了广泛的研究，该方法将文本指令与基于示例的FENETUNING结合起来。我们表明，如果正确配置，宠物在真正的几次拍摄设置中强烈执行，即，没有开发装置。这对这种强大的表现至关重要是宠物智能处理多个提示的能力。然后，我们通过在RAFT上运行PET来将我们的调查结果置于真实世界的测试中，直接从现实的NLP应用程序采取的任务的基准，没有标记的开发或测试集。宠物在筏上实现了新的艺术状态，并且在11个任务中靠近非专家人类进行了近距离进行。这些结果表明，基于及时的学习者像宠物Excel这样的真正的几次拍摄学习和支持我们的信念，即从指示中学习的信念将在人类少量学习能力的路径上发挥重要作用。

The remarkable success of pretrained language models has motivated the study of what kinds of knowledge these models learn during pretraining. Reformulating tasks as fillin-the-blanks problems (e.g., cloze tests) is a natural approach for gauging such knowledge, however, its usage is limited by the manual effort and guesswork required to write suitable prompts. To address this, we develop AUTOPROMPT, an automated method to create prompts for a diverse set of tasks, based on a gradient-guided search. Using AUTO-PROMPT, we show that masked language models (MLMs) have an inherent capability to perform sentiment analysis and natural language inference without additional parameters or finetuning, sometimes achieving performance on par with recent state-of-the-art supervised models. We also show that our prompts elicit more accurate factual knowledge from MLMs than the manually created prompts on the LAMA benchmark, and that MLMs can be used as relation extractors more effectively than supervised relation extraction models. These results demonstrate that automatically generated prompts are a viable parameter-free alternative to existing probing methods, and as pretrained LMs become more sophisticated and capable, potentially a replacement for finetuning.

When scaled to hundreds of billions of parameters, pretrained language models such as GPT-3 (Brown et al., 2020) achieve remarkable few-shot performance. However, enormous amounts of compute are required for training and applying such big models, resulting in a large carbon footprint and making it difficult for researchers and practitioners to use them. We show that performance similar to GPT-3 can be obtained with language models that are much "greener" in that their parameter count is several orders of magnitude smaller. This is achieved by converting textual inputs into cloze questions that contain a task description, combined with gradient-based optimization; exploiting unlabeled data gives further improvements. We identify key factors required for successful natural language understanding with small language models. 1

本文探讨了提高语言模型的零次学习能力的简单方法。我们表明，指令调整 - 通过对说明书中所述的任务集合微调语言模型 - 大幅提升零射门上看不见任务中的表现。我们采取预训练的语言模型和指令调整它通过自然语言指令模板语言表达了60NLP任务137B参数。我们评估这种指令调整模型，我们称之为FLAN，在看不见的任务类型。FLAN显着改善其未修饰的对应的性能和超过25的20个任务，我们评估零射门175BGPT-3。FLAN甚至GPT-3通过在安利，RTE，BoolQ，AI2-ARC，OpenbookQA和StoryCloze大比分胜过几拍。消融研究显示任务和模型的规模，这个数字是指令调整取得成功的关键组成部分。