Contrastive learning has become a new paradigm for unsupervised sentence embeddings. Previous studies focus on instance-wise contrastive learning, attempting to construct positive pairs with textual data augmentation. In this paper, we propose a novel Contrastive learning method with Prompt-derived Virtual semantic Prototypes (ConPVP). Specifically, with the help of prompts, we construct virtual semantic prototypes to each instance, and derive negative prototypes by using the negative form of the prompts. Using a prototypical contrastive loss, we enforce the anchor sentence embedding to be close to its corresponding semantic prototypes, and far apart from the negative prototypes as well as the prototypes of other sentences. Extensive experimental results on semantic textual similarity, transfer, and clustering tasks demonstrate the effectiveness of our proposed model compared to strong baselines. Code is available at https://github.com/lemon0830/promptCSE.

在以前的作品中广泛讨论了句子语义相似性的原始伯特的表现不佳。我们发现不满意的性能主要是由于静态令牌嵌入偏差和无效的伯特层，而不是姓氏的高余弦相似性。为此，我们提出了一个迅速的句子嵌入方法，可以减少令牌嵌入偏差，使原始伯特层更有效。通过将句子嵌入式任务重新塑造为填充空白问题，我们的方法显着提高了原始伯特的性能。我们讨论了两个提示符，表示基于及时的句子嵌入的三个提示搜索方法。此外，我们提出了一种通过模板去噪技术的新型无监督培训目标，这大大缩短了监督和无人监督的环境之间的性能差距。对于实验，我们评估我们在非微调和微调的设置上的方法。即使是非微调方法也可以优于STS任务上的无监督服务器等微调的方法。我们的微调方法在无监督和监督设置中优于最先进的方法SIMCSE。与SIMCSE相比，我们分别在无监督环境下实现了2.29和2.58点的伯特和罗伯塔的改进。

对比学习一直吸引着学习无监督的句子嵌入。当前的最新无监督方法是无监督的SIMCSE（UNSUP-SIMCSE）。 Unsup-Simcse将辍学作为最小数据增强方法，并将相同的输入句子传递给预训练的变压器编码器（带有掉落的掉落）两次，以获取两个相应的嵌入式以构建正对。由于句子的长度信息通常会由于使用嵌入变压器中的位置嵌入而编码到句子嵌入中，因此Unsup-Simcse中的每个正对实际上包含相同的长度信息。因此，接受这些正面对训练的Unsup-Simcse可能是有偏见的，这往往会考虑到语义上相同长度或相似长度的句子更相似。通过统计观察，我们发现Unsup-Simcse确实存在这样的问题。为了减轻它，我们应用了一个简单的重复操作来修改输入句子，然后分别将输入句子及其修改后的对应物传递给预训练的变压器编码器，以获取阳性对。此外，我们从计算机视觉社区中汲取灵感，并引入动量对比度，从而扩大了负面对的数量，而没有其他计算。提出的两种修改分别应用于正和负对，并构建一种新的句子嵌入方法，称为增强的Unsup-Simcse（ESIMCSE）。我们在几个基准数据集W.R.T上评估了所提出的ESIMCSE，语义文本相似性（STS）任务。实验结果表明，ESIMCSE的表现优于最先进的undup-Simcse，而Bert基碱的平均长矛相关性为2.02％。

This paper presents SimCSE, a simple contrastive learning framework that greatly advances state-of-the-art sentence embeddings. We first describe an unsupervised approach, which takes an input sentence and predicts itself in a contrastive objective, with only standard dropout used as noise. This simple method works surprisingly well, performing on par with previous supervised counterparts. We find that dropout acts as minimal data augmentation, and removing it leads to a representation collapse. Then, we propose a supervised approach, which incorporates annotated pairs from natural language inference datasets into our contrastive learning framework by using "entailment" pairs as positives and "contradiction" pairs as hard negatives. We evaluate SimCSE on standard semantic textual similarity (STS) tasks, and our unsupervised and supervised models using BERT base achieve an average of 76.3% and 81.6% Spearman's correlation respectively, a 4.2% and 2.2% improvement compared to the previous best results. We also show-both theoretically and empirically-that the contrastive learning objective regularizes pre-trained embeddings' anisotropic space to be more uniform, and it better aligns positive pairs when supervised signals are available. 1 2 We randomly sample 10 6 sentences from English Wikipedia and fine-tune BERTbase with learning rate = 3e-5, N = 64. In all our experiments, no STS training sets are used.

With the success of the prompt-tuning paradigm in Natural Language Processing (NLP), various prompt templates have been proposed to further stimulate specific knowledge for serving downstream tasks, e.g., machine translation, text generation, relation extraction, and so on. Existing prompt templates are mainly shared among all training samples with the information of task description. However, training samples are quite diverse. The sharing task description is unable to stimulate the unique task-related information in each training sample, especially for tasks with the finite-label space. To exploit the unique task-related information, we imitate the human decision process which aims to find the contrastive attributes between the objective factual and their potential counterfactuals. Thus, we propose the \textbf{C}ounterfactual \textbf{C}ontrastive \textbf{Prompt}-Tuning (CCPrompt) approach for many-class classification, e.g., relation classification, topic classification, and entity typing. Compared with simple classification tasks, these tasks have more complex finite-label spaces and are more rigorous for prompts. First of all, we prune the finite label space to construct fact-counterfactual pairs. Then, we exploit the contrastive attributes by projecting training instances onto every fact-counterfactual pair. We further set up global prototypes corresponding with all contrastive attributes for selecting valid contrastive attributes as additional tokens in the prompt template. Finally, a simple Siamese representation learning is employed to enhance the robustness of the model. We conduct experiments on relation classification, topic classification, and entity typing tasks in both fully supervised setting and few-shot setting. The results indicate that our model outperforms former baselines.

Prompt learning recently become an effective linguistic tool to motivate the PLMs' knowledge on few-shot-setting tasks. However, studies have shown the lack of robustness still exists in prompt learning, since suitable initialization of continuous prompt and expert-first manual prompt are essential in fine-tuning process. What is more, human also utilize their comparative ability to motivate their existing knowledge for distinguishing different examples. Motivated by this, we explore how to use contrastive samples to strengthen prompt learning. In detail, we first propose our model ConsPrompt combining with prompt encoding network, contrastive sampling module, and contrastive scoring module. Subsequently, two sampling strategies, similarity-based and label-based strategies, are introduced to realize differential contrastive learning. The effectiveness of proposed ConsPrompt is demonstrated in five different few-shot learning tasks and shown the similarity-based sampling strategy is more effective than label-based in combining contrastive learning. Our results also exhibits the state-of-the-art performance and robustness in different few-shot settings, which proves that the ConsPrompt could be assumed as a better knowledge probe to motivate PLMs.

We introduce TeSS (Text Similarity Comparison using Sentence Encoder), a framework for zero-shot classification where the assigned label is determined by the embedding similarity between the input text and each candidate label prompt. We leverage representations from sentence encoders optimized to locate semantically similar samples closer to each other in embedding space during pre-training. The label prompt embeddings serve as prototypes of their corresponding class clusters. Furthermore, to compensate for the potentially poorly descriptive labels in their original format, we retrieve semantically similar sentences from external corpora and additionally use them with the original label prompt (TeSS-R). TeSS outperforms strong baselines on various closed-set and open-set classification datasets under zero-shot setting, with further gains when combined with label prompt diversification through retrieval. These results are robustly attained to verbalizer variations, an ancillary benefit of using a bi-encoder. Altogether, our method serves as a reliable baseline for zero-shot classification and a simple interface to assess the quality of sentence encoders.

预先训练的蒙版语言模型通过将下游任务作为文本填充来成功执行几次学习。但是，作为全镜头环境中的强大替代方案，诸如Electra之类的判别预训练模型不适合范式。在这项工作中，我们调整了基于及时的几次学习来进行电信，并表明它在广泛的任务中优于蒙面的语言模型。Electra是预先训练的，以区分令牌是产生还是原始。我们自然地将其扩展到基于迅速的几次学习，通过培训来评分目标选项的原创性，而无需引入新参数。我们的方法很容易适应涉及多token预测的任务，而无需额外的计算开销。分析表明，Electra学习分布与下游任务更好。

学习高质量的对话表示对于解决各种面向对话的任务至关重要，尤其是考虑到对话系统通常会遇到数据稀缺。在本文中，我们介绍了对话句子嵌入（DSE），这是一种自我监督的对比学习方法，它学习有效的对话表示，适合各种对话任务。 DSE通过连续进行与对比度学习的正面对话的连续对话来从对话中学习。尽管它很简单，但DSE的表现能力比其他对话表示和普遍的句子表示模型要好得多。我们评估DSE的五个下游对话任务，这些任务检查了不同语义粒度的对话表示。几次射击和零射击设置的实验表明，DSE的表现要优于基线。例如，它在6个数据集中的1-Shot意图分类中比最强的无监督基线实现了13％的平均绩效提高。我们还提供了有关模型的好处和局限性的分析。

How can we extend a pre-trained model to many language understanding tasks, without labeled or additional unlabeled data? Pre-trained language models (PLMs) have been effective for a wide range of NLP tasks. However, existing approaches either require fine-tuning on downstream labeled datasets or manually constructing proper prompts. In this paper, we propose nonparametric prompting PLM (NPPrompt) for fully zero-shot language understanding. Unlike previous methods, NPPrompt uses only pre-trained language models and does not require any labeled data or additional raw corpus for further fine-tuning, nor does it rely on humans to construct a comprehensive set of prompt label words. We evaluate NPPrompt against previous major few-shot and zero-shot learning methods on diverse NLP tasks: including text classification, text entailment, similar text retrieval, and paraphrasing. Experimental results demonstrate that our NPPrompt outperforms the previous best fully zero-shot method by big margins, with absolute gains of 12.8% in accuracy on text classification and 18.9% on the GLUE benchmark.

最近，与“预训练，及时和预测”的新范式相比，与“预训练，微调”范式相比，新的范式“预训练，及时和预测”取得了显着的成就。在基于及时的GPT-3成功之后，一系列基于蒙版的语言模型（MLM）（例如Bert，Roberta）及时学习方法变得流行并广泛使用。但是，另一个有效的预训练的判别模型Electra可能被忽略了。在本文中，我们尝试使用拟议的替换代替令牌检测（RTD）基于基于的及时学习方法来完成零摄像的几个NLP任务。实验结果表明，基于RTD-Prompt学习的Electra模型可达到令人惊讶的最先进的零拍性能。在数字上，与MLM-Roberta-Large和MLM-Bert-Large相比，我们的RTD-Electra-Large在所有15个任务上平均提高了约8.4％和13.7％。特别是在SST-2任务上，我们的RTD-Electra-Large在没有任何培训数据的情况下达到了令人惊讶的90.1％精度。总体而言，与预先训练的蒙版语言模型相比，预先训练的代替令牌检测模型在零拍学习中的性能更好。因此，Electra是一位出色的零球学习者。源代码可在以下网址获得：https：//github.com/nishiwen1214/rtd-electra。

大型预训练的语言模型（PLM）的最新进展导致了自然语言理解（NLU）任务的令人印象深刻的增长，并具有特定于任务的微调。但是，直接调整PLM在很大程度上依赖大量的标记实例，这些实例通常很难获得。迅速对PLM的调整已被证明对各种少数次任务很有价值。现有的作品研究基于迅速的NLU任务的基于及时的调整，主要集中于用语言器来得出正确的标签单词或生成及时的模板，以从PLM中启发语义。此外，还对常规数据增强方法进行了验证，可用于少量射击任务。但是，目前几乎没有针对基于及时的调整范式设计的数据增强方法。因此，我们研究了迅速的少数射击学习者的新数据增强问题。由于标签语义对于迅速的调整至关重要，因此我们提出了一种新颖的标签引导数据增强方法促进DA，该方法利用了丰富的标签语义信息以进行数据增强。很少的文本分类任务的广泛实验结果表明，我们提出的框架通过有效利用标签语义和数据扩展来实现自然语言理解来实现卓越的性能。

We present Relational Sentence Embedding (RSE), a new paradigm to further discover the potential of sentence embeddings. Prior work mainly models the similarity between sentences based on their embedding distance. Because of the complex semantic meanings conveyed, sentence pairs can have various relation types, including but not limited to entailment, paraphrasing, and question-answer. It poses challenges to existing embedding methods to capture such relational information. We handle the problem by learning associated relational embeddings. Specifically, a relation-wise translation operation is applied to the source sentence to infer the corresponding target sentence with a pre-trained Siamese-based encoder. The fine-grained relational similarity scores can be computed from learned embeddings. We benchmark our method on 19 datasets covering a wide range of tasks, including semantic textual similarity, transfer, and domain-specific tasks. Experimental results show that our method is effective and flexible in modeling sentence relations and outperforms a series of state-of-the-art sentence embedding methods. https://github.com/BinWang28/RSE

The recent GPT-3 model (Brown et al., 2020) achieves remarkable few-shot performance solely by leveraging a natural-language prompt and a few task demonstrations as input context. Inspired by their findings, we study few-shot learning in a more practical scenario, where we use smaller language models for which fine-tuning is computationally efficient. We present LM-BFF-better few-shot fine-tuning of language models 1 -a suite of simple and complementary techniques for finetuning language models on a small number of annotated examples. Our approach includes (1) prompt-based fine-tuning together with a novel pipeline for automating prompt generation; and (2) a refined strategy for dynamically and selectively incorporating demonstrations into each context. Finally, we present a systematic evaluation for analyzing few-shot performance on a range of NLP tasks, including classification and regression. Our experiments demonstrate that our methods combine to dramatically outperform standard fine-tuning procedures in this low resource setting, achieving up to 30% absolute improvement, and 11% on average across all tasks. Our approach makes minimal assumptions on task resources and domain expertise, and hence constitutes a strong task-agnostic method for few-shot learning. 2 * The first two authors contributed equally. 1 Alternatively, language models' best friends forever. 2 Our implementation is publicly available at https:// github.com/princeton-nlp/LM-BFF.

我们提供了从文本到文本变换器（T5）的第一次探索句子嵌入式。句子嵌入式广泛适用于语言处理任务。虽然T5在作为序列到序列映射问题的语言任务上实现令人印象深刻的性能，但目前尚不清楚如何从编码器解码器模型生成陈列嵌入的句子。我们调查三种方法提取T5句子嵌入方法：两个仅利用T5编码器，一个使用全T5编码器解码器模型。为了支持我们的调查，我们建立了一个新的句子代表转移基准，SentGlue，它将Senteval Toolkit扩展到粘合基准的九个任务。我们的编码器的型号优于Senteval和SentGlue传输任务的句子 - BERT和SIMCSE句子嵌入，包括语义文本相似性（STS）。发现从数百万到数十亿参数的缩放T5产生一致的进一步改进。最后，我们的编码器 - 解码器方法在使用句子嵌入时在STS上实现了新的最先进的。我们的模型在https://tfhub.dev/google/collections/sentence-t5/1发布。

对比度学习已逐渐应用于学习高质量的无监督句子嵌入。据我们所知，在以前的无监督方法中，最新的最新方法是无监督的SIMCSE（Unsup-Simcse）。 Unsup-Simcse在训练阶段使用Infonce1Loss功能，通过将语义上相似的句子拉在一起并分开不相似。从理论上讲，我们希望在Unsup-Simcse中使用较大的批次，以在样本中进行更充分的比较并避免过度拟合。但是，增加批量的大小并不总是会导致改进，而是在批处理大小超过阈值时会导致性能降解。通过统计观察，我们发现这可能是由于在批量生产大小后引入了低信心负对。为了减轻这个问题，我们在Infonce损失函数上引入了一种简单的平滑策略，称为Gaussian平滑infonce（GS-Infonce）。特别是，我们将随机的高斯噪声向量添加为负样品，它们的负面样品空间的平滑性。简单，提出的平滑策略为Unsup-Simcse带来了重大改进。我们评估GS-INFONCEON标准语义文本相似性（STS）任务。 GS-Infonce的平均长矛人相关性优于最先进的Unsup-Simcse，在Bert-Base，Bert-Large，Roberta-Base的基础上，长矛人的相关性为1.38％，0.72％，1.17％和0.28％和罗伯塔·洛尔格（Roberta-Large）。

我们提出了Patron，这是一种新方法，它使用基于及时的不确定性估计，用于在冷启动场景下进行预训练的语言模型进行微调的数据选择，即，没有初始标记的数据可用。在顾客中，我们设计（1）一种基于迅速的不确定性传播方法来估计数据点的重要性和（2）分区 - 然后 - 剥离（PTR）策略，以促进对注释的样品多样性。六个文本分类数据集的实验表明，赞助人的表现优于最强的冷启动数据选择基准，高达6.9％。此外，仅具有128个标签，顾客分别基于香草微调和及时的学习，获得了91.0％和92.1％的全面监督性能。我们的赞助人实施可在\ url {https://github.com/yueyu1030/patron}上获得。

无监督的句子嵌入学习最近由对比度学习方法（例如SIMCSE）主导，该方法保持积极对相似，并将负面对拆开。对比操作旨在通过在积极实例之间最大化相互信息来保持尽可能多的信息，从而导致句子嵌入中的冗余信息。为了解决这个问题，我们提出了一个基于信息最小化的对比度学习（Informin-CL）模型，以保留有用的信息并通过最大化相互信息并最大程度地减少无监督句子表示学习的正面实例之间的信息熵，从而丢弃冗余信息。具体而言，我们发现信息最小化可以通过简单的对比度和重建目标来实现。重建操作通过另一个正实例重构积极实例，以最大程度地减少正实例之间的信息熵。我们在下游任务中评估了我们的模型，包括受监督和无监督的（语义文本相似性）任务。广泛的实验结果表明，我们的Informin-CL获得了最先进的性能。

及时调整是将预训练模型调整到下游任务的极其有效的工具。但是，基于标准及时的方法主要考虑下游任务的足够数据的情况。目前尚不清楚是否可以将优势传输到几杆式制度，在每个下游任务中只有有限的数据。尽管有些作品证明了在几次弹奏设置下及时调整的潜力，但通过搜索离散提示或使用有限数据调整软提示的主流方法仍然非常具有挑战性。通过广泛的实证研究，我们发现迅速调整和完全微调之间的学习差距仍然存在差距。为了弥合差距，我们提出了一个新的及时调整框架，称为软模板调整（STT）。 STT结合了手册和自动提示，并将下游分类任务视为掩盖语言建模任务。对不同设置的全面评估表明，STT可以在不引入其他参数的情况下缩小微调和基于及时的方法之间的差距。值得注意的是，它甚至可以胜过情感分类任务的时间和资源消耗的微调方法。

已显示迅速学习可以在大多数文本分类任务中实现近调调节性能，但很少有培训示例。对于样品稀缺的NLP任务是有利的。在本文中，我们试图将其应用于实际情况，即恢复信息提取，并增强现有方法，以使其更适用于简历信息提取任务。特别是，我们根据简历的文本特征创建了多组手动模板和语言器。此外，我们比较了蒙版语言模型（MLM）预培训语言模型（PLM）和SEQ2SEQ PLM在此任务上的性能。此外，我们改进了口头设计的设计方法，用于知识渊博的及时调整，以便为其他基于应用程序的NLP任务的迅速模板和语言设计的设计提供了示例。在这种情况下，我们提出了手动知识渊博的语言器（MKV）的概念。构造与应用程序方案相对应的知识渊博的口头表的规则。实验表明，基于我们的规则设计的模板和言语器比现有的手动模板更有效，更强大，并自动生成及时方法。已经确定，当前可用的自动提示方法无法与手动设计的及时模板竞争一些现实的任务方案。最终混淆矩阵的结果表明，我们提出的MKV显着解决了样本不平衡问题。