The explosion of e-commerce has caused the need for processing and analysis of product titles, like entity typing in product titles. However, the rapid activity in e-commerce has led to the rapid emergence of new entities, which is difficult to be solved by general entity typing. Besides, product titles in e-commerce have very different language styles from text data in general domain. In order to handle new entities in product titles and address the special language styles problem of product titles in e-commerce domain, we propose our textual entailment model with continuous prompt tuning based hypotheses and fusion embeddings for e-commerce entity typing. First, we reformulate the entity typing task into a textual entailment problem to handle new entities that are not present during training. Second, we design a model to automatically generate textual entailment hypotheses using a continuous prompt tuning method, which can generate better textual entailment hypotheses without manual design. Third, we utilize the fusion embeddings of BERT embedding and CharacterBERT embedding with a two-layer MLP classifier to solve the problem that the language styles of product titles in e-commerce are different from that of general domain. To analyze the effect of each contribution, we compare the performance of entity typing and textual entailment model, and conduct ablation studies on continuous prompt tuning and fusion embeddings. We also evaluate the impact of different prompt template initialization for the continuous prompt tuning. We show our proposed model improves the average F1 score by around 2% compared to the baseline BERT entity typing model.

及时调整是将预训练模型调整到下游任务的极其有效的工具。但是，基于标准及时的方法主要考虑下游任务的足够数据的情况。目前尚不清楚是否可以将优势传输到几杆式制度，在每个下游任务中只有有限的数据。尽管有些作品证明了在几次弹奏设置下及时调整的潜力，但通过搜索离散提示或使用有限数据调整软提示的主流方法仍然非常具有挑战性。通过广泛的实证研究，我们发现迅速调整和完全微调之间的学习差距仍然存在差距。为了弥合差距，我们提出了一个新的及时调整框架，称为软模板调整（STT）。 STT结合了手册和自动提示，并将下游分类任务视为掩盖语言建模任务。对不同设置的全面评估表明，STT可以在不引入其他参数的情况下缩小微调和基于及时的方法之间的差距。值得注意的是，它甚至可以胜过情感分类任务的时间和资源消耗的微调方法。

Pre-trained Language Models (PLMs) have been applied in NLP tasks and achieve promising results. Nevertheless, the fine-tuning procedure needs labeled data of the target domain, making it difficult to learn in low-resource and non-trivial labeled scenarios. To address these challenges, we propose Prompt-based Text Entailment (PTE) for low-resource named entity recognition, which better leverages knowledge in the PLMs. We first reformulate named entity recognition as the text entailment task. The original sentence with entity type-specific prompts is fed into PLMs to get entailment scores for each candidate. The entity type with the top score is then selected as final label. Then, we inject tagging labels into prompts and treat words as basic units instead of n-gram spans to reduce time complexity in generating candidates by n-grams enumeration. Experimental results demonstrate that the proposed method PTE achieves competitive performance on the CoNLL03 dataset, and better than fine-tuned counterparts on the MIT Movie and Few-NERD dataset in low-resource settings.

The recent GPT-3 model (Brown et al., 2020) achieves remarkable few-shot performance solely by leveraging a natural-language prompt and a few task demonstrations as input context. Inspired by their findings, we study few-shot learning in a more practical scenario, where we use smaller language models for which fine-tuning is computationally efficient. We present LM-BFF-better few-shot fine-tuning of language models 1 -a suite of simple and complementary techniques for finetuning language models on a small number of annotated examples. Our approach includes (1) prompt-based fine-tuning together with a novel pipeline for automating prompt generation; and (2) a refined strategy for dynamically and selectively incorporating demonstrations into each context. Finally, we present a systematic evaluation for analyzing few-shot performance on a range of NLP tasks, including classification and regression. Our experiments demonstrate that our methods combine to dramatically outperform standard fine-tuning procedures in this low resource setting, achieving up to 30% absolute improvement, and 11% on average across all tasks. Our approach makes minimal assumptions on task resources and domain expertise, and hence constitutes a strong task-agnostic method for few-shot learning. 2 * The first two authors contributed equally. 1 Alternatively, language models' best friends forever. 2 Our implementation is publicly available at https:// github.com/princeton-nlp/LM-BFF.

我们研究了很少的细粒实体键入（FET）的问题，其中只有几个带注释的实体对每种实体类型提供了上下文。最近，基于及时的调整通过将实体类型分类任务作为“填补空白”的问题来表明在几次射击方案中表现出优越的性能。这允许有效利用预训练的语言模型（PLM）的强语建模能力。尽管当前基于及时的调整方法成功了，但仍有两个主要挑战：（1）提示中的口头化器要么是由外部知识基础手动设计或构建的，而无需考虑目标语料库和标签层次结构信息，而且（2）当前方法主要利用PLM的表示能力，但没有通过广泛的通用域预训练来探索其产生的功率。在这项工作中，我们为由两个模块组成的几个弹药fet提出了一个新颖的框架：（1）实体类型标签解释模块自动学习将类型标签与词汇联系起来，通过共同利用几个播放实例和标签层次结构和标签层次结构，以及（2）基于类型的上下文化实例生成器根据给定实例生成新实例，以扩大培训集以更好地概括。在三个基准数据集上，我们的模型优于大量利润的现有方法。可以在https://github.com/teapot123/fine-graining-entity-typing上找到代码。

How can we extend a pre-trained model to many language understanding tasks, without labeled or additional unlabeled data? Pre-trained language models (PLMs) have been effective for a wide range of NLP tasks. However, existing approaches either require fine-tuning on downstream labeled datasets or manually constructing proper prompts. In this paper, we propose nonparametric prompting PLM (NPPrompt) for fully zero-shot language understanding. Unlike previous methods, NPPrompt uses only pre-trained language models and does not require any labeled data or additional raw corpus for further fine-tuning, nor does it rely on humans to construct a comprehensive set of prompt label words. We evaluate NPPrompt against previous major few-shot and zero-shot learning methods on diverse NLP tasks: including text classification, text entailment, similar text retrieval, and paraphrasing. Experimental results demonstrate that our NPPrompt outperforms the previous best fully zero-shot method by big margins, with absolute gains of 12.8% in accuracy on text classification and 18.9% on the GLUE benchmark.

Neural language representation models such as BERT pre-trained on large-scale corpora can well capture rich semantic patterns from plain text, and be fine-tuned to consistently improve the performance of various NLP tasks. However, the existing pre-trained language models rarely consider incorporating knowledge graphs (KGs), which can provide rich structured knowledge facts for better language understanding. We argue that informative entities in KGs can enhance language representation with external knowledge. In this paper, we utilize both large-scale textual corpora and KGs to train an enhanced language representation model (ERNIE), which can take full advantage of lexical, syntactic, and knowledge information simultaneously. The experimental results have demonstrated that ERNIE achieves significant improvements on various knowledge-driven tasks, and meanwhile is comparable with the state-of-the-art model BERT on other common NLP tasks. The source code and experiment details of this paper can be obtained from https:// github.com/thunlp/ERNIE.

Controlled text generation is a very important task in the arena of natural language processing due to its promising applications. In order to achieve this task we mainly introduce the novel soft prompt tuning method of using soft prompts at both encoder and decoder levels together in a T5 model and investigate the performance as the behaviour of an additional soft prompt related to the decoder of a T5 model in controlled text generation remained unexplored. Then we also investigate the feasibility of steering the output of this extended soft prompted T5 model at decoder level and finally analyse the utility of generated text to be used in AI related tasks such as training AI models with an interpretability analysis of the classifier trained with synthetic text, as there is a lack of proper analysis of methodologies in generating properly labelled data to be utilized in AI tasks. Through the performed in-depth intrinsic and extrinsic evaluations of this generation model along with the artificially generated data, we found that this model produced better results compared to the T5 model with a single soft prompt at encoder level and the sentiment classifier trained using this artificially generated data can produce comparable classification results to the results of a classifier trained with real labelled data and also the classifier decision is interpretable with respect to the input text content.

快速学习已成为现代自然语言处理的新范式，它直接适应培训的语言模型（PLMS）到$ CLOZE $ -Style预测，自回归建模或序列到序列生成，从而导致各种任务的表现。但是，尚未提出及时学习的标准实施框架，以及大多数现有的及时学习码条，通常是不受管制的，仅为特定方案提供有限的实现。由于有许多细节，例如模板策略，初始化策略和语言化策略等，因此需要在快速学习中考虑，从业者面临障碍，以便快速调整所需的迅速学习方法到他们的应用程序。在本文中，我们展示了{OpenPrompt}，一个统一的易于使用的工具包，可以通过PLMS快速学习。 OpenPrompt是一项研究型框架，配备了效率，模块化和可扩展性，其组合性允许自由地将不同的PLMS，任务格式和提示模块组合在统一的范例中。用户可以宽松地部署快速学习框架，并在没有约束的情况下在不同的NLP任务上评估它们的泛化。 OpenPrompt在{\ url {https://github.com/thunlp/openprompt}}上公开发布。

With the success of the prompt-tuning paradigm in Natural Language Processing (NLP), various prompt templates have been proposed to further stimulate specific knowledge for serving downstream tasks, e.g., machine translation, text generation, relation extraction, and so on. Existing prompt templates are mainly shared among all training samples with the information of task description. However, training samples are quite diverse. The sharing task description is unable to stimulate the unique task-related information in each training sample, especially for tasks with the finite-label space. To exploit the unique task-related information, we imitate the human decision process which aims to find the contrastive attributes between the objective factual and their potential counterfactuals. Thus, we propose the \textbf{C}ounterfactual \textbf{C}ontrastive \textbf{Prompt}-Tuning (CCPrompt) approach for many-class classification, e.g., relation classification, topic classification, and entity typing. Compared with simple classification tasks, these tasks have more complex finite-label spaces and are more rigorous for prompts. First of all, we prune the finite label space to construct fact-counterfactual pairs. Then, we exploit the contrastive attributes by projecting training instances onto every fact-counterfactual pair. We further set up global prototypes corresponding with all contrastive attributes for selecting valid contrastive attributes as additional tokens in the prompt template. Finally, a simple Siamese representation learning is employed to enhance the robustness of the model. We conduct experiments on relation classification, topic classification, and entity typing tasks in both fully supervised setting and few-shot setting. The results indicate that our model outperforms former baselines.

语言模型（LMS）已被证明在各种下游应用程序中很有用，例如摘要，翻译，问答和文本分类。由于它们可以存储的大量信息，LMS正在成为人工智能中越来越重要的工具。在这项工作中，我们提出了道具（提示为探测），该道具利用GPT-3（最初由OpenAI在2020年提出的大型语言模型）来执行知识基础构建任务（KBC）。 Prop实施了一种多步骤方法，该方法结合了各种提示技术来实现这一目标。我们的结果表明，手动提示策划是必不可少的，必须鼓励LM给出可变长度的答案集，特别是包括空的答案集，True/False问题是提高LM生成的建议精度的有用设备。 LM的大小是至关重要的因素，并且实体字典别名提高了LM评分。我们的评估研究表明，这些提出的技术可以大大提高最终预测的质量：Prop赢得了LM-KBC竞争的轨道2，表现优于基线36.4个百分点。我们的实施可在https://github.com/hemile/iswc-challenge上获得。

已显示迅速学习可以在大多数文本分类任务中实现近调调节性能，但很少有培训示例。对于样品稀缺的NLP任务是有利的。在本文中，我们试图将其应用于实际情况，即恢复信息提取，并增强现有方法，以使其更适用于简历信息提取任务。特别是，我们根据简历的文本特征创建了多组手动模板和语言器。此外，我们比较了蒙版语言模型（MLM）预培训语言模型（PLM）和SEQ2SEQ PLM在此任务上的性能。此外，我们改进了口头设计的设计方法，用于知识渊博的及时调整，以便为其他基于应用程序的NLP任务的迅速模板和语言设计的设计提供了示例。在这种情况下，我们提出了手动知识渊博的语言器（MKV）的概念。构造与应用程序方案相对应的知识渊博的口头表的规则。实验表明，基于我们的规则设计的模板和言语器比现有的手动模板更有效，更强大，并自动生成及时方法。已经确定，当前可用的自动提示方法无法与手动设计的及时模板竞争一些现实的任务方案。最终混淆矩阵的结果表明，我们提出的MKV显着解决了样本不平衡问题。

In this work, we explore "prompt tuning," a simple yet effective mechanism for learning "soft prompts" to condition frozen language models to perform specific downstream tasks. Unlike the discrete text prompts used by GPT-3, soft prompts are learned through backpropagation and can be tuned to incorporate signals from any number of labeled examples. Our end-to-end learned approach outperforms GPT-3's few-shot learning by a large margin. More remarkably, through ablations on model size using T5, we show that prompt tuning becomes more competitive with scale: as models exceed billions of parameters, our method "closes the gap" and matches the strong performance of model tuning (where all model weights are tuned). This finding is especially relevant because large models are costly to share and serve and the ability to reuse one frozen model for multiple downstream tasks can ease this burden. Our method can be seen as a simplification of the recently proposed "prefix tuning" of Li and Liang (2021) and we provide a comparison to this and other similar approaches. Finally, we show that conditioning a frozen model with soft prompts confers benefits in robustness to domain transfer and enables efficient "prompt ensembling." * Work done as a Google AI Resident.

预训练模型已在许多代码智能任务中有效。这些模型在大规模未标记的语料库中进行了预训练，然后在下游任务中进行了微调。但是，由于预训练和下游任务的输入是不同的形式，因此很难充分探索预训练模型的知识。此外，微调的性能强烈依赖于下游数据的量，而实际上，具有稀缺数据的场景很常见。自然语言处理（NLP）领域的最新研究表明，迅速调整，一种调整的新范式，减轻上述问题并在各种NLP任务中实现了有希望的结果。在迅速调整中，在调整过程中插入的提示提供了特定于任务的知识，这对于具有相对较少数据的任务特别有益。在本文中，我们凭经验评估了代码智能任务中迅速调整的用法和效果。我们对流行的预训练模型Codebert和codet5进行及时调整，并尝试三个代码智能任务，包括缺陷预测，代码摘要和代码翻译。我们的实验结果表明，在所有三个任务中，迅速调整始终优于微调。此外，及时调整在低资源场景中显示出很大的潜力，例如，对于代码摘要，平均将微调的BLEU分数提高了26％以上。我们的结果表明，我们可以调整代码智能任务的迅速调整，以实现更好的性能，尤其是在缺乏特定于任务的数据时，我们可以调整及时调整。

生物重建VII Track-2挑战包括命名实体识别，实体链接（或实体 - 归一化），主题索引任务 - 与实体和主题限制为这项挑战的化学品。命名实体识别是一个完善的问题，我们通过基于Bert的生物群体模型实现了我们的最佳性能。我们将基于BERT的方法扩展到实体链接任务。在预先预订Biobert的第二阶段，通过称为自对准预先训练（SAP）的度量学习损失策略，我们将基于其SAP-Biobert Word Embeddings之间的余弦相似性链接实体。尽管我们的命名实体识别实验取得了成功，但我们发现化学指数任务一般更具挑战性。除了传统的NER方法之外，我们还尝试使用基于新颖的文本或“提示”方法的命名实体识别和实体链接，该方法使用生成语言模型，例如T5和GPT。我们通过这种新方法实现了令人鼓舞的结果。

由于开放的社交平台允许大量未经验证的信息流动，因此谣言可以出乎意料地出现并迅速传播。但是，现有的谣言检测（RD）模型通常会采用相同的培训和测试分布，并且无法应对不断变化的社交网络环境。本文提出了一个持续的及时调整RD（CPT-RD）框架，该框架避免了在顺序任务学习过程中上游任务的灾难性遗忘（CF），并使域任务之间的双向知识转移。具体而言，我们提出以下策略：（a）我们的设计明确地将共享和特定于领域的知识分解，从而减少了优化过程中不同领域的干扰； （b）几种技术旨在转移上游任务的知识以应对紧急情况； （c）任务条件的及时性超网（TPHNET）用于合并过去的域。此外，CPT-RD避免了CF，而无需进行排练缓冲区。

对于自然语言处理中的许多任务，将知识从一个域转移到另一个领域至关重要，尤其是当目标域中的可用数据量受到限制时。在这项工作中，我们在指定实体识别（NER）的背景下提出了一种新颖的域适应方法。我们提出了一种两步方法，该方法由可变基本模块和模板模块组成，该模块在简单的描述模式的帮助下利用了预训练的语言模型中捕获的知识。我们的方法简单而通用，可以在几次射击和零拍设置中应用。评估我们在许多不同数据集中的轻量级方法表明，它可以将最新基准的性能提高2-5％的F1分数。

预先接受的语言模型实现了最先进的导致各种自然语言处理（NLP）任务。 GPT-3表明，缩放预先训练的语言模型可以进一步利用它们的巨大潜力。最近提出了一个名为Ernie 3.0的统一框架，以预先培训大型知识增强型号，并培训了具有10亿参数的模型。 Ernie 3.0在各种NLP任务上表现出最先进的模型。为了探讨缩放的表现，我们培养了百卢比的3.0泰坦参数型号，在PaddlePaddle平台上有高达260亿参数的泰坦。此外，我们设计了一种自我监督的对抗性损失和可控语言建模损失，以使ERNIE 3.0 TITAN产生可信和可控的文本。为了减少计算开销和碳排放，我们向Ernie 3.0泰坦提出了一个在线蒸馏框架，教师模型将同时教授学生和培训。埃塞尼3.0泰坦是迄今为止最大的中国密集预训练模型。经验结果表明，Ernie 3.0泰坦在68个NLP数据集中优于最先进的模型。

立场检测旨在确定文本的作者是否赞成，反对或中立。这项任务的主要挑战是两个方面的：由于不同目标以及缺乏目标的上下文信息而产生的几乎没有学习。现有作品主要通过设计基于注意力的模型或引入嘈杂的外部知识来解决第二期，而第一个问题仍未探索。在本文中，受到预训练的语言模型（PLM）的潜在能力（PLM）的启发，我们建议介绍基于立场检测的及时基于迅速的微调。 PLM可以为目标提供基本的上下文信息，并通过提示启用几次学习。考虑到目标在立场检测任务中的关键作用，我们设计了目标感知的提示并提出了一种新颖的语言。我们的语言器不会将每个标签映射到具体单词，而是将每个标签映射到矢量，并选择最能捕获姿势与目标之间相关性的标签。此外，为了减轻通过单人工提示来处理不同目标的可能缺陷，我们建议将信息从多个提示中学到的信息提炼。实验结果表明，我们提出的模型在全数据和少数场景中的表现出色。

在本文中，我们描述了我们参与Case-2022的子任务1，即与休闲新闻语料库的事件因果关系识别。我们通过在少数带注释的示例（即几次配置）上利用一组简单但互补的技术来解决因果关系识别（CRI）任务。我们遵循一种基于迅速的预测方法，用于微调LMS，其中CRI任务被视为掩盖语言建模问题（MLM）。这种方法允许LMS在MLM问题上进行本地预先训练，可以直接生成对CRI特异性提示的文本响应。我们将此方法的性能与在整个数据集中训练的集合技术进行比较。我们表现​​最佳的提交仅接受了每班256个实例，整个数据集的一小部分培训，但能够获得第二好的精度（0.82），第三好的精度（0.82）和F1得分。 （0.85）非常接近获胜者团队（0.86）的报道。