GPT-3 (Generative Pre-trained Transformer 3) is a large-scale autoregressive language model developed by OpenAI, which has demonstrated impressive few-shot performance on a wide range of natural language processing (NLP) tasks. Hence, an intuitive application is to use it for data annotation. In this paper, we investigate whether GPT-3 can be used as a good data annotator for NLP tasks. Data annotation is the process of labeling data that could be used to train machine learning models. It is a crucial step in the development of NLP systems, as it allows the model to learn the relationship between the input data and the desired output. Given the impressive language capabilities of GPT-3, it is natural to wonder whether it can be used to effectively annotate data for NLP tasks. In this paper, we evaluate the performance of GPT-3 as a data annotator by comparing it with traditional data annotation methods and analyzing its output on a range of tasks. Through this analysis, we aim to provide insight into the potential of GPT-3 as a general-purpose data annotator in NLP.

GPT-3等大型语言模型是优秀的几次学习者，允许他们通过自然文本提示来控制。最近的研究报告称，基于及时的直接分类消除了对微调的需求，但缺乏数据和推理可扩展性。本文提出了一种新的数据增强技术，利用大规模语言模型来生成来自真实样本的混合的现实文本样本。我们还建议利用语言模型预测的软标签，从大规模语言模型中有效地蒸馏知识并同时创建文本扰动。我们对各种分类任务进行数据增强实验，并显示我们的方法非常优于现有的文本增强方法。消融研究和定性分析为我们的方法提供了更多的见解。

The strong few-shot in-context learning capability of large pre-trained language models (PLMs) such as GPT-3 is highly appealing for application domains such as biomedicine, which feature high and diverse demands of language technologies but also high data annotation costs. In this paper, we present the first systematic and comprehensive study to compare the few-shot performance of GPT-3 in-context learning with fine-tuning smaller (i.e., BERT-sized) PLMs on two highly representative biomedical information extraction tasks, named entity recognition and relation extraction. We follow the true few-shot setting to avoid overestimating models' few-shot performance by model selection over a large validation set. We also optimize GPT-3's performance with known techniques such as contextual calibration and dynamic in-context example retrieval. However, our results show that GPT-3 still significantly underperforms compared to simply fine-tuning a smaller PLM. In addition, GPT-3 in-context learning also yields smaller gains in accuracy when more training data becomes available. Our in-depth analyses further reveal issues of the in-context learning setting that may be detrimental to information extraction tasks in general. Given the high cost of experimenting with GPT-3, we hope our study provides guidance for biomedical researchers and practitioners towards more promising directions such as fine-tuning small PLMs.

近年来，预制语言模型彻底改变了NLP世界，同时在各种下游任务中实现了最先进的性能。但是，在许多情况下，当标记数据稀缺时，这些模型不会表现良好，并且预计模型将在零或几秒钟内执行。最近，有几项工作表明，与下游任务更好地对准的预先预测或执行第二阶段，可以导致改进的结果，尤其是在稀缺数据设置中。在此，我们建议利用携带的情绪话语标记来产生大规模的弱标记数据，这又可以用于适应语言模型进行情感分析。广泛的实验结果显示了我们在各种基准数据集中的方法的价值，包括金融域。在https://github.com/ibm/tslm-discourse-markers上提供代码，模型和数据。

获得具有语义注释的文本数据集是一个艰苦的过程，但对于自然语言过程（NLP）的监督培训至关重要。通常，在特定于域的上下文中开发和应用新的NLP管道通常需要定制设计的数据集来以监督机器学习方式解决NLP任务。当使用非英语语言进行医学数据处理时，这会暴露出几个次要和主要的相互联系的问题，例如缺乏任务匹配数据集以及特定于任务的预训练模型。在我们的工作中，我们建议利用审计的语言模型来培训数据获取，以便检索足够大的数据集，以训练更小，更有效的模型，以便使用特定的特定任务。为了证明您的方法的有效性，我们创建了一个自定义数据集，我们用来培训用于德国文本的医学模型，但在原则上我们的方法仍然不依赖语言。我们获得的数据集以及我们的预培训模型可在以下网址公开获取：https：//github.com/frankkramer-lab/gptnermed

最近的自然语言理解进展（NLU）已经被驱动，部分是由胶水，超级格，小队等的基准。事实上，许多NLU模型现在在许多任务中匹配或超过“人类水平”性能这些基准。然而，大多数这些基准测试都提供模型访问相对大量的标记数据进行培训。因此，该模型提供了比人类所需的更多数据，以实现强大的性能。这有动机侧重于侧重于改善NLU模型的少量学习性能。然而，缺乏少量射门的标准化评估基准，导致不同纸张中的不同实验设置。为了帮助加速这一工作的工作，我们介绍了线索（受限制的语言理解评估标准），这是评估NLU模型的几次拍摄学习功能的基准。我们证明，虽然最近的模型在获得大量标记数据时达到人类性能，但对于大多数任务，少量拍摄设置中的性能存在巨大差距。我们还展示了几个拍摄设置中替代模型家族和适应技术之间的差异。最后，我们讨论了在设计实验设置时讨论了评估真实少量学习绩效的实验设置，并提出了统一的标准化方法，以获得少量学习评估。我们的目标是鼓励对NLU模型的研究，可以概括为具有少数示例的新任务。线索的代码和数据可以在https://github.com/microsoft/clues提供。

最近，与“预训练，及时和预测”的新范式相比，与“预训练，微调”范式相比，新的范式“预训练，及时和预测”取得了显着的成就。在基于及时的GPT-3成功之后，一系列基于蒙版的语言模型（MLM）（例如Bert，Roberta）及时学习方法变得流行并广泛使用。但是，另一个有效的预训练的判别模型Electra可能被忽略了。在本文中，我们尝试使用拟议的替换代替令牌检测（RTD）基于基于的及时学习方法来完成零摄像的几个NLP任务。实验结果表明，基于RTD-Prompt学习的Electra模型可达到令人惊讶的最先进的零拍性能。在数字上，与MLM-Roberta-Large和MLM-Bert-Large相比，我们的RTD-Electra-Large在所有15个任务上平均提高了约8.4％和13.7％。特别是在SST-2任务上，我们的RTD-Electra-Large在没有任何培训数据的情况下达到了令人惊讶的90.1％精度。总体而言，与预先训练的蒙版语言模型相比，预先训练的代替令牌检测模型在零拍学习中的性能更好。因此，Electra是一位出色的零球学习者。源代码可在以下网址获得：https：//github.com/nishiwen1214/rtd-electra。

A long-running goal of the clinical NLP community is the extraction of important variables trapped in clinical notes. However, roadblocks have included dataset shift from the general domain and a lack of public clinical corpora and annotations. In this work, we show that large language models, such as InstructGPT, perform well at zero- and few-shot information extraction from clinical text despite not being trained specifically for the clinical domain. Whereas text classification and generation performance have already been studied extensively in such models, here we additionally demonstrate how to leverage them to tackle a diverse set of NLP tasks which require more structured outputs, including span identification, token-level sequence classification, and relation extraction. Further, due to the dearth of available data to evaluate these systems, we introduce new datasets for benchmarking few-shot clinical information extraction based on a manual re-annotation of the CASI dataset for new tasks. On the clinical extraction tasks we studied, the GPT-3 systems significantly outperform existing zero- and few-shot baselines.

大型预训练的语言模型（PLM）的最新进展导致了自然语言理解（NLU）任务的令人印象深刻的增长，并具有特定于任务的微调。但是，直接调整PLM在很大程度上依赖大量的标记实例，这些实例通常很难获得。迅速对PLM的调整已被证明对各种少数次任务很有价值。现有的作品研究基于迅速的NLU任务的基于及时的调整，主要集中于用语言器来得出正确的标签单词或生成及时的模板，以从PLM中启发语义。此外，还对常规数据增强方法进行了验证，可用于少量射击任务。但是，目前几乎没有针对基于及时的调整范式设计的数据增强方法。因此，我们研究了迅速的少数射击学习者的新数据增强问题。由于标签语义对于迅速的调整至关重要，因此我们提出了一种新颖的标签引导数据增强方法促进DA，该方法利用了丰富的标签语义信息以进行数据增强。很少的文本分类任务的广泛实验结果表明，我们提出的框架通过有效利用标签语义和数据扩展来实现自然语言理解来实现卓越的性能。

几乎没有命名的实体识别（NER）对于在有限的资源领域中标记的实体标记至关重要，因此近年来受到了适当的关注。现有的几声方法主要在域内设置下进行评估。相比之下，对于这些固有的忠实模型如何使用一些标记的域内示例在跨域NER中执行的方式知之甚少。本文提出了一种两步以理性为中心的数据增强方法，以提高模型的泛化能力。几个数据集中的结果表明，与先前的最新方法相比，我们的模型无形方法可显着提高跨域NER任务的性能，包括反事实数据增强和及时调用方法。我们的代码可在\ url {https://github.com/lifan-yuan/factmix}上获得。

Pre-trained Language Models (PLMs) have been applied in NLP tasks and achieve promising results. Nevertheless, the fine-tuning procedure needs labeled data of the target domain, making it difficult to learn in low-resource and non-trivial labeled scenarios. To address these challenges, we propose Prompt-based Text Entailment (PTE) for low-resource named entity recognition, which better leverages knowledge in the PLMs. We first reformulate named entity recognition as the text entailment task. The original sentence with entity type-specific prompts is fed into PLMs to get entailment scores for each candidate. The entity type with the top score is then selected as final label. Then, we inject tagging labels into prompts and treat words as basic units instead of n-gram spans to reduce time complexity in generating candidates by n-grams enumeration. Experimental results demonstrate that the proposed method PTE achieves competitive performance on the CoNLL03 dataset, and better than fine-tuned counterparts on the MIT Movie and Few-NERD dataset in low-resource settings.

The recent GPT-3 model (Brown et al., 2020) achieves remarkable few-shot performance solely by leveraging a natural-language prompt and a few task demonstrations as input context. Inspired by their findings, we study few-shot learning in a more practical scenario, where we use smaller language models for which fine-tuning is computationally efficient. We present LM-BFF-better few-shot fine-tuning of language models 1 -a suite of simple and complementary techniques for finetuning language models on a small number of annotated examples. Our approach includes (1) prompt-based fine-tuning together with a novel pipeline for automating prompt generation; and (2) a refined strategy for dynamically and selectively incorporating demonstrations into each context. Finally, we present a systematic evaluation for analyzing few-shot performance on a range of NLP tasks, including classification and regression. Our experiments demonstrate that our methods combine to dramatically outperform standard fine-tuning procedures in this low resource setting, achieving up to 30% absolute improvement, and 11% on average across all tasks. Our approach makes minimal assumptions on task resources and domain expertise, and hence constitutes a strong task-agnostic method for few-shot learning. 2 * The first two authors contributed equally. 1 Alternatively, language models' best friends forever. 2 Our implementation is publicly available at https:// github.com/princeton-nlp/LM-BFF.

通过自我监督的学习预先训练的大型语言模型在各种各样的任务上表现出令人印象深刻的零击功能。在这项工作中，我们介绍了Welm：一种针对中文的精心读取的预训练的语言模型，能够无缝执行不同类型的任务，以零或几次演示。 Welm通过“阅读”涵盖广泛主题的精选高质量语料库来接受10b参数的培训。我们表明，韦尔姆拥有有关各种领域和语言的广泛知识。在18个单语（中文）任务中，WELM可以大大优于现有的预训练模型，尺寸相似，并匹配高达25倍大的模型的性能。韦尔姆还表现出强大的多种语言和代码转换理解的能力，优于预先对30种语言进行预培训的现有多语言模型。此外，我们收集了人工编写的提示，并通过多次培训进行了大量的中文和微调韦尔姆的监督数据集。最终的模型可以实现对看不见的任务类型的强烈概括，并在零射门学习中优于无监督的韦尔姆。最后，我们证明韦尔姆具有解释和校准自己的决策的基本技能，这可能是未来研究的有希望的方向。我们的模型可以从https://welm.weixin.qq.com/docs/api/应用。

Controllable Text Generation (CTG) is emerging area in the field of natural language generation (NLG). It is regarded as crucial for the development of advanced text generation technologies that are more natural and better meet the specific constraints in practical applications. In recent years, methods using large-scale pre-trained language models (PLMs), in particular the widely used transformer-based PLMs, have become a new paradigm of NLG, allowing generation of more diverse and fluent text. However, due to the lower level of interpretability of deep neural networks, the controllability of these methods need to be guaranteed. To this end, controllable text generation using transformer-based PLMs has become a rapidly growing yet challenging new research hotspot. A diverse range of approaches have emerged in the recent 3-4 years, targeting different CTG tasks which may require different types of controlled constraints. In this paper, we present a systematic critical review on the common tasks, main approaches and evaluation methods in this area. Finally, we discuss the challenges that the field is facing, and put forward various promising future directions. To the best of our knowledge, this is the first survey paper to summarize CTG techniques from the perspective of PLMs. We hope it can help researchers in related fields to quickly track the academic frontier, providing them with a landscape of the area and a roadmap for future research.

随着系统变得更大，更复杂，从开源的收集网络威胁智能对于维持和实现高水平的安全性变得越来越重要。但是，这些开源通常会受到信息过载的约束。因此，应用机器学习模型将信息量凝结到必要的内容很有用。然而，以前的研究和应用表明，由于其概括能力低，现有的分类器无法提取有关新兴网络安全事件的特定信息。因此，我们建议通过为每个新事件培训新的分类器来克服这个问题的系统。由于这需要使用标准培训方法进行大量标记的数据，因此我们结合了三种不同的低数据制度技术 - 转移学习，数据增强和很少的学习学习 - 从很少的标记实例中培训高质量的分类器。我们使用从2021年的Microsoft Exchange Server数据泄露中得出的新型数据集评估了我们的方法，该数据集由三名专家标记。与标准训练方法相比，与标准训练方法相比，与标准训练方法相比，F1得分的增加超过21分，与几次学习中的最新方法相比，F1得分的增加超过18分。此外，经过此方法培训的分类器和32个实例的分类器仅比接受1800个实例的分类器少于5 F1分数。

GPT-3显示了培训的大规模语言模型（LMS）的卓越情调学习能力，培训数十亿规模数据。在这里，我们解决了GPT-3纸张报告的一些剩余问题，例如非英语LM，不同大小模型的性能，以及最近引入的迅速优化对上下文学习的效果。为实现这一目标，我们介绍了HyperClova，一个韩国VPT-3的韩国变体训练在一个以韩国为中心的560b标准的令牌。通过我们的韩国特定标记化，HyperClova与我们的培训配置增强，显示了韩国各种下游任务的最先进的上下游零射击和几秒钟学习表演。此外，我们展示了基于及时的学习的性能优势，并演示如何集成到迅速的工程管道中。然后，我们讨论了通过引入Hyperclova Studio，互动提示工程界面向ML的非专家提供AI原型设计能力来实现No Code AI范例的可能性。最后，我们展示了我们具有三个成功的内部应用程序的方法的潜力。

大型预先训练的语言模型已经显示了几次拍摄学习的承诺，只提供了几个任务特定示例给出了基于文本的任务。款式将很快解决到目前为止为人类研究助理保留的分类任务吗？现有的基准标记不设计用于衡量应用设置的进度，因此不要直接回答这个问题。 RAFT基准（现实世界注释的少量拍摄任务）侧重于自然发生的任务，并使用镜像部署的评估设置。 RAFT的基线评估揭示了当前技术斗争的地区：推理在许多班级的长篇文章和任务上。人类基线表明，非专家人类难以反映出一些分类任务，反映了现实世界的价值有时依赖于域名专业知识。甚至非专业人类基线F1分数超过GPT-3平均为0.11。 RAFT DataSets和排行榜将跟踪哪些模型改进在https://raft.elict.org中转化为现实世界的优势。

In this work, we present an evaluation of smaller BLOOM model variants (350m/560m and 1b3/1b7) on various natural language processing tasks. This includes GLUE - language understanding, prompt-based zero-shot and few-shot text classification and extraction, question answering, prompt-based text generation, and multi-lingual text classification to understand model strengths/weaknesses and behavior. Empirical results show that BLOOM variants under-perform on all GLUE tasks (except WNLI), question-answering, and text generation. The variants bloom for WNLI, with an accuracy of 56.3%, and for prompt-based few-shot text extraction on MIT Movies and ATIS datasets. The BLOOM variants on average have 7% greater accuracy over GPT-2 and GPT-Neo models on Director and Airline Name extraction from MIT Movies and ATIS datasets, respectively.

We present, Naamapadam, the largest publicly available Named Entity Recognition (NER) dataset for the 11 major Indian languages from two language families. In each language, it contains more than 400k sentences annotated with a total of at least 100k entities from three standard entity categories (Person, Location and Organization) for 9 out of the 11 languages. The training dataset has been automatically created from the Samanantar parallel corpus by projecting automatically tagged entities from an English sentence to the corresponding Indian language sentence. We also create manually annotated testsets for 8 languages containing approximately 1000 sentences per language. We demonstrate the utility of the obtained dataset on existing testsets and the Naamapadam-test data for 8 Indic languages. We also release IndicNER, a multilingual mBERT model fine-tuned on the Naamapadam training set. IndicNER achieves the best F1 on the Naamapadam-test set compared to an mBERT model fine-tuned on existing datasets. IndicNER achieves an F1 score of more than 80 for 7 out of 11 Indic languages. The dataset and models are available under open-source licenses at https://ai4bharat.iitm.ac.in/naamapadam.

Natural language prompts have been shown to facilitate cross-task generalization for large language models. However, with no or limited labeled examples, the cross-task performance is highly sensitive to the choice of prompts, while selecting a high-performing prompt is challenging given the scarcity of labels. To address the issue, we propose a Zero-Label Prompt Selection (ZPS) method that selects prompts without any labeled data or gradient update. Specifically, given the candidate human-written prompts for a task, ZPS labels a set of unlabeled data with a prompt ensemble and uses the pseudo-labels for prompt selection. Experiments show that ZPS improves over prior methods by a sizeable margin in zero-label performance. We also extend ZPS to a few-shot setting and show its advantages over strong baselines such as prompt tuning and model tuning.