Ultra-fine entity typing (UFET) predicts extremely free-formed types (e.g., president, politician) of a given entity mention (e.g., Joe Biden) in context. State-of-the-art (SOTA) methods use the cross-encoder (CE) based architecture. CE concatenates the mention (and its context) with each type and feeds the pairs into a pretrained language model (PLM) to score their relevance. It brings deeper interaction between mention and types to reach better performance but has to perform N (type set size) forward passes to infer types of a single mention. CE is therefore very slow in inference when the type set is large (e.g., N = 10k for UFET). To this end, we propose to perform entity typing in a recall-expand-filter manner. The recall and expand stages prune the large type set and generate K (K is typically less than 256) most relevant type candidates for each mention. At the filter stage, we use a novel model called MCCE to concurrently encode and score these K candidates in only one forward pass to obtain the final type prediction. We investigate different variants of MCCE and extensive experiments show that MCCE under our paradigm reaches SOTA performance on ultra-fine entity typing and is thousands of times faster than the cross-encoder. We also found MCCE is very effective in fine-grained (130 types) and coarse-grained (9 types) entity typing. Our code is available at \url{https://github.com/modelscope/AdaSeq/tree/master/examples/MCCE}.

Ultra-fine entity typing (UFET) aims to predict a wide range of type phrases that correctly describe the categories of a given entity mention in a sentence. Most recent works infer each entity type independently, ignoring the correlations between types, e.g., when an entity is inferred as a president, it should also be a politician and a leader. To this end, we use an undirected graphical model called pairwise conditional random field (PCRF) to formulate the UFET problem, in which the type variables are not only unarily influenced by the input but also pairwisely relate to all the other type variables. We use various modern backbones for entity typing to compute unary potentials, and derive pairwise potentials from type phrase representations that both capture prior semantic information and facilitate accelerated inference. We use mean-field variational inference for efficient type inference on very large type sets and unfold it as a neural network module to enable end-to-end training. Experiments on UFET show that the Neural-PCRF consistently outperforms its backbones with little cost and results in a competitive performance against cross-encoder based SOTA while being thousands of times faster. We also find Neural- PCRF effective on a widely used fine-grained entity typing dataset with a smaller type set. We pack Neural-PCRF as a network module that can be plugged onto multi-label type classifiers with ease and release it in https://github.com/modelscope/adaseq/tree/master/examples/NPCRF.

我们研究了很少的细粒实体键入（FET）的问题，其中只有几个带注释的实体对每种实体类型提供了上下文。最近，基于及时的调整通过将实体类型分类任务作为“填补空白”的问题来表明在几次射击方案中表现出优越的性能。这允许有效利用预训练的语言模型（PLM）的强语建模能力。尽管当前基于及时的调整方法成功了，但仍有两个主要挑战：（1）提示中的口头化器要么是由外部知识基础手动设计或构建的，而无需考虑目标语料库和标签层次结构信息，而且（2）当前方法主要利用PLM的表示能力，但没有通过广泛的通用域预训练来探索其产生的功率。在这项工作中，我们为由两个模块组成的几个弹药fet提出了一个新颖的框架：（1）实体类型标签解释模块自动学习将类型标签与词汇联系起来，通过共同利用几个播放实例和标签层次结构和标签层次结构，以及（2）基于类型的上下文化实例生成器根据给定实例生成新实例，以扩大培训集以更好地概括。在三个基准数据集上，我们的模型优于大量利润的现有方法。可以在https://github.com/teapot123/fine-graining-entity-typing上找到代码。

Many NLP tasks can be regarded as a selection problem from a set of options, such as classification tasks, multi-choice question answering, etc. Textual entailment (TE) has been shown as the state-of-the-art (SOTA) approach to dealing with those selection problems. TE treats input texts as premises (P), options as hypotheses (H), then handles the selection problem by modeling (P, H) pairwise. Two limitations: first, the pairwise modeling is unaware of other options, which is less intuitive since humans often determine the best options by comparing competing candidates; second, the inference process of pairwise TE is time-consuming, especially when the option space is large. To deal with the two issues, this work first proposes a contextualized TE model (Context-TE) by appending other k options as the context of the current (P, H) modeling. Context-TE is able to learn more reliable decision for the H since it considers various context. Second, we speed up Context-TE by coming up with Parallel-TE, which learns the decisions of multiple options simultaneously. Parallel-TE significantly improves the inference speed while keeping comparable performance with Context-TE. Our methods are evaluated on three tasks (ultra-fine entity typing, intent detection and multi-choice QA) that are typical selection problems with different sizes of options. Experiments show our models set new SOTA performance; particularly, Parallel-TE is faster than the pairwise TE by k times in inference. Our code is publicly available at https://github.com/jiangshdd/LearningToSelect.

我们介绍了精致的，这是一种有效的端到端实体链接模型，该模型使用精细的实体类型和实体描述来执行链接。该模型执行提及的检测，细粒实体键入以及单个向前传球中文档中所有提及的实体歧义，使其比现有方法快60倍以上。精制还超过了标准实体链接数据集的最先进性能，平均比3.7 F1。该模型能够将其推广到大规模的知识库，例如Wikidata（其实体是Wikipedia的15倍）和零拍的实体链接。速度，准确性和规模的结合使精制成为从网络规模数据集中提取实体的有效且具有成本效益的系统，该数据集已成功部署该模型。我们的代码和预培训模型可在https://github.com/alexa/refined上找到

We present Pre-trained Machine Reader (PMR), a novel method to retrofit Pre-trained Language Models (PLMs) into Machine Reading Comprehension (MRC) models without acquiring labeled data. PMR is capable of resolving the discrepancy between model pre-training and downstream fine-tuning of existing PLMs, and provides a unified solver for tackling various extraction tasks. To achieve this, we construct a large volume of general-purpose and high-quality MRC-style training data with the help of Wikipedia hyperlinks and design a Wiki Anchor Extraction task to guide the MRC-style pre-training process. Although conceptually simple, PMR is particularly effective in solving extraction tasks including Extractive Question Answering and Named Entity Recognition, where it shows tremendous improvements over previous approaches especially under low-resource settings. Moreover, viewing sequence classification task as a special case of extraction task in our MRC formulation, PMR is even capable to extract high-quality rationales to explain the classification process, providing more explainability of the predictions.

Neural language representation models such as BERT pre-trained on large-scale corpora can well capture rich semantic patterns from plain text, and be fine-tuned to consistently improve the performance of various NLP tasks. However, the existing pre-trained language models rarely consider incorporating knowledge graphs (KGs), which can provide rich structured knowledge facts for better language understanding. We argue that informative entities in KGs can enhance language representation with external knowledge. In this paper, we utilize both large-scale textual corpora and KGs to train an enhanced language representation model (ERNIE), which can take full advantage of lexical, syntactic, and knowledge information simultaneously. The experimental results have demonstrated that ERNIE achieves significant improvements on various knowledge-driven tasks, and meanwhile is comparable with the state-of-the-art model BERT on other common NLP tasks. The source code and experiment details of this paper can be obtained from https:// github.com/thunlp/ERNIE.

Long document retrieval aims to fetch query-relevant documents from a large-scale collection, where knowledge distillation has become de facto to improve a retriever by mimicking a heterogeneous yet powerful cross-encoder. However, in contrast to passages or sentences, retrieval on long documents suffers from the scope hypothesis that a long document may cover multiple topics. This maximizes their structure heterogeneity and poses a granular-mismatch issue, leading to an inferior distillation efficacy. In this work, we propose a new learning framework, fine-grained distillation (FGD), for long-document retrievers. While preserving the conventional dense retrieval paradigm, it first produces global-consistent representations crossing different fine granularity and then applies multi-granular aligned distillation merely during training. In experiments, we evaluate our framework on two long-document retrieval benchmarks, which show state-of-the-art performance.

预训练的语言模型（PLM）在自然语言理解中的许多下游任务中取得了显着的性能增长。已提出了各种中文PLM，以学习更好的中文表示。但是，大多数当前模型都使用中文字符作为输入，并且无法编码中文单词中包含的语义信息。虽然最近的预训练模型同时融合了单词和字符，但它们通常会遭受不足的语义互动，并且无法捕获单词和字符之间的语义关系。为了解决上述问题，我们提出了一个简单而有效的PLM小扣手，该小扣子采用了对单词和性格表示的对比度学习。特别是，Clower通过对多透明信息的对比学习将粗粒的信息（即单词）隐式编码为细粒度表示（即字符）。在现实的情况下，小电动器具有很大的价值，因为它可以轻松地将其纳入任何现有的基于细粒的PLM中而无需修改生产管道。在一系列下游任务上进行的扩展实验表明，小动物的卓越性能超过了几个最先进的实验 - 艺术基线。

跨度提取，旨在从纯文本中提取文本跨度（如单词或短语），是信息提取中的基本过程。最近的作品介绍了通过将跨度提取任务正式化为问题（QA正式化）的跨度提取任务来提高文本表示，以实现最先进的表现。然而，QA正规化并没有充分利用标签知识并遭受培训/推理的低效率。为了解决这些问题，我们介绍了一种新的范例来整合标签知识，并进一步提出一个小说模型，明确有效地将标签知识集成到文本表示中。具体而言，它独立地编码文本和标签注释，然后将标签知识集成到文本表示中，并使用精心设计的语义融合模块进行文本表示。我们在三个典型的跨度提取任务中进行广泛的实验：扁平的网，嵌套网和事件检测。实证结果表明，我们的方法在四个基准测试中实现了最先进的性能，而且分别将培训时间和推理时间降低76％和77％，与QA形式化范例相比。我们的代码和数据可在https://github.com/apkepers/lear中获得。

Relation extraction (RE), which has relied on structurally annotated corpora for model training, has been particularly challenging in low-resource scenarios and domains. Recent literature has tackled low-resource RE by self-supervised learning, where the solution involves pretraining the relation embedding by RE-based objective and finetuning on labeled data by classification-based objective. However, a critical challenge to this approach is the gap in objectives, which prevents the RE model from fully utilizing the knowledge in pretrained representations. In this paper, we aim at bridging the gap and propose to pretrain and finetune the RE model using consistent objectives of contrastive learning. Since in this kind of representation learning paradigm, one relation may easily form multiple clusters in the representation space, we further propose a multi-center contrastive loss that allows one relation to form multiple clusters to better align with pretraining. Experiments on two document-level RE datasets, BioRED and Re-DocRED, demonstrate the effectiveness of our method. Particularly, when using 1% end-task training data, our method outperforms PLM-based RE classifier by 10.5% and 5.8% on the two datasets, respectively.

事实证明，将先验知识纳入预训练的语言模型中对知识驱动的NLP任务有效，例如实体键入和关系提取。当前的培训程序通常通过使用知识掩盖，知识融合和知识更换将外部知识注入模型。但是，输入句子中包含的事实信息尚未完全开采，并且尚未严格检查注射的外部知识。结果，无法完全利用上下文信息，并将引入额外的噪音，或者注入的知识量受到限制。为了解决这些问题，我们提出了MLRIP，该MLRIP修改了Ernie-Baidu提出的知识掩盖策略，并引入了两阶段的实体替代策略。进行全面分析的广泛实验说明了MLRIP在军事知识驱动的NLP任务中基于BERT的模型的优势。

Automatic International Classification of Diseases (ICD) coding aims to assign multiple ICD codes to a medical note with an average of 3,000+ tokens. This task is challenging due to the high-dimensional space of multi-label assignment (155,000+ ICD code candidates) and the long-tail challenge - Many ICD codes are infrequently assigned yet infrequent ICD codes are important clinically. This study addresses the long-tail challenge by transforming this multi-label classification task into an autoregressive generation task. Specifically, we first introduce a novel pretraining objective to generate free text diagnoses and procedure using the SOAP structure, the medical logic physicians use for note documentation. Second, instead of directly predicting the high dimensional space of ICD codes, our model generates the lower dimension of text descriptions, which then infer ICD codes. Third, we designed a novel prompt template for multi-label classification. We evaluate our Generation with Prompt model with the benchmark of all code assignment (MIMIC-III-full) and few shot ICD code assignment evaluation benchmark (MIMIC-III-few). Experiments on MIMIC-III-few show that our model performs with a marco F1 30.2, which substantially outperforms the previous MIMIC-III-full SOTA model (marco F1 4.3) and the model specifically designed for few/zero shot setting (marco F1 18.7). Finally, we design a novel ensemble learner, a cross attention reranker with prompts, to integrate previous SOTA and our best few-shot coding predictions. Experiments on MIMIC-III-full show that our ensemble learner substantially improves both macro and micro F1, from 10.4 to 14.6 and from 58.2 to 59.1, respectively.

We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models (Peters et al., 2018a;Radford et al., 2018), BERT is designed to pretrain deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be finetuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial taskspecific architecture modifications.BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

该报告描述了一个预先训练的语言模型Erlangshen，其倾向校正损失是线索语义匹配挑战中的第一名。在预训练阶段，我们基于掩盖语言建模（MLM）的知识构建动态掩盖策略，并具有整个单词掩盖。此外，通过观察数据集的特定结构，预先训练的Erlangshen在微调阶段应用了经倾向校正的损失（PCL）。总体而言，我们在F1得分中获得72.54分，测试集的准确性为78.90分。我们的代码可在以下网址公开获取：https：//github.com/idea-ccnl/fengshenbang-lm/tree/hf-ds/fengshen/examples/clue_sim。

迅速的学习方法通​​过诱导更好的几次表现，在他们仍然遵循基于参数的学习范式的同时，引起了自然语言处理的波动。学习中的遗忘和死记硬背的记忆问题可能会遇到不稳定的概括问题。具体而言，香草及时的学习可能难以利用死记硬背的非典型实例，在完全监督的培训或过度贴身模式的情况下使用低射击数据。为了减轻此类局限性，我们以将知识从记忆中解耦的动机发展为有助于模型在概括和记忆之间取得平衡。与香草及时学习相反，重新启动构造了培训实例中的开放式知识店，并在输入，培训和推理过程中实现检索机制，从而使该模型能够从培训语料库中检索相关环境作为能力为提示增强。广泛的实验表明，Retroppt可以在几次射击和零拍设置中获得更好的性能。此外，我们进一步说明，我们提出的撤退可以通过新数据集获得更好的概括能力。对记忆的详细分析确实显示逆转可以减少语言模型对记忆的依赖；因此，改善下游任务的概括。

Pre-trained Language Models (PLMs) have been applied in NLP tasks and achieve promising results. Nevertheless, the fine-tuning procedure needs labeled data of the target domain, making it difficult to learn in low-resource and non-trivial labeled scenarios. To address these challenges, we propose Prompt-based Text Entailment (PTE) for low-resource named entity recognition, which better leverages knowledge in the PLMs. We first reformulate named entity recognition as the text entailment task. The original sentence with entity type-specific prompts is fed into PLMs to get entailment scores for each candidate. The entity type with the top score is then selected as final label. Then, we inject tagging labels into prompts and treat words as basic units instead of n-gram spans to reduce time complexity in generating candidates by n-grams enumeration. Experimental results demonstrate that the proposed method PTE achieves competitive performance on the CoNLL03 dataset, and better than fine-tuned counterparts on the MIT Movie and Few-NERD dataset in low-resource settings.

近年来，基于变压器的预训练模型已获得了很大的进步，成为自然语言处理中最重要的骨干之一。最近的工作表明，变压器内部的注意力机制可能不需要，卷积神经网络和基于多层感知器的模型也已被研究为变压器替代方案。在本文中，我们考虑了一个用于语言模型预训练的图形循环网络，该网络通过本地令牌级通信为每个序列构建一个图形结构，以及与其他代币解耦的句子级表示。原始模型在受监督培训下的特定领域特定文本分类中表现良好，但是，其通过自我监督的方式学习转移知识的潜力尚未得到充分利用。我们通过优化体系结构并验证其在更通用的语言理解任务（英语和中文）中的有效性来填补这一空白。至于模型效率，我们的模型在基于变压器的模型中而不是二次复杂性，而是具有线性复杂性，并且在推断过程中的性能更有效。此外，我们发现与现有基于注意力的模型相比，我们的模型可以生成更多样化的输出，而背景化的功能冗余性较小。

链接的语音实体旨在识别和消除语言中的命名实体。常规方法严重遭受了不受限制的语音样式和ASR系统产生的嘈杂笔录。在本文中，我们提出了一种名为“知识增强命名实体识别”（KENER）的新颖方法，该方法致力于通过在实体识别阶段无痛地纳入适当的知识来改善鲁棒性，从而改善实体联系的整体性能。肯纳（Kener）首先检索未提及的句子的候选实体，然后利用实体描述作为额外的信息来帮助识别提及。当输入短或嘈杂时，由密集检索模块检索的候选实体特别有用。此外，我们研究了各种数据采样策略和设计有效的损失功能，以提高识别和歧义阶段中检索实体的质量。最后，将与过滤模块的链接作为最终保障措施应用，从而可以过滤出错误认可的提及。我们的系统在NLPCC-2022共享任务2的轨道1中获得第一名，并在轨道1中获得第一名。

大型预训练的语言模型（PLM）的最新进展导致了自然语言理解（NLU）任务的令人印象深刻的增长，并具有特定于任务的微调。但是，直接调整PLM在很大程度上依赖大量的标记实例，这些实例通常很难获得。迅速对PLM的调整已被证明对各种少数次任务很有价值。现有的作品研究基于迅速的NLU任务的基于及时的调整，主要集中于用语言器来得出正确的标签单词或生成及时的模板，以从PLM中启发语义。此外，还对常规数据增强方法进行了验证，可用于少量射击任务。但是，目前几乎没有针对基于及时的调整范式设计的数据增强方法。因此，我们研究了迅速的少数射击学习者的新数据增强问题。由于标签语义对于迅速的调整至关重要，因此我们提出了一种新颖的标签引导数据增强方法促进DA，该方法利用了丰富的标签语义信息以进行数据增强。很少的文本分类任务的广泛实验结果表明，我们提出的框架通过有效利用标签语义和数据扩展来实现自然语言理解来实现卓越的性能。