本文介绍了我们对CRAC 2022关于多语言核心分辨率的共享任务的方法。我们的模型基于最新的端到端核心分辨率系统。除了加入多语言培训之外，我们还通过提及头部预测提高了结果。我们还试图将依赖性信息集成到我们的模型中。我们的系统最终以$ 3^{rd} $ place。此外，我们在13个数据集中达到了最佳性能。

本文概述了与CRAC 2022研讨会相关的多语言核心分辨率的共享任务。共同的任务参与者应该开发能够识别提及并根据身份核心重点聚集的训练系统。Corefud 1.0的公共版本包含10种语言的13个数据集，被用作培训和评估数据的来源。先前面向核心共享任务中使用的串联分数用作主要评估度量。5个参与团队提交了8个核心预测系统；此外，组织者在共享任务开始时提供了一个基于竞争变压器的基线系统。获胜者系统的表现优于基线12个百分点（就所有语言的所有数据集而言，在所有数据集中平均得分）。

我们描述了关于多语言核心分辨率的CRAC 2022共享任务的获胜提交。我们的系统首先求解了提及检测，然后使用先进的最大化方法在检索到的跨度上链接，并且这两个任务均与共享变压器的权重进行微调。我们报告了微调各种预审预告额的结果。此贡献的中心是微调的多语言模型。我们发现了一个具有足够大的编码器的大型多语言模型，可以全面提高所有数据集的性能，因此不仅限于代表性不足的语言或类型上相对语言的群体。源代码可在https://github.com/ufal/crac2022-corpipe上获得。

Coreference resolution (CR) is one of the most challenging areas of natural language processing. This task seeks to identify all textual references to the same real-world entity. Research in this field is divided into coreference resolution and anaphora resolution. Due to its application in textual comprehension and its utility in other tasks such as information extraction systems, document summarization, and machine translation, this field has attracted considerable interest. Consequently, it has a significant effect on the quality of these systems. This article reviews the existing corpora and evaluation metrics in this field. Then, an overview of the coreference algorithms, from rule-based methods to the latest deep learning techniques, is provided. Finally, coreference resolution and pronoun resolution systems in Persian are investigated.

与伯特（Bert）等语言模型相比，已证明知识增强语言表示的预培训模型在知识基础构建任务（即〜关系提取）中更有效。这些知识增强的语言模型将知识纳入预训练中，以生成实体或关系的表示。但是，现有方法通常用单独的嵌入表示每个实体。结果，这些方法难以代表播出的实体和大量参数，在其基础代币模型之上（即〜变压器），必须使用，并且可以处理的实体数量为由于内存限制，实践限制。此外，现有模型仍然难以同时代表实体和关系。为了解决这些问题，我们提出了一个新的预培训模型，该模型分别从图书中学习实体和关系的表示形式，并分别在文本中跨越跨度。通过使用SPAN模块有效地编码跨度，我们的模型可以代表实体及其关系，但所需的参数比现有模型更少。我们通过从Wikipedia中提取的知识图对我们的模型进行了预训练，并在广泛的监督和无监督的信息提取任务上进行了测试。结果表明，我们的模型比基线学习对实体和关系的表现更好，而在监督的设置中，微调我们的模型始终优于罗伯塔，并在信息提取任务上取得了竞争成果。

许多自然语言处理任务，例如核心解决方案和语义角色标签，都需要选择文本跨度并就其做出决定。此类任务的典型方法是为所有可能的跨度评分，并贪婪地选择特定任务的下游处理的跨度。然而，这种方法并未纳入有关应选择哪种跨度的诱导偏见，例如，选定的跨度倾向于是句法成分。在本文中，我们提出了一种新型的基于语法的结构化选择模型，该模型学会了利用为此类问题提供的部分跨度注释。与以前的方法相比，我们的方法摆脱了启发式贪婪的跨度选择方案，使我们能够在一组最佳跨度上对下游任务进行建模。我们在两个流行的跨度预测任务上评估我们的模型：核心分辨率和语义角色标签。我们对两者都展示了经验改进。

我们提出了文件的实体级关系联合模型。与其他方法形成鲜明对比 - 重点关注本地句子中的对，因此需要提及级别的注释 - 我们的模型在实体级别运行。为此，遵循多任务方法，它在Coreference分辨率上建立并通过多级别表示结合全局实体和本地提到信息来聚集相关信号。我们在积木数据集中实现最先进的关系提取结果，并报告了未来参考的第一个实体级端到端关系提取结果。最后，我们的实验结果表明，联合方法与特定于任务专用的学习相提并论，虽然由于共享参数和培训步骤而言更有效。

Transformer language models (TLMs) are critical for most NLP tasks, but they are difficult to create for low-resource languages because of how much pretraining data they require. In this work, we investigate two techniques for training monolingual TLMs in a low-resource setting: greatly reducing TLM size, and complementing the masked language modeling objective with two linguistically rich supervised tasks (part-of-speech tagging and dependency parsing). Results from 7 diverse languages indicate that our model, MicroBERT, is able to produce marked improvements in downstream task evaluations relative to a typical monolingual TLM pretraining approach. Specifically, we find that monolingual MicroBERT models achieve gains of up to 18% for parser LAS and 11% for NER F1 compared to a multilingual baseline, mBERT, while having less than 1% of its parameter count. We conclude reducing TLM parameter count and using labeled data for pretraining low-resource TLMs can yield large quality benefits and in some cases produce models that outperform multilingual approaches.

Identification of named entities from legal texts is an essential building block for developing other legal Artificial Intelligence applications. Named Entities in legal texts are slightly different and more fine-grained than commonly used named entities like Person, Organization, Location etc. In this paper, we introduce a new corpus of 46545 annotated legal named entities mapped to 14 legal entity types. The Baseline model for extracting legal named entities from judgment text is also developed.

随着数据集可用性的增加，从各种数据源的学习可能增加了。从多个数据源改进学习的一个特定方法是在训练期间嵌入数据源。这允许模型来学习概括的功能以及区分数据集之间的功能。但是，在自然语言处理领域引入了基于Contextualized变换器的嵌入之前，这些数据集嵌入物主要使用。在这项工作中，我们将两种方法与基于变换器的多语言依赖性解析器进行了比较，并执行了广泛的评估。我们展示：1）嵌入数据集仍然有益于这些模型2）在编码器级别3的数据集中嵌入数据集的性能增加最高），我们确认表现增加对于具有低基线分数的小型数据集和数据集的性能增加最高。 4）我们显示所有数据集的组合的培训类似地执行基于语言相关性的较小群集。

We adapt Lee et al.'s (2018) span-based entity coreference model to the task of end-to-end discourse deixis resolution in dialogue, specifically by proposing extensions to their model that exploit task-specific characteristics. The resulting model, dd-utt, achieves state-of-the-art results on the four datasets in the CODI-CRAC 2021 shared task.

Multiconer共享的任务旨在检测在多种语言的简短和低文本设置中，在语义上模棱两可且复杂的命名实体。缺乏上下文使人们对歧义的命名实体的认识充满挑战。为了减轻此问题，我们的团队Damo-NLP提出了一个基于知识的系统，我们在其中建立了基于Wikipedia的多语言知识基础，以向指定的实体识别（NER）模型提供相关的上下文信息。给定输入句子，我们的系统有效地从知识库中检索了相关上下文。然后，将原始输入句子加强此类上下文信息，从而可以捕获明显更好的上下文化令牌表示。我们的系统在Multiconer共享任务中赢得了13个曲目中的10个。

The BERT family of neural language models have become highly popular due to their ability to provide sequences of text with rich context-sensitive token encodings which are able to generalise well to many NLP tasks. We introduce gaBERT, a monolingual BERT model for the Irish language. We compare our gaBERT model to multilingual BERT and the monolingual Irish WikiBERT, and we show that gaBERT provides better representations for a downstream parsing task. We also show how different filtering criteria, vocabulary size and the choice of subword tokenisation model affect downstream performance. We compare the results of fine-tuning a gaBERT model with an mBERT model for the task of identifying verbal multiword expressions, and show that the fine-tuned gaBERT model also performs better at this task. We release gaBERT and related code to the community.

大型预用屏蔽语言模型已成为许多NLP问题的最先进的解决方案。虽然研究表明，单晶模型产生比多语言模型产生更好的结果，但训练数据集必须足够大。我们培训了立陶宛，拉脱维亚语和英语的三种语言Litlat Bert样模型，以及爱沙尼亚的单语Est-Roberta模型。我们在四个下游任务中评估它们的性能：命名实体识别，依赖解析，词语标记和单词类比。为了分析对单一语言的重要性以及大型培训集的重要性，我们将创建的模型与爱沙尼亚，拉脱维亚和立陶宛人进行了现有的单语和多语言伯特模型。结果表明，新创建的Litlat Bert和Est-Roberta模型在大多数情况下改善了所有测试任务的现有模型的结果。

现有的多方对话数据集用于核心分辨率是新生的，许多挑战仍然没有解决。我们根据电视成绩单为此任务创建了一个大规模数据集，多语言多方CoreF（MMC）。由于使用多种语言的黄金质量字幕可用，我们建议重复注释以通过注释投影以其他语言（中文和Farsi）创建银色核心数据。在黄金（英语）数据上，现成的模型在MMC上的性能相对较差，这表明MMC比以前的数据集更广泛地覆盖多方核心。在银数据上，我们发现成功使用它进行数据增强和从头开始训练，这有效地模拟了零击的跨语性设置。

We propose a transition-based approach that, by training a single model, can efficiently parse any input sentence with both constituent and dependency trees, supporting both continuous/projective and discontinuous/non-projective syntactic structures. To that end, we develop a Pointer Network architecture with two separate task-specific decoders and a common encoder, and follow a multitask learning strategy to jointly train them. The resulting quadratic system, not only becomes the first parser that can jointly produce both unrestricted constituent and dependency trees from a single model, but also proves that both syntactic formalisms can benefit from each other during training, achieving state-of-the-art accuracies in several widely-used benchmarks such as the continuous English and Chinese Penn Treebanks, as well as the discontinuous German NEGRA and TIGER datasets.

We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text. Our approach extends BERT by (1) masking contiguous random spans, rather than random tokens, and (2) training the span boundary representations to predict the entire content of the masked span, without relying on the individual token representations within it. Span-BERT consistently outperforms BERT and our better-tuned baselines, with substantial gains on span selection tasks such as question answering and coreference resolution. In particular, with the same training data and model size as BERT large , our single model obtains 94.6% and 88.7% F1 on SQuAD 1.1 and 2.0 respectively. We also achieve a new state of the art on the OntoNotes coreference resolution task (79.6% F1), strong performance on the TACRED relation extraction benchmark, and even gains on GLUE. 1 * Equal contribution. 1 Our code and pre-trained models are available at https://github.com/facebookresearch/ SpanBERT.

We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models (Peters et al., 2018a;Radford et al., 2018), BERT is designed to pretrain deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be finetuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial taskspecific architecture modifications.BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

我们通过纳入通用依赖性（UD）的句法特征来瞄准直接零射击设置中的跨语言机器阅读理解（MRC）的任务，以及我们使用的关键功能是每个句子中的语法关系。虽然以前的工作已经证明了有效的语法引导MRC模型，但我们建议采用句子际句法关系，除了基本的句子关系外，还可以进一步利用MRC任务的多句子输入中的句法依赖性。在我们的方法中，我们构建了句子间依赖图（ISDG）连接依赖树以形成横跨句子的全局句法关系。然后，我们提出了编码全局依赖关系图的ISDG编码器，通过明确地通过一个跳和多跳依赖性路径来解决句子间关系。三个多语言MRC数据集（XQUAD，MLQA，Tydiqa-Goldp）的实验表明，我们仅对英语培训的编码器能够在涵盖8种语言的所有14个测试集中提高零射性能，最高可达3.8 F1 / 5.2 EM平均改善，以及某些语言的5.2 F1 / 11.2 em。进一步的分析表明，改进可以归因于跨语言上一致的句法路径上的注意力。

Large pre-trained neural networks such as BERT have had great recent success in NLP, motivating a growing body of research investigating what aspects of language they are able to learn from unlabeled data. Most recent analysis has focused on model outputs (e.g., language model surprisal) or internal vector representations (e.g., probing classifiers). Complementary to these works, we propose methods for analyzing the attention mechanisms of pre-trained models and apply them to BERT. BERT's attention heads exhibit patterns such as attending to delimiter tokens, specific positional offsets, or broadly attending over the whole sentence, with heads in the same layer often exhibiting similar behaviors. We further show that certain attention heads correspond well to linguistic notions of syntax and coreference. For example, we find heads that attend to the direct objects of verbs, determiners of nouns, objects of prepositions, and coreferent mentions with remarkably high accuracy. Lastly, we propose an attention-based probing classifier and use it to further demonstrate that substantial syntactic information is captured in BERT's attention. 1 Code will be released at https://github.com/ clarkkev/attention-analysis.2 We use the English base-sized model.