In order to achieve deep natural language understanding, syntactic constituent parsing is a vital step, highly demanded by many artificial intelligence systems to process both text and speech. One of the most recent proposals is the use of standard sequence-to-sequence models to perform constituent parsing as a machine translation task, instead of applying task-specific parsers. While they show a competitive performance, these text-to-parse transducers are still lagging behind classic techniques in terms of accuracy, coverage and speed. To close the gap, we here extend the framework of sequence-to-sequence models for constituent parsing, not only by providing a more powerful neural architecture for improving their performance, but also by enlarging their coverage to handle the most complex syntactic phenomena: discontinuous structures. To that end, we design several novel linearizations that can fully produce discontinuities and, for the first time, we test a sequence-to-sequence model on the main discontinuous benchmarks, obtaining competitive results on par with task-specific discontinuous constituent parsers and achieving state-of-the-art scores on the (discontinuous) English Penn Treebank.

We propose a transition-based approach that, by training a single model, can efficiently parse any input sentence with both constituent and dependency trees, supporting both continuous/projective and discontinuous/non-projective syntactic structures. To that end, we develop a Pointer Network architecture with two separate task-specific decoders and a common encoder, and follow a multitask learning strategy to jointly train them. The resulting quadratic system, not only becomes the first parser that can jointly produce both unrestricted constituent and dependency trees from a single model, but also proves that both syntactic formalisms can benefit from each other during training, achieving state-of-the-art accuracies in several widely-used benchmarks such as the continuous English and Chinese Penn Treebanks, as well as the discontinuous German NEGRA and TIGER datasets.

在本文中，我们试图通过引入深度学习模型的句法归纳偏见来建立两所学校之间的联系。我们提出了两个归纳偏见的家族，一个家庭用于选区结构，另一个用于依赖性结构。选区归纳偏见鼓励深度学习模型使用不同的单位（或神经元）分别处理长期和短期信息。这种分离为深度学习模型提供了一种方法，可以从顺序输入中构建潜在的层次表示形式，即更高级别的表示由高级表示形式组成，并且可以分解为一系列低级表示。例如，在不了解地面实际结构的情况下，我们提出的模型学会通过根据其句法结构组成变量和运算符的表示来处理逻辑表达。另一方面，依赖归纳偏置鼓励模型在输入序列中找到实体之间的潜在关系。对于自然语言，潜在关系通常被建模为一个定向依赖图，其中一个单词恰好具有一个父节点和零或几个孩子的节点。将此约束应用于类似变压器的模型之后，我们发现该模型能够诱导接近人类专家注释的有向图，并且在不同任务上也优于标准变压器模型。我们认为，这些实验结果为深度学习模型的未来发展展示了一个有趣的选择。

理解和生成语言的能力使人类认知与其他已知的生命形式不同。我们研究一种在语义解析的任务中，将两种最成功的途径（统计语言模型和象征性语义形式主义）梳理到语言的意义。我们基于基于过渡的抽象表示（AMR）解析器，AMREAGER，我们探索了融合预验证的上下文感知的单词嵌入的实用性 - 例如Bert和Roberta，在AMR解析的问题中，我们为新的解析器做出了贡献。 Dub作为Amrberger。实验发现，与非上下文对应物相比，这些丰富的词汇特征对改善解析器的总体表现并不特别有助于改善解析器的整体性能，而其他概念信息则赋予了系统以优于基准的能力。通过病变研究，我们发现上下文嵌入的使用有助于使系统更强大，以消除显式句法特征。这些发现揭示了上下文嵌入的优势和劣势，并以当前形式揭示了语言模型，并激发了更深入的理解。

Transformer-based models have pushed state of the art in many areas of NLP, but our understanding of what is behind their success is still limited. This paper is the first survey of over 150 studies of the popular BERT model. We review the current state of knowledge about how BERT works, what kind of information it learns and how it is represented, common modifications to its training objectives and architecture, the overparameterization issue and approaches to compression. We then outline directions for future research.

The word alignment task, despite its prominence in the era of statistical machine translation (SMT), is niche and under-explored today. In this two-part tutorial, we argue for the continued relevance for word alignment. The first part provides a historical background to word alignment as a core component of the traditional SMT pipeline. We zero-in on GIZA++, an unsupervised, statistical word aligner with surprising longevity. Jumping forward to the era of neural machine translation (NMT), we show how insights from word alignment inspired the attention mechanism fundamental to present-day NMT. The second part shifts to a survey approach. We cover neural word aligners, showing the slow but steady progress towards surpassing GIZA++ performance. Finally, we cover the present-day applications of word alignment, from cross-lingual annotation projection, to improving translation.

数据饥饿的深度神经网络已经将自己作为许多NLP任务的标准建立为包括传统序列标记的标准。尽管他们在高资源语言上表现最先进的表现，但它们仍然落后于低资源场景的统计计数器。一个方法来反击攻击此问题是文本增强，即，从现有数据生成新的合成训练数据点。虽然NLP最近目睹了一种文本增强技术的负载，但该领域仍然缺乏对多种语言和序列标记任务的系统性能分析。为了填补这一差距，我们调查了三类文本增强方法，其在语法（例如，裁剪子句子），令牌（例如，随机字插入）和字符（例如，字符交换）级别上执行更改。我们系统地将它们与语音标记，依赖解析和语义角色标记的分组进行了比较，用于使用各种模型的各种语言系列，包括依赖于诸如MBERT的普赖金的多语言语境化语言模型的架构。增强最显着改善了解析，然后是语音标记和语义角色标记的依赖性解析。我们发现实验技术通常在形态上丰富的语言，而不是越南语等分析语言。我们的研究结果表明，增强技术可以进一步改善基于MBERT的强基线。我们将字符级方法标识为最常见的表演者，而同义词替换和语法增强仪提供不一致的改进。最后，我们讨论了最大依赖于任务，语言对和模型类型的结果。

Transfer learning, where a model is first pre-trained on a data-rich task before being finetuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new "Colossal Clean Crawled Corpus", we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.

The rapid advancement of AI technology has made text generation tools like GPT-3 and ChatGPT increasingly accessible, scalable, and effective. This can pose serious threat to the credibility of various forms of media if these technologies are used for plagiarism, including scientific literature and news sources. Despite the development of automated methods for paraphrase identification, detecting this type of plagiarism remains a challenge due to the disparate nature of the datasets on which these methods are trained. In this study, we review traditional and current approaches to paraphrase identification and propose a refined typology of paraphrases. We also investigate how this typology is represented in popular datasets and how under-representation of certain types of paraphrases impacts detection capabilities. Finally, we outline new directions for future research and datasets in the pursuit of more effective paraphrase detection using AI.

We propose a unified neural network architecture and learning algorithm that can be applied to various natural language processing tasks including: part-of-speech tagging, chunking, named entity recognition, and semantic role labeling. This versatility is achieved by trying to avoid task-specific engineering and therefore disregarding a lot of prior knowledge. Instead of exploiting man-made input features carefully optimized for each task, our system learns internal representations on the basis of vast amounts of mostly unlabeled training data. This work is then used as a basis for building a freely available tagging system with good performance and minimal computational requirements.

我用Hunglish2语料库训练神经电脑翻译任务的模型。这项工作的主要贡献在培训NMT模型期间评估不同的数据增强方法。我提出了5种不同的增强方法，这些方法是结构感知的，这意味着而不是随机选择用于消隐或替换的单词，句子的依赖树用作增强的基础。我首先关于神经网络的详细文献综述，顺序建模，神经机翻译，依赖解析和数据增强。经过详细的探索性数据分析和Hunglish2语料库的预处理之后，我使用所提出的数据增强技术进行实验。匈牙利语的最佳型号达到了33.9的BLEU得分，而英国匈牙利最好的模型达到了28.6的BLEU得分。

We introduce Transformer Grammars (TGs), a novel class of Transformer language models that combine (i) the expressive power, scalability, and strong performance of Transformers and (ii) recursive syntactic compositions, which here are implemented through a special attention mask and deterministic transformation of the linearized tree. We find that TGs outperform various strong baselines on sentence-level language modeling perplexity, as well as on multiple syntax-sensitive language modeling evaluation metrics. Additionally, we find that the recursive syntactic composition bottleneck which represents each sentence as a single vector harms perplexity on document-level language modeling, providing evidence that a different kind of memory mechanism -- one that is independent of composed syntactic representations -- plays an important role in current successful models of long text.

基于变压器的语言模型最近在许多自然语言任务中取得了显着的结果。但是，通常通过利用大量培训数据来实现排行榜的性能，并且很少通过将明确的语言知识编码为神经模型。这使许多人质疑语言学对现代自然语言处理的相关性。在本文中，我介绍了几个案例研究，以说明理论语言学和神经语言模型仍然相互关联。首先，语言模型通过提供一个客观的工具来测量语义距离，这对语言学家很有用，语义距离很难使用传统方法。另一方面，语言理论通过提供框架和数据源来探究我们的语言模型，以了解语言理解的特定方面，从而有助于语言建模研究。本论文贡献了三项研究，探讨了语言模型中语法 - 听觉界面的不同方面。在论文的第一部分中，我将语言模型应用于单词类灵活性的问题。我将Mbert作为语义距离测量的来源，我提供了有利于将单词类灵活性分析为方向过程的证据。在论文的第二部分中，我提出了一种方法来测量语言模型中间层的惊奇方法。我的实验表明，包含形态句法异常的句子触发了语言模型早期的惊喜，而不是语义和常识异常。最后，在论文的第三部分中，我适应了一些心理语言学研究，以表明语言模型包含了论证结构结构的知识。总而言之，我的论文在自然语言处理，语言理论和心理语言学之间建立了新的联系，以为语言模型的解释提供新的观点。

State-of-the-art named entity recognition systems rely heavily on hand-crafted features and domain-specific knowledge in order to learn effectively from the small, supervised training corpora that are available. In this paper, we introduce two new neural architectures-one based on bidirectional LSTMs and conditional random fields, and the other that constructs and labels segments using a transition-based approach inspired by shift-reduce parsers. Our models rely on two sources of information about words: character-based word representations learned from the supervised corpus and unsupervised word representations learned from unannotated corpora. Our models obtain state-of-the-art performance in NER in four languages without resorting to any language-specific knowledge or resources such as gazetteers. 1

AMR到文本是NLP社区中旨在从抽象含义表示（AMR）图生成句子的关键技术之一。自2013年提出AMR以来，有关AMR到文本的研究越来越普遍，因为AMR作为自然语言的高级语义描述，由于AMR具有独特的优势，因此作为结构化数据的重要分支变得越来越普遍。在本文中，我们简要介绍了AMR到文本。首先，我们介绍了此技术的当前情况，并指出了它的困难。其次，根据先前研究中使用的方法，我们根据它们各自的机制将它们大致分为五个类别和预先训练的语言模型（PLM）。特别是，我们详细介绍了基于神经网络的方法，并介绍了AMR到文本的最新进展，该方法指的是AMR重建，解码器优化等。此外，我们介绍了AMR-TOXT的基准和评估方法。最终，我们提供了当前技术和未来研究的前景的摘要。

Deep Learning and Machine Learning based models have become extremely popular in text processing and information retrieval. However, the non-linear structures present inside the networks make these models largely inscrutable. A significant body of research has focused on increasing the transparency of these models. This article provides a broad overview of research on the explainability and interpretability of natural language processing and information retrieval methods. More specifically, we survey approaches that have been applied to explain word embeddings, sequence modeling, attention modules, transformers, BERT, and document ranking. The concluding section suggests some possible directions for future research on this topic.

Recent progress in pre-trained neural language models has significantly improved the performance of many natural language processing (NLP) tasks. In this paper we propose a new model architecture DeBERTa (Decoding-enhanced BERT with disentangled attention) that improves the BERT and RoBERTa models using two novel techniques. The first is the disentangled attention mechanism, where each word is represented using two vectors that encode its content and position, respectively, and the attention weights among words are computed using disentangled matrices on their contents and relative positions, respectively. Second, an enhanced mask decoder is used to incorporate absolute positions in the decoding layer to predict the masked tokens in model pre-training. In addition, a new virtual adversarial training method is used for fine-tuning to improve models' generalization. We show that these techniques significantly improve the efficiency of model pre-training and the performance of both natural language understand (NLU) and natural langauge generation (NLG) downstream tasks. Compared to RoBERTa-Large, a DeBERTa model trained on half of the training data performs consistently better on a wide range of NLP tasks, achieving improvements on MNLI by +0.9% (90.2% vs. 91.1%), on SQuAD v2.0 by +2.3% (88.4% vs. 90.7%) and RACE by +3.6% (83.2% vs. 86.8%). Notably, we scale up DeBERTa by training a larger version that consists of 48 Transform layers with 1.5 billion parameters. The significant performance boost makes the single DeBERTa model surpass the human performance on the SuperGLUE benchmark (Wang et al., 2019a) for the first time in terms of macro-average score (89.9 versus 89.8), and the ensemble DeBERTa model sits atop the SuperGLUE leaderboard as of January 6, 2021, outperforming the human baseline by a decent margin (90.3 versus 89.8). The pre-trained DeBERTa models and the source code were released at: https://github.com/microsoft/DeBERTa 1 .

目前，用于训练语言模型的最广泛的神经网络架构是所谓的BERT，导致各种自然语言处理（NLP）任务的改进。通常，BERT模型中的参数的数量越大，这些NLP任务中获得的结果越好。不幸的是，内存消耗和训练持续时间随着这些模型的大小而大大增加。在本文中，我们调查了较小的BERT模型的各种训练技术：我们将不同的方法与Albert，Roberta和相对位置编码等其他BERT变体相结合。此外，我们提出了两个新的微调修改，导致更好的性能：类开始终端标记和修改形式的线性链条条件随机字段。此外，我们介绍了整个词的注意力，从而降低了伯特存储器的使用，并导致性能的小幅增加，与古典的多重关注相比。我们评估了这些技术的五个公共德国命名实体识别（NER）任务，其中两条由这篇文章引入了两项任务。

使用诸如BERT，ELMO和FLAIR等模型建模上下文信息的成立具有显着改善了文字的表示学习。它还给出了几乎每个NLP任务机器翻译，文本摘要和命名实体识别的Sota结果，以命名为少。在这项工作中，除了使用这些主导的上下文感知的表示之外，我们还提出了一种用于命名实体识别（NER）的知识意识表示学习（KARL）网络。我们讨论了利用现有方法在纳入世界知识方面的挑战，并展示了如何利用我们所提出的方法来克服这些挑战。 KARL基于变压器编码器，该变压器编码器利用表示为事实三元组的大知识库，将它们转换为图形上下文，并提取驻留在内部的基本实体信息以生成用于特征增强的上下文化三联表示。实验结果表明，使用卡尔的增强可以大大提升我们的内部系统的性能，并在三个公共网络数据集中的文献中的现有方法，即Conll 2003，Conll ++和Ontonotes V5实现了比文献中现有方法的显着更好的结果。我们还观察到更好的概括和应用于从Karl上看不见的实体的真实环境。

诸如变形金刚和LSTMS之类的流行模型将令牌用作其信息单位。也就是说，每个令牌都被编码为向量表示，这些向量直接在计算中使用。但是，人类经常考虑跨令牌（即短语）而不是其组成代币。在本文中，我们介绍了TreeFormer，这是一个受CKY算法和变压器启发的体系结构，该体系结构学习了组成操作员和汇总功能，以构建针对短语和句子的层次编码。我们的广泛实验证明了将层次结构纳入变压器的好处，并且与机器翻译，抽象性摘要和各种自然语言理解任务相比，与基线变压器相比显示出重大改进。