预训练的语言模型的目的是学习文本数据的上下文表示。预训练的语言模型已成为自然语言处理和代码建模的主流。使用探针，一种研究隐藏矢量空间的语言特性的技术，以前的作品表明，这些预训练的语言模型在其隐藏表示中编码简单的语言特性。但是，以前的工作都没有评估这些模型是否编码编程语言的整个语法结构。在本文中，我们证明了\ textit {句法子空间}的存在，该{语法子空间}位于预训练的语言模型的隐藏表示中，其中包含编程语言的句法信息。我们表明，可以从模型的表示形式中提取此子空间，并定义一种新颖的探测方法AST-Probe，该方法可以恢复输入代码段的整个抽象语法树（AST）。在我们的实验中，我们表明这种句法子空间存在于五个最先进的预训练的语言模型中。此外，我们强调说，模型的中间层是编码大多数AST信息的模型。最后，我们估计该句法子空间的最佳大小，并表明其尺寸大大低于模型的表示空间。这表明，预训练的语言模型使用其表示空间的一小部分来编码编程语言的句法信息。

在NLP社区中有一个正在进行的辩论，无论现代语言模型是否包含语言知识，通过所谓的探针恢复。在本文中，我们研究了语言知识是否是现代语言模型良好表现的必要条件，我们称之为\ Texit {重新发现假设}。首先，我们展示了语言模型，这是显着压缩的，但在预先磨普目标上表现良好，以便在语言结构探讨时保持良好的分数。这一结果支持重新发现的假设，并导致我们的论文的第二款贡献：一个信息 - 理论框架，与语言建模目标相关。该框架还提供了测量语言信息对字词预测任务的影响的度量标准。我们通过英语综合和真正的NLP任务加固我们的分析结果。

Pre-trained language models for programming languages have shown a powerful ability on processing many Software Engineering (SE) tasks, e.g., program synthesis, code completion, and code search. However, it remains to be seen what is behind their success. Recent studies have examined how pre-trained models can effectively learn syntax information based on Abstract Syntax Trees. In this paper, we figure out what role the self-attention mechanism plays in understanding code syntax and semantics based on AST and static analysis. We focus on a well-known representative code model, CodeBERT, and study how it can learn code syntax and semantics by the self-attention mechanism and Masked Language Modelling (MLM) at the token level. We propose a group of probing tasks to analyze CodeBERT. Based on AST and static analysis, we establish the relationships among the code tokens. First, Our results show that CodeBERT can acquire syntax and semantics knowledge through self-attention and MLM. Second, we demonstrate that the self-attention mechanism pays more attention to dependence-relationship tokens than to other tokens. Different attention heads play different roles in learning code semantics; we show that some of them are weak at encoding code semantics. Different layers have different competencies to represent different code properties. Deep CodeBERT layers can encode the semantic information that requires some complex inference in the code context. More importantly, we show that our analysis is helpful and leverage our conclusions to improve CodeBERT. We show an alternative approach for pre-training models, which makes fully use of the current pre-training strategy, i.e, MLM, to learn code syntax and semantics, instead of combining features from different code data formats, e.g., data-flow, running-time states, and program outputs.

Transformer-based models have pushed state of the art in many areas of NLP, but our understanding of what is behind their success is still limited. This paper is the first survey of over 150 studies of the popular BERT model. We review the current state of knowledge about how BERT works, what kind of information it learns and how it is represented, common modifications to its training objectives and architecture, the overparameterization issue and approaches to compression. We then outline directions for future research.

基于变压器的语言模型最近在许多自然语言任务中取得了显着的结果。但是，通常通过利用大量培训数据来实现排行榜的性能，并且很少通过将明确的语言知识编码为神经模型。这使许多人质疑语言学对现代自然语言处理的相关性。在本文中，我介绍了几个案例研究，以说明理论语言学和神经语言模型仍然相互关联。首先，语言模型通过提供一个客观的工具来测量语义距离，这对语言学家很有用，语义距离很难使用传统方法。另一方面，语言理论通过提供框架和数据源来探究我们的语言模型，以了解语言理解的特定方面，从而有助于语言建模研究。本论文贡献了三项研究，探讨了语言模型中语法 - 听觉界面的不同方面。在论文的第一部分中，我将语言模型应用于单词类灵活性的问题。我将Mbert作为语义距离测量的来源，我提供了有利于将单词类灵活性分析为方向过程的证据。在论文的第二部分中，我提出了一种方法来测量语言模型中间层的惊奇方法。我的实验表明，包含形态句法异常的句子触发了语言模型早期的惊喜，而不是语义和常识异常。最后，在论文的第三部分中，我适应了一些心理语言学研究，以表明语言模型包含了论证结构结构的知识。总而言之，我的论文在自然语言处理，语言理论和心理语言学之间建立了新的联系，以为语言模型的解释提供新的观点。

Large language models (LLMs) have been reported to have strong performance on natural language processing tasks. However, performance metrics such as accuracy do not measure the quality of the model in terms of its ability to robustly represent complex linguistic structure. In this work, we propose a framework to evaluate the robustness of linguistic representations using probing tasks. We leverage recent advances in extracting emergent linguistic constructs from LLMs and apply syntax-preserving perturbations to test the stability of these constructs in order to better understand the representations learned by LLMs. Empirically, we study the performance of four LLMs across six different corpora on the proposed robustness measures. We provide evidence that context-free representation (e.g., GloVe) are in some cases competitive with context-dependent representations from modern LLMs (e.g., BERT), yet equally brittle to syntax-preserving manipulations. Emergent syntactic representations in neural networks are brittle, thus our work poses the attention on the risk of comparing such structures to those that are object of a long lasting debate in linguistics.

In this paper, we seek to measure how much information a component in a neural network could extract from the representations fed into it. Our work stands in contrast to prior probing work, most of which investigates how much information a model's representations contain. This shift in perspective leads us to propose a new principle for probing, the architectural bottleneck principle: In order to estimate how much information a given component could extract, a probe should look exactly like the component. Relying on this principle, we estimate how much syntactic information is available to transformers through our attentional probe, a probe that exactly resembles a transformer's self-attention head. Experimentally, we find that, in three models (BERT, ALBERT, and RoBERTa), a sentence's syntax tree is mostly extractable by our probe, suggesting these models have access to syntactic information while composing their contextual representations. Whether this information is actually used by these models, however, remains an open question.

Deep Learning and Machine Learning based models have become extremely popular in text processing and information retrieval. However, the non-linear structures present inside the networks make these models largely inscrutable. A significant body of research has focused on increasing the transparency of these models. This article provides a broad overview of research on the explainability and interpretability of natural language processing and information retrieval methods. More specifically, we survey approaches that have been applied to explain word embeddings, sequence modeling, attention modules, transformers, BERT, and document ranking. The concluding section suggests some possible directions for future research on this topic.

预先接受训练的语言模型的进展导致了对自然语言理解的下游任务的令人印象深刻的结果。探索预先训练的语言模型的最新工作揭示了在其上下围化表示中编码的广泛的语言属性。然而，目前尚不清楚他们是否编码对符号推理方法至关重要的语义知识。我们提出了一种用于探测预先接受训练的语言模型表示的逻辑推断的语言信息的方法。我们的探测数据集涵盖主要符号推理系统所需的语言现象列表。我们发现（i）预先接受的语言模型为推断编码几种类型的语言信息，但是还有一些类型的信息弱编码，（ii）语言模型可以通过微调有效地学习语言信息缺少语言信息。总体而言，我们的调查结果提供了逻辑推理语言模型的语言信息的洞察力，以及他们的预训练程序捕获。此外，我们已经证明了语言模型作为语义和背景知识库的潜力，用于支持符号推断方法。

Contextual word representations derived from large-scale neural language models are successful across a diverse set of NLP tasks, suggesting that they encode useful and transferable features of language. To shed light on the linguistic knowledge they capture, we study the representations produced by several recent pretrained contextualizers (variants of ELMo, the OpenAI transformer language model, and BERT) with a suite of seventeen diverse probing tasks. We find that linear models trained on top of frozen contextual representations are competitive with state-of-the-art task-specific models in many cases, but fail on tasks requiring fine-grained linguistic knowledge (e.g., conjunct identification). To investigate the transferability of contextual word representations, we quantify differences in the transferability of individual layers within contextualizers, especially between recurrent neural networks (RNNs) and transformers. For instance, higher layers of RNNs are more taskspecific, while transformer layers do not exhibit the same monotonic trend. In addition, to better understand what makes contextual word representations transferable, we compare language model pretraining with eleven supervised pretraining tasks. For any given task, pretraining on a closely related task yields better performance than language model pretraining (which is better on average) when the pretraining dataset is fixed. However, language model pretraining on more data gives the best results.

在源代码处理的领域中，基于变压器的表示模型表现出强大的功能，并在许多任务中都实现了最先进的（SOTA）性能。尽管变压器模型处理了顺序源代码，但证据表明，它们也可以捕获结构信息（\ eg，在语法树，数据流，控制流，\等）。我们提出了汇总的注意力评分，这是一种研究变压器学到的结构信息的方法。我们还提出了汇总的注意图，这是一种从预训练模型中提取程序图的新方法。我们从多个角度测量我们的方法。此外，根据我们的经验发现，我们使用自动提取的图形来替换可变滥用任务中那些巧妙的手动设计图。实验结果表明，我们自动提取的语义图非常有意义且有效，这为我们提供了一个新的观点，可以理解和使用模型中包含的信息。

探测是一种流行的方法，可以辨别预先训练的语言模型表示中包含哪些语言信息。但是，选择探针模型的机制最近受到了激烈的争论，因为尚不清楚探针是否只是在提取信息或对语言属性进行建模。为了应对这一挑战，本文通过将探测作为提示任务提出探测来介绍一种新颖的探测方法。我们对五个探测任务进行实验，并表明我们的方法在提取信息方面比诊断探针更为可比或更好，而自行学习得更少。我们通过提示方法与注意力头修剪进一步结合探测，以分析模型将语言信息存储在其体系结构中的位置。然后，我们通过删除对该属性至关重要的头部并评估所得模型在语言建模上的性能来检查特定语言属性对预训练的有用性。

在本文中，我们试图通过引入深度学习模型的句法归纳偏见来建立两所学校之间的联系。我们提出了两个归纳偏见的家族，一个家庭用于选区结构，另一个用于依赖性结构。选区归纳偏见鼓励深度学习模型使用不同的单位（或神经元）分别处理长期和短期信息。这种分离为深度学习模型提供了一种方法，可以从顺序输入中构建潜在的层次表示形式，即更高级别的表示由高级表示形式组成，并且可以分解为一系列低级表示。例如，在不了解地面实际结构的情况下，我们提出的模型学会通过根据其句法结构组成变量和运算符的表示来处理逻辑表达。另一方面，依赖归纳偏置鼓励模型在输入序列中找到实体之间的潜在关系。对于自然语言，潜在关系通常被建模为一个定向依赖图，其中一个单词恰好具有一个父节点和零或几个孩子的节点。将此约束应用于类似变压器的模型之后，我们发现该模型能够诱导接近人类专家注释的有向图，并且在不同任务上也优于标准变压器模型。我们认为，这些实验结果为深度学习模型的未来发展展示了一个有趣的选择。

In order to achieve deep natural language understanding, syntactic constituent parsing is a vital step, highly demanded by many artificial intelligence systems to process both text and speech. One of the most recent proposals is the use of standard sequence-to-sequence models to perform constituent parsing as a machine translation task, instead of applying task-specific parsers. While they show a competitive performance, these text-to-parse transducers are still lagging behind classic techniques in terms of accuracy, coverage and speed. To close the gap, we here extend the framework of sequence-to-sequence models for constituent parsing, not only by providing a more powerful neural architecture for improving their performance, but also by enlarging their coverage to handle the most complex syntactic phenomena: discontinuous structures. To that end, we design several novel linearizations that can fully produce discontinuities and, for the first time, we test a sequence-to-sequence model on the main discontinuous benchmarks, obtaining competitive results on par with task-specific discontinuous constituent parsers and achieving state-of-the-art scores on the (discontinuous) English Penn Treebank.

Contextualized representation models such as ELMo (Peters et al., 2018a) and BERT (Devlin et al., 2018) have recently achieved state-of-the-art results on a diverse array of downstream NLP tasks. Building on recent token-level probing work, we introduce a novel edge probing task design and construct a broad suite of sub-sentence tasks derived from the traditional structured NLP pipeline. We probe word-level contextual representations from four recent models and investigate how they encode sentence structure across a range of syntactic, semantic, local, and long-range phenomena. We find that existing models trained on language modeling and translation produce strong representations for syntactic phenomena, but only offer comparably small improvements on semantic tasks over a non-contextual baseline.

Large pre-trained neural networks such as BERT have had great recent success in NLP, motivating a growing body of research investigating what aspects of language they are able to learn from unlabeled data. Most recent analysis has focused on model outputs (e.g., language model surprisal) or internal vector representations (e.g., probing classifiers). Complementary to these works, we propose methods for analyzing the attention mechanisms of pre-trained models and apply them to BERT. BERT's attention heads exhibit patterns such as attending to delimiter tokens, specific positional offsets, or broadly attending over the whole sentence, with heads in the same layer often exhibiting similar behaviors. We further show that certain attention heads correspond well to linguistic notions of syntax and coreference. For example, we find heads that attend to the direct objects of verbs, determiners of nouns, objects of prepositions, and coreferent mentions with remarkably high accuracy. Lastly, we propose an attention-based probing classifier and use it to further demonstrate that substantial syntactic information is captured in BERT's attention. 1 Code will be released at https://github.com/ clarkkev/attention-analysis.2 We use the English base-sized model.

尽管在理解深度NLP模型中学到的表示形式以及他们所捕获的知识方面已经做了很多工作，但对单个神经元的关注很少。我们提出了一种称为语言相关性分析的技术，可在任何外部特性中提取模型中的显着神经元 - 目的是了解如何保留这种知识在神经元中。我们进行了细粒度的分析以回答以下问题：（i）我们可以识别网络中捕获特定语言特性的神经元子集吗？ （ii）整个网络中的局部或分布式神经元如何？ iii）信息保留了多么冗余？ iv）针对下游NLP任务的微调预训练模型如何影响学习的语言知识？ iv）架构在学习不同的语言特性方面有何不同？我们的数据驱动的定量分析阐明了有趣的发现：（i）我们发现了可以预测不同语言任务的神经元的小亚集，ii）捕获基本的词汇信息（例如后缀），而这些神经元位于较低的大多数层中，iii，iii），而这些神经元，而那些神经元，而那些神经元则可以预测。学习复杂的概念（例如句法角色）主要是在中间和更高层中，iii），在转移学习过程中，显着的语言神经元从较高到较低的层移至较低的层，因为网络保留了较高的层以特定于任务信息，iv）我们发现很有趣在培训预训练模型之间的差异，关于如何保留语言信息，V）我们发现概念在多语言变压器模型中跨不同语言表现出相似的神经元分布。我们的代码作为Neurox工具包的一部分公开可用。

Code completion aims to help improve developers' productivity by suggesting the next code tokens from a given context. Various approaches have been proposed to incorporate abstract syntax tree (AST) information for model training, ensuring that code completion is aware of the syntax of the programming languages. However, existing syntax-aware code completion approaches are not on-the-fly, as we found that for every two-thirds of characters that developers type, AST fails to be extracted because it requires the syntactically correct source code, limiting its practicality in real-world scenarios. On the other hand, existing on-the-fly code completion does not consider syntactic information yet. In this paper, we propose PyCoder to leverage token types, a kind of lightweight syntactic information, which is readily available and aligns with the natural order of source code. Our PyCoder is trained in a multi-task training manner so that by learning the supporting task of predicting token types during the training phase, the models achieve better performance on predicting tokens and lines of code without the need for token types in the inference phase. Comprehensive experiments show that PyCoder achieves the first rank on the CodeXGLUE leaderboard with an accuracy of 77.12% for the token-level predictions, which is 0.43%-24.25% more accurate than baselines. In addition, PyCoder achieves an exact match of 43.37% for the line-level predictions, which is 3.63%-84.73% more accurate than baselines. These results lead us to conclude that token type information (an alternative to syntactic information) that is rarely used in the past can greatly improve the performance of code completion approaches, without requiring the syntactically correct source code like AST-based approaches do. Our PyCoder is publicly available on HuggingFace.

We introduce Transformer Grammars (TGs), a novel class of Transformer language models that combine (i) the expressive power, scalability, and strong performance of Transformers and (ii) recursive syntactic compositions, which here are implemented through a special attention mask and deterministic transformation of the linearized tree. We find that TGs outperform various strong baselines on sentence-level language modeling perplexity, as well as on multiple syntax-sensitive language modeling evaluation metrics. Additionally, we find that the recursive syntactic composition bottleneck which represents each sentence as a single vector harms perplexity on document-level language modeling, providing evidence that a different kind of memory mechanism -- one that is independent of composed syntactic representations -- plays an important role in current successful models of long text.

We present CodeBERT, a bimodal pre-trained model for programming language (PL) and natural language (NL). CodeBERT learns general-purpose representations that support downstream NL-PL applications such as natural language code search, code documentation generation, etc. We develop Code-BERT with Transformer-based neural architecture, and train it with a hybrid objective function that incorporates the pre-training task of replaced token detection, which is to detect plausible alternatives sampled from generators. This enables us to utilize both "bimodal" data of NL-PL pairs and "unimodal" data, where the former provides input tokens for model training while the latter helps to learn better generators. We evaluate CodeBERT on two NL-PL applications by fine-tuning model parameters. Results show that CodeBERT achieves state-of-the-art performance on both natural language code search and code documentation generation. Furthermore, to investigate what type of knowledge is learned in CodeBERT, we construct a dataset for NL-PL probing, and evaluate in a zero-shot setting where parameters of pre-trained models are fixed. Results show that CodeBERT performs better than previous pre-trained models on NL-PL probing. 1