Automated software debugging is a crucial task for improving the productivity of software developers. Many neural-based techniques have been proven effective for debugging-related tasks such as bug localization and program repair (or bug fixing). However, these techniques often focus only on either one of them or approach them in a stage-wise manner, ignoring the mutual benefits between them. In this work, we propose a novel unified \emph{Detect-Localize-Repair} framework based on a pretrained programming language model CodeT5 to seamlessly address these tasks, named CodeT5-DLR. Specifically, we propose three objectives to adapt the generic CodeT5 for debugging: a bug detection objective to determine whether a given code snippet is buggy or not, a bug localization objective to identify the buggy lines, and a program repair objective to translate the buggy code to its fixed version. We evaluate it on each of these tasks and their combined setting on two newly collected line-level debugging datasets in Java and Python. Extensive results show that our model significantly outperforms existing baselines from both NLP and software engineering domains.

在开源存储库中发现的真正错误修复似乎是学习本地化和修复实际错误的理想来源。但是，缺乏大规模的错误修复集合使过去难以有效利用过去的较大神经模型的真正错误修复。相比之下，人工错误 - 通过突变现有源代码产生的人为错误可以轻松地以足够的规模获得，因此在培训现有方法时通常是首选的。尽管如此，在面对真正的错误时，经过对人造错误的培训的本地化和维修模型通常在表现不佳。这就提出了一个问题，是否在实际错误修复程序上培训的错误本地化和维修模型在本地化和维修实际错误方面更有效。我们通过引入Realit，这是一种预先培训和预先计算方法，以有效地学习从真正的错误修复中进行本地化和修复真实的错误来解决这个问题。 Realit首先是在传统突变操作员产生的大量人造错误上进行的，然后在较小的一组实际错误修复程序上进行了微调。微调不需要对学习算法进行任何修改，因此可以轻松地在各种培训方案中用于错误定位或维修（即使实际培训数据很少）。此外，我们发现，对使用真实错误修复的培训在经验上几乎使现有模型在实际错误上的本地化性能翻了一番，同时维护甚至改善了维修性能。

图形神经网络已被证明可以为各种软件工程任务产生令人印象深刻的结果。但是，现有技术仍然有两个问题：（1）长期依赖性和（2）不同的代码组件在不应该的情况下被视为平等。为了解决这些问题，我们提出了一种表示代码为层次结构（代码层次结构）的方法，其中不同的代码组件在各个粒度级别分别表示。然后，为了处理每个表示级别的表示，我们设计了一个新颖的网络体系结构Echelon，它结合了异质图形变压器网络和基于树的卷积神经网络的优势，以学习具有代码依赖性信息丰富的抽象语法树。我们还提出了一个新颖的预处理目标，称为缺失子树预测以补充我们的代码层次结构。评估结果表明，我们的方法在三个任务中大大优于其他基准：任何代码完成，代码分类和代码克隆检测。

As the complexity of modern software continues to escalate, software engineering has become an increasingly daunting and error-prone endeavor. In recent years, the field of Neural Code Intelligence (NCI) has emerged as a promising solution, leveraging the power of deep learning techniques to tackle analytical tasks on source code with the goal of improving programming efficiency and minimizing human errors within the software industry. Pretrained language models have become a dominant force in NCI research, consistently delivering state-of-the-art results across a wide range of tasks, including code summarization, generation, and translation. In this paper, we present a comprehensive survey of the NCI domain, including a thorough review of pretraining techniques, tasks, datasets, and model architectures. We hope this paper will serve as a bridge between the natural language and programming language communities, offering insights for future research in this rapidly evolving field.

在本文中，我们解决了深入学习的软件漏洞自动修复问题。数据驱动漏洞修复的主要问题是已知确认漏洞的少数现有数据集仅由几千例组成。然而，培训深度学习模型通常需要数十万例的例子。在这项工作中，我们利用了错误修复任务和漏洞修复任务的直觉相关，并且可以传输来自错误修复的知识可以传输到修复漏洞。在机器学习界中，这种技术称为转移学习。在本文中，我们提出了一种修复名为Vreepair的安全漏洞的方法，该方法是基于转移学习。 vreepair首先在大型错误修复语料库上培训，然后在漏洞修复数据集上调整，这是一个较小的数量级。在我们的实验中，我们表明，仅在错误修复语料库上培训的模型可能已经修复了一些漏洞。然后，我们证明转移学习改善了修复易受攻击的C功能的能力。我们还表明，转移学习模型比具有去噪任务训练的模型更好，并在漏洞固定任务上进行微调。总而言之，本文表明，与在小型数据集上的学习相比，转移学习适用于修复C中的安全漏洞。

在源代码中自动定位易受攻击的陈述至关重要，以确保软件安全性和缓解开发人员的调试工作。这在当今软件生态系统中变得更加重要，其中易受攻击的代码可以在像GitHub这样的软件存储库中轻松且无意中流动。在这类数百万的代码行中，传统的静态和动态方法争取缩放。虽然基于机器学习的方法在这样的设置中看起来很有希望，但大多数工作都在较高的粒度下检测到脆弱的代码 - 在方法或文件级别。因此，开发人员仍然需要检查大量代码以找到需要修复的弱势陈述。本文提出了一种新的集合学习方法来定位脆弱的陈述。我们的模型结合了基于图形的基于序列的神经网络，以成功捕获程序图的本地和全局上下文，并有效地了解代码语义和易受攻击的模式。为了研究天鹅绒的效果，我们使用了一个现成的合成数据集和最近发布的现实世界数据集。在静态分析设置中，未提前检测到易受攻击功能，Velvet可以实现4.5倍的性能，而不是真实世界数据上的基线静态分析仪。对于孤立的漏洞本地化任务，在我们假设特定漏洞声明未知的同时知道函数的漏洞，我们将天鹅绒与几个神经网络进行比较，这些内部网络也参加了本地和全局代码背景。天鹅绒分别达到99.6％和43.6％的13.6％，分别在合成数据和现实世界数据上实现了高精度，优于基线深度学习模型5.3-29.0％。

源代码存储库由大型代码库组成，通常包含容易发生的程序。软件的复杂性日益增加导致时间和识别这些缺陷的时间和成本急剧上升。存在各种方法可以自动生成错误代码的修复程序。但是，由于特定错误的可能解决方案的组合空间很大，因此没有很多工具和数据集可以有效地评估生成的代码。在这项工作中，我们介绍了FixeVal，这是一个基准，其中包括竞争性编程问题及其各自修复程序的基准。我们引入了丰富的测试套件，以评估和评估模型生成程序修复的正确性。我们将两种在编程语言上鉴定的变压器语言模型视为我们的基准，并使用基于匹配和基于执行的评估指标对其进行比较。我们的实验表明，基于匹配的指标不能准确反映模型生成的程序修复，而基于执行的方法通过专门为该解决方案设计的所有情况和场景评估程序。因此，我们认为FixeVal提供了朝着实际自动错误修复和模型生成的代码评估的步骤。

GitHub提交的记录，该代码随着自然语言消息的描述而变化，对于软件开发人员来说，在理解软件演变方面起着至关重要的作用。为了促进开源软件社区的开发，我们收集了一个提交基准，包括790万次跨7种编程语言的投入。基于此基准测试，我们提出了Citsbart，这是GitHub提交的大型预训练的编码器变压器模型。该模型由三个类别（即，为了学习提交碎片表示的六个预训练任务）预先培训（即，剥夺目标，跨模式生成和对比度学习）。此外，我们将一个“委托智能”框架与一项理解任务和提交的三个世代任务统一。这些任务的综合实验表明，提案巴特大大优于以前的代码预先培训作品。进一步的分析还揭示了每个预训练任务可增强模型性能。我们鼓励后续研究人员在将来为我们的框架贡献更多与承诺相关的下游任务。

在编程中，学习代码表示有各种应用程序，包括代码分类，代码搜索，注释生成，错误预测等。已经提出了在令牌，语法树，依赖图，代码导航路径或其变体组合方面的各种代码表示，但是，现有的vanilla学习技术具有鲁棒性的主要限制，即，型号很容易当输入以微妙的方式改变输入时，要进行错误的预测。为了增强稳健性，现有方法专注于识别对抗性样本，而不是在落在给定分布之外的有效样品上，我们将其称为分配（OOD）样本。识别出这样的ood样本是本文研究的新问题。为此，我们建议首先使用分发的样本进行in =分发数据集，使得当培训在一起时，它们将增强模型的鲁棒性。我们建议使用能量有界学习的目标函数来将更高的分数分配给分布式样本和较低的分数，以便将这种分布式样品纳入源的培训过程中代码模型。在检测和逆势样本检测方面，我们的评估结果表明，现有源代码模型的稳健性更加准确，在识别ood数据时，同时在同时对对抗性攻击更具抵抗力。此外，所提出的能量有限评分优于大幅的余量，包括Softmax置信度评分，Mahalanobis评分和Odin。

In software development, it is common for programmers to copy-paste or port code snippets and then adapt them to their use case. This scenario motivates the code adaptation task -- a variant of program repair which aims to adapt variable identifiers in a pasted snippet of code to the surrounding, preexisting source code. However, no existing approach has been shown to effectively address this task. In this paper, we introduce AdaptivePaste, a learning-based approach to source code adaptation, based on transformers and a dedicated dataflow-aware deobfuscation pre-training task to learn meaningful representations of variable usage patterns. We evaluate AdaptivePaste on a dataset of code snippets in Python. Results suggest that our model can learn to adapt source code with 79.8% accuracy. To evaluate how valuable is AdaptivePaste in practice, we perform a user study with 10 Python developers on a hundred real-world copy-paste instances. The results show that AdaptivePaste reduces the dwell time to nearly half the time it takes for manual code adaptation, and helps to avoid bugs. In addition, we utilize the participant feedback to identify potential avenues for improvement of AdaptivePaste.

改善软件性能是软件开发周期中重要但充满挑战的部分。如今，大多数性能效率低下是由绩效专家确定和修补的。深度学习方法的最新进展和开源数据的广泛可用性为自动化绩效问题的识别和修补提供了一个绝佳的机会。在本文中，我们提出了Deepperf，这是一种基于变压器的方法，以建议针对C＃应用程序进行性能改进。我们在英语和源代码语料库上预告了Deepperf，然后进行了Finetuning的任务，以生成C＃应用程序的性能改进补丁。我们的评估表明，我们的模型可以在约53％的案例中生成与开发人员修复相同的性能改进建议，在我们专家验证的C＃开发人员进行的绩效更改的数据集中，逐字化约34％。此外，我们使用基准测试和单元测试在GitHub上在50个开源C＃存储库上评估Deepperf，并发现我们的模型能够提出有效的性能改进，以改善CPU使用和内存分配。到目前为止，我们已经提交了19个带有28种不同性能优化的拉装重新要求，其中11个PR已获得项目所有者的批准。

The problem of reversing the compilation process, decompilation, is an important tool in reverse engineering of computer software. Recently, researchers have proposed using techniques from neural machine translation to automate the process in decompilation. Although such techniques hold the promise of targeting a wider range of source and assembly languages, to date they have primarily targeted C code. In this paper we argue that existing neural decompilers have achieved higher accuracy at the cost of requiring language-specific domain knowledge such as tokenizers and parsers to build an abstract syntax tree (AST) for the source language, which increases the overhead of supporting new languages. We explore a different tradeoff that, to the extent possible, treats the assembly and source languages as plain text, and show that this allows us to build a decompiler that is easily retargetable to new languages. We evaluate our prototype decompiler, Beyond The C (BTC), on Go, Fortran, OCaml, and C, and examine the impact of parameters such as tokenization and training data selection on the quality of decompilation, finding that it achieves comparable decompilation results to prior work in neural decompilation with significantly less domain knowledge. We will release our training data, trained decompilation models, and code to help encourage future research into language-agnostic decompilation.

Automated Program Repair (APR) is defined as the process of fixing a bug/defect in the source code, by an automated tool. APR tools have recently experienced promising results by leveraging state-of-the-art Neural Language Processing (NLP) techniques. APR tools such as TFix and CodeXGLUE combine text-to-text transformers with software-specific techniques are outperforming alternatives, these days. However, in most APR studies the train and test sets are chosen from the same set of projects. In reality, however, APR models are meant to be generalizable to new and different projects. Therefore, there is a potential threat that reported APR models with high effectiveness perform poorly when the characteristics of the new project or its bugs are different than the training set's(Domain Shift). In this study, we first define and measure the domain shift problem in automated program repair. Then, we then propose a domain adaptation framework that can adapt an APR model for a given target project. We conduct an empirical study with three domain adaptation methods FullFineTuning, TuningWithLightWeightAdapterLayers, and CurriculumLearning using two state-of-the-art domain adaptation tools (TFix and CodeXGLUE) and two APR models on 611 bugs from 19 projects. The results show that our proposed framework can improve the effectiveness of TFix by 13.05% and CodeXGLUE by 23.4%. Another contribution of this study is the proposal of a data synthesis method to address the lack of labelled data in APR. We leverage transformers to create a bug generator model. We use the generated synthetic data to domain adapt TFix and CodeXGLUE on the projects with no data (Zero-shot learning), which results in an average improvement of 5.76% and 24.42% for TFix and CodeXGLUE, respectively.

While pre-trained language models (LM) for code have achieved great success in code completion, they generate code conditioned only on the contents within the file, i.e., in-file context, but ignore the rich semantics in other files within the same project, i.e., cross-file context, a critical source of information that is especially useful in modern modular software development. Such overlooking constrains code language models' capacity in code completion, leading to unexpected behaviors such as generating hallucinated class member functions or function calls with unexpected arguments. In this work, we develop a cross-file context finder tool, CCFINDER, that effectively locates and retrieves the most relevant cross-file context. We propose CoCoMIC, a framework that incorporates cross-file context to learn the in-file and cross-file context jointly on top of pretrained code LMs. CoCoMIC successfully improves the existing code LM with a 19.30% relative increase in exact match and a 15.41% relative increase in identifier matching for code completion when the cross-file context is provided.

In this paper, we propose a novel technique, namely INVALIDATOR, to automatically assess the correctness of APR-generated patches via semantic and syntactic reasoning. INVALIDATOR reasons about program semantic via program invariants while it also captures program syntax via language semantic learned from large code corpus using the pre-trained language model. Given a buggy program and the developer-patched program, INVALIDATOR infers likely invariants on both programs. Then, INVALIDATOR determines that a APR-generated patch overfits if: (1) it violates correct specifications or (2) maintains errors behaviors of the original buggy program. In case our approach fails to determine an overfitting patch based on invariants, INVALIDATOR utilizes a trained model from labeled patches to assess patch correctness based on program syntax. The benefit of INVALIDATOR is three-fold. First, INVALIDATOR is able to leverage both semantic and syntactic reasoning to enhance its discriminant capability. Second, INVALIDATOR does not require new test cases to be generated but instead only relies on the current test suite and uses invariant inference to generalize the behaviors of a program. Third, INVALIDATOR is fully automated. We have conducted our experiments on a dataset of 885 patches generated on real-world programs in Defects4J. Experiment results show that INVALIDATOR correctly classified 79% overfitting patches, accounting for 23% more overfitting patches being detected by the best baseline. INVALIDATOR also substantially outperforms the best baselines by 14% and 19% in terms of Accuracy and F-Measure, respectively.

Machine Learning for Source Code (ML4Code) is an active research field in which extensive experimentation is needed to discover how to best use source code's richly structured information. With this in mind, we introduce JEMMA, an Extensible Java Dataset for ML4Code Applications, which is a large-scale, diverse, and high-quality dataset targeted at ML4Code. Our goal with JEMMA is to lower the barrier to entry in ML4Code by providing the building blocks to experiment with source code models and tasks. JEMMA comes with a considerable amount of pre-processed information such as metadata, representations (e.g., code tokens, ASTs, graphs), and several properties (e.g., metrics, static analysis results) for 50,000 Java projects from the 50KC dataset, with over 1.2 million classes and over 8 million methods. JEMMA is also extensible allowing users to add new properties and representations to the dataset, and evaluate tasks on them. Thus, JEMMA becomes a workbench that researchers can use to experiment with novel representations and tasks operating on source code. To demonstrate the utility of the dataset, we also report results from two empirical studies on our data, ultimately showing that significant work lies ahead in the design of context-aware source code models that can reason over a broader network of source code entities in a software project, the very task that JEMMA is designed to help with.

程序合成或代码生成旨在生成满足问题规范的程序。使用大规模预处理的语言模型（LMS）的最新方法显示出令人鼓舞的结果，但它们有一些关键的局限性。特别是，他们经常遵循标准监督的微调程序，仅从对自然语言问题描述和基础真相计划对培训代码生成模型。这种范式在很大程度上忽略了问题规范中的一些重要但潜在的信号，例如单位测试，因此在求解复杂的看不见的编码任务时通常会导致性能差。为了解决这些局限性，我们提出了“ Coderl”，这是通过验证的LMS和深入强化学习（RL）实现程序合成任务的新框架。具体而言，在培训期间，我们将代码生成的LM视为参与者网络，并引入批评网络，该网络经过培训，以预测生成的程序的功能正确性，并为演员提供密集的反馈信号。在推理期间，我们引入了一种新一代程序，具有关键的抽样策略，该过程允许模型根据示例单位测试和评论家分数的反馈自动重新生成程序。对于模型骨架，我们扩展了Codet5的编码器架构，具有增强的学习目标，更大的模型大小和更好的预处理数据。我们的方法不仅在具有挑战性的应用程序基准上实现了新的SOTA结果，而且还显示出强大的零弹性传输能力，并在简单的MBPP基准上具有新的SOTA结果。

迄今为止，统计类型推理系统彻底依赖于监督的学习方法，这些方法需要艰苦的手动努力来收集和标记大量数据。大多数图灵完整的命令式语言共享相似的控制和数据流结构，这使得将知识从一种语言转移到另一种语言。在本文中，我们提出了一个跨语言转移学习框架，即柏拉图，用于统计类型推理，这使我们能够利用一种从一种语言的标签数据集中学到的先验知识并将其转移到另一种语言的数据集中，例如Python，将其转移到JavaScript，Java，Java对于JavaScript等。柏拉图由一种新颖的核心注意机制提供动力，以限制主干变压器模型的注意范围，以便模型被迫将其预测基于语言之间普遍共享的特征。此外，我们提出了语法增强功能，以增强语言域之间的特征重叠的学习。此外，柏拉图还可以通过引入跨语言扩展来用于提高常规监督类型推理的性能，这使该模型能够学习多种语言的更多一般功能。我们在两种设置下评估了柏拉图：1）在跨域方案下，目标语言数据未标记或标记部分，结果表明，柏拉图的表现优于最先进的域传输技术，例如。 ，它通过+14.6%@em，+18.6%@weighted-f1和2）在传统单语言监督场景下改善了Python的打字稿基线，Plato将python的基线改进了+4.10%@em，+1.90%@weighted em -f1引入了跨语性增强。

深层神经网络（DNN）越来越多地用于软件工程和代码智能任务。这些是强大的工具，能够通过数百万参数从大型数据集中学习高度概括的模式。同时，它们的大容量可以使他们容易记住数据点。最近的工作表明，当训练数据集嘈杂，涉及许多模棱两可或可疑的样本时，记忆风险特别强烈表现出来，而记忆是唯一的追索权。本文的目的是评估和比较神经代码智能模型中的记忆和概括程度。它旨在提供有关记忆如何影响神经模型在代码智能系统中的学习行为的见解。为了观察模型中的记忆程度，我们为原始训练数据集增加了随机噪声，并使用各种指标来量化噪声对训练和测试各个方面的影响。我们根据Java，Python和Ruby Codebase评估了几种最先进的神经代码智能模型和基准。我们的结果突出了重要的风险：数百万可训练的参数允许神经网络记住任何包括嘈杂数据，并提供错误的概括感。我们观察到所有模型都表现出某些形式的记忆。在大多数代码智能任务中，这可能会很麻烦，因为它们依赖于相当容易发生噪声和重复性数据源，例如GitHub的代码。据我们所知，我们提供了第一个研究，以量化软件工程和代码智能系统领域的记忆效应。这项工作提高了人们的意识，并为训练神经模型的重要问题提供了新的见解，这些问题通常被软件工程研究人员忽略。

软件开发人员将源代码内的日志记录嵌入为现代软件开发中的命令占空税，因为日志文件是跟踪运行时系统问题和故障排除系统管理任务所必需的。但是，当前的日志记录过程主要是手动，因此，日志语句的适当放置和内容仍然是挑战。为了克服这些挑战，旨在自动化日志放置并预测其内容的方法，即“来到哪里以及登录的地方”，具有很高的兴趣。因此，我们专注于通过利用源代码克隆和自然语言处理（NLP）来预测日志语句的位置（即，其中）和描述（即，什么），因为这些方法为日志预测提供了额外的上下文和优点。具体而言，我们指导我们的研究三项研究问题（RQS）:( RQ1）如何利用代码片段，即代码克隆，用于日志语句预测如何？ （RQ2）如何扩展方法以自动执行日志语句的描述？ （RQ3）所提出的方法是如何有效的日志位置和描述预测？为了追求我们的RQ，我们对七个开源Java项目进行了实验研究。我们介绍了更新和改进的日志感知代码克隆检测方法，以预测日志记录语句（RQ1）的位置。然后，我们纳入自然语言处理（NLP）和深度学习方法，以自动化日志语句的描述预测（RQ2）。我们的分析表明，我们的混合NLP和Code-CC'd检测方法（NLP CC'd）优于常规克隆探测器，平均地查找日志声明位置，并在Bleu和Rouge分数上实现了40.86％的性能，以预测伐木的描述与先前研究（RQ3）相比的陈述。