Automated Program Repair (APR) is defined as the process of fixing a bug/defect in the source code, by an automated tool. APR tools have recently experienced promising results by leveraging state-of-the-art Neural Language Processing (NLP) techniques. APR tools such as TFix and CodeXGLUE combine text-to-text transformers with software-specific techniques are outperforming alternatives, these days. However, in most APR studies the train and test sets are chosen from the same set of projects. In reality, however, APR models are meant to be generalizable to new and different projects. Therefore, there is a potential threat that reported APR models with high effectiveness perform poorly when the characteristics of the new project or its bugs are different than the training set's(Domain Shift). In this study, we first define and measure the domain shift problem in automated program repair. Then, we then propose a domain adaptation framework that can adapt an APR model for a given target project. We conduct an empirical study with three domain adaptation methods FullFineTuning, TuningWithLightWeightAdapterLayers, and CurriculumLearning using two state-of-the-art domain adaptation tools (TFix and CodeXGLUE) and two APR models on 611 bugs from 19 projects. The results show that our proposed framework can improve the effectiveness of TFix by 13.05% and CodeXGLUE by 23.4%. Another contribution of this study is the proposal of a data synthesis method to address the lack of labelled data in APR. We leverage transformers to create a bug generator model. We use the generated synthetic data to domain adapt TFix and CodeXGLUE on the projects with no data (Zero-shot learning), which results in an average improvement of 5.76% and 24.42% for TFix and CodeXGLUE, respectively.
translated by 谷歌翻译
在本文中,我们解决了深入学习的软件漏洞自动修复问题。数据驱动漏洞修复的主要问题是已知确认漏洞的少数现有数据集仅由几千例组成。然而,培训深度学习模型通常需要数十万例的例子。在这项工作中,我们利用了错误修复任务和漏洞修复任务的直觉相关,并且可以传输来自错误修复的知识可以传输到修复漏洞。在机器学习界中,这种技术称为转移学习。在本文中,我们提出了一种修复名为Vreepair的安全漏洞的方法,该方法是基于转移学习。 vreepair首先在大型错误修复语料库上培训,然后在漏洞修复数据集上调整,这是一个较小的数量级。在我们的实验中,我们表明,仅在错误修复语料库上培训的模型可能已经修复了一些漏洞。然后,我们证明转移学习改善了修复易受攻击的C功能的能力。我们还表明,转移学习模型比具有去噪任务训练的模型更好,并在漏洞固定任务上进行微调。总而言之,本文表明,与在小型数据集上的学习相比,转移学习适用于修复C中的安全漏洞。
translated by 谷歌翻译
Machine Learning for Source Code (ML4Code) is an active research field in which extensive experimentation is needed to discover how to best use source code's richly structured information. With this in mind, we introduce JEMMA, an Extensible Java Dataset for ML4Code Applications, which is a large-scale, diverse, and high-quality dataset targeted at ML4Code. Our goal with JEMMA is to lower the barrier to entry in ML4Code by providing the building blocks to experiment with source code models and tasks. JEMMA comes with a considerable amount of pre-processed information such as metadata, representations (e.g., code tokens, ASTs, graphs), and several properties (e.g., metrics, static analysis results) for 50,000 Java projects from the 50KC dataset, with over 1.2 million classes and over 8 million methods. JEMMA is also extensible allowing users to add new properties and representations to the dataset, and evaluate tasks on them. Thus, JEMMA becomes a workbench that researchers can use to experiment with novel representations and tasks operating on source code. To demonstrate the utility of the dataset, we also report results from two empirical studies on our data, ultimately showing that significant work lies ahead in the design of context-aware source code models that can reason over a broader network of source code entities in a software project, the very task that JEMMA is designed to help with.
translated by 谷歌翻译
The International Workshop on Reading Music Systems (WoRMS) is a workshop that tries to connect researchers who develop systems for reading music, such as in the field of Optical Music Recognition, with other researchers and practitioners that could benefit from such systems, like librarians or musicologists. The relevant topics of interest for the workshop include, but are not limited to: Music reading systems; Optical music recognition; Datasets and performance evaluation; Image processing on music scores; Writer identification; Authoring, editing, storing and presentation systems for music scores; Multi-modal systems; Novel input-methods for music to produce written music; Web-based Music Information Retrieval services; Applications and projects; Use-cases related to written music. These are the proceedings of the 3rd International Workshop on Reading Music Systems, held in Alicante on the 23rd of July 2021.
translated by 谷歌翻译
我们提出了Pangu-Coder,这是一种仅预读的解码器语言模型,该模型采用pangu-alpha架构进行文本到代码生成,即给定自然语言问题描述的编程语言解决方案的合成。我们使用两阶段策略训练Pangu-Coder:第一阶段采用因果语言建模(CLM)来预先培训原始编程语言数据,而第二阶段则使用因果语言建模和掩盖语言建模(MLM)的组合培训目标,专注于文本到代码生成的下游任务,并培训松散的自然语言程序定义和代码功能。最后,我们讨论了pangu-coder-ft,该pander the是通过竞争性编程问题和代码与持续集成测试的结合进行了微调的。我们评估了pangu-coder,重点是它是否生成功能上正确的程序,并证明它在参加较小的上下文窗口和较少的数据培训的同时,它比诸如Codex之类的类似大小的模型(例如Codex)实现等效性或更好的性能。
translated by 谷歌翻译
软件工程(ML4SE)的机器学习是一个积极发展的研究领域,专注于帮助程序员工作的方法。为了在实践中应用开发的方法,他们需要实现合理的质量,以帮助而不是分散开发人员的注意力。尽管开发新方法来代码表示和数据收集可以提高模型的整体质量,但它没有考虑到我们可以从手头项目中获得的信息。在这项工作中,我们研究了如果我们针对特定项目,则如何提高模型的质量。我们开发一个框架来评估质量改进,模型可以在特定项目上的方法名称预测任务进行微调后获得。我们评估了三种不同复杂性的模型,并在三个设置中进行了比较它们的质量:在大型Java项目的大型数据集上进行培训,进一步对特定项目的数据进行了微调,并从头开始训练了此数据。我们表明,每项项目的微调可以极大地提高模型的质量,因为它们捕获了项目的领域和命名约定。我们开放用于数据收集的工具以及运行实验的代码:https://zenodo.org/record/6040745。
translated by 谷歌翻译
Transfer learning, where a model is first pre-trained on a data-rich task before being finetuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new "Colossal Clean Crawled Corpus", we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.
translated by 谷歌翻译
深层神经网络(DNN)越来越多地用于软件工程和代码智能任务。这些是强大的工具,能够通过数百万参数从大型数据集中学习高度概括的模式。同时,它们的大容量可以使他们容易记住数据点。最近的工作表明,当训练数据集嘈杂,涉及许多模棱两可或可疑的样本时,记忆风险特别强烈表现出来,而记忆是唯一的追索权。本文的目的是评估和比较神经代码智能模型中的记忆和概括程度。它旨在提供有关记忆如何影响神经模型在代码智能系统中的学习行为的见解。为了观察模型中的记忆程度,我们为原始训练数据集增加了随机噪声,并使用各种指标来量化噪声对训练和测试各个方面的影响。我们根据Java,Python和Ruby Codebase评估了几种最先进的神经代码智能模型和基准。我们的结果突出了重要的风险:数百万可训练的参数允许神经网络记住任何包括嘈杂数据,并提供错误的概括感。我们观察到所有模型都表现出某些形式的记忆。在大多数代码智能任务中,这可能会很麻烦,因为它们依赖于相当容易发生噪声和重复性数据源,例如GitHub的代码。据我们所知,我们提供了第一个研究,以量化软件工程和代码智能系统领域的记忆效应。这项工作提高了人们的意识,并为训练神经模型的重要问题提供了新的见解,这些问题通常被软件工程研究人员忽略。
translated by 谷歌翻译
大多数自动化软件测试任务可以从测试用例的抽象表示中受益。传统上,这是通过基于测试案例的代码覆盖范围来完成的。规范级别的标准可以替换代码覆盖范围以更好地表示测试用例的行为,但通常不具有成本效益。在本文中,我们假设测试用例的执行痕迹可以使其在自动测试任务中抽象其行为的好选择。我们提出了一种新颖的嵌入方法Test2VEC,该方法将测试执行映射到潜在空间。我们在测试案例的优先级(TP)任务中评估了此表示形式。我们的默认TP方法基于嵌入式向量与历史失败测试向量的相似性。我们还根据测试向量的多样性研究了一种替代方案。最后,我们提出了一种决定给定测试套件的方法,以决定选择哪种TP。该实验基于几个真实和种子故障,具有超过一百万个执行痕迹。结果表明,就第一个失败测试案例(FFR)的中位数等级而言,我们提议的TP将最佳替代品提高了41.80%。就中位数APFD和中位数归一化FFR而言,它的表现优于传统代码覆盖范围的方法25.05%和59.25%。
translated by 谷歌翻译
随着预先训练模型的巨大成功,Pretrain-Then-Finetune范式已被广泛采用下游任务,以获得源代码的理解。但是,与昂贵的培训从头开始培训,如何将预先训练的模型从划痕进行有效地调整到新任务的训练模型尚未完全探索。在本文中,我们提出了一种桥接预先训练的模型和与代码相关任务的方法。我们利用语义保留的转换来丰富下游数据分集,并帮助预先接受的模型学习语义特征不变于这些语义上等效的转换。此外,我们介绍课程学习以易于努力的方式组织转换的数据,以微调现有的预先训练的模型。我们将我们的方法应用于一系列预先训练的型号,它们在源代码理解的任务中显着优于最先进的模型,例如算法分类,代码克隆检测和代码搜索。我们的实验甚至表明,在没有重量训练的代码数据上,自然语言预先训练的模型罗伯塔微调我们的轻质方法可以优于或竞争现有的代码,在上述任务中进行微调,如Codebert和Codebert和GraphCodebert。这一发现表明,代码预训练模型中仍有很大的改进空间。
translated by 谷歌翻译
本次调查绘制了用于分析社交媒体数据的生成方法的研究状态的广泛的全景照片(Sota)。它填补了空白,因为现有的调查文章在其范围内或被约会。我们包括两个重要方面,目前正在挖掘和建模社交媒体的重要性:动态和网络。社会动态对于了解影响影响或疾病的传播,友谊的形成,友谊的形成等,另一方面,可以捕获各种复杂关系,提供额外的洞察力和识别否则将不会被注意的重要模式。
translated by 谷歌翻译
源代码的预训练的生成语言模型(例如PLBART,CODET5,SPT-CODE)在过去几年中对多个任务(包括代码生成和翻译)产生了强劲的结果。这些模型采用了不同的训练前目标,以自我监督的方式从非常大规模的语料库中学习代码构建的统计数据。预训练模型的成功很大程度上取决于这些预训练的目标。本文提出了一个新的预训练目标,即“归化”源代码,利用代码的双峰,双通道(正式和自然渠道)性质。与自然语言不同,代码的双峰,双通道的性质使我们能够大规模生成语义上等效的代码。我们介绍了六类的语义保存转换,以引入非自然的代码形式,然后强迫我们的模型制作开发人员编写的更自然的原创程序。学习在没有明确的手动监督的情况下,通过大型的开源代码来生成等效但更自然的代码,有助于模型学习摄入和生成代码。我们将模型在三个生成软件工程任务中微调:代码生成,代码翻译和代码改进,具有有限的人类策划标记数据并实现最先进的性能与CODET5。我们表明,我们的预训练模型在零射门和少数学习方面特别有竞争力,并且在学习代码属性(例如语法,数据流)方面更好。
translated by 谷歌翻译
源代码存储库由大型代码库组成,通常包含容易发生的程序。软件的复杂性日益增加导致时间和识别这些缺陷的时间和成本急剧上升。存在各种方法可以自动生成错误代码的修复程序。但是,由于特定错误的可能解决方案的组合空间很大,因此没有很多工具和数据集可以有效地评估生成的代码。在这项工作中,我们介绍了FixeVal,这是一个基准,其中包括竞争性编程问题及其各自修复程序的基准。我们引入了丰富的测试套件,以评估和评估模型生成程序修复的正确性。我们将两种在编程语言上鉴定的变压器语言模型视为我们的基准,并使用基于匹配和基于执行的评估指标对其进行比较。我们的实验表明,基于匹配的指标不能准确反映模型生成的程序修复,而基于执行的方法通过专门为该解决方案设计的所有情况和场景评估程序。因此,我们认为FixeVal提供了朝着实际自动错误修复和模型生成的代码评估的步骤。
translated by 谷歌翻译
许多利用移动设备中的传感器的应用以及应用机器学习以提供新颖的服务。然而,诸如不同的用户,设备,环境和超参数之类的各种因素影响了这种应用的性能,从而使域移位(即,来自训练源数据集的目标用户的分发偏移)是一个重要问题。虽然最近的域适应技术试图解决这个问题,但各种因素之间的复杂相互作用通常会限制其有效性。我们认为,准确估算未训练的域中的性能可能会显着降低性能不确定性。我们呈现Dapper(域适配性能估计器),其估计目标域中的适应性能,只有未标记的目标数据。我们的直觉是目标数据上模型的输出提供了模型在目标域中的实际性能的线索。 Dapper不需要昂贵的标签成本,也不需要在部署后涉及额外的培训。与四个基线相比,我们与四个真实世界传感数据集进行了评估,表明,估计精度平均17%平均占据了基线的表现。此外,我们的On-Device实验表明,与基线相比,Dapper达到了多达216倍的计算开销。
translated by 谷歌翻译
State-of-the-art computer vision systems are trained to predict a fixed set of predetermined object categories. This restricted form of supervision limits their generality and usability since additional labeled data is needed to specify any other visual concept. Learning directly from raw text about images is a promising alternative which leverages a much broader source of supervision. We demonstrate that the simple pre-training task of predicting which caption goes with which image is an efficient and scalable way to learn SOTA image representations from scratch on a dataset of 400 million (image, text) pairs collected from the internet. After pre-training, natural language is used to reference learned visual concepts (or describe new ones) enabling zero-shot transfer of the model to downstream tasks. We study the performance of this approach by benchmarking on over 30 different existing computer vision datasets, spanning tasks such as OCR, action recognition in videos, geo-localization, and many types of fine-grained object classification. The model transfers non-trivially to most tasks and is often competitive with a fully supervised baseline without the need for any dataset specific training. For instance, we match the accuracy of the original ResNet-50 on ImageNet zero-shot without needing to use any of the 1.28 million training examples it was trained on. We release our code and pre-trained model weights at https://github.com/OpenAI/CLIP.
translated by 谷歌翻译
训练有素的机器学习模型,利用大量的开源软件数据,现在已经成为自动化许多软件工程任务的有趣方法。几个硒任务都受到这种方法,在过去的几年里,性能逐渐改善,具有更好的模型和培训方法。更多,更多样化,清洁,标记数据更好的培训;但构建高质量的数据集是耗时和挑战。增强清洁量和多样性的方法,标记数据通常具有广泛的适用性。对于某些语言(例如,Ruby)标记的数据不那么丰富;在其他(例如,JavaScript)中,可用数据可能更多地关注某些应用域,从而更加多样化。作为围绕此类数据瓶颈,我们提出了证据表明,不同语言(执行相同功能)的人写代码相当相似,特别是保留标识符命名模式;我们进一步提出了证据表明标识符是软件工程任务培训数据的一个非常重要的要素。我们利用这种相当偶然的现象来查找可用的多语言训练数据(跨不同语言)的证据可用于放大性能。我们研究这一点3个不同的任务:代码摘要,代码检索和功能命名。我们注意到,这种数据增强方法与不同的任务,语言和机器学习模型广泛兼容。
translated by 谷歌翻译
Natural Language Generation (NLG) has improved exponentially in recent years thanks to the development of sequence-to-sequence deep learning technologies such as Transformer-based language models. This advancement has led to more fluent and coherent NLG, leading to improved development in downstream tasks such as abstractive summarization, dialogue generation and data-to-text generation. However, it is also apparent that deep learning based generation is prone to hallucinate unintended text, which degrades the system performance and fails to meet user expectations in many real-world scenarios. To address this issue, many studies have been presented in measuring and mitigating hallucinated texts, but these have never been reviewed in a comprehensive manner before. In this survey, we thus provide a broad overview of the research progress and challenges in the hallucination problem in NLG. The survey is organized into two parts: (1) a general overview of metrics, mitigation methods, and future directions; and (2) an overview of task-specific research progress on hallucinations in the following downstream tasks, namely abstractive summarization, dialogue generation, generative question answering, data-to-text generation, machine translation, and visual-language generation. This survey serves to facilitate collaborative efforts among researchers in tackling the challenge of hallucinated texts in NLG.
translated by 谷歌翻译
可选类型的注释允许通过静态打字功能来丰富动态编程语言,例如更好的集成开发环境(IDE)支持,更精确的程序分析以及与类型相关的运行时错误的早期检测和预防。基于机器学习的类型推理有望自动执行此任务的有趣结果。但是,此类系统的实际用法取决于它们在跨不同领域概括的能力,因为它们通常在训练领域之外应用。在这项工作中,我们通过进行广泛的跨域实验来研究Type4py作为最先进的基于深度学习类型推理系统的代表性的概括能力。因此,我们解决了以下问题:数据集偏移,播音外词,未知类别和稀有类。为了执行此类实验,我们使用数据集nytypes4py和crossdomaintypes4py。我们在本文中介绍的后者。我们的数据集具有超过1,000,000个类型的注释,并可以使用来自两个域Web开发和科学计算的数据的数据的不同域中的类型推理系统进行跨域评估。通过我们的实验,我们检测到数据集中的变化,并具有长尾巴分布,并具有许多稀有和未知的数据类型,从而大大降低了基于深度学习的推理系统的性能。在这种情况下,我们测试了无监督的域适应方法和微调以克服问题。此外,我们研究了量量表的单词的影响。
translated by 谷歌翻译
背景:机器学习(ML)可以实现有效的自动测试生成。目的:我们表征了新兴研究,检查测试实践,研究人员目标,应用的ML技术,评估和挑战。方法:我们对97个出版物的样本进行系统文献综述。结果:ML生成系统,GUI,单位,性能和组合测试的输入或改善现有生成方法的性能。 ML还用于生成测试判决,基于属性的和预期的输出序列。经常基于神经网络和强化学习的监督学习通常是基于Q学习的 - 很普遍,并且某些出版物还采用了无监督或半监督的学习。使用传统的测试指标和与ML相关的指标(例如准确性)评估(半/非 - )监督方法,而经常使用与奖励功能相关的测试指标来评估强化学习。结论:工作到尽头表现出巨大的希望,但是在培训数据,再探术,可伸缩性,评估复杂性,所采用的ML算法以及如何应用 - 基准和可复制性方面存在公开挑战。我们的发现可以作为该领域研究人员的路线图和灵感。
translated by 谷歌翻译
Labeling a module defective or non-defective is an expensive task. Hence, there are often limits on how much-labeled data is available for training. Semi-supervised classifiers use far fewer labels for training models, but there are numerous semi-supervised methods, including self-labeling, co-training, maximal-margin, and graph-based methods, to name a few. Only a handful of these methods have been tested in SE for (e.g.) predicting defects and even that, those tests have been on just a handful of projects. This paper takes a wide range of 55 semi-supervised learners and applies these to over 714 projects. We find that semi-supervised "co-training methods" work significantly better than other approaches. However, co-training needs to be used with caution since the specific choice of co-training methods needs to be carefully selected based on a user's specific goals. Also, we warn that a commonly-used co-training method ("multi-view"-- where different learners get different sets of columns) does not improve predictions (while adding too much to the run time costs 11 hours vs. 1.8 hours). Those cautions stated, we find using these "co-trainers," we can label just 2.5% of data, then make predictions that are competitive to those using 100% of the data. It is an open question worthy of future work to test if these reductions can be seen in other areas of software analytics. All the codes used and datasets analyzed during the current study are available in the https://GitHub.com/Suvodeep90/Semi_Supervised_Methods.
translated by 谷歌翻译