We present CodeBERT, a bimodal pre-trained model for programming language (PL) and natural language (NL). CodeBERT learns general-purpose representations that support downstream NL-PL applications such as natural language code search, code documentation generation, etc. We develop Code-BERT with Transformer-based neural architecture, and train it with a hybrid objective function that incorporates the pre-training task of replaced token detection, which is to detect plausible alternatives sampled from generators. This enables us to utilize both "bimodal" data of NL-PL pairs and "unimodal" data, where the former provides input tokens for model training while the latter helps to learn better generators. We evaluate CodeBERT on two NL-PL applications by fine-tuning model parameters. Results show that CodeBERT achieves state-of-the-art performance on both natural language code search and code documentation generation. Furthermore, to investigate what type of knowledge is learned in CodeBERT, we construct a dataset for NL-PL probing, and evaluate in a zero-shot setting where parameters of pre-trained models are fixed. Results show that CodeBERT performs better than previous pre-trained models on NL-PL probing. 1

Masked language modeling (MLM) pre-training methods such as BERT corrupt the input by replacing some tokens with [MASK] and then train a model to reconstruct the original tokens. While they produce good results when transferred to downstream NLP tasks, they generally require large amounts of compute to be effective. As an alternative, we propose a more sample-efficient pre-training task called replaced token detection. Instead of masking the input, our approach corrupts it by replacing some tokens with plausible alternatives sampled from a small generator network. Then, instead of training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that predicts whether each token in the corrupted input was replaced by a generator sample or not. Thorough experiments demonstrate this new pre-training task is more efficient than MLM because the task is defined over all input tokens rather than just the small subset that was masked out. As a result, the contextual representations learned by our approach substantially outperform the ones learned by BERT given the same model size, data, and compute. The gains are particularly strong for small models; for example, we train a model on one GPU for 4 days that outperforms GPT (trained using 30x more compute) on the GLUE natural language understanding benchmark. Our approach also works well at scale, where it performs comparably to RoBERTa and XLNet while using less than 1/4 of their compute and outperforms them when using the same amount of compute.

训练有素的机器学习模型，利用大量的开源软件数据，现在已经成为自动化许多软件工程任务的有趣方法。几个硒任务都受到这种方法，在过去的几年里，性能逐渐改善，具有更好的模型和培训方法。更多，更多样化，清洁，标记数据更好的培训;但构建高质量的数据集是耗时和挑战。增强清洁量和多样性的方法，标记数据通常具有广泛的适用性。对于某些语言（例如，Ruby）标记的数据不那么丰富;在其他（例如，JavaScript）中，可用数据可能更多地关注某些应用域，从而更加多样化。作为围绕此类数据瓶颈，我们提出了证据表明，不同语言（执行相同功能）的人写代码相当相似，特别是保留标识符命名模式;我们进一步提出了证据表明标识符是软件工程任务培训数据的一个非常重要的要素。我们利用这种相当偶然的现象来查找可用的多语言训练数据（跨不同语言）的证据可用于放大性能。我们研究这一点3个不同的任务：代码摘要，代码检索和功能命名。我们注意到，这种数据增强方法与不同的任务，语言和机器学习模型广泛兼容。

We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models (Peters et al., 2018a;Radford et al., 2018), BERT is designed to pretrain deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be finetuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial taskspecific architecture modifications.BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

GitHub提交的记录，该代码随着自然语言消息的描述而变化，对于软件开发人员来说，在理解软件演变方面起着至关重要的作用。为了促进开源软件社区的开发，我们收集了一个提交基准，包括790万次跨7种编程语言的投入。基于此基准测试，我们提出了Citsbart，这是GitHub提交的大型预训练的编码器变压器模型。该模型由三个类别（即，为了学习提交碎片表示的六个预训练任务）预先培训（即，剥夺目标，跨模式生成和对比度学习）。此外，我们将一个“委托智能”框架与一项理解任务和提交的三个世代任务统一。这些任务的综合实验表明，提案巴特大大优于以前的代码预先培训作品。进一步的分析还揭示了每个预训练任务可增强模型性能。我们鼓励后续研究人员在将来为我们的框架贡献更多与承诺相关的下游任务。

Pre-trained language models have achieved promising success in code retrieval tasks, where a natural language documentation query is given to find the most relevant existing code snippet. However, existing models focus only on optimizing the documentation code pairs by embedding them into latent space, without the association of external knowledge. In this paper, we propose a generation-augmented query expansion framework. Inspired by the human retrieval process - sketching an answer before searching, in this work, we utilize the powerful code generation model to benefit the code retrieval task. Specifically, we demonstrate that rather than merely retrieving the target code snippet according to the documentation query, it would be helpful to augment the documentation query with its generation counterpart - generated code snippets from the code generation model. To the best of our knowledge, this is the first attempt that leverages the code generation model to enhance the code retrieval task. We achieve new state-of-the-art results on the CodeSearchNet benchmark and surpass the baselines significantly.

我们提出了Pangu-Coder，这是一种仅预读的解码器语言模型，该模型采用pangu-alpha架构进行文本到代码生成，即给定自然语言问题描述的编程语言解决方案的合成。我们使用两阶段策略训练Pangu-Coder：第一阶段采用因果语言建模（CLM）来预先培训原始编程语言数据，而第二阶段则使用因果语言建模和掩盖语言建模（MLM）的组合培训目标，专注于文本到代码生成的下游任务，并培训松散的自然语言程序定义和代码功能。最后，我们讨论了pangu-coder-ft，该pander the是通过竞争性编程问题和代码与持续集成测试的结合进行了微调的。我们评估了pangu-coder，重点是它是否生成功能上正确的程序，并证明它在参加较小的上下文窗口和较少的数据培训的同时，它比诸如Codex之类的类似大小的模型（例如Codex）实现等效性或更好的性能。

Pre-trained models have achieved remarkable success in natural language processing (NLP). However, existing pre-training methods underutilize the benefits of language understanding for generation. Inspired by the idea of Generative Adversarial Networks (GANs), we propose a GAN-style model for encoder-decoder pre-training by introducing an auxiliary discriminator, unifying the ability of language understanding and generation in a single model. Our model, named as GanLM, is trained with two pre-training objectives: replaced token detection and replaced token denoising. Specifically, given masked source sentences, the generator outputs the target distribution and the discriminator predicts whether the target sampled tokens from distribution are incorrect. The target sentence is replaced with misclassified tokens to construct noisy previous context, which is used to generate the gold sentence. In general, both tasks improve the ability of language understanding and generation by selectively using the denoising data. Extensive experiments in language generation benchmarks show that GanLM with the powerful language understanding capability outperforms various strong pre-trained language models (PLMs) and achieves state-of-the-art performance.

在培训数据中拟合复杂的模式，例如推理和争议，是语言预训练的关键挑战。根据最近的研究和我们的经验观察，一种可能的原因是训练数据中的一些易于适应的模式，例如经常共同发生的单词组合，主导和伤害预训练，使模型很难适合更复杂的信息。我们争辩说，错误预测可以帮助找到危害语言理解的这种主导模式。当发生错误预测时，应该经常与导致MIS预测的模型拟合的MIS预测字相同的模式。如果我们可以添加正规化以培训模型，当MIS预测发生并更多地对待更微妙的模式时，可以在更多信息上缩小到这种主导模式时，可以在预训练中有效地安装更多信息。在此动机之后，我们提出了一种新的语言预培训方法，错误预测作为伤害警报（MPA）。在MPA中，当在预训练期间发生错误预测时，我们使用其共同发生信息来指导自我关注模块的多个头部。变压器模块中的一些自我关注头经过优化，以将更低的注意重量分配给频繁地在误报中的输入句子中的单词，同时将更高权重分配给另一个单词。通过这样做，变压器模型训练，以依赖于主导的频繁共同发生模式，而在误报中，当发生错误预测时，在剩余更复杂的信息上更加关注更多。我们的实验表明，MPA加快了伯特和电器的预训练，并提高了他们对下游任务的表现。

As the complexity of modern software continues to escalate, software engineering has become an increasingly daunting and error-prone endeavor. In recent years, the field of Neural Code Intelligence (NCI) has emerged as a promising solution, leveraging the power of deep learning techniques to tackle analytical tasks on source code with the goal of improving programming efficiency and minimizing human errors within the software industry. Pretrained language models have become a dominant force in NCI research, consistently delivering state-of-the-art results across a wide range of tasks, including code summarization, generation, and translation. In this paper, we present a comprehensive survey of the NCI domain, including a thorough review of pretraining techniques, tasks, datasets, and model architectures. We hope this paper will serve as a bridge between the natural language and programming language communities, offering insights for future research in this rapidly evolving field.

来自变压器（BERT）的双向编码器表示显示了各种NLP任务的奇妙改进，并且已经提出了其连续的变体来进一步提高预先训练的语言模型的性能。在本文中，我们的目标是首先介绍中国伯特的全文掩蔽（WWM）策略，以及一系列中国预培训的语言模型。然后我们还提出了一种简单但有效的型号，称为Macbert，这在几种方面提高了罗伯塔。特别是，我们提出了一种称为MLM作为校正（MAC）的新掩蔽策略。为了展示这些模型的有效性，我们创建了一系列中国预先培训的语言模型，作为我们的基线，包括BERT，Roberta，Electra，RBT等。我们对十个中国NLP任务进行了广泛的实验，以评估创建的中国人托管语言模型以及提议的麦克白。实验结果表明，Macbert可以在许多NLP任务上实现最先进的表演，我们还通过几种可能有助于未来的研究的调查结果来消融细节。我们开源我们的预先培训的语言模型，以进一步促进我们的研究界。资源可用：https://github.com/ymcui/chinese-bert-wwm

在过去的几年中，使用机器学习的编程语言处理（PLP）取得了广泛的改进。越来越多的人有兴趣探索这个有前途的领域。但是，对于新的研究人员和开发人员来说，要找到合适的组件来构建自己的机器学习管道，鉴于要解决的多样化的PLP任务，已发布大量数据集和模型，以及一组复杂的编译器或工具。涉及。为了改善机器学习组件的可发现性，可访问性，互操作性和可重复性（公平性），我们在基于机器学习的PLP领域中收集和分析了一组代表性论文。然后，我们确定并表征关键概念，包括PLP任务，模型架构和支持工具。最后，我们展示了一些利用可重复使用的组件来构建机器学习管道以解决一组PLP任务的例子。

本文提出了一种新的预先接受训练的语言模型Debertav3，它通过用更换的令牌检测（RTD）更换掩模语言建模（MLM）来改善原始的Deberta模型，更高的预训练任务。我们的分析表明，Vanilla嵌入了电力中的共享损害培训效率和模型性能。这是因为鉴别器的培训损失和发电机的销售损失在不同的方向上拉动令牌嵌入，从而创造“拔河”动态。因此，我们提出了一种新的梯度 - 解开嵌入共享方法，避免了战争动态，提高了训练效率和预训练模型的质量。我们使用与Deberta相同的设置预先接受了培训的Debertav3，以展示其在广泛的下游自然语言理解（NLU）任务上的特殊表现。以八个任务为例，Debertav3大型模型以八个任务为例，平均得分为91.37％，杜伯塔省的1.37％和电力1.91％，在模型中设置新的最先进（SOTA）具有类似的结构。此外，我们预先培训了多语思伯类Mdeberta，与英语模型相比，对强基线的更大改善。例如，Mdeberta基地达到XNLI的79.8％零射频精度和超过XLM-R基础的3.6％的改进，在此基准上创建了一个新的Sota。我们在HTTPS://github.com/microsoft/deberta公开提供我们预先接受的模型和推理码。

源代码的预训练的生成语言模型（例如PLBART，CODET5，SPT-CODE）在过去几年中对多个任务（包括代码生成和翻译）产生了强劲的结果。这些模型采用了不同的训练前目标，以自我监督的方式从非常大规模的语料库中学习代码构建的统计数据。预训练模型的成功很大程度上取决于这些预训练的目标。本文提出了一个新的预训练目标，即“归化”源代码，利用代码的双峰，双通道（正式和自然渠道）性质。与自然语言不同，代码的双峰，双通道的性质使我们能够大规模生成语义上等效的代码。我们介绍了六类的语义保存转换，以引入非自然的代码形式，然后强迫我们的模型制作开发人员编写的更自然的原创程序。学习在没有明确的手动监督的情况下，通过大型的开源代码来生成等效但更自然的代码，有助于模型学习摄入和生成代码。我们将模型在三个生成软件工程任务中微调：代码生成，代码翻译和代码改进，具有有限的人类策划标记数据并实现最先进的性能与CODET5。我们表明，我们的预训练模型在零射门和少数学习方面特别有竞争力，并且在学习代码属性（例如语法，数据流）方面更好。

最近，与“预训练，及时和预测”的新范式相比，与“预训练，微调”范式相比，新的范式“预训练，及时和预测”取得了显着的成就。在基于及时的GPT-3成功之后，一系列基于蒙版的语言模型（MLM）（例如Bert，Roberta）及时学习方法变得流行并广泛使用。但是，另一个有效的预训练的判别模型Electra可能被忽略了。在本文中，我们尝试使用拟议的替换代替令牌检测（RTD）基于基于的及时学习方法来完成零摄像的几个NLP任务。实验结果表明，基于RTD-Prompt学习的Electra模型可达到令人惊讶的最先进的零拍性能。在数字上，与MLM-Roberta-Large和MLM-Bert-Large相比，我们的RTD-Electra-Large在所有15个任务上平均提高了约8.4％和13.7％。特别是在SST-2任务上，我们的RTD-Electra-Large在没有任何培训数据的情况下达到了令人惊讶的90.1％精度。总体而言，与预先训练的蒙版语言模型相比，预先训练的代替令牌检测模型在零拍学习中的性能更好。因此，Electra是一位出色的零球学习者。源代码可在以下网址获得：https：//github.com/nishiwen1214/rtd-electra。

Recent progress in pre-trained neural language models has significantly improved the performance of many natural language processing (NLP) tasks. In this paper we propose a new model architecture DeBERTa (Decoding-enhanced BERT with disentangled attention) that improves the BERT and RoBERTa models using two novel techniques. The first is the disentangled attention mechanism, where each word is represented using two vectors that encode its content and position, respectively, and the attention weights among words are computed using disentangled matrices on their contents and relative positions, respectively. Second, an enhanced mask decoder is used to incorporate absolute positions in the decoding layer to predict the masked tokens in model pre-training. In addition, a new virtual adversarial training method is used for fine-tuning to improve models' generalization. We show that these techniques significantly improve the efficiency of model pre-training and the performance of both natural language understand (NLU) and natural langauge generation (NLG) downstream tasks. Compared to RoBERTa-Large, a DeBERTa model trained on half of the training data performs consistently better on a wide range of NLP tasks, achieving improvements on MNLI by +0.9% (90.2% vs. 91.1%), on SQuAD v2.0 by +2.3% (88.4% vs. 90.7%) and RACE by +3.6% (83.2% vs. 86.8%). Notably, we scale up DeBERTa by training a larger version that consists of 48 Transform layers with 1.5 billion parameters. The significant performance boost makes the single DeBERTa model surpass the human performance on the SuperGLUE benchmark (Wang et al., 2019a) for the first time in terms of macro-average score (89.9 versus 89.8), and the ensemble DeBERTa model sits atop the SuperGLUE leaderboard as of January 6, 2021, outperforming the human baseline by a decent margin (90.3 versus 89.8). The pre-trained DeBERTa models and the source code were released at: https://github.com/microsoft/DeBERTa 1 .

预训练模型已在许多代码智能任务中有效。这些模型在大规模未标记的语料库中进行了预训练，然后在下游任务中进行了微调。但是，由于预训练和下游任务的输入是不同的形式，因此很难充分探索预训练模型的知识。此外，微调的性能强烈依赖于下游数据的量，而实际上，具有稀缺数据的场景很常见。自然语言处理（NLP）领域的最新研究表明，迅速调整，一种调整的新范式，减轻上述问题并在各种NLP任务中实现了有希望的结果。在迅速调整中，在调整过程中插入的提示提供了特定于任务的知识，这对于具有相对较少数据的任务特别有益。在本文中，我们凭经验评估了代码智能任务中迅速调整的用法和效果。我们对流行的预训练模型Codebert和codet5进行及时调整，并尝试三个代码智能任务，包括缺陷预测，代码摘要和代码翻译。我们的实验结果表明，在所有三个任务中，迅速调整始终优于微调。此外，及时调整在低资源场景中显示出很大的潜力，例如，对于代码摘要，平均将微调的BLEU分数提高了26％以上。我们的结果表明，我们可以调整代码智能任务的迅速调整，以实现更好的性能，尤其是在缺乏特定于任务的数据时，我们可以调整及时调整。

近年来，在应用预训练的语言模型（例如Bert）上，取得了巨大进展，以获取信息检索（IR）任务。在网页中通常使用的超链接已被利用用于设计预训练目标。例如，超链接的锚文本已用于模拟查询，从而构建了巨大的查询文档对以进行预训练。但是，作为跨越两个网页的桥梁，尚未完全探索超链接的潜力。在这项工作中，我们专注于建模通过超链接连接的两个文档之间的关系，并为临时检索设计一个新的预训练目标。具体而言，我们将文档之间的关系分为四组：无链接，单向链接，对称链接和最相关的对称链接。通过比较从相邻组采样的两个文档，该模型可以逐渐提高其捕获匹配信号的能力。我们提出了一个渐进的超链接预测（{php}）框架，以探索预训练中超链接的利用。对两个大规模临时检索数据集和六个提问数据集的实验结果证明了其优于现有的预训练方法。

Software engineers working with the same programming language (PL) may speak different natural languages (NLs) and vice versa, erecting huge barriers to communication and working efficiency. Recent studies have demonstrated the effectiveness of generative pre-training in computer programs, yet they are always English-centric. In this work, we step towards bridging the gap between multilingual NLs and multilingual PLs for large language models (LLMs). We release ERNIE-Code, a unified pre-trained language model for 116 NLs and 6 PLs. We employ two methods for universal cross-lingual pre-training: span-corruption language modeling that learns patterns from monolingual NL or PL; and pivot-based translation language modeling that relies on parallel data of many NLs and PLs. Extensive results show that ERNIE-Code outperforms previous multilingual LLMs for PL or NL across a wide range of end tasks of code intelligence, including multilingual code-to-text, text-to-code, code-to-code, and text-to-text generation. We further show its advantage of zero-shot prompting on multilingual code summarization and text-to-text translation. We will make our code and pre-trained models publicly available.

基于变压器的大型语言模型在自然语言处理中表现出色。通过考虑这些模型在一个领域中获得的知识的可传递性，以及自然语言与高级编程语言（例如C/C ++）的亲密关系，这项工作研究了如何利用（大）基于变压器语言模型检测软件漏洞以及这些模型在漏洞检测任务方面的良好程度。在这方面，首先提出了一个系统的（凝聚）框架，详细介绍了源代码翻译，模型准备和推理。然后，使用具有多个漏洞的C/C ++源代码的软件漏洞数据集进行经验分析，该数据集对应于库功能调用，指针使用，数组使用情况和算术表达式。我们的经验结果证明了语言模型在脆弱性检测中的良好性能。此外，这些语言模型具有比当代模型更好的性能指标，例如F1得分，即双向长期记忆和双向封闭式复发单元。由于计算资源，平台，库和依赖项的要求，对语言模型进行实验始终是具有挑战性的。因此，本文还分析了流行的平台，以有效地微调这些模型并在选择平台时提出建议。