大型语言模型,例如OpenAI的法典和DeepMind的字母,可以生成代码来解决以自然语言表达的各种问题。这项技术已经在至少一项广泛使用的编程编辑器扩展程序中进行了商业化:Github Copilot。在本文中,我们探讨了具有大型语言模型(LLM辅助编程)的编程与程序员协助的先前概念化相似,并且与众不同。我们借鉴了公开可用的经验报告,有关LLM辅助编程以及先前的可用性和设计研究。我们发现,尽管LLM辅助编程通过搜索和重用分享了一些编译,配对编程和编程的属性,但技术可能性和实践经验都存在根本差异。因此,应该将LLM辅助编程视为具有自己独特的属性和挑战的新方法。最后,我们借鉴了用户研究的观察结果,在该观察中,非专家最终用户程序员使用LLM辅助工具来求解电子表格中的数据任务。我们讨论可能出现的问题,并在将大型语言模型应用于最终用户编程时,尤其是对于几乎没有编程专业知识的用户。
translated by 谷歌翻译
The introductory programming sequence has been the focus of much research in computing education. The recent advent of several viable and freely-available AI-driven code generation tools present several immediate opportunities and challenges in this domain. In this position paper we argue that the community needs to act quickly in deciding what possible opportunities can and should be leveraged and how, while also working on how to overcome or otherwise mitigate the possible challenges. Assuming that the effectiveness and proliferation of these tools will continue to progress rapidly, without quick, deliberate, and concerted efforts, educators will lose advantage in helping shape what opportunities come to be, and what challenges will endure. With this paper we aim to seed this discussion within the computing education community.
translated by 谷歌翻译
We are currently unable to specify human goals and societal values in a way that reliably directs AI behavior. Law-making and legal interpretation form a computational engine that converts opaque human values into legible directives. "Law Informs Code" is the research agenda capturing complex computational legal processes, and embedding them in AI. Similar to how parties to a legal contract cannot foresee every potential contingency of their future relationship, and legislators cannot predict all the circumstances under which their proposed bills will be applied, we cannot ex ante specify rules that provably direct good AI behavior. Legal theory and practice have developed arrays of tools to address these specification problems. For instance, legal standards allow humans to develop shared understandings and adapt them to novel situations. In contrast to more prosaic uses of the law (e.g., as a deterrent of bad behavior through the threat of sanction), leveraged as an expression of how humans communicate their goals, and what society values, Law Informs Code. We describe how data generated by legal processes (methods of law-making, statutory interpretation, contract drafting, applications of legal standards, legal reasoning, etc.) can facilitate the robust specification of inherently vague human goals. This increases human-AI alignment and the local usefulness of AI. Toward society-AI alignment, we present a framework for understanding law as the applied philosophy of multi-agent alignment. Although law is partly a reflection of historically contingent political power - and thus not a perfect aggregation of citizen preferences - if properly parsed, its distillation offers the most legitimate computational comprehension of societal values available. If law eventually informs powerful AI, engaging in the deliberative political process to improve law takes on even more meaning.
translated by 谷歌翻译
人类开发人员可以使用网络安全缺陷生产代码。可以新兴'智能'代码完成工具有助于修复这些缺点吗?在这项工作中,我们研究了对零拍摄漏洞修复的代码(如Openai的Codex和AI21的侏罗纪J-1)使用大型语言模型(如Openai的Codex和AI21的J-1)。我们调查设计方面的挑战,提示将Coax LLMS进入生成不安全代码的修复版本。由于许多方法来短语和句法 - 具有自然语言,这很困难。通过对四个商业,黑盒子,“现成的”典型的模型进行大规模研究,以及局部训练的模型,在合成,手工制作和现实世界的安全错误场景的混合中,我们的实验表明,LLMS可以共同修复100%的综合生成和手工制作的情景,以及58%的脆弱性,在真实的开源项目中的历史错误中选择。
translated by 谷歌翻译
在设计基于AI的系统中,有蓬勃发展的兴趣,以帮助人类设计计算系统,包括自动生成计算机代码的工具。这些最值得注意的是,以第一个自我描述的“Ai对程序员”,GitHub Copilot,一种在开源GitHub代码上培训的语言模型。但是,代码通常包含错误 - 因此,鉴于Copilot处理的大量未曝避代码,肯定是语言模型将从可利用的错误代码中学到。这提出了对Copilot代码捐助的安全的担忧。在这项工作中,我们系统地调查了可能导致Github CopIlot推荐不安全代码的普遍存在和条件。为了执行此分析,我们提示CopIlot在与高风险CWE相关的方案中生成代码(例如,从吉利的“前25名”列表中的方案)。我们探索了三个不同代码生成轴上的Copilot的表现 - 检查它如何表现为特定的弱点多样性,提示的多样性以及域的多样性。总共生产89个不同的Copilot方案,以完成,生产1,689个计划。其中,我们发现大约40%的脆弱。
translated by 谷歌翻译
Many real-world applications of language models (LMs), such as code autocomplete and writing assistance, involve human-LM interaction, but the main LM benchmarks are non-interactive, where a system produces output without human intervention. To evaluate human-LM interaction, we develop a framework, Human-AI Language-based Interaction Evaluation (H-LINE), that expands non-interactive evaluation along three dimensions, capturing (i) the interactive process, not only the final output; (ii) the first-person subjective experience, not just a third-party assessment; and (iii) notions of preference beyond quality. We then design five tasks ranging from goal-oriented to open-ended to capture different forms of interaction. On four state-of-the-art LMs (three variants of OpenAI's GPT-3 and AI21's J1-Jumbo), we find that non-interactive performance does not always result in better human-LM interaction and that first-person and third-party metrics can diverge, suggesting the importance of examining the nuances of human-LM interaction.
translated by 谷歌翻译
使用计算笔记本(例如,Jupyter Notebook),数据科学家根据他们的先前经验和外部知识(如在线示例)合理化他们的探索性数据分析(EDA)。对于缺乏关于数据集或问题的具体了解的新手或数据科学家,有效地获得和理解外部信息对于执行EDA至关重要。本文介绍了eDassistant,一个jupyterlab扩展,支持EDA的原位搜索示例笔记本电脑和有用的API的推荐,由搜索结果的新颖交互式可视化供电。代码搜索和推荐是由最先进的机器学习模型启用的,培训在线收集的EDA笔记本电脑的大型语料库。进行用户学习,以调查埃迪卡斯特和数据科学家的当前实践(即,使用外部搜索引擎)。结果证明了埃迪斯坦特的有效性和有用性,与会者赞赏其对EDA的顺利和环境支持。我们还报告了有关代码推荐工具的几种设计意义。
translated by 谷歌翻译
自然语言处理(NLP)已成为当前人工智能繁荣中的主要应用领域之一。转移学习已经启用了大量深入学习的神经网络,接受了语言建模任务,以大大提高了所有语言任务的性能。有趣的是,当模型培训使用包含软件代码的数据培训时,它们在从自然语言规范中生成功能计算机代码时展示了显着的能力。我们认为这是一种难题,用于神经模型为生成词组结构语法提供了一种替代理论,以说明语言有效。由于编程语言的语法由短语结构语法决定,因此成功的神经模型显然是对编程语言的理论基础的理论基础,以及通过扩展,自然语言来实现。我们认为语言模型的术语模型是误导性的,因为深度学习模型不是语言的理论模型,并提出采用语料库模型,这更好地反映了模型的成因和内容。
translated by 谷歌翻译
本次调查绘制了用于分析社交媒体数据的生成方法的研究状态的广泛的全景照片(Sota)。它填补了空白,因为现有的调查文章在其范围内或被约会。我们包括两个重要方面,目前正在挖掘和建模社交媒体的重要性:动态和网络。社会动态对于了解影响影响或疾病的传播,友谊的形成,友谊的形成等,另一方面,可以捕获各种复杂关系,提供额外的洞察力和识别否则将不会被注意的重要模式。
translated by 谷歌翻译
我们提出了一项探索性定性研究,以了解作家如何与下一页建议相互作用。尽管对建议系统对写作的影响进行了一些定量研究,但几乎没有定性的工作来理解作家如何与建议系统互动及其如何影响他们的写作过程 - 特别是针对非本地但英国作家的。我们进行了一项研究,要求业余作家分别写两部电影评论,一本没有建议。我们发现作家以各种复杂的方式与下一页建议互动 - 作家能够抽象建议的多个部分并将其纳入他们的写作中 - 即使他们不同意整个建议。建议系统对写作过程也有各种影响 - 以独特的方式为写作过程的不同方面做出了影响。我们提出了一种用于与GPT-2写作的作家 - 探索互动模型,用于电影评论写作任务,然后是该模型可用于未来研究的方式,并概述了研究和设计的机会。
translated by 谷歌翻译
我们介绍了Sparrow,这是一个寻求信息的对话代理,与提示的语言模型基线相比,训练有素,更有帮助,正确和无害。我们使用从人类反馈中的强化学习来培训我们的模型,以帮助人类评估者判断代理人的行为。首先,为了使我们的代理人更有帮助和无害,我们将良好对话的要求分解为代理人应遵循的自然语言规则,并分别向评估者询问每个规则。我们证明,这种崩溃使我们能够收集对代理行为的更多针对性的人类判断,并允许更有效的规则条件奖励模型。其次,我们的代理商在收集对模型声明的偏好判决时提供了支持事实主张的来源的证据。对于事实问题,麻雀提供的证据支持了78%的时间。比基线比基线更享受麻雀,同时对人类的对抗性探测更具弹性,在探测时只有8%的时间违反了我们的规则。最后,我们进行了广泛的分析,表明尽管我们的模型学会遵守我们的规则,但它可以表现出分布偏见。
translated by 谷歌翻译
Recent developments in natural language generation (NLG) using neural language models have brought us closer than ever to the goal of building AI-powered creative writing tools. However, most prior work on human-AI collaboration in the creative writing domain has evaluated new systems with amateur writers, typically in contrived user studies of limited scope. In this work, we commissioned 13 professional, published writers from a diverse set of creative writing backgrounds to craft stories using Wordcraft, a text editor with built-in AI-powered writing assistance tools. Using interviews and participant journals, we discuss the potential of NLG to have significant impact in the creative writing domain--especially with respect to brainstorming, generation of story details, world-building, and research assistance. Experienced writers, more so than amateurs, typically have well-developed systems and methodologies for writing, as well as distinctive voices and target audiences. Our work highlights the challenges in building for these writers; NLG technologies struggle to preserve style and authorial voice, and they lack deep understanding of story contents. In order for AI-powered writing assistants to realize their full potential, it is essential that they take into account the diverse goals and expertise of human writers.
translated by 谷歌翻译
本文探讨了大语言模型的自然语言生成能力,并应用于编程课程中常见的两种学习资源类型。使用OpenAI Codex作为大语言模型,我们创建编程练习(包括示例解决方案和测试用例)和代码说明,从定性和定量上评估这些练习。我们的结果表明,大多数自动生成的内容既新颖又明智,在某些情况下可以按原样使用。在创建练习时,我们发现仅通过提供关键字作为模型输入来影响编程概念和它们所包含的上下文主题非常容易。我们的分析表明,大规模生成机器学习模型是指导者的工具,尽管仍然需要进行一些监督以确保生成的内容的质量在传递给学生之前。我们进一步讨论了OpenAI Codex和类似工具对入门编程教育的含义,并强调了未来的研究流,这些研究流有可能提高教师和学生的教育体验质量。
translated by 谷歌翻译
如今,由于最近在人工智能(AI)和机器学习(ML)中的近期突破,因此,智能系统和服务越来越受欢迎。然而,机器学习不仅满足软件工程,不仅具有有希望的潜力,而且还具有一些固有的挑战。尽管最近的一些研究努力,但我们仍然没有明确了解开发基于ML的申请和当前行业实践的挑战。此外,目前尚不清楚软件工程研究人员应将其努力集中起来,以更好地支持ML应用程序开发人员。在本文中,我们报告了一个旨在了解ML应用程序开发的挑战和最佳实践的调查。我们合成从80名从业者(以不同的技能,经验和应用领域)获得的结果为17个调查结果;概述ML应用程序开发的挑战和最佳实践。参与基于ML的软件系统发展的从业者可以利用总结最佳实践来提高其系统的质量。我们希望报告的挑战将通知研究界有关需要调查的主题,以改善工程过程和基于ML的申请的质量。
translated by 谷歌翻译
Incivility remains a major challenge for online discussion platforms, to such an extent that even conversations between well-intentioned users can often derail into uncivil behavior. Traditionally, platforms have relied on moderators to -- with or without algorithmic assistance -- take corrective actions such as removing comments or banning users. In this work we propose a complementary paradigm that directly empowers users by proactively enhancing their awareness about existing tension in the conversation they are engaging in and actively guides them as they are drafting their replies to avoid further escalation. As a proof of concept for this paradigm, we design an algorithmic tool that provides such proactive information directly to users, and conduct a user study in a popular discussion platform. Through a mixed methods approach combining surveys with a randomized controlled experiment, we uncover qualitative and quantitative insights regarding how the participants utilize and react to this information. Most participants report finding this proactive paradigm valuable, noting that it helps them to identify tension that they may have otherwise missed and prompts them to further reflect on their own replies and to revise them. These effects are corroborated by a comparison of how the participants draft their reply when our tool warns them that their conversation is at risk of derailing into uncivil behavior versus in a control condition where the tool is disabled. These preliminary findings highlight the potential of this user-centered paradigm and point to concrete directions for future implementations.
translated by 谷歌翻译
Computational notebooks, such as Jupyter notebooks, are interactive computing environments that are ubiquitous among data scientists to perform data wrangling and analytic tasks. To measure the performance of AI pair programmers that automatically synthesize programs for those tasks given natural language (NL) intents from users, we build ARCADE, a benchmark of 1082 code generation problems using the pandas data analysis framework in data science notebooks. ARCADE features multiple rounds of NL-to-code problems from the same notebook. It requires a model to understand rich multi-modal contexts, such as existing notebook cells and their execution states as well as previous turns of interaction. To establish a strong baseline on this challenging task, we develop PaChiNCo, a 62B code language model (LM) for Python computational notebooks, which significantly outperforms public code LMs. Finally, we explore few-shot prompting strategies to elicit better code with step-by-step decomposition and NL explanation, showing the potential to improve the diversity and explainability of model predictions.
translated by 谷歌翻译
评论是源代码的重要组成部分,是文档的主要来源。这引起了人们对使用大量注释的兴趣训练或评估消耗或生产它们的工具,例如生成甲骨文,甚至是从注释中生成代码,或自动生成代码摘要。这项工作大部分对评论的结构和质量做出了强烈的假设,例如假设它们主要由适当的英语句子组成。但是,我们对这些用例的现有评论的实际质量知之甚少。评论通常包含在其他类型的文本中看不到的独特结构和元素,并且从中过滤或提取信息需要额外的谨慎。本文探讨了来自GitHub的840个最受欢迎的开源项目和Srilab数据集的8422个项目的Python评论的内容和质量,并且Na \“ Ive vs.深入过滤的影响都可以使用现有注释来用于使用现有注释。培训和评估产生评论的系统。
translated by 谷歌翻译
Drawing from the resources of psychoanalysis and critical media studies, in this paper we develop an analysis of Large Language Models (LLMs) as automated subjects. We argue the intentional fictional projection of subjectivity onto LLMs can yield an alternate frame through which AI behaviour, including its productions of bias and harm, can be analysed. First, we introduce language models, discuss their significance and risks, and outline our case for interpreting model design and outputs with support from psychoanalytic concepts. We trace a brief history of language models, culminating with the releases, in 2022, of systems that realise state-of-the-art natural language processing performance. We engage with one such system, OpenAI's InstructGPT, as a case study, detailing the layers of its construction and conducting exploratory and semi-structured interviews with chatbots. These interviews probe the model's moral imperatives to be helpful, truthful and harmless by design. The model acts, we argue, as the condensation of often competing social desires, articulated through the internet and harvested into training data, which must then be regulated and repressed. This foundational structure can however be redirected via prompting, so that the model comes to identify with, and transfer, its commitments to the immediate human subject before it. In turn, these automated productions of language can lead to the human subject projecting agency upon the model, effecting occasionally further forms of countertransference. We conclude that critical media methods and psychoanalytic theory together offer a productive frame for grasping the powerful new capacities of AI-driven language systems.
translated by 谷歌翻译
随着人工智能(AI)技术在社会中变得越来越强大和突出,他们的滥用就是日益关注的问题。在教育环境中,学生可以使用AI技术来欺骗作业和考试。在本文中,我们探讨了变形金刚是否可以用于求解介绍级的编程作业,同时绕过常用的AI工具来检测软件部分之间的相似性。我们发现使用GPT-J [Wang和Komatsuzaki,2021]的学生可以完成入门级的编程作业,而无需触发Moss的怀疑[Aiken,2000],这是一种广泛使用的软件相似性和窃探测工具。尽管事实上GPT-J没有接受有关问题的培训,也没有提供任何示例可供工作。我们进一步发现,GPT-J编写的代码在结构上是多种多样的,缺乏任何特定的告诉未来的pla窃检测技术可能会用来尝试识别算法生成的代码。最后,我们讨论了大语言模型的道德和教育含义以及未来研究的方向。
translated by 谷歌翻译
我们的工作是使用自然语言处理以自动化对物流至关重要的系统模型的开发的首次尝试。我们证明,基于变压器的语言模型的微型GPT-3 Codex的顶部建立的框架可以在给定口头描述的情况下对排队和库存控制系统进行功能有效的模拟。在进行的实验中,GPT-3 Codex在Python上表现出令人信服的专业知识,并了解了特定于域的词汇。结果,语言模型可以在给定特定于域的上下文,详细描述过程以及具有相应值的变量列表的情况下,生成单产品库存控制系统和单服务器排队系统的模拟。展示的结果以及语言模型的快速改进,为模拟模型开发背后的工作流程的重大简化打开了大门,这将使专家可以专注于对问题和整体思维的高级考虑。
translated by 谷歌翻译