来自科幻小说的普通愿景是机器人将有一天居住在我们的物理空间中，感知世界，才能协助我们的物理劳动力，并通过自然语言与我们沟通。在这里，我们研究如何使用虚拟环境的简化设计如何与人类自然交互的人工代理。我们表明，与自我监督学习的模拟世界中的人类交互的模仿学习足以产生我们称之为MIA的多模式互动剂，这成功与非对抗人类互动75％的时间。我们进一步确定了提高性能的架构和算法技术，例如分层动作选择。完全，我们的结果表明，模仿多模态，实时人类行为可以提供具有丰富的行为的富含性的令人生意的和令人惊讶的有效手段，然后可以为特定目的进行微调，从而铺设基础用于培训互动机器人或数字助理的能力。可以在https://youtu.be/zfgrif7my找到MIA的行为的视频

创建可以自然与人类互动的代理是人工智能（AI）研究中的共同目标。但是，评估这些互动是具有挑战性的：收集在线人类代理相互作用缓慢而昂贵，但更快的代理指标通常与交互式评估相关。在本文中，我们评估了这些现有评估指标的优点，并提出了一种新颖的评估方法，称为标准化测试套件（STS）。 STS使用从真实人类交互数据中挖掘出的行为方案。代理商请参阅重播方案上下文，接收指令，然后将控制权控制以脱机完成交互。记录这些代理的延续并将其发送给人类注释者以将其标记为成功或失败，并且根据其成功的连续性比例对代理进行排名。最终的ST是自然主义相互作用的快速，控制，可解释的和代表的。总的来说，STS巩固了我们许多标准评估指标中所需的许多值，从而使我们能够加速研究进展，以生产可以自然与人类互动的代理。可以在https://youtu.be/yr1tnggorgq上找到视频。

Inspired by progress in large-scale language modeling, we apply a similar approach towards building a single generalist agent beyond the realm of text outputs. The agent, which we refer to as Gato, works as a multi-modal, multi-task, multi-embodiment generalist policy. The same network with the same weights can play Atari, caption images, chat, stack blocks with a real robot arm and much more, deciding based on its context whether to output text, joint torques, button presses, or other tokens. In this report we describe the model and the data, and document the current capabilities of Gato.

We present a retrospective on the state of Embodied AI research. Our analysis focuses on 13 challenges presented at the Embodied AI Workshop at CVPR. These challenges are grouped into three themes: (1) visual navigation, (2) rearrangement, and (3) embodied vision-and-language. We discuss the dominant datasets within each theme, evaluation metrics for the challenges, and the performance of state-of-the-art models. We highlight commonalities between top approaches to the challenges and identify potential future directions for Embodied AI research.

By transferring knowledge from large, diverse, task-agnostic datasets, modern machine learning models can solve specific downstream tasks either zero-shot or with small task-specific datasets to a high level of performance. While this capability has been demonstrated in other fields such as computer vision, natural language processing or speech recognition, it remains to be shown in robotics, where the generalization capabilities of the models are particularly critical due to the difficulty of collecting real-world robotic data. We argue that one of the keys to the success of such general robotic models lies with open-ended task-agnostic training, combined with high-capacity architectures that can absorb all of the diverse, robotic data. In this paper, we present a model class, dubbed Robotics Transformer, that exhibits promising scalable model properties. We verify our conclusions in a study of different model classes and their ability to generalize as a function of the data size, model size, and data diversity based on a large-scale data collection on real robots performing real-world tasks. The project's website and videos can be found at robotics-transformer.github.io

It would be useful for machines to use computers as humans do so that they can aid us in everyday tasks. This is a setting in which there is also the potential to leverage large-scale expert demonstrations and human judgements of interactive behaviour, which are two ingredients that have driven much recent success in AI. Here we investigate the setting of computer control using keyboard and mouse, with goals specified via natural language. Instead of focusing on hand-designed curricula and specialized action spaces, we focus on developing a scalable method centered on reinforcement learning combined with behavioural priors informed by actual human-computer interactions. We achieve state-of-the-art and human-level mean performance across all tasks within the MiniWob++ benchmark, a challenging suite of computer control problems, and find strong evidence of cross-task transfer. These results demonstrate the usefulness of a unified human-agent interface when training machines to use computers. Altogether our results suggest a formula for achieving competency beyond MiniWob++ and towards controlling computers, in general, as a human would.

建立能够参与与人类社会互动的自治代理是AI的主要挑战之一。在深度加强学习（DRL）领域内，这一目标激励了多种作品上体现语言使用。然而，目前的方法在非常简化和非多样化的社交场合中关注语言作为通信工具：语言的“自然”减少到高词汇大小和变异性的概念。在本文中，我们认为针对人类级别的AI需要更广泛的关键社交技能：1）语言在复杂和可变的社会环境中使用; 2）超越语言，在不断发展的社会世界内的多模式设置中的复杂体现通信。我们解释了认知科学的概念如何帮助AI向人类智力绘制路线图，重点关注其社会方面。作为第一步，我们建议将目前的研究扩大到更广泛的核心社交技能。为此，我们展示了使用其他（脚本）社会代理商的多个网格世界环境来评估DRL代理商社交技能的基准。然后，我们研究了最近的Sota DRL方法的限制，当时在Sowisai上进行测试并讨论熟练社会代理商的重要下一步。视频和代码可在https://sites.google.com/view/socialai找到。

在嘈杂的互联网规模数据集上进行了预测，已对具有广泛的文本，图像和其他模式能力的培训模型进行了大量研究。但是，对于许多顺序决策域，例如机器人技术，视频游戏和计算机使用，公开可用的数据不包含以相同方式训练行为先验所需的标签。我们通过半监督的模仿学习将互联网规模的预处理扩展到顺序的决策域，其中代理通过观看在线未标记的视频来学习行动。具体而言，我们表明，使用少量标记的数据，我们可以训练一个足够准确的反向动力学模型，可以标记一个巨大的未标记在线数据来源 - 在这里，在线播放Minecraft的在线视频 - 然后我们可以从中训练一般行为先验。尽管使用了本地人类界面（鼠标和键盘为20Hz），但我们表明，这种行为先验具有非平凡的零射击功能，并且可以通过模仿学习和加强学习，可以对其进行微调，以进行硬探索任务。不可能通过增强学习从头开始学习。对于许多任务，我们的模型都表现出人类水平的性能，我们是第一个报告可以制作钻石工具的计算机代理，这些工具可以花费超过20分钟（24,000个环境动作）的游戏玩法来实现。

Our situated environment is full of uncertainty and highly dynamic, thus hindering the widespread adoption of machine-led Intelligent Decision-Making (IDM) in real world scenarios. This means IDM should have the capability of continuously learning new skills and efficiently generalizing across wider applications. IDM benefits from any new approaches and theoretical breakthroughs that exhibit Artificial General Intelligence (AGI) breaking the barriers between tasks and applications. Recent research has well-examined neural architecture, Transformer, as a backbone foundation model and its generalization to various tasks, including computer vision, natural language processing, and reinforcement learning. We therefore argue that a foundation decision model (FDM) can be established by formulating various decision-making tasks as a sequence decoding task using the Transformer architecture; this would be a promising solution to advance the applications of IDM in more complex real world tasks. In this paper, we elaborate on how a foundation decision model improves the efficiency and generalization of IDM. We also discuss potential applications of a FDM in multi-agent game AI, production scheduling, and robotics tasks. Finally, through a case study, we demonstrate our realization of the FDM, DigitalBrain (DB1) with 1.2 billion parameters, which achieves human-level performance over 453 tasks, including text generation, images caption, video games playing, robotic control, and traveling salesman problems. As a foundation decision model, DB1 would be a baby step towards more autonomous and efficient real world IDM applications.

人类语言学习者暴露于信息丰富的上下文敏感语言，但要大量的原始感觉数据。通过社会语言的使用和彩排和实践的内部过程，语言学习者能够建立高级的语义表示，以解释他们的看法。在这里，我们从人类中的“内在语音”过程中汲取灵感（Vygotsky，1934年），以更好地理解代理内语言在体现行为中的作用。首先，我们正式地将代理语音作为半监督问题，并开发了两种算法，这些算法能够以很少的标记语言数据进行视觉接地字幕。然后，我们通过实验计算不同量的标记数据的缩放曲线，并将数据效率与监督学习基线进行比较。最后，我们将演讲内部的语音纳入3D虚拟世界中运行的体现的移动操纵剂代理，并表明，只需多达150个附加图像标题，代理语音就可以操纵和回答有关的问题。一个没有任何相关任务经验的新对象（零射）。综上所述，我们的实验表明，对代理内部的语音进行建模有效，可以使体现的代理有效地学习新任务，而无需直接互动经验。

从“Internet AI”的时代到“体现AI”的时代，AI算法和代理商出现了一个新兴范式转变，其中不再从主要来自Internet策划的图像，视频或文本的数据集。相反，他们通过与与人类类似的Enocentric感知来通过与其环境的互动学习。因此，对体现AI模拟器的需求存在大幅增长，以支持各种体现的AI研究任务。这种越来越多的体现AI兴趣是有利于对人工综合情报（AGI）的更大追求，但对这一领域并无一直存在当代和全面的调查。本文旨在向体现AI领域提供百科全书的调查，从其模拟器到其研究。通过使用我们提出的七种功能评估九个当前体现的AI模拟器，旨在了解模拟器，以其在体现AI研究和其局限性中使用。最后，本文调查了体现AI - 视觉探索，视觉导航和体现问题的三个主要研究任务（QA），涵盖了最先进的方法，评估指标和数据集。最后，随着通过测量该领域的新见解，本文将为仿真器 - 任务选择和建议提供关于该领域的未来方向的建议。

与人类在环境中共存的通用机器人必须学会将人类语言与其在一系列日常任务中有用的看法和行动联系起来。此外，他们需要获取各种曲目的一般专用技能，允许通过遵循无约束语言指示来组成长地平任务。在本文中，我们呈现了凯文（从语言和愿景撰写的行动），是一个露天模拟基准，用于学习Long-Horizo​​ n语言条件的任务。我们的目的是使可以开发能够通过船上传感器解决许多机器人操纵任务的代理商，并且仅通过人类语言指定。 Calvin任务在序列长度，动作空间和语言方面更复杂，而不是现有的视觉和语言任务数据集，并支持灵活的传感器套件规范。我们评估零拍摄的代理商以新颖的语言指示以及新的环境和对象。我们表明，基于多语境模仿学习的基线模型在凯文中表现不佳，表明有很大的空间，用于开发创新代理，了解学习将人类语言与这款基准相关的世界模型。

Transformer, originally devised for natural language processing, has also attested significant success in computer vision. Thanks to its super expressive power, researchers are investigating ways to deploy transformers to reinforcement learning (RL) and the transformer-based models have manifested their potential in representative RL benchmarks. In this paper, we collect and dissect recent advances on transforming RL by transformer (transformer-based RL or TRL), in order to explore its development trajectory and future trend. We group existing developments in two categories: architecture enhancement and trajectory optimization, and examine the main applications of TRL in robotic manipulation, text-based games, navigation and autonomous driving. For architecture enhancement, these methods consider how to apply the powerful transformer structure to RL problems under the traditional RL framework, which model agents and environments much more precisely than deep RL methods, but they are still limited by the inherent defects of traditional RL algorithms, such as bootstrapping and "deadly triad". For trajectory optimization, these methods treat RL problems as sequence modeling and train a joint state-action model over entire trajectories under the behavior cloning framework, which are able to extract policies from static datasets and fully use the long-sequence modeling capability of the transformer. Given these advancements, extensions and challenges in TRL are reviewed and proposals about future direction are discussed. We hope that this survey can provide a detailed introduction to TRL and motivate future research in this rapidly developing field.

Recent progress in artificial intelligence (AI) has renewed interest in building systems that learn and think like people. Many advances have come from using deep neural networks trained end-to-end in tasks such as object recognition, video games, and board games, achieving performance that equals or even beats humans in some respects. Despite their biological inspiration and performance achievements, these systems differ from human intelligence in crucial ways. We review progress in cognitive science suggesting that truly human-like learning and thinking machines will have to reach beyond current engineering trends in both what they learn, and how they learn it. Specifically, we argue that these machines should (a) build causal models of the world that support explanation and understanding, rather than merely solving pattern recognition problems; (b) ground learning in intuitive theories of physics and psychology, to support and enrich the knowledge that is learned; and (c) harness compositionality and learning-to-learn to rapidly acquire and generalize knowledge to new tasks and situations. We suggest concrete challenges and promising routes towards these goals that can combine the strengths of recent neural network advances with more structured cognitive models.

机器人技术中的一个长期目标是建立可以从使用其板载传感器获得的感知中执行各种日常任务的机器人，并且仅通过自然语言指定。尽管最近通过利用从像素的端到端学习来实现了在语言驱动的机器人技术中的实质性进步，但由于设置的基本差异，没有明确且妥善理解的过程来做出各种设计选择。在本文中，我们对从离线自由模仿数据集中学习语言条件政策的最关键挑战进行了广泛的研究。我们进一步确定了改善性能的架构和算法技术，例如机器人控制学习的层次分解，多模式变压器编码器，离散的潜在计划以及与视频和语言表示一致的自我监视的对比损失。通过将调查的结果与改进的模型组件相结合，我们能够提出一种新颖的方法，该方法在具有挑战性的语言条件长的长摩托器机器人操纵Calvin基准上大大优于最新技术。我们已经开源的实施方式，以促进未来的研究，以学习自然语言连续指定的许多复杂的操纵技能。 http://hulc.cs.uni-freiburg.de可用代码库和训练有素的模型

最近围绕语言处理模型的复杂性的最新炒作使人们对机器获得了类似人类自然语言的指挥的乐观情绪。人工智能中自然语言理解的领域声称在这一领域取得了长足的进步，但是，在这方面和其他学科中使用“理解”的概念性清晰，使我们很难辨别我们实际上有多近的距离。目前的方法和剩余挑战的全面，跨学科的概述尚待进行。除了语言知识之外，这还需要考虑我们特定于物种的能力，以对，记忆，标签和传达我们（足够相似的）体现和位置经验。此外，测量实际约束需要严格分析当前模型的技术能力，以及对理论可能性和局限性的更深入的哲学反思。在本文中，我将所有这些观点（哲学，认知语言和技术）团结在一起，以揭开达到真实（人类般的）语言理解所涉及的挑战。通过解开当前方法固有的理论假设，我希望说明我们距离实现这一目标的实际程度，如果确实是目标。

与人类合作需要迅速适应他们的个人优势，缺点和偏好。遗憾的是，大多数标准的多智能经纪增强学习技术，如自助（SP）或人口剧（PP），产生培训合作伙伴的代理商，并且对人类不完全概括。或者，研究人员可以使用行为克隆收集人体数据，培训人类模型，然后使用该模型培训“人类感知”代理（“行为克隆播放”或BCP）。虽然这种方法可以改善代理商的概括到新的人类共同球员，但它涉及首先收集大量人体数据的繁重和昂贵的步骤。在这里，我们研究如何培训与人类合作伙伴合作的代理的问题，而无需使用人类数据。我们认为这个问题的症结是制作各种培训伙伴。从竞争域中取得成功的多智能经纪人方法绘制灵感，我们发现令人惊讶的简单方法非常有效。我们培养我们的代理商合作伙伴作为对自行发行代理人口的最佳反应及其过去培训的过去检查点，这是我们呼叫虚构共同扮演（FCP）的方法。我们的实验专注于两位运动员协作烹饪模拟器，最近被提议作为与人类协调的挑战问题。我们发现，与新的代理商和人类合作伙伴配对时，FCP代理商会显着高于SP，PP和BCP。此外，人类还报告了强烈的主观偏好，以与所有基线与FCP代理合作。

大型语言模型可以编码有关世界的大量语义知识。这种知识对于旨在采取自然语言表达的高级，时间扩展的指示的机器人可能非常有用。但是，语言模型的一个重大弱点是，它们缺乏现实世界的经验，这使得很难利用它们在给定的体现中进行决策。例如，要求语言模型描述如何清洁溢出物可能会导致合理的叙述，但是它可能不适用于需要在特定环境中执行此任务的特定代理商（例如机器人）。我们建议通过预处理的技能来提供现实世界的基础，这些技能用于限制模型以提出可行且在上下文上适当的自然语言动作。机器人可以充当语​​言模型的“手和眼睛”，而语言模型可以提供有关任务的高级语义知识。我们展示了如何将低级技能与大语言模型结合在一起，以便语言模型提供有关执行复杂和时间扩展说明的过程的高级知识，而与这些技能相关的价值功能则提供了连接必要的基础了解特定的物理环境。我们在许多现实世界的机器人任务上评估了我们的方法，我们表明了对现实世界接地的需求，并且这种方法能够在移动操纵器上完成长远，抽象的自然语言指令。该项目的网站和视频可以在https://say-can.github.io/上找到。

Imitation learning techniques aim to mimic human behavior in a given task. An agent (a learning machine) is trained to perform a task from demonstrations by learning a mapping between observations and actions. The idea of teaching by imitation has been around for many years, however, the field is gaining attention recently due to advances in computing and sensing as well as rising demand for intelligent applications. The paradigm of learning by imitation is gaining popularity because it facilitates teaching complex tasks with minimal expert knowledge of the tasks. Generic imitation learning methods could potentially reduce the problem of teaching a task to that of providing demonstrations; without the need for explicit programming or designing reward functions specific to the task. Modern sensors are able to collect and transmit high volumes of data rapidly, and processors with high computational power allow fast processing that maps the sensory data to actions in a timely manner. This opens the door for many potential AI applications that require real-time perception and reaction such as humanoid robots, self-driving vehicles, human computer interaction and computer games to name a few. However, specialized algorithms are needed to effectively and robustly learn models as learning by imitation poses its own set of challenges. In this paper, we survey imitation learning methods and present design options in different steps of the learning process. We introduce a background and motivation for the field as well as highlight challenges specific to the imitation problem. Methods for designing and evaluating imitation learning tasks are categorized and reviewed. Special attention is given to learning methods in robotics and games as these domains are the most popular in the literature and provide a wide array of problems and methodologies. We extensively discuss combining imitation learning approaches using different sources and methods, as well as incorporating other motion learning methods to enhance imitation. We also discuss the potential impact on industry, present major applications and highlight current and future research directions.

While large-scale sequence modeling from offline data has led to impressive performance gains in natural language and image generation, directly translating such ideas to robotics has been challenging. One critical reason for this is that uncurated robot demonstration data, i.e. play data, collected from non-expert human demonstrators are often noisy, diverse, and distributionally multi-modal. This makes extracting useful, task-centric behaviors from such data a difficult generative modeling problem. In this work, we present Conditional Behavior Transformers (C-BeT), a method that combines the multi-modal generation ability of Behavior Transformer with future-conditioned goal specification. On a suite of simulated benchmark tasks, we find that C-BeT improves upon prior state-of-the-art work in learning from play data by an average of 45.7%. Further, we demonstrate for the first time that useful task-centric behaviors can be learned on a real-world robot purely from play data without any task labels or reward information. Robot videos are best viewed on our project website: https://play-to-policy.github.io