We present a retrospective on the state of Embodied AI research. Our analysis focuses on 13 challenges presented at the Embodied AI Workshop at CVPR. These challenges are grouped into three themes: (1) visual navigation, (2) rearrangement, and (3) embodied vision-and-language. We discuss the dominant datasets within each theme, evaluation metrics for the challenges, and the performance of state-of-the-art models. We highlight commonalities between top approaches to the challenges and identify potential future directions for Embodied AI research.

从“Internet AI”的时代到“体现AI”的时代，AI算法和代理商出现了一个新兴范式转变，其中不再从主要来自Internet策划的图像，视频或文本的数据集。相反，他们通过与与人类类似的Enocentric感知来通过与其环境的互动学习。因此，对体现AI模拟器的需求存在大幅增长，以支持各种体现的AI研究任务。这种越来越多的体现AI兴趣是有利于对人工综合情报（AGI）的更大追求，但对这一领域并无一直存在当代和全面的调查。本文旨在向体现AI领域提供百科全书的调查，从其模拟器到其研究。通过使用我们提出的七种功能评估九个当前体现的AI模拟器，旨在了解模拟器，以其在体现AI研究和其局限性中使用。最后，本文调查了体现AI - 视觉探索，视觉导航和体现问题的三个主要研究任务（QA），涵盖了最先进的方法，评估指标和数据集。最后，随着通过测量该领域的新见解，本文将为仿真器 - 任务选择和建议提供关于该领域的未来方向的建议。

We present Habitat, a platform for research in embodied artificial intelligence (AI). Habitat enables training embodied agents (virtual robots) in highly efficient photorealistic 3D simulation. Specifically, Habitat consists of: (i) Habitat-Sim: a flexible, high-performance 3D simulator with configurable agents, sensors, and generic 3D dataset handling. Habitat-Sim is fast -when rendering a scene from Matterport3D, it achieves several thousand frames per second (fps) running single-threaded, and can reach over 10,000 fps multi-process on a single GPU. (ii) Habitat-API: a modular high-level library for end-toend development of embodied AI algorithms -defining tasks (e.g. navigation, instruction following, question answering), configuring, training, and benchmarking embodied agents.These large-scale engineering contributions enable us to answer scientific questions requiring experiments that were till now impracticable or 'merely' impractical. Specifically, in the context of point-goal navigation: (1) we revisit the comparison between learning and SLAM approaches from two recent works [20,16] and find evidence for the opposite conclusion -that learning outperforms SLAM if scaled to an order of magnitude more experience than previous investigations, and (2) we conduct the first cross-dataset generalization experiments {train, test} × {Matterport3D, Gibson} for multiple sensors {blind, RGB, RGBD, D} and find that only agents with depth (D) sensors generalize across datasets. We hope that our open-source platform and these findings will advance research in embodied AI.

对象看起来和声音的方式提供了对其物理特性的互补反射。在许多设置中，视觉和试听的线索都异步到达，但必须集成，就像我们听到一个物体掉落在地板上，然后必须找到它时。在本文中，我们介绍了一个设置，用于研究3D虚拟环境中的多模式对象定位。一个物体在房间的某个地方掉落。配备了摄像头和麦克风的具体机器人剂必须通过将音频和视觉信号与知识的基础物理学结合来确定已删除的对象以及位置。为了研究此问题，我们生成了一个大规模数据集 - 倒下的对象数据集 - 其中包括64个房间中30个物理对象类别的8000个实例。该数据集使用Threedworld平台，该平台可以模拟基于物理的影响声音和在影片设置中对象之间的复杂物理交互。作为解决这一挑战的第一步，我们基于模仿学习，强化学习和模块化计划，开发了一组具体的代理基线，并对这项新任务的挑战进行了深入的分析。

Semantic navigation is necessary to deploy mobile robots in uncontrolled environments like our homes, schools, and hospitals. Many learning-based approaches have been proposed in response to the lack of semantic understanding of the classical pipeline for spatial navigation, which builds a geometric map using depth sensors and plans to reach point goals. Broadly, end-to-end learning approaches reactively map sensor inputs to actions with deep neural networks, while modular learning approaches enrich the classical pipeline with learning-based semantic sensing and exploration. But learned visual navigation policies have predominantly been evaluated in simulation. How well do different classes of methods work on a robot? We present a large-scale empirical study of semantic visual navigation methods comparing representative methods from classical, modular, and end-to-end learning approaches across six homes with no prior experience, maps, or instrumentation. We find that modular learning works well in the real world, attaining a 90% success rate. In contrast, end-to-end learning does not, dropping from 77% simulation to 23% real-world success rate due to a large image domain gap between simulation and reality. For practitioners, we show that modular learning is a reliable approach to navigate to objects: modularity and abstraction in policy design enable Sim-to-Real transfer. For researchers, we identify two key issues that prevent today's simulators from being reliable evaluation benchmarks - (A) a large Sim-to-Real gap in images and (B) a disconnect between simulation and real-world error modes - and propose concrete steps forward.

我们介绍了栖息地2.0（H2.0），这是一个模拟平台，用于培训交互式3D环境和复杂物理的场景中的虚拟机器人。我们为体现的AI堆栈 - 数据，仿真和基准任务做出了全面的贡献。具体来说，我们提出：（i）复制：一个由艺术家的，带注释的，可重新配置的3D公寓（匹配真实空间）与铰接对象（例如可以打开/关闭的橱柜和抽屉）； （ii）H2.0：一个高性能物理学的3D模拟器，其速度超过8-GPU节点上的每秒25,000个模拟步骤（实时850x实时），代表先前工作的100倍加速；和（iii）家庭助理基准（HAB）：一套辅助机器人（整理房屋，准备杂货，设置餐桌）的一套常见任务，以测试一系列移动操作功能。这些大规模的工程贡献使我们能够系统地比较长期结构化任务中的大规模加固学习（RL）和经典的感官平面操作（SPA）管道，并重点是对新对象，容器和布局的概括。 。我们发现（1）与层次结构相比，（1）平面RL政策在HAB上挣扎； （2）具有独立技能的层次结构遭受“交接问题”的困扰，（3）水疗管道比RL政策更脆。

我们介绍了互动室（Thor），这是一个视觉AI研究的框架，可在http://ai2thor.allenai.org上找到。AI2-这是由几乎逼真的3D室内场景组成的，在该场景中，AI代理可以在场景中导航并与对象进行交互以执行任务。AI2-这可以在许多不同的领域进行研究，包括但不限于深入强化学习，模仿学习，通过互动，计划，视觉问答答案，无监督的表示学习，对象检测和细分以及认知模型。AI2的目的是促进构建视觉上智能模型，并将研究推向该领域。

最近的作品表明，如何将大语言模型（LLM）的推理能力应用于自然语言处理以外的领域，例如机器人的计划和互动。这些具体的问题要求代理商了解世界上许多语义方面：可用技能的曲目，这些技能如何影响世界以及对世界的变化如何映射回该语言。在体现环境中规划的LLMS不仅需要考虑要做什么技能，还需要考虑如何以及何时进行操作 - 答案随着时间的推移而变化，以响应代理商自己的选择。在这项工作中，我们调查了在这种体现的环境中使用的LLM在多大程度上可以推论通过自然语言提供的反馈来源，而无需任何其他培训。我们建议，通过利用环境反馈，LLM能够形成内部独白，使他们能够在机器人控制方案中进行更丰富的处理和计划。我们研究了各种反馈来源，例如成功检测，场景描述和人类互动。我们发现，闭环语言反馈显着改善了三个领域的高级指导完成，包括模拟和真实的桌面顶部重新排列任务以及现实世界中厨房环境中的长途移动操作任务。

最近的视听导航工作是无噪音音频环境中的单一静态声音，并努力推广到闻名声音。我们介绍了一种新型动态视听导航基准测试，其中一个体现的AI代理必须在存在分散的人和嘈杂的声音存在下在未映射的环境中捕获移动声源。我们提出了一种依赖于多模态架构的端到端增强学习方法，该方法依赖于融合来自双耳音频信号和空间占用映射的空间视听信息，以编码为我们的新的稳健导航策略进行编码所需的功能复杂的任务设置。我们展示了我们的方法优于当前的最先进状态，以更好地推广到闻名声音以及对嘈杂的3D扫描现实世界数据集副本和TASTPORT3D上的嘈杂情景更好地对嘈杂的情景进行了更好的稳健性，以实现静态和动态的视听导航基准。我们的小型基准将在http://dav-nav.cs.uni-freiburg.de提供。

大量数据集和高容量模型推动了计算机视觉和自然语言理解方面的许多最新进步。这项工作提出了一个平台，可以在体现的AI中实现类似的成功案例。我们提出了Procthor，这是一个程序生成体现的AI环境的框架。 Procthor使我们能够采样多种，交互式，可自定义和性能的虚拟环境的任意大型数据集，以训练和评估在导航，互动和操纵任务中的体现代理。我们通过10,000个生成的房屋和简单的神经模型的样本来证明procthor的能力和潜力。仅在Procthor上仅使用RGB图像训练的模型，没有明确的映射，并且没有人类任务监督在6个体现的AI基准中产生最先进的结果，用于导航，重排和手臂操纵，包括目前正在运行的Habitat 2022，AI2-- Thor重新安排2022，以及机器人挑战。我们还通过对procthor进行预训练，在下游基准测试上没有进行微调，通常会击败以前的最先进的系统，从而访问下游训练数据。

我们介绍了ThreedWorld（TDW），是交互式多模态物理模拟的平台。 TDW能够模拟高保真感官数据和富裕的3D环境中的移动代理和对象之间的物理交互。独特的属性包括：实时近光 - 真实图像渲染;对象和环境库，以及他们定制的例程;有效构建新环境课程的生成程序;高保真音频渲染;各种材料类型的现实物理相互作用，包括布料，液体和可变形物体;可定制的代理体现AI代理商;并支持与VR设备的人类交互。 TDW的API使多个代理能够在模拟中进行交互，并返回一系列表示世界状态的传感器和物理数据。我们在计算机视觉，机器学习和认知科学中的新兴的研究方向上提供了通过TDW的初始实验，包括多模态物理场景理解，物理动态预测，多代理交互，像孩子一样学习的模型，并注意研究人类和神经网络。

与人类在环境中共存的通用机器人必须学会将人类语言与其在一系列日常任务中有用的看法和行动联系起来。此外，他们需要获取各种曲目的一般专用技能，允许通过遵循无约束语言指示来组成长地平任务。在本文中，我们呈现了凯文（从语言和愿景撰写的行动），是一个露天模拟基准，用于学习Long-Horizo​​ n语言条件的任务。我们的目的是使可以开发能够通过船上传感器解决许多机器人操纵任务的代理商，并且仅通过人类语言指定。 Calvin任务在序列长度，动作空间和语言方面更复杂，而不是现有的视觉和语言任务数据集，并支持灵活的传感器套件规范。我们评估零拍摄的代理商以新颖的语言指示以及新的环境和对象。我们表明，基于多语境模仿学习的基线模型在凯文中表现不佳，表明有很大的空间，用于开发创新代理，了解学习将人类语言与这款基准相关的世界模型。

Recent progress in artificial intelligence (AI) has renewed interest in building systems that learn and think like people. Many advances have come from using deep neural networks trained end-to-end in tasks such as object recognition, video games, and board games, achieving performance that equals or even beats humans in some respects. Despite their biological inspiration and performance achievements, these systems differ from human intelligence in crucial ways. We review progress in cognitive science suggesting that truly human-like learning and thinking machines will have to reach beyond current engineering trends in both what they learn, and how they learn it. Specifically, we argue that these machines should (a) build causal models of the world that support explanation and understanding, rather than merely solving pattern recognition problems; (b) ground learning in intuitive theories of physics and psychology, to support and enrich the knowledge that is learned; and (c) harness compositionality and learning-to-learn to rapidly acquire and generalize knowledge to new tasks and situations. We suggest concrete challenges and promising routes towards these goals that can combine the strengths of recent neural network advances with more structured cognitive models.

这项工作研究了图像目标导航问题，需要通过真正拥挤的环境引导具有嘈杂传感器和控制的机器人。最近的富有成效的方法依赖于深度加强学习，并学习模拟环境中的导航政策，这些环境比真实环境更简单。直接将这些训练有素的策略转移到真正的环境可能非常具有挑战性甚至危险。我们用由四个解耦模块组成的分层导航方法来解决这个问题。第一模块在机器人导航期间维护障碍物映射。第二个将定期预测实时地图上的长期目标。第三个计划碰撞命令集以导航到长期目标，而最终模块将机器人正确靠近目标图像。四个模块是单独开发的，以适应真实拥挤的情景中的图像目标导航。此外，分层分解对导航目标规划，碰撞避免和导航结束预测的学习进行了解耦，这在导航训练期间减少了搜索空间，并有助于改善以前看不见的真实场景的概括。我们通过移动机器人评估模拟器和现实世界中的方法。结果表明，我们的方法优于多种导航基线，可以在这些方案中成功实现导航任务。

移动机器人的视觉导航经典通过SLAM加上最佳规划，最近通过实现作为深网络的端到端培训。虽然前者通常仅限于航点计划，但即使在真实的物理环境中已经证明了它们的效率，后一种解决方案最常用于模拟中，但已被证明能够学习更复杂的视觉推理，涉及复杂的语义规则。通过实际机器人在物理环境中导航仍然是一个开放问题。端到端的培训方法仅在模拟中进行了彻底测试，实验涉及实际机器人的实际机器人在简化的实验室条件下限制为罕见的性能评估。在这项工作中，我们对真实物理代理的性能和推理能力进行了深入研究，在模拟中培训并部署到两个不同的物理环境。除了基准测试之外，我们提供了对不同条件下不同代理商培训的泛化能力的见解。我们可视化传感器使用以及不同类型信号的重要性。我们展示了，对于Pointgoal Task，一个代理在各种任务上进行预先培训，并在目标环境的模拟版本上进行微调，可以达到竞争性能，而无需建模任何SIM2重传，即通过直接从仿真部署培训的代理即可一个真正的物理机器人。

A robot that can carry out a natural-language instruction has been a dream since before the Jetsons cartoon series imagined a life of leisure mediated by a fleet of attentive robot helpers. It is a dream that remains stubbornly distant. However, recent advances in vision and language methods have made incredible progress in closely related areas. This is significant because a robot interpreting a naturallanguage navigation instruction on the basis of what it sees is carrying out a vision and language process that is similar to Visual Question Answering. Both tasks can be interpreted as visually grounded sequence-to-sequence translation problems, and many of the same methods are applicable. To enable and encourage the application of vision and language methods to the problem of interpreting visuallygrounded navigation instructions, we present the Matter-port3D Simulator -a large-scale reinforcement learning environment based on real imagery [11]. Using this simulator, which can in future support a range of embodied vision and language tasks, we provide the first benchmark dataset for visually-grounded natural language navigation in real buildings -the Room-to-Room (R2R) dataset 1 .1 https://bringmeaspoon.org Instruction: Head upstairs and walk past the piano through an archway directly in front. Turn right when the hallway ends at pictures and table. Wait by the moose antlers hanging on the wall.

对比语言图像预测（剪辑）编码器已被证明是有利于对分类和检测到标题和图像操纵的一系列视觉任务。我们调查剪辑视觉骨干网的有效性，以实现AI任务。我们构建令人难以置信的简单基线，名为Emplip，没有任务特定的架构，归纳偏差（如使用语义地图），培训期间的辅助任务，或深度映射 - 但我们发现我们的改进的基线在范围内表现得非常好任务和模拟器。 empclip将Robothor ObjectNav排行榜上面的20分的巨额边缘（成功率）。它使ithor 1相重新安排排行榜上面，击败了采用主动神经映射的下一个最佳提交，而且多于固定的严格度量（0.08至0.17）。它还击败了2021年栖息地对象挑战的获奖者，该挑战采用辅助任务，深度地图和人类示范以及2019年栖息地进程挑战的挑战。我们评估剪辑视觉表示在捕获有关输入观测的语义信息时的能力 - 用于导航沉重的体现任务的基元 - 并且发现剪辑的表示比想象成掠过的骨干更有效地编码这些基元。最后，我们扩展了我们的一个基线，产生了能够归零对象导航的代理，该导航可以导航到在训练期间未被用作目标的对象。

当代机器人主义者的主要目标之一是使智能移动机器人能够在共享的人类机器人环境中平稳运行。为此目标服务的最基本必要的功能之一是在这种“社会”背景下有效的导航。结果，最近的一般社会导航的研究激增，尤其是如何处理社会导航代理之间的冲突。这些贡献介绍了各种模型，算法和评估指标，但是由于该研究领域本质上是跨学科的，因此许多相关论文是不可比较的，并且没有共同的标准词汇。这项调查的主要目标是通过引入这种通用语言，使用它来调查现有工作并突出开放问题来弥合这一差距。它首先定义社会导航的冲突，并提供其组成部分的详细分类学。然后，这项调查将现有工作映射到了本分类法中，同时使用其框架讨论论文。最后，本文提出了一些未来的研究方向和开放问题，这些方向目前正在社会导航的边界，以帮助集中于正在进行的和未来的研究。

We are currently unable to specify human goals and societal values in a way that reliably directs AI behavior. Law-making and legal interpretation form a computational engine that converts opaque human values into legible directives. "Law Informs Code" is the research agenda capturing complex computational legal processes, and embedding them in AI. Similar to how parties to a legal contract cannot foresee every potential contingency of their future relationship, and legislators cannot predict all the circumstances under which their proposed bills will be applied, we cannot ex ante specify rules that provably direct good AI behavior. Legal theory and practice have developed arrays of tools to address these specification problems. For instance, legal standards allow humans to develop shared understandings and adapt them to novel situations. In contrast to more prosaic uses of the law (e.g., as a deterrent of bad behavior through the threat of sanction), leveraged as an expression of how humans communicate their goals, and what society values, Law Informs Code. We describe how data generated by legal processes (methods of law-making, statutory interpretation, contract drafting, applications of legal standards, legal reasoning, etc.) can facilitate the robust specification of inherently vague human goals. This increases human-AI alignment and the local usefulness of AI. Toward society-AI alignment, we present a framework for understanding law as the applied philosophy of multi-agent alignment. Although law is partly a reflection of historically contingent political power - and thus not a perfect aggregation of citizen preferences - if properly parsed, its distillation offers the most legitimate computational comprehension of societal values available. If law eventually informs powerful AI, engaging in the deliberative political process to improve law takes on even more meaning.

By transferring knowledge from large, diverse, task-agnostic datasets, modern machine learning models can solve specific downstream tasks either zero-shot or with small task-specific datasets to a high level of performance. While this capability has been demonstrated in other fields such as computer vision, natural language processing or speech recognition, it remains to be shown in robotics, where the generalization capabilities of the models are particularly critical due to the difficulty of collecting real-world robotic data. We argue that one of the keys to the success of such general robotic models lies with open-ended task-agnostic training, combined with high-capacity architectures that can absorb all of the diverse, robotic data. In this paper, we present a model class, dubbed Robotics Transformer, that exhibits promising scalable model properties. We verify our conclusions in a study of different model classes and their ability to generalize as a function of the data size, model size, and data diversity based on a large-scale data collection on real robots performing real-world tasks. The project's website and videos can be found at robotics-transformer.github.io