我们介绍了一个目标驱动的导航系统，以改善室内场景中的Fapless视觉导航。我们的方法在每次步骤中都将机器人和目标的多视图观察为输入，以提供将机器人移动到目标的一系列动作，而不依赖于运行时在运行时。通过优化包含三个关键设计的组合目标来了解该系统。首先，我们建议代理人在做出行动决定之前构建下一次观察。这是通过从专家演示中学习变分生成模块来实现的。然后，我们提出预测预先预测静态碰撞，作为辅助任务，以改善导航期间的安全性。此外，为了减轻终止动作预测的训练数据不平衡问题，我们还介绍了一个目标检查模块来区分与终止动作的增强导航策略。这三种建议的设计都有助于提高培训数据效率，静态冲突避免和导航泛化性能，从而产生了一种新颖的目标驱动的FLASES导航系统。通过对Turtlebot的实验，我们提供了证据表明我们的模型可以集成到机器人系统中并在现实世界中导航。视频和型号可以在补充材料中找到。

为了基于深度加强学习（RL）来增强目标驱动的视觉导航的交叉目标和跨场景，我们将信息理论正则化术语引入RL目标。正则化最大化导航动作与代理的视觉观察变换之间的互信息，从而促进更明智的导航决策。这样，代理通过学习变分生成模型来模拟动作观察动态。基于该模型，代理生成（想象）从其当前观察和导航目标的下一次观察。这样，代理学会了解导航操作与其观察变化之间的因果关系，这允许代理通过比较当前和想象的下一个观察来预测导航的下一个动作。 AI2-Thor框架上的交叉目标和跨场景评估表明，我们的方法在某些最先进的模型上获得了平均成功率的10美元。我们进一步评估了我们的模型在两个现实世界中：来自离散的活动视觉数据集（AVD）和带有TurtleBot的连续现实世界环境中的看不见的室内场景导航。我们证明我们的导航模型能够成功实现导航任务这些情景。视频和型号可以在补充材料中找到。

我们建议通过学习通过构思它预期看到的下一个观察来引导的代理来改善视觉导航的跨目标和跨场景概括。这是通过学习变分贝叶斯模型来实现的，称为Neonav，该模型产生了在试剂和目标视图的当前观察中的下一个预期观察（Neo）。我们的生成模式是通过优化包含两个关键设计的变分目标来了解。首先，潜在分布在当前观察和目标视图上进行调节，导致基于模型的目标驱动导航。其次，潜伏的空间用在当前观察和下一个最佳动作上的高斯的混合物建模。我们使用后医混合物的用途能够有效地减轻过正规化的潜在空间的问题，从而大大提高了新目标和新场景的模型概括。此外，Neo Generation模型代理环境交互的前向动态，从而提高了近似推断的质量，因此提高了数据效率。我们对现实世界和合成基准进行了广泛的评估，并表明我们的模型在成功率，数据效率和泛化方面始终如一地优于最先进的模型。

这项工作研究了图像目标导航问题，需要通过真正拥挤的环境引导具有嘈杂传感器和控制的机器人。最近的富有成效的方法依赖于深度加强学习，并学习模拟环境中的导航政策，这些环境比真实环境更简单。直接将这些训练有素的策略转移到真正的环境可能非常具有挑战性甚至危险。我们用由四个解耦模块组成的分层导航方法来解决这个问题。第一模块在机器人导航期间维护障碍物映射。第二个将定期预测实时地图上的长期目标。第三个计划碰撞命令集以导航到长期目标，而最终模块将机器人正确靠近目标图像。四个模块是单独开发的，以适应真实拥挤的情景中的图像目标导航。此外，分层分解对导航目标规划，碰撞避免和导航结束预测的学习进行了解耦，这在导航训练期间减少了搜索空间，并有助于改善以前看不见的真实场景的概括。我们通过移动机器人评估模拟器和现实世界中的方法。结果表明，我们的方法优于多种导航基线，可以在这些方案中成功实现导航任务。

We present a retrospective on the state of Embodied AI research. Our analysis focuses on 13 challenges presented at the Embodied AI Workshop at CVPR. These challenges are grouped into three themes: (1) visual navigation, (2) rearrangement, and (3) embodied vision-and-language. We discuss the dominant datasets within each theme, evaluation metrics for the challenges, and the performance of state-of-the-art models. We highlight commonalities between top approaches to the challenges and identify potential future directions for Embodied AI research.

在这项工作中，我们提出了一种用于图像目标导航的内存调格方法。早期的尝试，包括基于RL的基于RL的方法和基于SLAM的方法的概括性能差，或者在姿势/深度传感器上稳定稳定。我们的方法基于一个基于注意力的端到端模型，该模型利用情节记忆来学习导航。首先，我们以自我监督的方式训练一个国家安置的网络，然后将其嵌入以前访问的状态中的代理商的记忆中。我们的导航政策通过注意机制利用了此信息。我们通过广泛的评估来验证我们的方法，并表明我们的模型在具有挑战性的吉布森数据集上建立了新的最新技术。此外，与相关工作形成鲜明对比的是，我们仅凭RGB输入就实现了这种令人印象深刻的性能，而无需访问其他信息，例如位置或深度。

Two less addressed issues of deep reinforcement learning are (1) lack of generalization capability to new target goals, and (2) data inefficiency i.e., the model requires several (and often costly) episodes of trial and error to converge, which makes it impractical to be applied to real-world scenarios. In this paper, we address these two issues and apply our model to the task of target-driven visual navigation. To address the first issue, we propose an actor-critic model whose policy is a function of the goal as well as the current state, which allows to better generalize. To address the second issue, we propose AI2-THOR framework, which provides an environment with highquality 3D scenes and physics engine. Our framework enables agents to take actions and interact with objects. Hence, we can collect a huge number of training samples efficiently.We show that our proposed method (1) converges faster than the state-of-the-art deep reinforcement learning methods, (2) generalizes across targets and across scenes, (3) generalizes to a real robot scenario with a small amount of fine-tuning (although the model is trained in simulation), ( 4) is end-to-end trainable and does not need feature engineering, feature matching between frames or 3D reconstruction of the environment.The supplementary video can be accessed at the following link: https://youtu.be/SmBxMDiOrvs.

从“Internet AI”的时代到“体现AI”的时代，AI算法和代理商出现了一个新兴范式转变，其中不再从主要来自Internet策划的图像，视频或文本的数据集。相反，他们通过与与人类类似的Enocentric感知来通过与其环境的互动学习。因此，对体现AI模拟器的需求存在大幅增长，以支持各种体现的AI研究任务。这种越来越多的体现AI兴趣是有利于对人工综合情报（AGI）的更大追求，但对这一领域并无一直存在当代和全面的调查。本文旨在向体现AI领域提供百科全书的调查，从其模拟器到其研究。通过使用我们提出的七种功能评估九个当前体现的AI模拟器，旨在了解模拟器，以其在体现AI研究和其局限性中使用。最后，本文调查了体现AI - 视觉探索，视觉导航和体现问题的三个主要研究任务（QA），涵盖了最先进的方法，评估指标和数据集。最后，随着通过测量该领域的新见解，本文将为仿真器 - 任务选择和建议提供关于该领域的未来方向的建议。

Underwater navigation presents several challenges, including unstructured unknown environments, lack of reliable localization systems (e.g., GPS), and poor visibility. Furthermore, good-quality obstacle detection sensors for underwater robots are scant and costly; and many sensors like RGB-D cameras and LiDAR only work in-air. To enable reliable mapless underwater navigation despite these challenges, we propose a low-cost end-to-end navigation system, based on a monocular camera and a fixed single-beam echo-sounder, that efficiently navigates an underwater robot to waypoints while avoiding nearby obstacles. Our proposed method is based on Proximal Policy Optimization (PPO), which takes as input current relative goal information, estimated depth images, echo-sounder readings, and previous executed actions, and outputs 3D robot actions in a normalized scale. End-to-end training was done in simulation, where we adopted domain randomization (varying underwater conditions and visibility) to learn a robust policy against noise and changes in visibility conditions. The experiments in simulation and real-world demonstrated that our proposed method is successful and resilient in navigating a low-cost underwater robot in unknown underwater environments. The implementation is made publicly available at https://github.com/dartmouthrobotics/deeprl-uw-robot-navigation.

移动机器人的视觉导航经典通过SLAM加上最佳规划，最近通过实现作为深网络的端到端培训。虽然前者通常仅限于航点计划，但即使在真实的物理环境中已经证明了它们的效率，后一种解决方案最常用于模拟中，但已被证明能够学习更复杂的视觉推理，涉及复杂的语义规则。通过实际机器人在物理环境中导航仍然是一个开放问题。端到端的培训方法仅在模拟中进行了彻底测试，实验涉及实际机器人的实际机器人在简化的实验室条件下限制为罕见的性能评估。在这项工作中，我们对真实物理代理的性能和推理能力进行了深入研究，在模拟中培训并部署到两个不同的物理环境。除了基准测试之外，我们提供了对不同条件下不同代理商培训的泛化能力的见解。我们可视化传感器使用以及不同类型信号的重要性。我们展示了，对于Pointgoal Task，一个代理在各种任务上进行预先培训，并在目标环境的模拟版本上进行微调，可以达到竞争性能，而无需建模任何SIM2重传，即通过直接从仿真部署培训的代理即可一个真正的物理机器人。

With the development of deep representation learning, the domain of reinforcement learning (RL) has become a powerful learning framework now capable of learning complex policies in high dimensional environments. This review summarises deep reinforcement learning (DRL) algorithms and provides a taxonomy of automated driving tasks where (D)RL methods have been employed, while addressing key computational challenges in real world deployment of autonomous driving agents. It also delineates adjacent domains such as behavior cloning, imitation learning, inverse reinforcement learning that are related but are not classical RL algorithms. The role of simulators in training agents, methods to validate, test and robustify existing solutions in RL are discussed.

Training effective embodied AI agents often involves manual reward engineering, expert imitation, specialized components such as maps, or leveraging additional sensors for depth and localization. Another approach is to use neural architectures alongside self-supervised objectives which encourage better representation learning. In practice, there are few guarantees that these self-supervised objectives encode task-relevant information. We propose the Scene Graph Contrastive (SGC) loss, which uses scene graphs as general-purpose, training-only, supervisory signals. The SGC loss does away with explicit graph decoding and instead uses contrastive learning to align an agent's representation with a rich graphical encoding of its environment. The SGC loss is generally applicable, simple to implement, and encourages representations that encode objects' semantics, relationships, and history. Using the SGC loss, we attain significant gains on three embodied tasks: Object Navigation, Multi-Object Navigation, and Arm Point Navigation. Finally, we present studies and analyses which demonstrate the ability of our trained representation to encode semantic cues about the environment.

尽管移动操作在工业和服务机器人技术方面都重要，但仍然是一个重大挑战，因为它需要将最终效应轨迹的无缝整合与导航技能以及对长匹马的推理。现有方法难以控制大型配置空间，并导航动态和未知环境。在先前的工作中，我们建议将移动操纵任务分解为任务空间中最终效果的简化运动生成器，并将移动设备分解为训练有素的强化学习代理，以说明移动基础的运动基础，以说明运动的运动可行性。在这项工作中，我们引入了移动操作的神经导航（n $^2 $ m $^2 $），该导航将这种分解扩展到复杂的障碍环境，并使其能够解决现实世界中的广泛任务。最终的方法可以在未探索的环境中执行看不见的长马任务，同时立即对动态障碍和环境变化做出反应。同时，它提供了一种定义新的移动操作任务的简单方法。我们证明了我们提出的方法在多个运动学上多样化的移动操纵器上进行的广泛模拟和现实实验的能力。代码和视频可在http://mobile-rl.cs.uni-freiburg.de上公开获得。

我们考虑将移动机器人导航到具有视觉传感器的未知环境中的问题，在该环境中，机器人和传感器都无法访问全局定位信息，并且仅使用第一人称视图图像。虽然基于传感器网络的先前工作使用明确的映射和计划技术，并且经常得到外部定位系统的帮助，但我们提出了一种基于视觉的学习方法，该方法利用图形神经网络（GNN）来编码和传达相关的视点信息到移动机器人。在导航期间，机器人以模型为指导，我们通过模仿学习训练以近似最佳的运动原语，从而预测有效的成本（目标）。在我们的实验中，我们首先证明了具有各种传感器布局的以前看不见的环境的普遍性。仿真结果表明，通过利用传感器和机器人之间的通信，我们可以达到$ 18.1 \％$ $的成功率，同时将路径弯路的平均值降低$ 29.3 \％$，并且可变性降低了$ 48.4 \％$ $。这是在不需要全局地图，定位数据或传感器网络预校准的情况下完成的。其次，我们将模型从模拟到现实世界进行零拍传输。为此，我们训练一个“翻译器”模型，该模型在{}真实图像和模拟图像之间转换，以便可以直接在真实的机器人上使用导航策略（完全在模拟中训练），而无需其他微调。 。物理实验证明了我们在各种混乱的环境中的有效性。

Transformer, originally devised for natural language processing, has also attested significant success in computer vision. Thanks to its super expressive power, researchers are investigating ways to deploy transformers to reinforcement learning (RL) and the transformer-based models have manifested their potential in representative RL benchmarks. In this paper, we collect and dissect recent advances on transforming RL by transformer (transformer-based RL or TRL), in order to explore its development trajectory and future trend. We group existing developments in two categories: architecture enhancement and trajectory optimization, and examine the main applications of TRL in robotic manipulation, text-based games, navigation and autonomous driving. For architecture enhancement, these methods consider how to apply the powerful transformer structure to RL problems under the traditional RL framework, which model agents and environments much more precisely than deep RL methods, but they are still limited by the inherent defects of traditional RL algorithms, such as bootstrapping and "deadly triad". For trajectory optimization, these methods treat RL problems as sequence modeling and train a joint state-action model over entire trajectories under the behavior cloning framework, which are able to extract policies from static datasets and fully use the long-sequence modeling capability of the transformer. Given these advancements, extensions and challenges in TRL are reviewed and proposals about future direction are discussed. We hope that this survey can provide a detailed introduction to TRL and motivate future research in this rapidly developing field.

深度强化学习在基于激光的碰撞避免有效的情况下取得了巨大的成功，因为激光器可以感觉到准确的深度信息而无需太多冗余数据，这可以在算法从模拟环境迁移到现实世界时保持算法的稳健性。但是，高成本激光设备不仅很难为大型机器人部署，而且还表现出对复杂障碍的鲁棒性，包括不规则的障碍，例如桌子，桌子，椅子和架子，以及复杂的地面和特殊材料。在本文中，我们提出了一个新型的基于单眼相机的复杂障碍避免框架。特别是，我们创新地将捕获的RGB图像转换为伪激光测量，以进行有效的深度强化学习。与在一定高度捕获的传统激光测量相比，仅包含距离附近障碍的一维距离信息，我们提议的伪激光测量融合了捕获的RGB图像的深度和语义信息，这使我们的方法有效地有效障碍。我们还设计了一个功能提取引导模块，以加重输入伪激光测量，并且代理对当前状态具有更合理的关注，这有利于提高障碍避免政策的准确性和效率。

我们解决了动态环境中感知力的问题。在这个问题中，四足动物的机器人必须对环境混乱和移动的障碍物表现出强大而敏捷的步行行为。我们提出了一个名为Prelude的分层学习框架，该框架将感知力的问题分解为高级决策，以预测导航命令和低级步态生成以实现目标命令。在此框架中，我们通过在可进入手推车上收集的人类示范和使用加固学习（RL）的低级步态控制器（RL）上收集的人类示范中的模仿学习来训练高级导航控制器。因此，我们的方法可以从人类监督中获取复杂的导航行为，并从反复试验中发现多功能步态。我们证明了方法在模拟和硬件实验中的有效性。可以在https://ut-aut-autin-rpl.github.io/prelude上找到视频和代码。

Despite some successful applications of goal-driven navigation, existing deep reinforcement learning-based approaches notoriously suffers from poor data efficiency issue. One of the reasons is that the goal information is decoupled from the perception module and directly introduced as a condition of decision-making, resulting in the goal-irrelevant features of the scene representation playing an adversary role during the learning process. In light of this, we present a novel Goal-guided Transformer-enabled reinforcement learning (GTRL) approach by considering the physical goal states as an input of the scene encoder for guiding the scene representation to couple with the goal information and realizing efficient autonomous navigation. More specifically, we propose a novel variant of the Vision Transformer as the backbone of the perception system, namely Goal-guided Transformer (GoT), and pre-train it with expert priors to boost the data efficiency. Subsequently, a reinforcement learning algorithm is instantiated for the decision-making system, taking the goal-oriented scene representation from the GoT as the input and generating decision commands. As a result, our approach motivates the scene representation to concentrate mainly on goal-relevant features, which substantially enhances the data efficiency of the DRL learning process, leading to superior navigation performance. Both simulation and real-world experimental results manifest the superiority of our approach in terms of data efficiency, performance, robustness, and sim-to-real generalization, compared with other state-of-art baselines. Demonstration videos are available at \colorb{https://youtu.be/93LGlGvaN0c.

在本文中，我们专注于在线学习主动视觉在未知室内环境中的对象的搜索（AVS）的最优策略问题。我们建议POMP++，规划战略，介绍了经典的部分可观察蒙特卡洛规划（POMCP）框架之上的新制剂，允许免费培训，在线政策在未知的环境中学习。我们提出了一个新的信仰振兴战略，允许使用POMCP与动态扩展状态空间来解决在线生成平面地图的。我们评估我们在两个公共标准数据集的方法，AVD由是从真正的3D场景渲染扫描真正的机器人平台和人居ObjectNav收购，用>10％，比国家的the-改善达到最佳的成功率技术方法。

Reinforcement learning can enable robots to navigate to distant goals while optimizing user-specified reward functions, including preferences for following lanes, staying on paved paths, or avoiding freshly mowed grass. However, online learning from trial-and-error for real-world robots is logistically challenging, and methods that instead can utilize existing datasets of robotic navigation data could be significantly more scalable and enable broader generalization. In this paper, we present ReViND, the first offline RL system for robotic navigation that can leverage previously collected data to optimize user-specified reward functions in the real-world. We evaluate our system for off-road navigation without any additional data collection or fine-tuning, and show that it can navigate to distant goals using only offline training from this dataset, and exhibit behaviors that qualitatively differ based on the user-specified reward function.