在视觉和语言导航（VLN）中，按照自然语言指令在现实的3D环境中需要具体的代理。现有VLN方法的一个主要瓶颈是缺乏足够的培训数据，从而导致对看不见的环境的概括不令人满意。虽然通常会手动收集VLN数据，但这种方法很昂贵，并且可以防止可扩展性。在这项工作中，我们通过建议从HM3D自动创建900个未标记的3D建筑物的大规模VLN数据集来解决数据稀缺问题。我们为每个建筑物生成一个导航图，并通过交叉视图一致性从2D传输对象预测，从2D传输伪3D对象标签。然后，我们使用伪对象标签来微调一个预处理的语言模型，作为减轻教学生成中跨模式差距的提示。在导航环境和说明方面，我们生成的HM3D-AUTOVLN数据集是比现有VLN数据集大的数量级。我们通过实验表明，HM3D-AUTOVLN显着提高了所得VLN模型的概括能力。在SPL指标上，我们的方法分别在Reverie和DataSet的看不见的验证分裂分别对艺术的状态提高了7.1％和8.1％。

Recent studies in Vision-and-Language Navigation (VLN) train RL agents to execute natural-language navigation instructions in photorealistic environments, as a step towards robots that can follow human instructions. However, given the scarcity of human instruction data and limited diversity in the training environments, these agents still struggle with complex language grounding and spatial language understanding. Pretraining on large text and image-text datasets from the web has been extensively explored but the improvements are limited. We investigate large-scale augmentation with synthetic instructions. We take 500+ indoor environments captured in densely-sampled 360 degree panoramas, construct navigation trajectories through these panoramas, and generate a visually-grounded instruction for each trajectory using Marky, a high-quality multilingual navigation instruction generator. We also synthesize image observations from novel viewpoints using an image-to-image GAN. The resulting dataset of 4.2M instruction-trajectory pairs is two orders of magnitude larger than existing human-annotated datasets, and contains a wider variety of environments and viewpoints. To efficiently leverage data at this scale, we train a simple transformer agent with imitation learning. On the challenging RxR dataset, our approach outperforms all existing RL agents, improving the state-of-the-art NDTW from 71.1 to 79.1 in seen environments, and from 64.6 to 66.8 in unseen test environments. Our work points to a new path to improving instruction-following agents, emphasizing large-scale imitation learning and the development of synthetic instruction generation capabilities.

Existing approaches for vision-and-language navigation (VLN) are mainly based on cross-modal reasoning over discrete views. However, this scheme may hamper an agent's spatial and numerical reasoning because of incomplete objects within a single view and duplicate observations across views. A potential solution is mapping discrete views into a unified birds's-eye view, which can aggregate partial and duplicate observations. Existing metric maps could achieve this goal, but they suffer from less expressive semantics (e.g. usually predefined labels) and limited map size, which weakens an agent's language grounding and long-term planning ability. Inspired by the robotics community, we introduce hybrid topo-metric maps into VLN, where a topological map is used for long-term planning and a metric map for short-term reasoning. Beyond mapping with more expressive deep features, we further design a pre-training framework via the hybrid map to learn language-informed map representations, which enhances cross-modal grounding and facilitates the final language-guided navigation goal. Extensive experiments demonstrate the effectiveness of the map-based route for VLN, and the proposed method sets the new state-of-the-art on three VLN benchmarks.

In this work, we study the problem of Embodied Referring Expression Grounding, where an agent needs to navigate in a previously unseen environment and localize a remote object described by a concise high-level natural language instruction. When facing such a situation, a human tends to imagine what the destination may look like and to explore the environment based on prior knowledge of the environmental layout, such as the fact that a bathroom is more likely to be found near a bedroom than a kitchen. We have designed an autonomous agent called Layout-aware Dreamer (LAD), including two novel modules, that is, the Layout Learner and the Goal Dreamer to mimic this cognitive decision process. The Layout Learner learns to infer the room category distribution of neighboring unexplored areas along the path for coarse layout estimation, which effectively introduces layout common sense of room-to-room transitions to our agent. To learn an effective exploration of the environment, the Goal Dreamer imagines the destination beforehand. Our agent achieves new state-of-the-art performance on the public leaderboard of the REVERIE dataset in challenging unseen test environments with improvement in navigation success (SR) by 4.02% and remote grounding success (RGS) by 3.43% compared to the previous state-of-the-art. The code is released at https://github.com/zehao-wang/LAD

A robot that can carry out a natural-language instruction has been a dream since before the Jetsons cartoon series imagined a life of leisure mediated by a fleet of attentive robot helpers. It is a dream that remains stubbornly distant. However, recent advances in vision and language methods have made incredible progress in closely related areas. This is significant because a robot interpreting a naturallanguage navigation instruction on the basis of what it sees is carrying out a vision and language process that is similar to Visual Question Answering. Both tasks can be interpreted as visually grounded sequence-to-sequence translation problems, and many of the same methods are applicable. To enable and encourage the application of vision and language methods to the problem of interpreting visuallygrounded navigation instructions, we present the Matter-port3D Simulator -a large-scale reinforcement learning environment based on real imagery [11]. Using this simulator, which can in future support a range of embodied vision and language tasks, we provide the first benchmark dataset for visually-grounded natural language navigation in real buildings -the Room-to-Room (R2R) dataset 1 .1 https://bringmeaspoon.org Instruction: Head upstairs and walk past the piano through an archway directly in front. Turn right when the hallway ends at pictures and table. Wait by the moose antlers hanging on the wall.

We present a retrospective on the state of Embodied AI research. Our analysis focuses on 13 challenges presented at the Embodied AI Workshop at CVPR. These challenges are grouped into three themes: (1) visual navigation, (2) rearrangement, and (3) embodied vision-and-language. We discuss the dominant datasets within each theme, evaluation metrics for the challenges, and the performance of state-of-the-art models. We highlight commonalities between top approaches to the challenges and identify potential future directions for Embodied AI research.

我们研究了在室内路线上捕获的360度图像中的自动生成导航指令。现有的发电机遭受较差的视觉接地，导致它们依赖语言前沿和幻觉对象。我们的Marky-MT5系统通过专注于视觉地标来解决这一点;它包括第一阶段地标检测器和第二级发生器 - 多峰，多语言，多任务编码器 - 解码器。要培训它，我们在房间顶部（RXR）数据集的顶部引导地标注释。使用文本解析器，来自RXR的姿势迹线的弱监督，以及在1.8B图像上培训的多语言图像文本编码器，我们识别1.1M英语，印地语和泰卢语的地标描述并将其接地为Panoramas的特定区域。在房间到室内，人类途径在Marky-MT5的指示之后获得了71％的成功率（SR），只害羞他们的75％SR在人类指令之后 - 以及与其他发电机的SR高于SRS。对RXR更长的评估，不同的路径上的三种语言获得61-64％的SRS。在新颖环境中生成这种高质量的导航指令是迈向对话导航工具的一步，可以促进对指令跟随代理的大规模培训。

视觉和语言导航（VLN）是一种任务，即遵循语言指令以导航到目标位置的语言指令，这依赖于在移动期间与环境的持续交互。最近的基于变压器的VLN方法取得了很大的进步，从视觉观测和语言指令之间的直接连接通过多模式跨关注机制。然而，这些方法通常代表通过使用LSTM解码器或使用手动设计隐藏状态来构建反复变压器的时间上下文作为固定长度矢量。考虑到单个固定长度向量通常不足以捕获长期时间上下文，在本文中，我们通过显式建模时间上下文来引入具有可变长度存储器（MTVM）的多模式变压器，通过模拟时间上下文。具体地，MTVM使代理能够通过直接存储在存储体中的先前激活来跟踪导航轨迹。为了进一步提高性能，我们提出了内存感知的一致性损失，以帮助学习随机屏蔽指令的时间上下文的更好关节表示。我们在流行的R2R和CVDN数据集上评估MTVM，我们的模型在R2R看不见的验证和测试中提高了2％的成功率，并在CVDN测试集上减少了1.6米的目标进程。

我们提出了一种可扩展的方法，用于学习开放世界对象目标导航（ObjectNAV） - 要求虚拟机器人（代理）在未探索的环境中找到对象的任何实例（例如，“查找接收器”）。我们的方法完全是零拍的 - 即，它不需要任何形式的objectNav奖励或演示。取而代之的是，我们训练图像目标导航（ImagenAv）任务，在该任务中，代理在其中找到了捕获图片（即目标图像）的位置。具体而言，我们将目标图像编码为多模式的语义嵌入空间，以在未注释的3D环境（例如HM3D）中以大规模训练语义目标导航（Senanticnav）代理。训练后，可以指示Semanticnav代理查找以自由形式的自然语言描述的对象（例如，“接收器”，“浴室水槽”等），通过将语言目标投射到相同的多模式，语义嵌入空间中。结果，我们的方法启用了开放世界的ObjectNAV。我们在三个ObjectNAV数据集（Gibson，HM3D和MP3D）上广泛评估了我们的代理商，并观察到成功的4.2％-20.0％的绝对改进。作为参考，这些收益与2020年至2021年Objectnav挑战赛竞争对手之间成功的5％改善相似或更好。在开放世界的环境中，我们发现我们的代理商可以概括为明确提到的房间（例如，“找到厨房水槽”）的复合说明，并且何时可以推断目标室（例如，”找到水槽和炉子”）。

Household environments are visually diverse. Embodied agents performing Vision-and-Language Navigation (VLN) in the wild must be able to handle this diversity, while also following arbitrary language instructions. Recently, Vision-Language models like CLIP have shown great performance on the task of zero-shot object recognition. In this work, we ask if these models are also capable of zero-shot language grounding. In particular, we utilize CLIP to tackle the novel problem of zero-shot VLN using natural language referring expressions that describe target objects, in contrast to past work that used simple language templates describing object classes. We examine CLIP's capability in making sequential navigational decisions without any dataset-specific finetuning, and study how it influences the path that an agent takes. Our results on the coarse-grained instruction following task of REVERIE demonstrate the navigational capability of CLIP, surpassing the supervised baseline in terms of both success rate (SR) and success weighted by path length (SPL). More importantly, we quantitatively show that our CLIP-based zero-shot approach generalizes better to show consistent performance across environments when compared to SOTA, fully supervised learning approaches when evaluated via Relative Change in Success (RCS).

愿景 - 语言导航（VLN）任务要求代理逐步导航，同时感知视觉观察并理解自然语言指令。大数据偏置，这是由小数据量表和大型导航空间之间的视差比率引起的，使得VLN任务具有挑战性。以前的作品提出了各种数据增强方法来减少数据偏差。但是，这些作品不会明确降低不同房间场景的数据偏差。因此，该代理将覆盖所见的场景，并在看不见的场景中实现较差的导航性能。为了解决这个问题，我们提出了随机环境混合（REM）方法，它通过混合环境作为增强数据生成交叉连接的房屋场景。具体而言，我们首先根据每个场景的房间连接图选择键视点。然后，我们交叉连接不同场景的关键视图，以构建增强场景。最后，我们在交叉连接场景中生成增强的指令路径对。基准数据集的实验结果表明，我们的增强数据通过REM帮助代理商会降低所见和看不见的环境之间的性能差距，提高整体性能，使我们的模型成为标准VLN基准的最佳现有方法。该代码已发布：https://github.com/lcfractal/vlnrem。

了解空间和视觉信息对于遵循自然语言说明的导航代理至关重要。当前的基于变压器的VLN代理纠缠了方向和视觉信息，这限制了每个信息源的学习中的增益。在本文中，我们设计了具有明确取向和视觉模块的神经药物。这些模块学会了将空间信息和地标在视觉环境中的说明中提及。为了加强代理的空间推理和视觉感知，我们设计了特定的预训练任务，以进食并更好地利用我们最终导航模型中的相应模块。我们在Room2Room（R2R）和Room4Room（R4R）数据集上评估我们的方法，并在两个基准测试中实现最新结果。

This study focuses on embodied agents that can follow natural language instructions to complete complex tasks in a visually-perceived environment. Existing methods rely on a large amount of (instruction, gold trajectory) pairs to learn a good policy. The high data cost and poor sample efficiency prevents the development of versatile agents that are capable of many tasks and can learn new tasks quickly. In this work, we propose a novel method, LLM-Planner, that harnesses the power of large language models (LLMs) such as GPT-3 to do few-shot planning for embodied agents. We further propose a simple but effective way to enhance LLMs with physical grounding to generate plans that are grounded in the current environment. Experiments on the ALFRED dataset show that our method can achieve very competitive few-shot performance, even outperforming several recent baselines that are trained using the full training data despite using less than 0.5% of paired training data. Existing methods can barely complete any task successfully under the same few-shot setting. Our work opens the door for developing versatile and sample-efficient embodied agents that can quickly learn many tasks.

我们研究了开发自主代理的问题，这些自主代理可以遵循人类的指示来推断和执行一系列行动以完成基础任务。近年来取得了重大进展，尤其是对于短范围的任务。但是，当涉及具有扩展动作序列的长匹马任务时，代理可以轻松忽略某些指令或陷入长长指令中间，并最终使任务失败。为了应对这一挑战，我们提出了一个基于模型的里程碑的任务跟踪器（M-Track），以指导代理商并监视其进度。具体而言，我们提出了一个里程碑构建器，该建筑商通过导航和交互里程碑标记指令，代理商需要逐步完成，以及一个系统地检查代理商当前里程碑的进度并确定何时继续进行下一个的里程碑检查器。在具有挑战性的Alfred数据集上，我们的M轨道在两个竞争基本模型中，未见成功率的相对成功率显着提高了33％和52％。

第一人称视频在其持续环境的背景下突出了摄影师的活动。但是，当前的视频理解方法是从短视频剪辑中的视觉特征的原因，这些视频片段与基础物理空间分离，只捕获直接看到的东西。我们提出了一种方法，该方法通过学习摄影师（潜在看不见的）本地环境来促进以人为中心的环境的了解来链接以自我为中心的视频和摄像机随着时间的推移而张开。我们使用来自模拟的3D环境中的代理商的视频进行训练，在该环境中，环境完全可以观察到，并在看不见的环境的房屋旅行的真实视频中对其进行测试。我们表明，通过将视频接地在其物理环境中，我们的模型超过了传统的场景分类模型，可以预测摄影师所处的哪个房间（其中帧级信息不足），并且可以利用这种基础来定位与环境相对应的视频瞬间 - 中心查询，优于先验方法。项目页面：http：//vision.cs.utexas.edu/projects/ego-scene-context/

从“Internet AI”的时代到“体现AI”的时代，AI算法和代理商出现了一个新兴范式转变，其中不再从主要来自Internet策划的图像，视频或文本的数据集。相反，他们通过与与人类类似的Enocentric感知来通过与其环境的互动学习。因此，对体现AI模拟器的需求存在大幅增长，以支持各种体现的AI研究任务。这种越来越多的体现AI兴趣是有利于对人工综合情报（AGI）的更大追求，但对这一领域并无一直存在当代和全面的调查。本文旨在向体现AI领域提供百科全书的调查，从其模拟器到其研究。通过使用我们提出的七种功能评估九个当前体现的AI模拟器，旨在了解模拟器，以其在体现AI研究和其局限性中使用。最后，本文调查了体现AI - 视觉探索，视觉导航和体现问题的三个主要研究任务（QA），涵盖了最先进的方法，评估指标和数据集。最后，随着通过测量该领域的新见解，本文将为仿真器 - 任务选择和建议提供关于该领域的未来方向的建议。

视觉和语言导航（VLN）任务要求代理根据语言说明浏览环境。在本文中，我们旨在解决此任务中的两个关键挑战：利用多语言指令改进教学路径接地，并在培训期间看不见的新环境中导航。为了应对这些挑战，我们提出了明确的：跨语性和环境不可屈服的表示。首先，我们的经纪人在室内室内数据集中学习了三种语言（英语，印地语和泰卢固语）的共享且视觉上的跨语言表示。我们的语言表示学习是由视觉信息对齐的文本对指导的。其次，我们的代理商通过从不同环境中最大化语义对齐的图像对（对象匹配的约束）之间的相似性来学习环境不足的视觉表示。我们的环境不可知的视觉表示可以减轻低级视觉信息引起的环境偏见。从经验上讲，在房间 - 室内数据集中，我们表明，当通过跨语性语言表示和环境 - 非局部视觉表示形式概括地看不见的环境时，我们的多语言代理在所有指标上都比强大的基线模型进行了巨大改进。此外，我们表明我们学到的语言和视觉表示可以成功地转移到房间和合作的视觉和二元式导航任务上，并提出详细的定性和定量的概括和基础分析。我们的代码可从https://github.com/jialuli-luka/clear获得

基于学习的培训方法的方法通常需要大量包含现实布局的高质量场景并支持有意义的互动。然而，用于体现AI（EAI）挑战的当前模拟器仅提供具有有限数量的布局的模拟室内场景。本文呈现出发光，第一研究框架采用最先进的室内场景综合算法，以在体现AI挑战的情况下生成大规模模拟场景。此外，我们通过支持复杂的家庭任务的能力自动和定量地评估生成的室内场景的质量。发光结合了一种新颖的场景生成算法（受限的随机现场生成（CSSG）），实现了具有人类设计的场景的竞争性能。在发光，EAI任务执行器，任务指令生成模块和视频呈现工具包中可以集体为实现的AI代理商的培训和评估集体为新场景产生大量多模式数据集。广泛的实验结果表明了发光产生的数据的有效性，使对泛化和鲁棒性的体现特性进行全面评估。

机器人导航的目标条件政策可以在大型未注释的数据集上进行培训，从而为现实世界中的设置提供了良好的概括。但是，尤其是在指定目标需要图像的基于视觉的设置中，这是一个不自然的界面。语言为与机器人的通信提供了一种更方便的方式，但是现代方法通常需要以语言描述注释的轨迹的形式进行昂贵的监督。我们提出了一个用于机器人导航的系统，该系统享受着未注释的大型轨迹数据集培训的好处，同时仍为用户提供高级接口。我们没有在数据集之后使用标记的指令，而是表明可以完全从预先训练的导航模型（VING），图像语言关联（剪辑）和语言建模（GPT-3）中构建这样的系统，而无需任何微调或语言宣布的机器人数据。我们将LM-NAV实例化在现实世界中的移动机器人上，并通过自然语言指令通过复杂的室外环境演示长途导航。有关我们的实验的视频，代码发布和在浏览器中运行的交互式COLAB笔记本，请查看我们的项目页面https://sites.google.com/view/lmnav

We present ALFRED (Action Learning From Realistic Environments and Directives), a benchmark for learning a mapping from natural language instructions and egocentric vision to sequences of actions for household tasks. ALFRED includes long, compositional tasks with nonreversible state changes to shrink the gap between research benchmarks and real-world applications. ALFRED consists of expert demonstrations in interactive visual environments for 25k natural language directives. These directives contain both high-level goals like "Rinse off a mug and place it in the coffee maker." and low-level language instructions like "Walk to the coffee maker on the right." ALFRED tasks are more complex in terms of sequence length, action space, and language than existing visionand-language task datasets. We show that a baseline model based on recent embodied vision-and-language tasks performs poorly on ALFRED, suggesting that there is significant room for developing innovative grounded visual language understanding models with this benchmark.