Are extralinguistic signals such as image pixels crucial for inducing constituency grammars? While past work has shown substantial gains from multimodal cues, we investigate whether such gains persist in the presence of rich information from large language models (LLMs). We find that our approach, LLM-based C-PCFG (LC-PCFG), outperforms previous multi-modal methods on the task of unsupervised constituency parsing, achieving state-of-the-art performance on a variety of datasets. Moreover, LC-PCFG results in an over 50% reduction in parameter count, and speedups in training time of 1.7x for image-aided models and more than 5x for video-aided models, respectively. These results challenge the notion that extralinguistic signals such as image pixels are needed for unsupervised grammar induction, and point to the need for better text-only baselines in evaluating the need of multi-modality for the task.

We present Masked Audio-Video Learners (MAViL) to train audio-visual representations. Our approach learns with three complementary forms of self-supervision: (1) reconstruction of masked audio and video input data, (2) intra- and inter-modal contrastive learning with masking, and (3) self-training by reconstructing joint audio-video contextualized features learned from the first two objectives. Pre-training with MAViL not only enables the model to perform well in audio-visual classification and retrieval tasks but also improves representations of each modality in isolation, without using information from the other modality for fine-tuning or inference. Empirically, MAViL sets a new state-of-the-art on AudioSet (53.1 mAP) and VGGSound (67.1% accuracy). For the first time, a self-supervised audio-visual model outperforms ones that use external supervision on these benchmarks. Code will be available soon.

Semantic navigation is necessary to deploy mobile robots in uncontrolled environments like our homes, schools, and hospitals. Many learning-based approaches have been proposed in response to the lack of semantic understanding of the classical pipeline for spatial navigation, which builds a geometric map using depth sensors and plans to reach point goals. Broadly, end-to-end learning approaches reactively map sensor inputs to actions with deep neural networks, while modular learning approaches enrich the classical pipeline with learning-based semantic sensing and exploration. But learned visual navigation policies have predominantly been evaluated in simulation. How well do different classes of methods work on a robot? We present a large-scale empirical study of semantic visual navigation methods comparing representative methods from classical, modular, and end-to-end learning approaches across six homes with no prior experience, maps, or instrumentation. We find that modular learning works well in the real world, attaining a 90% success rate. In contrast, end-to-end learning does not, dropping from 77% simulation to 23% real-world success rate due to a large image domain gap between simulation and reality. For practitioners, we show that modular learning is a reliable approach to navigate to objects: modularity and abstraction in policy design enable Sim-to-Real transfer. For researchers, we identify two key issues that prevent today's simulators from being reliable evaluation benchmarks - (A) a large Sim-to-Real gap in images and (B) a disconnect between simulation and real-world error modes - and propose concrete steps forward.

We consider the problem of embodied visual navigation given an image-goal (ImageNav) where an agent is initialized in an unfamiliar environment and tasked with navigating to a location 'described' by an image. Unlike related navigation tasks, ImageNav does not have a standardized task definition which makes comparison across methods difficult. Further, existing formulations have two problematic properties; (1) image-goals are sampled from random locations which can lead to ambiguity (e.g., looking at walls), and (2) image-goals match the camera specification and embodiment of the agent; this rigidity is limiting when considering user-driven downstream applications. We present the Instance-specific ImageNav task (InstanceImageNav) to address these limitations. Specifically, the goal image is 'focused' on some particular object instance in the scene and is taken with camera parameters independent of the agent. We instantiate InstanceImageNav in the Habitat Simulator using scenes from the Habitat-Matterport3D dataset (HM3D) and release a standardized benchmark to measure community progress.

In this work, we show how to learn a visual walking policy that only uses a monocular RGB camera and proprioception. Since simulating RGB is hard, we necessarily have to learn vision in the real world. We start with a blind walking policy trained in simulation. This policy can traverse some terrains in the real world but often struggles since it lacks knowledge of the upcoming geometry. This can be resolved with the use of vision. We train a visual module in the real world to predict the upcoming terrain with our proposed algorithm Cross-Modal Supervision (CMS). CMS uses time-shifted proprioception to supervise vision and allows the policy to continually improve with more real-world experience. We evaluate our vision-based walking policy over a diverse set of terrains including stairs (up to 19cm high), slippery slopes (inclination of 35 degrees), curbs and tall steps (up to 20cm), and complex discrete terrains. We achieve this performance with less than 30 minutes of real-world data. Finally, we show that our policy can adapt to shifts in the visual field with a limited amount of real-world experience. Video results and code at https://antonilo.github.io/vision_locomotion/.

我们探索一种以数据为基础的学习方法来优化神经网络。我们构建神经网络检查点的数据集，并培训有关参数的生成模型。特别是，我们的模型是一个条件扩散变压器，鉴于初始输入参数向量以及提示的丢失，误差或返回，可以预测实现所需度量的参数更新的分布。在测试时，它可以在一个更新中优化具有看不见的参数的神经网络。我们发现我们的方法成功地生成了各种损失提示的参数。此外，它可以采样多模式参数解决方案，并具有有利的缩放属性。我们将方法应用于监督和强化学习中的不同神经网络体系结构和任务。

本文提出了针对四方的通用自适应控制器，可以将其部署为零射击到具有截然不同的质量，手臂长度和运动常数的四轮驱动器，并且还显示出对运行时未知干扰的快速适应。核心算法的想法是学习一个单一的策略，该策略不仅可以在测试时间在线适应无人机的干扰，还可以在同一框架中适用于机器人动力学和硬件。我们通过训练神经网络来估计机器人和环境参数的潜在表示，该参数用于调节控制器的行为，也表示为神经网络。我们专门训练两个网络进行模拟，目的是将四轮驱动器飞往目标位置并避免撞击地面。我们直接在模拟中训练了相同的控制器，而没有对两个四肢旋转器进行任何修改，其中质量，惯性差异差异，最大电动机速度最大为4次。此外，我们显示了四肢和惯性的突然和大型干扰（最高35.7％）的快速适应。我们在模拟和物理世界中进行了广泛的评估，在该评估中，我们的表现优于最先进的基于学习的自适应控制器和专门针对每个平台的传统PID控制器。视频结果可以在https://dz298.github.io/universal-drone-controller/上找到。

我们研究了一种模块化方法，可以解决对象重排的长马移动操作任务，该任务将完整的任务分解为一系列子任务。为了解决整个任务，先前的工作将具有点目标导航技能的多个固定操作技巧，这些技巧是在子任务上单独学习的。尽管比整体端到端的RL政策更有效，但该框架遭受了技能链条的复杂错误，例如导航到一个不良位置，在这种情况下，固定操作技能无法达到其目标进行操纵。为此，我们建议操纵技巧应包括移动性，以便从多个位置与目标对象进行互动，同时导航技能可能具有多个终点，从而导致成功的操纵。我们通过实施移动操纵技能而不是固定技能来实现这些想法，并训练接受区域目标而不是积分目标的导航技能。我们在家庭助理基准（HAB）中评估了三个挑战性的长途移动操纵任务M3，并在3个挑战性的长途移动操纵任务上评估了我们的多技能，并且与基线相比表现出了出色的性能。

腿部运动的最新进展使四足动物在具有挑战性的地形上行走。但是，两足机器人本质上更加不稳定，因此很难为其设计步行控制器。在这项工作中，我们利用了对机车控制的快速适应的最新进展，并将其扩展到双皮亚机器人。与现有作品类似，我们从基本策略开始，该策略在将适应模块的输入中作为输入作为输入。该外部媒介包含有关环境的信息，并使步行控制器能够快速在线适应。但是，外部估计器可能是不完善的，这可能导致基本政策的性能不佳，这预计是一个完美的估计器。在本文中，我们提出了A-RMA（Adapting RMA），该A-RMA（适应RMA）还通过使用无模型RL对其进行了鉴定，从而适应了不完美的外部外部估计器的基本策略。我们证明，A-RMA在仿真中胜过许多基于RL的基线控制器和基于模型的控制器，并显示了单个A-RMA策略的零拍摄部署，以使双皮德机器人Cassie能够在各种各样的现实世界中的不同场景超出了培训期间所见。 https://ashish-kmr.github.io/a-rma/的视频和结果

对象目标导航的最新方法依赖于增强学习，通常需要大量的计算资源和学习时间。我们提出了使用无互动学习（PONI）的对象导航的潜在功能，这是一种模块化方法，可以散布“在哪里看？”的技能？对于对象和“如何导航到（x，y）？”。我们的主要见解是“在哪里看？”可以纯粹将其视为感知问题，而没有环境相互作用就可以学习。为了解决这个问题，我们提出了一个网络，该网络可以预测两个在语义图上的互补电位功能，并使用它们来决定在哪里寻找看不见的对象。我们使用在自上而下的语义图的被动数据集上使用受监督的学习来训练潜在的功能网络，并将其集成到模块化框架中以执行对象目标导航。 Gibson和MatterPort3D的实验表明，我们的方法可实现对象目标导航的最新方法，同时减少培训计算成本高达1,600倍。可以使用代码和预训练的模型：https：//vision.cs.utexas.edu/projects/poni/