我们为机器人素描剂提供了一个自动学习框架,该框架能够同时学习基于中风的渲染和运动控制。我们将机器人的草图问题提出为深度分离的分层增强学习;独立学习了基于中风的渲染和电动机控制的两种政策,以实现绘画的子任务,并在合作进行现实世界绘图时形成层次结构。没有手工制作的特征,绘制序列或轨迹以及逆运动学,该方法将从刮擦上训练机器人素描剂。我们用带有2F抓手的6多机器人臂进行了实验,以素描涂鸦。我们的实验结果表明,这两个策略成功地学习了子任务并合作绘制目标图像。此外,通过不同的绘图工具和表面来检查鲁棒性和灵活性。
translated by 谷歌翻译
Imitation learning techniques aim to mimic human behavior in a given task. An agent (a learning machine) is trained to perform a task from demonstrations by learning a mapping between observations and actions. The idea of teaching by imitation has been around for many years, however, the field is gaining attention recently due to advances in computing and sensing as well as rising demand for intelligent applications. The paradigm of learning by imitation is gaining popularity because it facilitates teaching complex tasks with minimal expert knowledge of the tasks. Generic imitation learning methods could potentially reduce the problem of teaching a task to that of providing demonstrations; without the need for explicit programming or designing reward functions specific to the task. Modern sensors are able to collect and transmit high volumes of data rapidly, and processors with high computational power allow fast processing that maps the sensory data to actions in a timely manner. This opens the door for many potential AI applications that require real-time perception and reaction such as humanoid robots, self-driving vehicles, human computer interaction and computer games to name a few. However, specialized algorithms are needed to effectively and robustly learn models as learning by imitation poses its own set of challenges. In this paper, we survey imitation learning methods and present design options in different steps of the learning process. We introduce a background and motivation for the field as well as highlight challenges specific to the imitation problem. Methods for designing and evaluating imitation learning tasks are categorized and reviewed. Special attention is given to learning methods in robotics and games as these domains are the most popular in the literature and provide a wide array of problems and methodologies. We extensively discuss combining imitation learning approaches using different sources and methods, as well as incorporating other motion learning methods to enhance imitation. We also discuss the potential impact on industry, present major applications and highlight current and future research directions.
translated by 谷歌翻译
Hierarchical Reinforcement Learning (HRL) algorithms have been demonstrated to perform well on high-dimensional decision making and robotic control tasks. However, because they solely optimize for rewards, the agent tends to search the same space redundantly. This problem reduces the speed of learning and achieved reward. In this work, we present an Off-Policy HRL algorithm that maximizes entropy for efficient exploration. The algorithm learns a temporally abstracted low-level policy and is able to explore broadly through the addition of entropy to the high-level. The novelty of this work is the theoretical motivation of adding entropy to the RL objective in the HRL setting. We empirically show that the entropy can be added to both levels if the Kullback-Leibler (KL) divergence between consecutive updates of the low-level policy is sufficiently small. We performed an ablative study to analyze the effects of entropy on hierarchy, in which adding entropy to high-level emerged as the most desirable configuration. Furthermore, a higher temperature in the low-level leads to Q-value overestimation and increases the stochasticity of the environment that the high-level operates on, making learning more challenging. Our method, SHIRO, surpasses state-of-the-art performance on a range of simulated robotic control benchmark tasks and requires minimal tuning.
translated by 谷歌翻译
在机器学习中使用大型数据集已导致出色的结果,在某些情况下,在机器上认为不可能的任务中的人数优于人类。但是,在处理身体上的互动任务时,实现人类水平的表现,例如,在接触良好的机器人操作中,仍然是一个巨大的挑战。众所周知,规范笛卡尔阻抗进行此类行动对于成功执行至关重要。加强学习(RL)之类的方法可能是解决此类问题的有希望的范式。更确切地说,在解决新任务具有巨大潜力时,使用任务不足的专家演示的方法可以利用大型数据集。但是,现有的数据收集系统是昂贵,复杂的,或者不允许进行阻抗调节。这项工作是朝着数据收集框架迈出的第一步,适合收集与使用新颖的动作空间的RL问题公式相容的基于阻抗的专家演示的大型数据集。该框架是根据对机器人操纵的可用数据收集框架进行广泛分析后根据要求设计的。结果是一个低成本且开放的远程阻抗框架,它使人类专家能够展示接触式任务。
translated by 谷歌翻译
机器人将机器人的无缝集成到人类环境需要机器人来学习如何使用现有的人类工具。学习工具操纵技能的目前方法主要依赖于目标机器人环境中提供的专家演示,例如,通过手动引导机器人操纵器或通过远程操作。在这项工作中,我们介绍了一种自动化方法,取代了一个专家演示,用YouTube视频来学习工具操纵策略。主要贡献是双重的。首先,我们设计一个对齐过程,使模拟环境与视频中观察到的真实世界。这是作为优化问题,找到刀具轨迹的空间对齐,以最大化环境给出的稀疏目标奖励。其次,我们描述了一种专注于工具的轨迹而不是人类的运动的模仿学习方法。为此,我们将加强学习与优化过程相结合,以基于对准环境中的工具运动来找到控制策略和机器人的放置。我们展示了仿真中的铲子,镰刀和锤子工具的建议方法,并展示了训练有素的政策对真正的弗兰卡·埃米卡熊猫机器人示范的卫生政策的有效性。
translated by 谷歌翻译
现实的操纵任务要求机器人与具有长时间运动动作序列的环境相互作用。尽管最近出现了深厚的强化学习方法,这是自动化操作行为的有希望的范式,但由于勘探负担,它们通常在长途任务中缺乏。这项工作介绍了操纵原始增强的强化学习(Maple),这是一个学习框架,可通过预定的行为原始库来增强标准强化学习算法。这些行为原始素是专门实现操纵目标(例如抓住和推动)的强大功能模块。为了使用这些异质原始素,我们制定了涉及原语的层次结构策略,并使用输入参数实例化执行。我们证明,枫树的表现优于基线方法,通过一系列模拟的操纵任务的大幅度。我们还量化了学习行为的组成结构,并突出了我们方法将策略转移到新任务变体和物理硬件的能力。视频和代码可从https://ut-aut-autin-rpl.github.io/maple获得
translated by 谷歌翻译
从意外的外部扰动中恢复的能力是双模型运动的基本机动技能。有效的答复包括不仅可以恢复平衡并保持稳定性的能力,而且在平衡恢复物质不可行时,也可以保证安全的方式。对于与双式运动有关的机器人,例如人形机器人和辅助机器人设备,可帮助人类行走,设计能够提供这种稳定性和安全性的控制器可以防止机器人损坏或防止伤害相关的医疗费用。这是一个具有挑战性的任务,因为它涉及用触点产生高维,非线性和致动系统的高动态运动。尽管使用基于模型和优化方法的前进方面,但诸如广泛领域知识的要求,诸如较大的计算时间和有限的动态变化的鲁棒性仍然会使这个打开问题。在本文中,为了解决这些问题,我们开发基于学习的算法,能够为两种不同的机器人合成推送恢复控制政策:人形机器人和有助于双模型运动的辅助机器人设备。我们的工作可以分为两个密切相关的指示:1)学习人形机器人的安全下降和预防策略,2)使用机器人辅助装置学习人类的预防策略。为实现这一目标,我们介绍了一套深度加强学习(DRL)算法,以学习使用这些机器人时提高安全性的控制策略。
translated by 谷歌翻译
通过加强学习(RL)掌握机器人操纵技巧通常需要设计奖励功能。该地区的最新进展表明,使用稀疏奖励,即仅在成功完成任务时奖励代理,可能会导致更好的政策。但是,在这种情况下,国家行动空间探索更困难。最近的RL与稀疏奖励学习的方法已经为任务提供了高质量的人类演示,但这些可能是昂贵的,耗时甚至不可能获得的。在本文中,我们提出了一种不需要人类示范的新颖有效方法。我们观察到,每个机器人操纵任务都可以被视为涉及从被操纵对象的角度来看运动的任务,即,对象可以了解如何自己达到目标状态。为了利用这个想法,我们介绍了一个框架,最初使用现实物理模拟器获得对象运动策略。然后,此策略用于生成辅助奖励,称为模拟的机器人演示奖励(SLDRS),使我们能够学习机器人操纵策略。拟议的方法已在增加复杂性的13个任务中进行了评估,与替代算法相比,可以实现更高的成功率和更快的学习率。 SLDRS对多对象堆叠和非刚性物体操作等任务特别有益。
translated by 谷歌翻译
在本次调查中,我们介绍了执行需要不同于环境的操作任务的机器人的当前状态,使得机器人必须隐含地或明确地控制与环境的接触力来完成任务。机器人可以执行越来越多的人体操作任务,并且在1)主题上具有越来越多的出版物,其执行始终需要联系的任务,并且通过利用完美的任务来减轻环境来缓解不确定性信息,可以在没有联系的情况下进行。最近的趋势已经看到机器人在留下的人类留给人类,例如按摩,以及诸如PEG孔的经典任务中,对其他类似任务的概率更有效,更好的误差容忍以及更快的规划或学习任务。因此,在本调查中,我们涵盖了执行此类任务的机器人的当前阶段,从调查开始所有不同的联系方式机器人可以执行,观察这些任务是如何控制和表示的,并且最终呈现所需技能的学习和规划完成这些任务。
translated by 谷歌翻译
Surgical robot automation has attracted increasing research interest over the past decade, expecting its huge potential to benefit surgeons, nurses and patients. Recently, the learning paradigm of embodied AI has demonstrated promising ability to learn good control policies for various complex tasks, where embodied AI simulators play an essential role to facilitate relevant researchers. However, existing open-sourced simulators for surgical robot are still not sufficiently supporting human interactions through physical input devices, which further limits effective investigations on how human demonstrations would affect policy learning. In this paper, we study human-in-the-loop embodied intelligence with a new interactive simulation platform for surgical robot learning. Specifically, we establish our platform based on our previously released SurRoL simulator with several new features co-developed to allow high-quality human interaction via an input device. With these, we further propose to collect human demonstrations and imitate the action patterns to achieve more effective policy learning. We showcase the improvement of our simulation environment with the designed new features and tasks, and validate state-of-the-art reinforcement learning algorithms using the interactive environment. Promising results are obtained, with which we hope to pave the way for future research on surgical embodied intelligence. Our platform is released and will be continuously updated in the website: https://med-air.github.io/SurRoL/
translated by 谷歌翻译
Real-world autonomous missions often require rich interaction with nearby objects, such as doors or switches, along with effective navigation. However, such complex behaviors are difficult to learn because they involve both high-level planning and low-level motor control. We present a novel framework, Cascaded Compositional Residual Learning (CCRL), which learns composite skills by recursively leveraging a library of previously learned control policies. Our framework learns multiplicative policy composition, task-specific residual actions, and synthetic goal information simultaneously while freezing the prerequisite policies. We further explicitly control the style of the motion by regularizing residual actions. We show that our framework learns joint-level control policies for a diverse set of motor skills ranging from basic locomotion to complex interactive navigation, including navigating around obstacles, pushing objects, crawling under a table, pushing a door open with its leg, and holding it open while walking through it. The proposed CCRL framework leads to policies with consistent styles and lower joint torques, which we successfully transfer to a real Unitree A1 robot without any additional fine-tuning.
translated by 谷歌翻译
Complex and contact-rich robotic manipulation tasks, particularly those that involve multi-fingered hands and underactuated object manipulation, present a significant challenge to any control method. Methods based on reinforcement learning offer an appealing choice for such settings, as they can enable robots to learn to delicately balance contact forces and dexterously reposition objects without strong modeling assumptions. However, running reinforcement learning on real-world dexterous manipulation systems often requires significant manual engineering. This negates the benefits of autonomous data collection and ease of use that reinforcement learning should in principle provide. In this paper, we describe a system for vision-based dexterous manipulation that provides a "programming-free" approach for users to define new tasks and enable robots with complex multi-fingered hands to learn to perform them through interaction. The core principle underlying our system is that, in a vision-based setting, users should be able to provide high-level intermediate supervision that circumvents challenges in teleoperation or kinesthetic teaching which allow a robot to not only learn a task efficiently but also to autonomously practice. Our system includes a framework for users to define a final task and intermediate sub-tasks with image examples, a reinforcement learning procedure that learns the task autonomously without interventions, and experimental results with a four-finger robotic hand learning multi-stage object manipulation tasks directly in the real world, without simulation, manual modeling, or reward engineering.
translated by 谷歌翻译
实现人类水平的灵活性是机器人技术中的重要开放问题。但是,即使在婴儿级别,灵巧的手动操纵任务也是通过增强学习(RL)的挑战。困难在于高度的自由度和异质因素(例如手指关节)之间所需的合作。在这项研究中,我们提出了双人灵感手基准(BI-DEXHANDS),这是一种模拟器,涉及两只灵巧的手,其中包含数十只双人操纵任务和数千个目标对象。具体而言,根据认知科学文献,BI-DEXHANDS中的任务旨在匹配不同级别的人类运动技能。我们在ISSAC体育馆里建造了Bi-Dexhands;这可以实现高效的RL培训,仅在一个NVIDIA RTX 3090中达到30,000+ fps。我们在不同的设置下为流行的RL算法提供了全面的基准;这包括单代理/多代理RL,离线RL,多任务RL和META RL。我们的结果表明,PPO类型的上车算法可以掌握简单的操纵任务,该任务等效到48个月的人类婴儿(例如,捕获飞行的物体,打开瓶子),而多代理RL可以进一步帮助掌握掌握需要熟练的双人合作的操作(例如,举起锅,堆叠块)。尽管每个任务都取得了成功,但在获得多个操纵技能方面,现有的RL算法无法在大多数多任务和少量学习设置中工作,这需要从RL社区进行更实质性的发展。我们的项目通过https://github.com/pku-marl/dexteroushands开放。
translated by 谷歌翻译
虽然对理解计算机视觉中的手对象交互进行了重大进展,但机器人执行复杂的灵巧操纵仍然非常具有挑战性。在本文中,我们提出了一种新的平台和管道DEXMV(来自视频的Dexerous操纵)以进行模仿学习。我们设计了一个平台:(i)具有多指机器人手和(ii)计算机视觉系统的复杂灵巧操纵任务的仿真系统,以记录进行相同任务的人类手的大规模示范。在我们的小说管道中,我们从视频中提取3D手和对象姿势,并提出了一种新颖的演示翻译方法,将人类运动转换为机器人示范。然后,我们将多个仿制学习算法与演示进行应用。我们表明,示威活动确实可以通过大幅度提高机器人学习,并解决独自增强学习无法解决的复杂任务。具有视频的项目页面:https://yzqin.github.io/dexmv
translated by 谷歌翻译
长摩根和包括一系列隐性子任务的日常任务仍然在离线机器人控制中构成了重大挑战。尽管许多先前的方法旨在通过模仿和离线增强学习的变体来解决这种设置,但学习的行为通常是狭窄的,并且经常努力实现可配置的长匹配目标。由于这两个范式都具有互补的优势和劣势,因此我们提出了一种新型的层次结构方法,结合了两种方法的优势,以从高维相机观察中学习任务无关的长胜压策略。具体而言,我们结合了一项低级政策,该政策通过模仿学习和从离线强化学习中学到的高级政策学习潜在的技能,以促进潜在的行为先验。各种模拟和真实机器人控制任务的实验表明,我们的配方使以前看不见的技能组合能够通过“缝制”潜在技能通过目标链条,并在绩效上提高绩效的顺序,从而实现潜在的目标。艺术基线。我们甚至还学习了一个多任务视觉运动策略,用于现实世界中25个不同的操纵任务,这既优于模仿学习和离线强化学习技术。
translated by 谷歌翻译
技能链是一种希望通过顺序结合以前学习的技能来合成复杂行为的有希望的方法。然而,当政策遭遇在培训期间从未见过的起始状态时,幼稚的技能组成失败。对于成功的技能链接,先前的方法试图扩大策略的起始状态分布。然而,这些方法需要覆盖更大的状态分布,因为更多的策略进行测序,因此仅限于短的技能序列。在本文中,我们通过在对抗学习框架中规范终端状态分布来提出连锁多个初始状态分布的多重政策。我们评估了我们对家具组件的两个复杂的长地平衡任务的方法。我们的结果表明,我们的方法建立了第一种无模型加强学习算法来解决这些任务;而先前的技能链接方法失败。代码和视频可在https://clvrai.com/skill-chaining上获得
translated by 谷歌翻译
最先进的多机构增强学习(MARL)方法为各种复杂问题提供了有希望的解决方案。然而,这些方法都假定代理执行同步的原始操作执行,因此它们不能真正可扩展到长期胜利的真实世界多代理/机器人任务,这些任务固有地要求代理/机器人以异步的理由,涉及有关高级动作选择的理由。不同的时间。宏观行动分散的部分可观察到的马尔可夫决策过程(MACDEC-POMDP)是在完全合作的多代理任务中不确定的异步决策的一般形式化。在本论文中,我们首先提出了MacDec-Pomdps的一组基于价值的RL方法,其中允许代理在三个范式中使用宏观成果功能执行异步学习和决策:分散学习和控制,集中学习,集中学习和控制,以及分散执行的集中培训(CTDE)。在上述工作的基础上,我们在三个训练范式下制定了一组基于宏观行动的策略梯度算法,在该训练范式下,允许代理以异步方式直接优化其参数化策略。我们在模拟和真实的机器人中评估了我们的方法。经验结果证明了我们在大型多代理问题中的方法的优势,并验证了我们算法在学习具有宏观actions的高质量和异步溶液方面的有效性。
translated by 谷歌翻译
Hierarchical decomposition of control is unavoidable in large dynamical systems. In reinforcement learning (RL), it is usually solved with subgoals defined at higher policy levels and achieved at lower policy levels. Reaching these goals can take a substantial amount of time, during which it is not verified whether they are still worth pursuing. However, due to the randomness of the environment, these goals may become obsolete. In this paper, we address this gap in the state-of-the-art approaches and propose a method in which the validity of higher-level actions (thus lower-level goals) is constantly verified at the higher level. If the actions, i.e. lower level goals, become inadequate, they are replaced by more appropriate ones. This way we combine the advantages of hierarchical RL, which is fast training, and flat RL, which is immediate reactivity. We study our approach experimentally on seven benchmark environments.
translated by 谷歌翻译
移动操作(MM)系统是在非结构化现实世界环境中扮演个人助理角色的理想候选者。除其他挑战外,MM需要有效协调机器人的实施例,以执行需要移动性和操纵的任务。强化学习(RL)的承诺是将机器人具有自适应行为,但是大多数方法都需要大量的数据来学习有用的控制策略。在这项工作中,我们研究了机器人可及先验在参与者批判性RL方法中的整合,以加速学习和获取任务的MM学习。也就是说,我们考虑了最佳基础位置的问题以及是否激活ARM达到6D目标的后续决定。为此,我们设计了一种新型的混合RL方法,该方法可以共同处理离散和连续的动作,从而诉诸Gumbel-Softmax重新聚集化。接下来,我们使用来自经典方法的操作机器人工作区中的数据训练可及性。随后,我们得出了增强的混合RL(BHYRL),这是一种通过将其建模为残留近似器的总和来学习Q功能的新型算法。每当需要学习新任务时,我们都可以转移我们学到的残差并了解特定于任务的Q功能的组成部分,从而从先前的行为中维护任务结构。此外,我们发现将目标政策与先前的策略正规化产生更多的表达行为。我们评估了我们在达到难度增加和提取任务的模拟方面的方法,并显示了Bhyrl在基线方法上的卓越性能。最后,我们用Bhyrl零转移了我们学到的6D提取政策,以归功于我们的MM机器人Tiago ++。有关更多详细信息和代码发布,请参阅我们的项目网站:irosalab.com/rlmmbp
translated by 谷歌翻译
我们介绍了栖息地2.0(H2.0),这是一个模拟平台,用于培训交互式3D环境和复杂物理的场景中的虚拟机器人。我们为体现的AI堆栈 - 数据,仿真和基准任务做出了全面的贡献。具体来说,我们提出:(i)复制:一个由艺术家的,带注释的,可重新配置的3D公寓(匹配真实空间)与铰接对象(例如可以打开/关闭的橱柜和抽屉); (ii)H2.0:一个高性能物理学的3D模拟器,其速度超过8-GPU节点上的每秒25,000个模拟步骤(实时850x实时),代表先前工作的100倍加速;和(iii)家庭助理基准(HAB):一套辅助机器人(整理房屋,准备杂货,设置餐桌)的一套常见任务,以测试一系列移动操作功能。这些大规模的工程贡献使我们能够系统地比较长期结构化任务中的大规模加固学习(RL)和经典的感官平面操作(SPA)管道,并重点是对新对象,容器和布局的概括。 。我们发现(1)与层次结构相比,(1)平面RL政策在HAB上挣扎; (2)具有独立技能的层次结构遭受“交接问题”的困扰,(3)水疗管道比RL政策更脆。
translated by 谷歌翻译