Reinforcement learning can acquire complex behaviors from high-level specifications. However, defining a cost function that can be optimized effectively and encodes the correct task is challenging in practice. We explore how inverse optimal control (IOC) can be used to learn behaviors from demonstrations, with applications to torque control of high-dimensional robotic systems. Our method addresses two key challenges in inverse optimal control: first, the need for informative features and effective regularization to impose structure on the cost, and second, the difficulty of learning the cost function under unknown dynamics for high-dimensional continuous systems. To address the former challenge, we present an algorithm capable of learning arbitrary nonlinear cost functions, such as neural networks, without meticulous feature engineering. To address the latter challenge, we formulate an efficient sample-based approximation for MaxEnt IOC. We evaluate our method on a series of simulated tasks and real-world robotic manipulation problems, demonstrating substantial improvement over prior methods both in terms of task complexity and sample efficiency.

Policy search methods can allow robots to learn control policies for a wide range of tasks, but practical applications of policy search often require hand-engineered components for perception, state estimation, and low-level control. In this paper, we aim to answer the following question: does training the perception and control systems jointly end-toend provide better performance than training each component separately? To this end, we develop a method that can be used to learn policies that map raw image observations directly to torques at the robot's motors. The policies are represented by deep convolutional neural networks (CNNs) with 92,000 parameters, and are trained using a guided policy search method, which transforms policy search into supervised learning, with supervision provided by a simple trajectory-centric reinforcement learning method. We evaluate our method on a range of real-world manipulation tasks that require close coordination between vision and control, such as screwing a cap onto a bottle, and present simulated comparisons to a range of prior policy search methods.

Direct policy search can effectively scale to high-dimensional systems, but complex policies with hundreds of parameters often present a challenge for such methods, requiring numerous samples and often falling into poor local optima. We present a guided policy search algorithm that uses trajectory optimization to direct policy learning and avoid poor local optima. We show how differential dynamic programming can be used to generate suitable guiding samples, and describe a regularized importance sampled policy optimization that incorporates these samples into the policy search. We evaluate the method by learning neural network controllers for planar swimming, hopping, and walking, as well as simulated 3D humanoid running.

Reinforcement learning holds the promise of enabling autonomous robots to learn large repertoires of behavioral skills with minimal human intervention. However, robotic applications of reinforcement learning often compromise the autonomy of the learning process in favor of achieving training times that are practical for real physical systems. This typically involves introducing hand-engineered policy representations and human-supplied demonstrations. Deep reinforcement learning alleviates this limitation by training general-purpose neural network policies, but applications of direct deep reinforcement learning algorithms have so far been restricted to simulated settings and relatively simple tasks, due to their apparent high sample complexity. In this paper, we demonstrate that a recent deep reinforcement learning algorithm based on offpolicy training of deep Q-functions can scale to complex 3D manipulation tasks and can learn deep neural network policies efficiently enough to train on real physical robots. We demonstrate that the training times can be further reduced by parallelizing the algorithm across multiple robots which pool their policy updates asynchronously. Our experimental evaluation shows that our method can learn a variety of 3D manipulation skills in simulation and a complex door opening skill on real robots without any prior demonstrations or manually designed representations.

Imitation learning techniques aim to mimic human behavior in a given task. An agent (a learning machine) is trained to perform a task from demonstrations by learning a mapping between observations and actions. The idea of teaching by imitation has been around for many years, however, the field is gaining attention recently due to advances in computing and sensing as well as rising demand for intelligent applications. The paradigm of learning by imitation is gaining popularity because it facilitates teaching complex tasks with minimal expert knowledge of the tasks. Generic imitation learning methods could potentially reduce the problem of teaching a task to that of providing demonstrations; without the need for explicit programming or designing reward functions specific to the task. Modern sensors are able to collect and transmit high volumes of data rapidly, and processors with high computational power allow fast processing that maps the sensory data to actions in a timely manner. This opens the door for many potential AI applications that require real-time perception and reaction such as humanoid robots, self-driving vehicles, human computer interaction and computer games to name a few. However, specialized algorithms are needed to effectively and robustly learn models as learning by imitation poses its own set of challenges. In this paper, we survey imitation learning methods and present design options in different steps of the learning process. We introduce a background and motivation for the field as well as highlight challenges specific to the imitation problem. Methods for designing and evaluating imitation learning tasks are categorized and reviewed. Special attention is given to learning methods in robotics and games as these domains are the most popular in the literature and provide a wide array of problems and methodologies. We extensively discuss combining imitation learning approaches using different sources and methods, as well as incorporating other motion learning methods to enhance imitation. We also discuss the potential impact on industry, present major applications and highlight current and future research directions.

从意外的外部扰动中恢复的能力是双模型运动的基本机动技能。有效的答复包括不仅可以恢复平衡并保持稳定性的能力，而且在平衡恢复物质不可行时，也可以保证安全的方式。对于与双式运动有关的机器人，例如人形机器人和辅助机器人设备，可帮助人类行走，设计能够提供这种稳定性和安全性的控制器可以防止机器人损坏或防止伤害相关的医疗费用。这是一个具有挑战性的任务，因为它涉及用触点产生高维，非线性和致动系统的高动态运动。尽管使用基于模型和优化方法的前进方面，但诸如广泛领域知识的要求，诸如较大的计算时间和有限的动态变化的鲁棒性仍然会使这个打开问题。在本文中，为了解决这些问题，我们开发基于学习的算法，能够为两种不同的机器人合成推送恢复控制政策：人形机器人和有助于双模型运动的辅助机器人设备。我们的工作可以分为两个密切相关的指示：1）学习人形机器人的安全下降和预防策略，2）使用机器人辅助装置学习人类的预防策略。为实现这一目标，我们介绍了一套深度加强学习（DRL）算法，以学习使用这些机器人时提高安全性的控制策略。

在本文中，我们关注将基于能量的模型（EBM）作为运动优化的指导先验的问题。 EBM是一组神经网络，可以用合适的能量函数参数为参数的GIBBS分布来表示表达概率密度分布。由于其隐含性，它们可以轻松地作为优化因素或运动优化问题中的初始采样分布整合在一起，从而使它们成为良好的候选者，以将数据驱动的先验集成在运动优化问题中。在这项工作中，我们提出了一组所需的建模和算法选择，以使EBMS适应运动优化。我们调查了将其他正规化器在学习EBM中的好处，以将它们与基于梯度的优化器一起使用，并提供一组EBM架构，以学习用于操纵任务的可通用分布。我们提出了多种情况，可以将EBM集成以进行运动优化，并评估学到的EBM的性能，以指导模拟和真实机器人实验的指导先验。

For an autonomous agent to fulfill a wide range of user-specified goals at test time, it must be able to learn broadly applicable and general-purpose skill repertoires. Furthermore, to provide the requisite level of generality, these skills must handle raw sensory input such as images. In this paper, we propose an algorithm that acquires such general-purpose skills by combining unsupervised representation learning and reinforcement learning of goal-conditioned policies. Since the particular goals that might be required at test-time are not known in advance, the agent performs a self-supervised "practice" phase where it imagines goals and attempts to achieve them. We learn a visual representation with three distinct purposes: sampling goals for self-supervised practice, providing a structured transformation of raw sensory inputs, and computing a reward signal for goal reaching. We also propose a retroactive goal relabeling scheme to further improve the sample-efficiency of our method. Our off-policy algorithm is efficient enough to learn policies that operate on raw image observations and goals for a real-world robotic system, and substantially outperforms prior techniques. * Equal contribution. Order was determined by coin flip.

在本次调查中，我们介绍了执行需要不同于环境的操作任务的机器人的当前状态，使得机器人必须隐含地或明确地控制与环境的接触力来完成任务。机器人可以执行越来越多的人体操作任务，并且在1）主题上具有越来越多的出版物，其执行始终需要联系的任务，并且通过利用完美的任务来减轻环境来缓解不确定性信息，可以在没有联系的情况下进行。最近的趋势已经看到机器人在留下的人类留给人类，例如按摩，以及诸如PEG孔的经典任务中，对其他类似任务的概率更有效，更好的误差容忍以及更快的规划或学习任务。因此，在本调查中，我们涵盖了执行此类任务的机器人的当前阶段，从调查开始所有不同的联系方式机器人可以执行，观察这些任务是如何控制和表示的，并且最终呈现所需技能的学习和规划完成这些任务。

逆强化学习（IRL）试图推断出一种成本函数，以解释专家演示的基本目标和偏好。本文介绍了向后的地平线逆增强学习（RHIRL），这是一种新的IRL算法，用于具有黑盒动态模型的高维，嘈杂，连续的系统。 Rhirl解决了IRL的两个主要挑战：可伸缩性和鲁棒性。为了处理高维的连续系统，Rhirl以退缩的地平线方式与当地的专家演示相匹配，并将其“针迹”一起“缝制”本地解决方案以学习成本；因此，它避免了“维度的诅咒”。这与早期的算法形成鲜明对比，这些算法与在整个高维状态空间中与全球范围内的专家示威相匹配。为了与不完美的专家示范和系统控制噪声保持强大的态度，Rhirl在轻度条件下学习了与系统动力学的状态依赖性成本函数。基准任务的实验表明，在大多数情况下，Rhirl的表现都优于几种领先的IRL算法。我们还证明，Rhirl的累积误差随任务持续时间线性增长。

Sampling-based Model Predictive Control (MPC) is a flexible control framework that can reason about non-smooth dynamics and cost functions. Recently, significant work has focused on the use of machine learning to improve the performance of MPC, often through learning or fine-tuning the dynamics or cost function. In contrast, we focus on learning to optimize more effectively. In other words, to improve the update rule within MPC. We show that this can be particularly useful in sampling-based MPC, where we often wish to minimize the number of samples for computational reasons. Unfortunately, the cost of computational efficiency is a reduction in performance; fewer samples results in noisier updates. We show that we can contend with this noise by learning how to update the control distribution more effectively and make better use of the few samples that we have. Our learned controllers are trained via imitation learning to mimic an expert which has access to substantially more samples. We test the efficacy of our approach on multiple simulated robotics tasks in sample-constrained regimes and demonstrate that our approach can outperform a MPC controller with the same number of samples.

Sampling-based methods have become a cornerstone of contemporary approaches to Model Predictive Control (MPC), as they make no restrictions on the differentiability of the dynamics or cost function and are straightforward to parallelize. However, their efficacy is highly dependent on the quality of the sampling distribution itself, which is often assumed to be simple, like a Gaussian. This restriction can result in samples which are far from optimal, leading to poor performance. Recent work has explored improving the performance of MPC by sampling in a learned latent space of controls. However, these methods ultimately perform all MPC parameter updates and warm-starting between time steps in the control space. This requires us to rely on a number of heuristics for generating samples and updating the distribution and may lead to sub-optimal performance. Instead, we propose to carry out all operations in the latent space, allowing us to take full advantage of the learned distribution. Specifically, we frame the learning problem as bi-level optimization and show how to train the controller with backpropagation-through-time. By using a normalizing flow parameterization of the distribution, we can leverage its tractable density to avoid requiring differentiability of the dynamics and cost function. Finally, we evaluate the proposed approach on simulated robotics tasks and demonstrate its ability to surpass the performance of prior methods and scale better with a reduced number of samples.

策略搜索和模型预测控制〜（MPC）是机器人控制的两个不同范式：策略搜索具有使用经验丰富的数据自动学习复杂策略的强度，而MPC可以使用模型和轨迹优化提供最佳控制性能。开放的研究问题是如何利用并结合两种方法的优势。在这项工作中，我们通过使用策略搜索自动选择MPC的高级决策变量提供答案，这导致了一种新的策略搜索 - 用于模型预测控制框架。具体地，我们将MPC作为参数化控制器配制，其中难以优化的决策变量表示为高级策略。这种制定允许以自我监督的方式优化政策。我们通过专注于敏捷无人机飞行中的具有挑战性的问题来验证这一框架：通过快速的盖茨飞行四轮车。实验表明，我们的控制器在模拟和现实世界中实现了鲁棒和实时的控制性能。拟议的框架提供了合并学习和控制的新视角。

机器人和与世界相互作用或互动的机器人和智能系统越来越多地被用来自动化各种任务。这些系统完成这些任务的能力取决于构成机器人物理及其传感器物体的机械和电气部件，例如，感知算法感知环境，并计划和控制算法以生产和控制算法来生产和控制算法有意义的行动。因此，通常有必要在设计具体系统时考虑这些组件之间的相互作用。本文探讨了以端到端方式对机器人系统进行任务驱动的合作的工作，同时使用推理或控制算法直接优化了系统的物理组件以进行任务性能。我们首先考虑直接优化基于信标的本地化系统以达到本地化准确性的问题。设计这样的系统涉及将信标放置在整个环境中，并通过传感器读数推断位置。在我们的工作中，我们开发了一种深度学习方法，以直接优化信标的放置和位置推断以达到本地化精度。然后，我们将注意力转移到了由任务驱动的机器人及其控制器优化的相关问题上。在我们的工作中，我们首先提出基于多任务增强学习的数据有效算法。我们的方法通过利用能够在物理设计的空间上概括设计条件的控制器，有效地直接优化了物理设计和控制参数，以直接优化任务性能。然后，我们对此进行跟进，以允许对离散形态参数（例如四肢的数字和配置）进行优化。最后，我们通过探索优化的软机器人的制造和部署来得出结论。

强化学习（RL）算法有望为机器人系统实现自主技能获取。但是，实际上，现实世界中的机器人RL通常需要耗时的数据收集和频繁的人类干预来重置环境。此外，当部署超出知识的设置超出其学习的设置时，使用RL学到的机器人政策通常会失败。在这项工作中，我们研究了如何通过从先前看到的任务中收集的各种离线数据集的有效利用来应对这些挑战。当面对一项新任务时，我们的系统会适应以前学习的技能，以快速学习执行新任务并将环境返回到初始状态，从而有效地执行自己的环境重置。我们的经验结果表明，将先前的数据纳入机器人增强学习中可以实现自主学习，从而大大提高了学习的样本效率，并可以更好地概括。

Deep reinforcement learning is poised to revolutionise the field of AI and represents a step towards building autonomous systems with a higher level understanding of the visual world. Currently, deep learning is enabling reinforcement learning to scale to problems that were previously intractable, such as learning to play video games directly from pixels. Deep reinforcement learning algorithms are also applied to robotics, allowing control policies for robots to be learned directly from camera inputs in the real world. In this survey, we begin with an introduction to the general field of reinforcement learning, then progress to the main streams of value-based and policybased methods. Our survey will cover central algorithms in deep reinforcement learning, including the deep Q-network, trust region policy optimisation, and asynchronous advantage actor-critic. In parallel, we highlight the unique advantages of deep neural networks, focusing on visual understanding via reinforcement learning. To conclude, we describe several current areas of research within the field.

Robots need to be able to adapt to unexpected changes in the environment such that they can autonomously succeed in their tasks. However, hand-designing feedback models for adaptation is tedious, if at all possible, making data-driven methods a promising alternative. In this paper we introduce a full framework for learning feedback models for reactive motion planning. Our pipeline starts by segmenting demonstrations of a complete task into motion primitives via a semi-automated segmentation algorithm. Then, given additional demonstrations of successful adaptation behaviors, we learn initial feedback models through learning from demonstrations. In the final phase, a sample-efficient reinforcement learning algorithm fine-tunes these feedback models for novel task settings through few real system interactions. We evaluate our approach on a real anthropomorphic robot in learning a tactile feedback task.

在现实世界中，教授多指的灵巧机器人在现实世界中掌握物体，这是一个充满挑战的问题，由于其高维状态和动作空间。我们提出了一个机器人学习系统，该系统可以进行少量的人类示范，并学会掌握在某些被遮挡的观察结果的情况下掌握看不见的物体姿势。我们的系统利用了一个小型运动捕获数据集，并为多指的机器人抓手生成具有多种多样且成功的轨迹的大型数据集。通过添加域随机化，我们表明我们的数据集提供了可以将其转移到策略学习者的强大抓地力轨迹。我们训练一种灵活的抓紧策略，该策略将对象的点云作为输入，并预测连续的动作以从不同初始机器人状态掌握对象。我们在模拟中评估了系统对22多伏的浮动手的有效性，并在现实世界中带有kuka手臂的23多杆Allegro机器人手。从我们的数据集中汲取的政策可以很好地概括在模拟和现实世界中的看不见的对象姿势

We adapt the ideas underlying the success of Deep Q-Learning to the continuous action domain. We present an actor-critic, model-free algorithm based on the deterministic policy gradient that can operate over continuous action spaces. Using the same learning algorithm, network architecture and hyper-parameters, our algorithm robustly solves more than 20 simulated physics tasks, including classic problems such as cartpole swing-up, dexterous manipulation, legged locomotion and car driving. Our algorithm is able to find policies whose performance is competitive with those found by a planning algorithm with full access to the dynamics of the domain and its derivatives. We further demonstrate that for many of the tasks the algorithm can learn policies "end-to-end": directly from raw pixel inputs.

We describe an iterative procedure for optimizing policies, with guaranteed monotonic improvement. By making several approximations to the theoretically-justified procedure, we develop a practical algorithm, called Trust Region Policy Optimization (TRPO). This algorithm is similar to natural policy gradient methods and is effective for optimizing large nonlinear policies such as neural networks. Our experiments demonstrate its robust performance on a wide variety of tasks: learning simulated robotic swimming, hopping, and walking gaits; and playing Atari games using images of the screen as input. Despite its approximations that deviate from the theory, TRPO tends to give monotonic improvement, with little tuning of hyperparameters.