从演示中学习的方法（LFD）通过模仿用户表现出在获取行为策略方面的成功。但是，即使对于一项任务，LFD也可能需要大量的演示。对于必须通过演示学习许多任务的多功能代理，如果孤立地学习每个任务，此过程将大大负担用户的负担。为了应对这一挑战，我们介绍了从演示中学习的新颖问题，该问题使代理商能够不断地基于从先前演示的任务中学到的知识，以加速学习新任务，从而减少所需的示范量。作为解决这个问题的一种解决方案，我们提出了第一种终身学习方法来进行逆强化学习，该方法通过演示学习连续的任务，不断地在任务之间转移知识以提高绩效。

The reinforcement learning paradigm is a popular way to address problems that have only limited environmental feedback, rather than correctly labeled examples, as is common in other machine learning contexts. While significant progress has been made to improve learning in a single task, the idea of transfer learning has only recently been applied to reinforcement learning tasks. The core idea of transfer is that experience gained in learning to perform one task can help improve learning performance in a related, but different, task. In this article we present a framework that classifies transfer learning methods in terms of their capabilities and goals, and then use it to survey the existing literature, as well as to suggest future directions for transfer learning work.

本文解决了逆增强学习（IRL）的问题 - 从观察其行为中推断出代理的奖励功能。 IRL可以为学徒学习提供可概括和紧凑的代表，并能够准确推断人的偏好以帮助他们。 ％并提供更准确的预测。但是，有效的IRL具有挑战性，因为许多奖励功能可以与观察到的行为兼容。我们专注于如何利用先前的强化学习（RL）经验，以使学习这些偏好更快，更高效。我们提出了IRL算法基础（通过样本中的连续功能意图推断行为获取行为），该算法利用多任务RL预培训和后继功能，使代理商可以为跨越可能的目标建立强大的基础，从而跨越可能的目标。给定的域。当仅接触一些专家演示以优化新颖目标时，代理商会使用其基础快速有效地推断奖励功能。我们的实验表明，我们的方法非常有效地推断和优化显示出奖励功能，从而准确地从少于100个轨迹中推断出奖励功能。

对于许多强化学习（RL）应用程序，指定奖励是困难的。本文考虑了一个RL设置，其中代理仅通过查询可以询问可以的专家来获取有关奖励的信息，例如，评估单个状态或通过轨迹提供二进制偏好。从如此昂贵的反馈中，我们的目标是学习奖励的模型，允许标准RL算法实现高预期的回报，尽可能少的专家查询。为此，我们提出了信息定向奖励学习（IDRL），它使用奖励的贝叶斯模型，然后选择要最大化信息增益的查询，这些查询是有关合理的最佳策略之间的返回差异的差异。与针对特定类型查询设计的先前主动奖励学习方法相比，IDRL自然地适应不同的查询类型。此外，它通过将焦点转移降低奖励近似误差来实现类似或更好的性能，从而降低奖励近似误差，以改善奖励模型引起的策略。我们支持我们的调查结果，在多个环境中进行广泛的评估，并具有不同的查询类型。

Imitation learning techniques aim to mimic human behavior in a given task. An agent (a learning machine) is trained to perform a task from demonstrations by learning a mapping between observations and actions. The idea of teaching by imitation has been around for many years, however, the field is gaining attention recently due to advances in computing and sensing as well as rising demand for intelligent applications. The paradigm of learning by imitation is gaining popularity because it facilitates teaching complex tasks with minimal expert knowledge of the tasks. Generic imitation learning methods could potentially reduce the problem of teaching a task to that of providing demonstrations; without the need for explicit programming or designing reward functions specific to the task. Modern sensors are able to collect and transmit high volumes of data rapidly, and processors with high computational power allow fast processing that maps the sensory data to actions in a timely manner. This opens the door for many potential AI applications that require real-time perception and reaction such as humanoid robots, self-driving vehicles, human computer interaction and computer games to name a few. However, specialized algorithms are needed to effectively and robustly learn models as learning by imitation poses its own set of challenges. In this paper, we survey imitation learning methods and present design options in different steps of the learning process. We introduce a background and motivation for the field as well as highlight challenges specific to the imitation problem. Methods for designing and evaluating imitation learning tasks are categorized and reviewed. Special attention is given to learning methods in robotics and games as these domains are the most popular in the literature and provide a wide array of problems and methodologies. We extensively discuss combining imitation learning approaches using different sources and methods, as well as incorporating other motion learning methods to enhance imitation. We also discuss the potential impact on industry, present major applications and highlight current and future research directions.

逆强化学习（IRL）试图推断出一种成本函数，以解释专家演示的基本目标和偏好。本文介绍了向后的地平线逆增强学习（RHIRL），这是一种新的IRL算法，用于具有黑盒动态模型的高维，嘈杂，连续的系统。 Rhirl解决了IRL的两个主要挑战：可伸缩性和鲁棒性。为了处理高维的连续系统，Rhirl以退缩的地平线方式与当地的专家演示相匹配，并将其“针迹”一起“缝制”本地解决方案以学习成本；因此，它避免了“维度的诅咒”。这与早期的算法形成鲜明对比，这些算法与在整个高维状态空间中与全球范围内的专家示威相匹配。为了与不完美的专家示范和系统控制噪声保持强大的态度，Rhirl在轻度条件下学习了与系统动力学的状态依赖性成本函数。基准任务的实验表明，在大多数情况下，Rhirl的表现都优于几种领先的IRL算法。我们还证明，Rhirl的累积误差随任务持续时间线性增长。

我们研究自主代理如何学会从不同领域（例如不同环境或不同代理）中的示范中执行任务。这样的跨域模仿学习需要例如从人类专家的演示中培训人造代理。我们提出了一个可扩展的框架，该框架可以实现跨域模仿学习，而无需访问其他演示或进一步的领域知识。我们共同培训学习者的政策，并通过对抗性培训学习学习者和专家领域的映射。我们通过使用共同信息标准来找到包含与任务相关的信息的专家状态空间的嵌入，并且对域细节不变。此步骤大大简化了估计学习者和专家领域之间的映射，因此有助于端到端学习。我们证明了在相当不同的域之间成功转移了政策，而没有额外的示范，以及其他方法失败的情况。

在过去的几年中，逆增强学习（\ textit {irl}）问题已经迅速发展，在机器人技术，认知和健康等领域中具有重要的应用。在这项工作中，我们探讨了当前IRL方法从描述长马，复杂的顺序任务的专家轨迹中学习代理奖励函数的效率低下。我们假设，将IRL模型带入捕获基本任务的结构图案可以实现和提高其性能。随后，我们提出了一种新颖的IRL方法Smirl，该方法首先学习任务的（近似）结构为有限状态-Satate-automaton（FSA），然后使用结构基序来解决IRL问题。我们在离散网格世界和高维连续域环境上测试我们的模型。我们从经验上表明，我们提出的方法成功地学习了所有四个复杂的任务，其中两个基础IRL基准失败了。我们的模型还优于简单的玩具任务中样本效率的基准。我们进一步在具有组成奖励函数的任务上的经过修改的连续域中显示了有希望的测试结果。

在许多顺序决策问题（例如，机器人控制，游戏播放，顺序预测），人类或专家数据可用包含有关任务的有用信息。然而，来自少量专家数据的模仿学习（IL）可能在具有复杂动态的高维环境中具有挑战性。行为克隆是一种简单的方法，由于其简单的实现和稳定的收敛而被广泛使用，但不利用涉及环境动态的任何信息。由于对奖励和政策近似器或偏差，高方差梯度估计器，难以在实践中难以在实践中努力训练的许多现有方法。我们介绍了一种用于动态感知IL的方法，它通过学习单个Q函数来避免对抗训练，隐含地代表奖励和策略。在标准基准测试中，隐式学习的奖励显示与地面真实奖励的高正面相关性，说明我们的方法也可以用于逆钢筋学习（IRL）。我们的方法，逆软Q学习（IQ-Learn）获得了最先进的结果，在离线和在线模仿学习设置中，显着优于现有的现有方法，这些方法都在所需的环境交互和高维空间中的可扩展性中，通常超过3倍。

In inverse reinforcement learning (IRL), a learning agent infers a reward function encoding the underlying task using demonstrations from experts. However, many existing IRL techniques make the often unrealistic assumption that the agent has access to full information about the environment. We remove this assumption by developing an algorithm for IRL in partially observable Markov decision processes (POMDPs). We address two limitations of existing IRL techniques. First, they require an excessive amount of data due to the information asymmetry between the expert and the learner. Second, most of these IRL techniques require solving the computationally intractable forward problem -- computing an optimal policy given a reward function -- in POMDPs. The developed algorithm reduces the information asymmetry while increasing the data efficiency by incorporating task specifications expressed in temporal logic into IRL. Such specifications may be interpreted as side information available to the learner a priori in addition to the demonstrations. Further, the algorithm avoids a common source of algorithmic complexity by building on causal entropy as the measure of the likelihood of the demonstrations as opposed to entropy. Nevertheless, the resulting problem is nonconvex due to the so-called forward problem. We solve the intrinsic nonconvexity of the forward problem in a scalable manner through a sequential linear programming scheme that guarantees to converge to a locally optimal policy. In a series of examples, including experiments in a high-fidelity Unity simulator, we demonstrate that even with a limited amount of data and POMDPs with tens of thousands of states, our algorithm learns reward functions and policies that satisfy the task while inducing similar behavior to the expert by leveraging the provided side information.

With the development of deep representation learning, the domain of reinforcement learning (RL) has become a powerful learning framework now capable of learning complex policies in high dimensional environments. This review summarises deep reinforcement learning (DRL) algorithms and provides a taxonomy of automated driving tasks where (D)RL methods have been employed, while addressing key computational challenges in real world deployment of autonomous driving agents. It also delineates adjacent domains such as behavior cloning, imitation learning, inverse reinforcement learning that are related but are not classical RL algorithms. The role of simulators in training agents, methods to validate, test and robustify existing solutions in RL are discussed.

我们研究逆增强学习（IRL）和模仿学习（IM），这是从专家所证明的轨迹中恢复奖励或政策功能的问题。我们提出了一种新的方法来通过在最大的熵框架中添加权重功能来改善学习过程，并具有学习和恢复专家政策的随机性（或有限理性）的动机。我们的框架和算法允许学习奖励（或政策）功能以及添加到马尔可夫决策过程中的熵条款的结构，从而增强了学习过程。我们使用人类和模拟演示以及通过离散和连续的IRL/IM任务进行的数值实验表明，我们的方法表现优于先前的算法。

Recent research has shown the benefit of framing problems of imitation learning as solutions to Markov Decision Problems. This approach reduces learning to the problem of recovering a utility function that makes the behavior induced by a near-optimal policy closely mimic demonstrated behavior. In this work, we develop a probabilistic approach based on the principle of maximum entropy. Our approach provides a well-defined, globally normalized distribution over decision sequences, while providing the same performance guarantees as existing methods. We develop our technique in the context of modeling realworld navigation and driving behaviors where collected data is inherently noisy and imperfect. Our probabilistic approach enables modeling of route preferences as well as a powerful new approach to inferring destinations and routes based on partial trajectories.

Inferring reward functions from human behavior is at the center of value alignment - aligning AI objectives with what we, humans, actually want. But doing so relies on models of how humans behave given their objectives. After decades of research in cognitive science, neuroscience, and behavioral economics, obtaining accurate human models remains an open research topic. This begs the question: how accurate do these models need to be in order for the reward inference to be accurate? On the one hand, if small errors in the model can lead to catastrophic error in inference, the entire framework of reward learning seems ill-fated, as we will never have perfect models of human behavior. On the other hand, if as our models improve, we can have a guarantee that reward accuracy also improves, this would show the benefit of more work on the modeling side. We study this question both theoretically and empirically. We do show that it is unfortunately possible to construct small adversarial biases in behavior that lead to arbitrarily large errors in the inferred reward. However, and arguably more importantly, we are also able to identify reasonable assumptions under which the reward inference error can be bounded linearly in the error in the human model. Finally, we verify our theoretical insights in discrete and continuous control tasks with simulated and human data.

This paper surveys the eld of reinforcement learning from a computer-science perspective. It is written to be accessible to researchers familiar with machine learning. Both the historical basis of the eld and a broad selection of current work are summarized. Reinforcement learning is the problem faced by an agent that learns behavior through trial-and-error interactions with a dynamic environment. The work described here has a resemblance to work in psychology, but di ers considerably in the details and in the use of the word \reinforcement." The paper discusses central issues of reinforcement learning, including trading o exploration and exploitation, establishing the foundations of the eld via Markov decision theory, learning from delayed reinforcement, constructing empirical models to accelerate learning, making use of generalization and hierarchy, and coping with hidden state. It concludes with a survey of some implemented systems and an assessment of the practical utility of current methods for reinforcement learning.

Safe Reinforcement Learning can be defined as the process of learning policies that maximize the expectation of the return in problems in which it is important to ensure reasonable system performance and/or respect safety constraints during the learning and/or deployment processes. We categorize and analyze two approaches of Safe Reinforcement Learning. The first is based on the modification of the optimality criterion, the classic discounted finite/infinite horizon, with a safety factor. The second is based on the modification of the exploration process through the incorporation of external knowledge or the guidance of a risk metric. We use the proposed classification to survey the existing literature, as well as suggesting future directions for Safe Reinforcement Learning.

A long-standing challenge in artificial intelligence is lifelong learning. In lifelong learning, many tasks are presented in sequence and learners must efficiently transfer knowledge between tasks while avoiding catastrophic forgetting over long lifetimes. On these problems, policy reuse and other multi-policy reinforcement learning techniques can learn many tasks. However, they can generate many temporary or permanent policies, resulting in memory issues. Consequently, there is a need for lifetime-scalable methods that continually refine a policy library of a pre-defined size. This paper presents a first approach to lifetime-scalable policy reuse. To pre-select the number of policies, a notion of task capacity, the maximal number of tasks that a policy can accurately solve, is proposed. To evaluate lifetime policy reuse using this method, two state-of-the-art single-actor base-learners are compared: 1) a value-based reinforcement learner, Deep Q-Network (DQN) or Deep Recurrent Q-Network (DRQN); and 2) an actor-critic reinforcement learner, Proximal Policy Optimisation (PPO) with or without Long Short-Term Memory layer. By selecting the number of policies based on task capacity, D(R)QN achieves near-optimal performance with 6 policies in a 27-task MDP domain and 9 policies in an 18-task POMDP domain; with fewer policies, catastrophic forgetting and negative transfer are observed. Due to slow, monotonic improvement, PPO requires fewer policies, 1 policy for the 27-task domain and 4 policies for the 18-task domain, but it learns the tasks with lower accuracy than D(R)QN. These findings validate lifetime-scalable policy reuse and suggest using D(R)QN for larger and PPO for smaller library sizes.

我们考虑通过序贯决策设置中的示范教学问题。特别是，我们研究如何在演示中设计一个个性化课程，以加快学习者的融合。我们为两种流行的学习者型号提供统一的课程策略：最大因果熵逆加强学习（MaxEnt-IRL）和交叉熵行为克隆（Crossent-BC）。我们的统一战略基于难度评分计算的概念来突出排名。教师的最佳政策和学习者的当前政策。与现有技术相比，我们的战略不需要访问学习者的内部动态，并且在轻度技术条件下仍然享有类似的收敛保证。此外，我们将我们的课程策略调整到使用特定于任务特定难度分数的教师代理的环境。在合成车驾驶环境和基于导航环境的实验证明了我们课程策略的有效性。

到目前为止，大多数关于推荐系统的研究专注于通过促进相关和个性化内容维持长期用户参与和满足感。但是，评估这种内容的质量和可靠性仍然非常具有挑战性。在本文中，我们提出了FEBR（基于专家的建议框架），是评估在线平台上建议内容的质量的学徒学习框架。该框架在推荐评估环境中挖掘专家（假设可靠）的演示轨迹，以恢复未知的实用程序功能。此功能用于学习描述专家行为的最佳策略，然后在框架中使用，以提供高质量和个性化的建议。我们通过用户兴趣模拟环境（使用RECSIM）评估我们的解决方案的性能。我们模拟了上述专家政策下的互动，以进行视频推荐，并将其效率与标准推荐方法进行比较。结果表明，我们的方法在内容质量方面提供了显着的收益，由专家评估并由用户观察，同时保持与基线方法几乎相同的表格。

Adequately assigning credit to actions for future outcomes based on their contributions is a long-standing open challenge in Reinforcement Learning. The assumptions of the most commonly used credit assignment method are disadvantageous in tasks where the effects of decisions are not immediately evident. Furthermore, this method can only evaluate actions that have been selected by the agent, making it highly inefficient. Still, no alternative methods have been widely adopted in the field. Hindsight Credit Assignment is a promising, but still unexplored candidate, which aims to solve the problems of both long-term and counterfactual credit assignment. In this thesis, we empirically investigate Hindsight Credit Assignment to identify its main benefits, and key points to improve. Then, we apply it to factored state representations, and in particular to state representations based on the causal structure of the environment. In this setting, we propose a variant of Hindsight Credit Assignment that effectively exploits a given causal structure. We show that our modification greatly decreases the workload of Hindsight Credit Assignment, making it more efficient and enabling it to outperform the baseline credit assignment method on various tasks. This opens the way to other methods based on given or learned causal structures.