System identification, also known as learning forward models, transfer functions, system dynamics, etc., has a long tradition both in science and engineering in different fields. Particularly, it is a recurring theme in Reinforcement Learning research, where forward models approximate the state transition function of a Markov Decision Process by learning a mapping function from current state and action to the next state. This problem is commonly defined as a Supervised Learning problem in a direct way. This common approach faces several difficulties due to the inherent complexities of the dynamics to learn, for example, delayed effects, high non-linearity, non-stationarity, partial observability and, more important, error accumulation when using bootstrapped predictions (predictions based on past predictions), over large time horizons. Here we explore the use of Reinforcement Learning in this problem. We elaborate on why and how this problem fits naturally and sound as a Reinforcement Learning problem, and present some experimental results that demonstrate RL is a promising technique to solve these kind of problems.

解决稀疏奖励的多目标强化学习（RL）问题通常是具有挑战性的。现有方法利用目标依赖收集的经验，以减轻稀疏奖励提出的问题。然而，这些方法仍然有效，无法充分利用经验。在本文中，我们提出了基于模型的后敏感体验重放（MIRH），通过利用环境动态来产生虚拟实现的目标，更有效地利用更有效的体验。用从训练有素的动态模型的交互中产生的虚拟目标替换原始目标导致一种新的重定相制方法，基于模型的重新标记（MBR）。基于MBR，MEHER执行加强学习和监督学习以获得高效的政策改进。从理论上讲，我们还证明了MBR数据的目标调节监督学习的监督部分，优化了多目标RL目标的下限。基于几个点的任务和模拟机器人环境的实验结果表明，MINHER比以前的无模型和基于模型的多目标方法实现显着更高的样本效率。

资产分配（或投资组合管理）是确定如何最佳将有限预算的资金分配给一系列金融工具/资产（例如股票）的任务。这项研究调查了使用无模型的深RL代理应用于投资组合管理的增强学习（RL）的性能。我们培训了几个RL代理商的现实股票价格，以学习如何执行资产分配。我们比较了这些RL剂与某些基线剂的性能。我们还比较了RL代理，以了解哪些类别的代理表现更好。从我们的分析中，RL代理可以执行投资组合管理的任务，因为它们的表现明显优于基线代理（随机分配和均匀分配）。四个RL代理（A2C，SAC，PPO和TRPO）总体上优于最佳基线MPT。这显示了RL代理商发现更有利可图的交易策略的能力。此外，基于价值和基于策略的RL代理之间没有显着的性能差异。演员批评者的表现比其他类型的药物更好。同样，在政策代理商方面的表现要好，因为它们在政策评估方面更好，样品效率在投资组合管理中并不是一个重大问题。这项研究表明，RL代理可以大大改善资产分配，因为它们的表现优于强基础。基于我们的分析，在政策上，参与者批评的RL药物显示出最大的希望。

强化学习中的固有问题是应对不确定要采取的行动（或状态价值）的政策。模型不确定性，更正式地称为认知不确定性，是指超出采样噪声的模型的预期预测误差。在本文中，我们提出了Q值函数中认知不确定性估计的度量，我们将其称为路线上的认知不确定性。我们进一步开发了一种计算其近似上限的方法，我们称之为f值。我们通过实验将后者应用于深Q-Networks（DQN），并表明增强学习中的不确定性估计是学习进步的有用指标。然后，我们提出了一种新的方法，通过从现有（以前学过的或硬编码）的甲骨文政策中学习不确定性的同时，旨在避免在训练过程中避免非生产性的随机操作，从而提高参与者批评算法的样本效率。我们认为这位评论家的信心指导了探索（CCGE）。我们使用我们的F-Value指标在软演奏者（SAC）上实施CCGE，我们将其应用于少数流行的健身环境，并表明它比有限的背景下的香草囊获得了更好的样本效率和全部情节奖励。

深度强化学习（RL）导致了许多最近和开创性的进步。但是，这些进步通常以培训的基础体系结构的规模增加以及用于训练它们的RL算法的复杂性提高，而均以增加规模的成本。这些增长反过来又使研究人员更难迅速原型新想法或复制已发表的RL算法。为了解决这些问题，这项工作描述了ACME，这是一个用于构建新型RL算法的框架，这些框架是专门设计的，用于启用使用简单的模块化组件构建的代理，这些组件可以在各种执行范围内使用。尽管ACME的主要目标是为算法开发提供一个框架，但第二个目标是提供重要或最先进算法的简单参考实现。这些实现既是对我们的设计决策的验证，也是对RL研究中可重复性的重要贡献。在这项工作中，我们描述了ACME内部做出的主要设计决策，并提供了有关如何使用其组件来实施各种算法的进一步详细信息。我们的实验为许多常见和最先进的算法提供了基准，并显示了如何为更大且更复杂的环境扩展这些算法。这突出了ACME的主要优点之一，即它可用于实现大型，分布式的RL算法，这些算法可以以较大的尺度运行，同时仍保持该实现的固有可读性。这项工作提出了第二篇文章的版本，恰好与模块化的增加相吻合，对离线，模仿和从演示算法学习以及作为ACME的一部分实现的各种新代理。

Reinforcement learning (RL) gained considerable attention by creating decision-making agents that maximize rewards received from fully observable environments. However, many real-world problems are partially or noisily observable by nature, where agents do not receive the true and complete state of the environment. Such problems are formulated as partially observable Markov decision processes (POMDPs). Some studies applied RL to POMDPs by recalling previous decisions and observations or inferring the true state of the environment from received observations. Nevertheless, aggregating observations and decisions over time is impractical for environments with high-dimensional continuous state and action spaces. Moreover, so-called inference-based RL approaches require large number of samples to perform well since agents eschew uncertainty in the inferred state for the decision-making. Active inference is a framework that is naturally formulated in POMDPs and directs agents to select decisions by minimising expected free energy (EFE). This supplies reward-maximising (exploitative) behaviour in RL, with an information-seeking (exploratory) behaviour. Despite this exploratory behaviour of active inference, its usage is limited to discrete state and action spaces due to the computational difficulty of the EFE. We propose a unified principle for joint information-seeking and reward maximization that clarifies a theoretical connection between active inference and RL, unifies active inference and RL, and overcomes their aforementioned limitations. Our findings are supported by strong theoretical analysis. The proposed framework's superior exploration property is also validated by experimental results on partial observable tasks with high-dimensional continuous state and action spaces. Moreover, the results show that our model solves reward-free problems, making task reward design optional.

近年来，深度加固学习（DRL）已经成功地进入了复杂的决策应用，例如机器人，自动驾驶或视频游戏。违规算法往往比其策略对应物更具样本效率，并且可以从存储在重放缓冲区中存储的任何违规数据中受益。专家演示是此类数据的流行来源：代理人接触到成功的国家和行动，可以加速学习过程并提高性能。在过去，已经提出了多种想法来充分利用缓冲区中的演示，例如仅在演示或最小化额外的成本函数的预先估算。我们继续进行研究，以孤立地评估这些想法中的几个想法，以了解哪一个具有最大的影响。我们还根据给予示范和成功集中的奖励奖金，为稀疏奖励任务提供了一种新的方法。首先，我们向来自示威活动的过渡提供奖励奖金，以鼓励代理商符合所证明的行为。然后，在收集成功的剧集时，我们将其在将其添加到重播缓冲区之前与相同的奖金转换，鼓励代理也与其先前的成功相匹配。我们的实验的基本算法是流行的软演员 - 评论家（SAC），用于连续动作空间的最先进的脱核算法。我们的实验专注于操纵机器人，特别是在模拟中的机器人手臂的3D到达任务。我们表明，我们的方法Sacr2根据奖励重新标记提高了此任务的性能，即使在没有示范的情况下也是如此。

Offline reinforcement learning (RL) refers to the problem of learning policies entirely from a large batch of previously collected data. This problem setting offers the promise of utilizing such datasets to acquire policies without any costly or dangerous active exploration. However, it is also challenging, due to the distributional shift between the offline training data and those states visited by the learned policy. Despite significant recent progress, the most successful prior methods are model-free and constrain the policy to the support of data, precluding generalization to unseen states. In this paper, we first observe that an existing model-based RL algorithm already produces significant gains in the offline setting compared to model-free approaches. However, standard model-based RL methods, designed for the online setting, do not provide an explicit mechanism to avoid the offline setting's distributional shift issue. Instead, we propose to modify the existing model-based RL methods by applying them with rewards artificially penalized by the uncertainty of the dynamics. We theoretically show that the algorithm maximizes a lower bound of the policy's return under the true MDP. We also characterize the trade-off between the gain and risk of leaving the support of the batch data. Our algorithm, Model-based Offline Policy Optimization (MOPO), outperforms standard model-based RL algorithms and prior state-of-the-art model-free offline RL algorithms on existing offline RL benchmarks and two challenging continuous control tasks that require generalizing from data collected for a different task. * equal contribution. † equal advising. Orders randomized.34th Conference on Neural Information Processing Systems (NeurIPS 2020),

深度加强学习（RL）的增长为该领域带来了多种令人兴奋的工具和方法。这种快速扩展使得了解RL工具箱的各个元素之间的相互作用。通过在连续控制环境中进行研究，我们从实证角度接近这项任务。我们提出了对基本性质的多个见解，包括：从相同数据培训的多个演员的平均值提升了性能;现有方法在培训运行，培训时期，培训时期和评估运行不稳定;有效培训不需要常用的添加剂动作噪声;基于后抽样的策略探讨比近似的UCB与加权Bellman备份相结合的探讨;单独加权的Bellman备份不能取代剪辑的双Q学习;批评者的初始化在基于集合的演员批评探索中起着重要作用。作为一个结论，我们展示了现有的工具如何以新颖的方式汇集，产生集合深度确定性政策梯度（ED2）方法，从Openai Gyem Mujoco的连续控制任务产生最先进的结果。从实际方面，ED2在概念上简单，易于编码，并且不需要在现有RL工具箱之外的知识。

Recent advances in batch (offline) reinforcement learning have shown promising results in learning from available offline data and proved offline reinforcement learning to be an essential toolkit in learning control policies in a model-free setting. An offline reinforcement learning algorithm applied to a dataset collected by a suboptimal non-learning-based algorithm can result in a policy that outperforms the behavior agent used to collect the data. Such a scenario is frequent in robotics, where existing automation is collecting operational data. Although offline learning techniques can learn from data generated by a sub-optimal behavior agent, there is still an opportunity to improve the sample complexity of existing offline reinforcement learning algorithms by strategically introducing human demonstration data into the training process. To this end, we propose a novel approach that uses uncertainty estimation to trigger the injection of human demonstration data and guide policy training towards optimal behavior while reducing overall sample complexity. Our experiments show that this approach is more sample efficient when compared to a naive way of combining expert data with data collected from a sub-optimal agent. We augmented an existing offline reinforcement learning algorithm Conservative Q-Learning with our approach and performed experiments on data collected from MuJoCo and OffWorld Gym learning environments.

离线强化学习在利用大型预采用的数据集进行政策学习方面表现出了巨大的希望，使代理商可以放弃经常廉价的在线数据收集。但是，迄今为止，离线强化学习的探索相对较小，并且缺乏对剩余挑战所在的何处的了解。在本文中，我们试图建立简单的基线以在视觉域中连续控制。我们表明，对两个基于最先进的在线增强学习算法，Dreamerv2和DRQ-V2进行了简单的修改，足以超越事先工作并建立竞争性的基准。我们在现有的离线数据集中对这些算法进行了严格的评估，以及从视觉观察结果中进行离线强化学习的新测试台，更好地代表现实世界中离线增强学习问题中存在的数据分布，并开放我们的代码和数据以促进此方面的进度重要领域。最后，我们介绍并分析了来自视觉观察的离线RL所独有的几个关键Desiderata，包括视觉分散注意力和动态视觉上可识别的变化。

Adversarial Imitation Learning (AIL) is a class of popular state-of-the-art Imitation Learning algorithms commonly used in robotics. In AIL, an artificial adversary's misclassification is used as a reward signal that is optimized by any standard Reinforcement Learning (RL) algorithm. Unlike most RL settings, the reward in AIL is $differentiable$ but current model-free RL algorithms do not make use of this property to train a policy. The reward is AIL is also shaped since it comes from an adversary. We leverage the differentiability property of the shaped AIL reward function and formulate a class of Actor Residual Critic (ARC) RL algorithms. ARC algorithms draw a parallel to the standard Actor-Critic (AC) algorithms in RL literature and uses a residual critic, $C$ function (instead of the standard $Q$ function) to approximate only the discounted future return (excluding the immediate reward). ARC algorithms have similar convergence properties as the standard AC algorithms with the additional advantage that the gradient through the immediate reward is exact. For the discrete (tabular) case with finite states, actions, and known dynamics, we prove that policy iteration with $C$ function converges to an optimal policy. In the continuous case with function approximation and unknown dynamics, we experimentally show that ARC aided AIL outperforms standard AIL in simulated continuous-control and real robotic manipulation tasks. ARC algorithms are simple to implement and can be incorporated into any existing AIL implementation with an AC algorithm. Video and link to code are available at: https://sites.google.com/view/actor-residual-critic.

强化学习（RL）通过与环境相互作用的试验过程解决顺序决策问题。尽管RL在玩复杂的视频游戏方面取得了巨大的成功，但在现实世界中，犯错误总是不希望的。为了提高样本效率并从而降低错误，据信基于模型的增强学习（MBRL）是一个有前途的方向，它建立了环境模型，在该模型中可以进行反复试验，而无需实际成本。在这项调查中，我们对MBRL进行了审查，重点是Deep RL的最新进展。对于非壮观环境，学到的环境模型与真实环境之间始终存在概括性错误。因此，非常重要的是分析环境模型中的政策培训与实际环境中的差异，这反过来又指导了更好的模型学习，模型使用和政策培训的算法设计。此外，我们还讨论了其他形式的RL，包括离线RL，目标条件RL，多代理RL和Meta-RL的最新进展。此外，我们讨论了MBRL在现实世界任务中的适用性和优势。最后，我们通过讨论MBRL未来发展的前景来结束这项调查。我们认为，MBRL在被忽略的现实应用程序中具有巨大的潜力和优势，我们希望这项调查能够吸引更多关于MBRL的研究。

经验重放机制允许代理多次使用经验。在以前的作品中，过渡的抽样概率根据其重要性进行调整。重新分配采样概率在每次迭代后的重传缓冲器的每个过渡是非常低效的。因此，经验重播优先算法重新计算时，相应的过渡进行采样，以获得计算效率转变的意义。然而，过渡的重要性水平动态变化的政策和代理人的价值函数被更新。此外，经验回放存储转换由可显著从代理的最新货币政策偏离剂的以前的政策产生。从代理引线的最新货币政策更关闭策略更新，这是有害的代理高偏差。在本文中，我们开发了一种新的算法，通过KL散度批次优先化体验重播（KLPER），其优先批次转换的，而不是直接优先每个过渡。此外，为了减少更新的截止policyness，我们的算法选择一个批次中的某一批次的数量和力量的通过很有可能是代理的最新货币政策所产生的一批学习代理。我们结合与深确定性政策渐变和Twin算法延迟深确定性政策渐变，并评估它在不同的连续控制任务。 KLPER提供培训期间的抽样效率，最终表现和政策的稳定性方面有前途的深确定性的连续控制算法的改进。

最近的工作表明，离线增强学习（RL）可以作为序列建模问题（Chen等，2021; Janner等，2021）配制，并通过类似于大规模语言建模的方法解决。但是，RL的任何实际实例化也涉及一个在线组件，在线组件中，通过与环境的任务规定相互作用对被动离线数据集进行了预测的策略。我们建议在线决策变压器（ODT），这是一种基于序列建模的RL算法，该算法将离线预处理与统一框架中的在线填充融为一体。我们的框架将序列级熵正规仪与自回归建模目标结合使用，用于样品效率探索和填充。从经验上讲，我们表明ODT在D4RL基准上的绝对性能中与最先进的表现具有竞争力，但在填充过程中显示出更大的收益。

深层确定性的非政策算法的类别有效地用于解决具有挑战性的连续控制问题。但是，当前的方法使用随机噪声作为一种常见的探索方法，该方法具有多个弱点，例如需要对给定任务进行手动调整以及在训练过程中没有探索性校准。我们通过提出一种新颖的指导探索方法来应对这些挑战，该方法使用差异方向控制器来结合可扩展的探索性动作校正。提供探索性方向的蒙特卡洛评论家合奏作为控制器。提出的方法通过动态改变勘探来改善传统探索方案。然后，我们提出了一种新颖的算法，利用拟议的定向控制器进行政策和评论家修改。所提出的算法在DMControl Suite的各种问题上都优于现代增强算法的现代增强算法。

Hierarchical Reinforcement Learning (HRL) algorithms have been demonstrated to perform well on high-dimensional decision making and robotic control tasks. However, because they solely optimize for rewards, the agent tends to search the same space redundantly. This problem reduces the speed of learning and achieved reward. In this work, we present an Off-Policy HRL algorithm that maximizes entropy for efficient exploration. The algorithm learns a temporally abstracted low-level policy and is able to explore broadly through the addition of entropy to the high-level. The novelty of this work is the theoretical motivation of adding entropy to the RL objective in the HRL setting. We empirically show that the entropy can be added to both levels if the Kullback-Leibler (KL) divergence between consecutive updates of the low-level policy is sufficiently small. We performed an ablative study to analyze the effects of entropy on hierarchy, in which adding entropy to high-level emerged as the most desirable configuration. Furthermore, a higher temperature in the low-level leads to Q-value overestimation and increases the stochasticity of the environment that the high-level operates on, making learning more challenging. Our method, SHIRO, surpasses state-of-the-art performance on a range of simulated robotic control benchmark tasks and requires minimal tuning.

在这项工作中，我们提出并评估了一种新的增强学习方法，紧凑体验重放（编者），它使用基于相似转换集的复发的预测目标值的时间差异学习，以及基于两个转换的经验重放的新方法记忆。我们的目标是减少在长期累计累计奖励的经纪人培训所需的经验。它与强化学习的相关性与少量观察结果有关，即它需要实现类似于文献中的相关方法获得的结果，这通常需要数百万视频框架来培训ATARI 2600游戏。我们举报了在八个挑战街机学习环境（ALE）挑战游戏中，为仅10万帧的培训试验和大约25,000次迭代的培训试验中报告了培训试验。我们还在与基线的同一游戏中具有相同的实验协议的DQN代理呈现结果。为了验证从较少数量的观察结果近似于良好的政策，我们还将其结果与从啤酒的基准上呈现的数百万帧中获得的结果进行比较。

一种被称为优先体验重播（PER）的广泛研究的深钢筋学习（RL）技术使代理可以从与其时间差异（TD）误差成正比的过渡中学习。尽管已经表明，PER是离散作用域中深度RL方法总体性能的最关键组成部分之一，但许多经验研究表明，在连续控制中，它的表现非常低于参与者 - 批评算法。从理论上讲，我们表明，无法有效地通过具有较大TD错误的过渡对演员网络进行训练。结果，在Q网络下计算的近似策略梯度与在最佳Q功能下计算的实际梯度不同。在此激励的基础上，我们引入了一种新颖的经验重播抽样框架，用于演员批评方法，该框架还认为稳定性和最新发现的问题是Per的经验表现不佳。引入的算法提出了对演员和评论家网络的有效和高效培训的改进的新分支。一系列广泛的实验验证了我们的理论主张，并证明了引入的方法显着优于竞争方法，并获得了与标准的非政策参与者 - 批评算法相比，获得最先进的结果。

Deep reinforcement learning is poised to revolutionise the field of AI and represents a step towards building autonomous systems with a higher level understanding of the visual world. Currently, deep learning is enabling reinforcement learning to scale to problems that were previously intractable, such as learning to play video games directly from pixels. Deep reinforcement learning algorithms are also applied to robotics, allowing control policies for robots to be learned directly from camera inputs in the real world. In this survey, we begin with an introduction to the general field of reinforcement learning, then progress to the main streams of value-based and policybased methods. Our survey will cover central algorithms in deep reinforcement learning, including the deep Q-network, trust region policy optimisation, and asynchronous advantage actor-critic. In parallel, we highlight the unique advantages of deep neural networks, focusing on visual understanding via reinforcement learning. To conclude, we describe several current areas of research within the field.