In classic reinforcement learning algorithms, agents make decisions at discrete and fixed time intervals. The physical duration between one decision and the next becomes a critical hyperparameter. When this duration is too short, the agent needs to make many decisions to achieve its goal, aggravating the problem's difficulty. But when this duration is too long, the agent becomes incapable of controlling the system. Physical systems, however, do not need a constant control frequency. For learning agents, it is desirable to operate with low frequency when possible and high frequency when necessary. We propose a framework called Continuous-Time Continuous-Options (CTCO), where the agent chooses options as sub-policies of variable durations. Such options are time-continuous and can interact with the system at any desired frequency providing a smooth change of actions. The empirical analysis shows that our algorithm is competitive w.r.t. other time-abstraction techniques, such as classic option learning and action repetition, and practically overcomes the difficult choice of the decision frequency.

In order to avoid conventional controlling methods which created obstacles due to the complexity of systems and intense demand on data density, developing modern and more efficient control methods are required. In this way, reinforcement learning off-policy and model-free algorithms help to avoid working with complex models. In terms of speed and accuracy, they become prominent methods because the algorithms use their past experience to learn the optimal policies. In this study, three reinforcement learning algorithms; DDPG, TD3 and SAC have been used to train Fetch robotic manipulator for four different tasks in MuJoCo simulation environment. All of these algorithms are off-policy and able to achieve their desired target by optimizing both policy and value functions. In the current study, the efficiency and the speed of these three algorithms are analyzed in a controlled environment.

Hierarchical Reinforcement Learning (HRL) algorithms have been demonstrated to perform well on high-dimensional decision making and robotic control tasks. However, because they solely optimize for rewards, the agent tends to search the same space redundantly. This problem reduces the speed of learning and achieved reward. In this work, we present an Off-Policy HRL algorithm that maximizes entropy for efficient exploration. The algorithm learns a temporally abstracted low-level policy and is able to explore broadly through the addition of entropy to the high-level. The novelty of this work is the theoretical motivation of adding entropy to the RL objective in the HRL setting. We empirically show that the entropy can be added to both levels if the Kullback-Leibler (KL) divergence between consecutive updates of the low-level policy is sufficiently small. We performed an ablative study to analyze the effects of entropy on hierarchy, in which adding entropy to high-level emerged as the most desirable configuration. Furthermore, a higher temperature in the low-level leads to Q-value overestimation and increases the stochasticity of the environment that the high-level operates on, making learning more challenging. Our method, SHIRO, surpasses state-of-the-art performance on a range of simulated robotic control benchmark tasks and requires minimal tuning.

Model-free deep reinforcement learning (RL) algorithms have been demonstrated on a range of challenging decision making and control tasks. However, these methods typically suffer from two major challenges: very high sample complexity and brittle convergence properties, which necessitate meticulous hyperparameter tuning. Both of these challenges severely limit the applicability of such methods to complex, real-world domains. In this paper, we propose soft actor-critic, an offpolicy actor-critic deep RL algorithm based on the maximum entropy reinforcement learning framework. In this framework, the actor aims to maximize expected reward while also maximizing entropy. That is, to succeed at the task while acting as randomly as possible. Prior deep RL methods based on this framework have been formulated as Q-learning methods. By combining off-policy updates with a stable stochastic actor-critic formulation, our method achieves state-of-the-art performance on a range of continuous control benchmark tasks, outperforming prior on-policy and off-policy methods. Furthermore, we demonstrate that, in contrast to other off-policy algorithms, our approach is very stable, achieving very similar performance across different random seeds.

现实的操纵任务要求机器人与具有长时间运动动作序列的环境相互作用。尽管最近出现了深厚的强化学习方法，这是自动化操作行为的有希望的范式，但由于勘探负担，它们通常在长途任务中缺乏。这项工作介绍了操纵原始增强的强化学习（Maple），这是一个学习框架，可通过预定的行为原始库来增强标准强化学习算法。这些行为原始素是专门实现操纵目标（例如抓住和推动）的强大功能模块。为了使用这些异质原始素，我们制定了涉及原语的层次结构策略，并使用输入参数实例化执行。我们证明，枫树的表现优于基线方法，通过一系列模拟的操纵任务的大幅度。我们还量化了学习行为的组成结构，并突出了我们方法将策略转移到新任务变体和物理硬件的能力。视频和代码可从https://ut-aut-autin-rpl.github.io/maple获得

我们提出了一种在该框架中的精细时间离散化和学习算法中的加强学习（RL）的框架。 RL的主要目标之一是为物理机器提供学习最佳行为而不是被编程的方法。然而，机器通常在精细时间离散化中控制。最常见的RL方法将独立的随机元素应用于每个操作，这不适合该设置。这是不可行的，因为它导致受控系统猛拉，而且没有确保足够的探索，因为单一动作不足以创造可能被翻译成政策改进的重要经验。在本文介绍的RL框架中，考虑了策略，以产生基于在后续时刻中自相关的状态和随机元素的动作。这里介绍的RL算法大致优化了这种策略。在不同的时间离散化中，在四个模拟学习控制问题（ANT，HALFCHETAH，HOPPER和WANKER2D）中验证了该算法的效率。在大多数情况下，这里介绍的算法优于竞争对手。

采用合理的策略是具有挑战性的，但对于智能代理商的智能代理人至关重要，其资源有限，在危险，非结构化和动态环境中工作，以改善系统实用性，降低整体成本并增加任务成功概率。深度强化学习（DRL）帮助组织代理的行为和基于其状态的行为，并代表复杂的策略（行动的组成）。本文提出了一种基于贝叶斯链条的新型分层策略分解方法，将复杂的政策分为几个简单的子手段，并将其作为贝叶斯战略网络（BSN）组织。我们将这种方法整合到最先进的DRL方法中，软演奏者 - 批评者（SAC），并通过组织几个子主管作为联合政策来构建相应的贝叶斯软演奏者（BSAC）模型。我们将建议的BSAC方法与标准连续控制基准（Hopper-V2，Walker2D-V2和Humanoid-V2）在SAC和其他最先进的方法（例如TD3，DDPG和PPO）中进行比较 - Mujoco与Openai健身房环境。结果表明，BSAC方法的有希望的潜力可显着提高训练效率。可以从https://github.com/herolab-uga/bsac访问BSAC的开源代码。

许多现实世界的应用程序都可以作为多机构合作问题进行配置，例如网络数据包路由和自动驾驶汽车的协调。深入增强学习（DRL）的出现为通过代理和环境的相互作用提供了一种有前途的多代理合作方法。但是，在政策搜索过程中，传统的DRL解决方案遭受了多个代理具有连续动作空间的高维度。此外，代理商政策的动态性使训练非平稳。为了解决这些问题，我们建议采用高级决策和低水平的个人控制，以进行有效的政策搜索，提出一种分层增强学习方法。特别是，可以在高级离散的动作空间中有效地学习多个代理的合作。同时，低水平的个人控制可以减少为单格强化学习。除了分层增强学习外，我们还建议对手建模网络在学习过程中对其他代理的政策进行建模。与端到端的DRL方法相反，我们的方法通过以层次结构将整体任务分解为子任务来降低学习的复杂性。为了评估我们的方法的效率，我们在合作车道变更方案中进行了现实世界中的案例研究。模拟和现实世界实验都表明我们的方法在碰撞速度和收敛速度中的优越性。

在钢筋学习中，连续时间通常是通过时间缩放$ \ delta $离散的，所以已知产生的性能是高度敏感的。在这项工作中，我们寻求找到一个$ \ delta $-invariant算法，用于策略渐变（pg）方法，无论$ \ delta $的值如何，它会效果良好。我们首先确定导致PG方法失败的潜在原因作为$ \ delta \ 0美元，证明了PG估计器的方差在随机性的某些假设下随机环境中的无限远。虽然可以使用持续行动或动作重复来拥有$ \ delta $-invariance，但之前的操作重复方法不能立即对随机环境中的意外情况作出反应。因此，我们提出了一种新的$ \ delta $-invariant方法，命名为适用于任何现有的pg算法的安全操作重复（sar）。 SAR可以通过自适应地反应在行动重复期间的状态变化来处理环境的随机性。我们经验表明，我们的方法不仅是$ \ delta $-invariant，而且对随机性的强大，表现出以前的八个Mujoco环境中的前一\ \ delta $-invariant方法，具有确定性和随机设置。我们的代码在https://vision.snu.ac.kr/projects/sar上获得。

由于数据量增加，金融业的快速变化已经彻底改变了数据处理和数据分析的技术，并带来了新的理论和计算挑战。与古典随机控制理论和解决财务决策问题的其他分析方法相比，解决模型假设的财务决策问题，强化学习（RL）的新发展能够充分利用具有更少模型假设的大量财务数据并改善复杂的金融环境中的决策。该调查纸目的旨在审查最近的资金途径的发展和使用RL方法。我们介绍了马尔可夫决策过程，这是许多常用的RL方法的设置。然后引入各种算法，重点介绍不需要任何模型假设的基于价值和基于策略的方法。连接是用神经网络进行的，以扩展框架以包含深的RL算法。我们的调查通过讨论了这些RL算法在金融中各种决策问题中的应用，包括最佳执行，投资组合优化，期权定价和对冲，市场制作，智能订单路由和Robo-Awaring。

近年来近年来，加固学习方法已经发展了一系列政策梯度方法，主要用于建模随机政策的高斯分布。然而，高斯分布具有无限的支持，而现实世界应用通常具有有限的动作空间。如果它提供有限支持，则该解剖会导致可以消除的估计偏差，因为它提出了有限的支持。在这项工作中，我们调查如何在Openai健身房的两个连续控制任务中训练该测试策略在训练时执行该测试策略。对于这两个任务来说，测试政策在代理人的最终预期奖励方面优于高斯政策，也显示出更多的稳定性和更快的培训过程融合。对于具有高维图像输入的卡路里环境，在高斯政策中，代理的成功率提高了63％。

Reinforcement learning (RL) gained considerable attention by creating decision-making agents that maximize rewards received from fully observable environments. However, many real-world problems are partially or noisily observable by nature, where agents do not receive the true and complete state of the environment. Such problems are formulated as partially observable Markov decision processes (POMDPs). Some studies applied RL to POMDPs by recalling previous decisions and observations or inferring the true state of the environment from received observations. Nevertheless, aggregating observations and decisions over time is impractical for environments with high-dimensional continuous state and action spaces. Moreover, so-called inference-based RL approaches require large number of samples to perform well since agents eschew uncertainty in the inferred state for the decision-making. Active inference is a framework that is naturally formulated in POMDPs and directs agents to select decisions by minimising expected free energy (EFE). This supplies reward-maximising (exploitative) behaviour in RL, with an information-seeking (exploratory) behaviour. Despite this exploratory behaviour of active inference, its usage is limited to discrete state and action spaces due to the computational difficulty of the EFE. We propose a unified principle for joint information-seeking and reward maximization that clarifies a theoretical connection between active inference and RL, unifies active inference and RL, and overcomes their aforementioned limitations. Our findings are supported by strong theoretical analysis. The proposed framework's superior exploration property is also validated by experimental results on partial observable tasks with high-dimensional continuous state and action spaces. Moreover, the results show that our model solves reward-free problems, making task reward design optional.

We present temporally layered architecture (TLA), a biologically inspired system for temporally adaptive distributed control. TLA layers a fast and a slow controller together to achieve temporal abstraction that allows each layer to focus on a different time-scale. Our design is biologically inspired and draws on the architecture of the human brain which executes actions at different timescales depending on the environment's demands. Such distributed control design is widespread across biological systems because it increases survivability and accuracy in certain and uncertain environments. We demonstrate that TLA can provide many advantages over existing approaches, including persistent exploration, adaptive control, explainable temporal behavior, compute efficiency and distributed control. We present two different algorithms for training TLA: (a) Closed-loop control, where the fast controller is trained over a pre-trained slow controller, allowing better exploration for the fast controller and closed-loop control where the fast controller decides whether to "act-or-not" at each timestep; and (b) Partially open loop control, where the slow controller is trained over a pre-trained fast controller, allowing for open loop-control where the slow controller picks a temporally extended action or defers the next n-actions to the fast controller. We evaluated our method on a suite of continuous control tasks and demonstrate the advantages of TLA over several strong baselines.

Deep reinforcement learning is poised to revolutionise the field of AI and represents a step towards building autonomous systems with a higher level understanding of the visual world. Currently, deep learning is enabling reinforcement learning to scale to problems that were previously intractable, such as learning to play video games directly from pixels. Deep reinforcement learning algorithms are also applied to robotics, allowing control policies for robots to be learned directly from camera inputs in the real world. In this survey, we begin with an introduction to the general field of reinforcement learning, then progress to the main streams of value-based and policybased methods. Our survey will cover central algorithms in deep reinforcement learning, including the deep Q-network, trust region policy optimisation, and asynchronous advantage actor-critic. In parallel, we highlight the unique advantages of deep neural networks, focusing on visual understanding via reinforcement learning. To conclude, we describe several current areas of research within the field.

强化学习和最近的深度增强学习是解决如Markov决策过程建模的顺序决策问题的流行方法。问题和选择算法和超参数的RL建模需要仔细考虑，因为不同的配置可能需要完全不同的性能。这些考虑因素主要是RL专家的任务;然而，RL在研究人员和系统设计师不是RL专家的其他领域中逐渐变得流行。此外，许多建模决策，例如定义状态和动作空间，批次的大小和批量更新的频率以及时间戳的数量通常是手动进行的。由于这些原因，RL框架的自动化不同组成部分具有重要意义，近年来它引起了很多关注。自动RL提供了一个框架，其中RL的不同组件包括MDP建模，算法选择和超参数优化是自动建模和定义的。在本文中，我们探讨了可以在自动化RL中使用的文献和目前的工作。此外，我们讨论了Autorl中的挑战，打开问题和研究方向。

人工智能（AI）的努力是设计能够完成复杂任务的自主代理。也就是说，加强学习（RL）提出了学习最佳行为的理论背景。实际上，RL算法依靠几何折扣来评估这种最优性。不幸的是，这并不涵盖未来回报并没有达到成倍价值的决策过程。根据问题的不同，此限制会引起样本信息（由于饲料后额定值是指数衰减），并且需要其他课程/探索机制（以处理稀疏，欺骗性或对抗性奖励）。在本文中，我们通过通过延迟目标功能将折现问题提出来解决这些问题。我们研究了得出的基本RL问题：1）最佳固定解和2）最佳非平稳控制的近似值。设计的算法解决了表格环境上的​​硬探索问题，并在经典的模拟机器人基准上提高了样品效率。

We adapt the ideas underlying the success of Deep Q-Learning to the continuous action domain. We present an actor-critic, model-free algorithm based on the deterministic policy gradient that can operate over continuous action spaces. Using the same learning algorithm, network architecture and hyper-parameters, our algorithm robustly solves more than 20 simulated physics tasks, including classic problems such as cartpole swing-up, dexterous manipulation, legged locomotion and car driving. Our algorithm is able to find policies whose performance is competitive with those found by a planning algorithm with full access to the dynamics of the domain and its derivatives. We further demonstrate that for many of the tasks the algorithm can learn policies "end-to-end": directly from raw pixel inputs.

我们提出了一种层次结构的增强学习方法Hidio，可以以自我监督的方式学习任务不合时宜的选项，同时共同学习利用它们来解决稀疏的奖励任务。与当前倾向于制定目标的低水平任务或预定临时的低级政策不同的层次RL方法不同，Hidio鼓励下级选项学习与手头任务无关，几乎不需要假设或很少的知识任务结构。这些选项是通过基于选项子对象的固有熵最小化目标来学习的。博学的选择是多种多样的，任务不可能的。在稀疏的机器人操作和导航任务的实验中，Hidio比常规RL基准和两种最先进的层次RL方法，其样品效率更高。

资产分配（或投资组合管理）是确定如何最佳将有限预算的资金分配给一系列金融工具/资产（例如股票）的任务。这项研究调查了使用无模型的深RL代理应用于投资组合管理的增强学习（RL）的性能。我们培训了几个RL代理商的现实股票价格，以学习如何执行资产分配。我们比较了这些RL剂与某些基线剂的性能。我们还比较了RL代理，以了解哪些类别的代理表现更好。从我们的分析中，RL代理可以执行投资组合管理的任务，因为它们的表现明显优于基线代理（随机分配和均匀分配）。四个RL代理（A2C，SAC，PPO和TRPO）总体上优于最佳基线MPT。这显示了RL代理商发现更有利可图的交易策略的能力。此外，基于价值和基于策略的RL代理之间没有显着的性能差异。演员批评者的表现比其他类型的药物更好。同样，在政策代理商方面的表现要好，因为它们在政策评估方面更好，样品效率在投资组合管理中并不是一个重大问题。这项研究表明，RL代理可以大大改善资产分配，因为它们的表现优于强基础。基于我们的分析，在政策上，参与者批评的RL药物显示出最大的希望。

最先进的多机构增强学习（MARL）方法为各种复杂问题提供了有希望的解决方案。然而，这些方法都假定代理执行同步的原始操作执行，因此它们不能真正可扩展到长期胜利的真实世界多代理/机器人任务，这些任务固有地要求代理/机器人以异步的理由，涉及有关高级动作选择的理由。不同的时间。宏观行动分散的部分可观察到的马尔可夫决策过程（MACDEC-POMDP）是在完全合作的多代理任务中不确定的异步决策的一般形式化。在本论文中，我们首先提出了MacDec-Pomdps的一组基于价值的RL方法，其中允许代理在三个范式中使用宏观成果功能执行异步学习和决策：分散学习和控制，集中学习，集中学习和控制，以及分散执行的集中培训（CTDE）。在上述工作的基础上，我们在三个训练范式下制定了一组基于宏观行动的策略梯度算法，在该训练范式下，允许代理以异步方式直接优化其参数化策略。我们在模拟和真实的机器人中评估了我们的方法。经验结果证明了我们在大型多代理问题中的方法的优势，并验证了我们算法在学习具有宏观actions的高质量和异步溶液方面的有效性。