One of the key challenges in deploying RL to real-world applications is to adapt to variations of unknown environment contexts, such as changing terrains in robotic tasks and fluctuated bandwidth in congestion control. Existing works on adaptation to unknown environment contexts either assume the contexts are the same for the whole episode or assume the context variables are Markovian. However, in many real-world applications, the environment context usually stays stable for a stochastic period and then changes in an abrupt and unpredictable manner within an episode, resulting in a segment structure, which existing works fail to address. To leverage the segment structure of piecewise stable context in real-world applications, in this paper, we propose a \textit{\textbf{Se}gmented \textbf{C}ontext \textbf{B}elief \textbf{A}ugmented \textbf{D}eep~(SeCBAD)} RL method. Our method can jointly infer the belief distribution over latent context with the posterior over segment length and perform more accurate belief context inference with observed data within the current context segment. The inferred belief context can be leveraged to augment the state, leading to a policy that can adapt to abrupt variations in context. We demonstrate empirically that SeCBAD can infer context segment length accurately and outperform existing methods on a toy grid world environment and Mujuco tasks with piecewise-stable context.

Deep reinforcement learning algorithms require large amounts of experience to learn an individual task. While in principle meta-reinforcement learning (meta-RL) algorithms enable agents to learn new skills from small amounts of experience, several major challenges preclude their practicality. Current methods rely heavily on on-policy experience, limiting their sample efficiency. The also lack mechanisms to reason about task uncertainty when adapting to new tasks, limiting their effectiveness in sparse reward problems. In this paper, we address these challenges by developing an offpolicy meta-RL algorithm that disentangles task inference and control. In our approach, we perform online probabilistic filtering of latent task variables to infer how to solve a new task from small amounts of experience. This probabilistic interpretation enables posterior sampling for structured and efficient exploration. We demonstrate how to integrate these task variables with off-policy RL algorithms to achieve both metatraining and adaptation efficiency. Our method outperforms prior algorithms in sample efficiency by 20-100X as well as in asymptotic performance on several meta-RL benchmarks.

Planning has been very successful for control tasks with known environment dynamics. To leverage planning in unknown environments, the agent needs to learn the dynamics from interactions with the world. However, learning dynamics models that are accurate enough for planning has been a long-standing challenge, especially in image-based domains. We propose the Deep Planning Network (PlaNet), a purely model-based agent that learns the environment dynamics from images and chooses actions through fast online planning in latent space. To achieve high performance, the dynamics model must accurately predict the rewards ahead for multiple time steps. We approach this using a latent dynamics model with both deterministic and stochastic transition components. Moreover, we propose a multi-step variational inference objective that we name latent overshooting. Using only pixel observations, our agent solves continuous control tasks with contact dynamics, partial observability, and sparse rewards, which exceed the difficulty of tasks that were previously solved by planning with learned models. PlaNet uses substantially fewer episodes and reaches final performance close to and sometimes higher than strong model-free algorithms.

有效的探索仍然是强化学习中有挑战性的问题，特别是对于来自环境的外在奖励稀疏甚至完全忽视的任务。基于内在动机的重要进展显示了在简单环境中的有希望的结果，但通常会在具有多式联运和随机动力学的环境中陷入困境。在这项工作中，我们提出了一种基于条件变分推理的变分动力模型来模拟多模和随机性。通过在当前状态，动作和潜在变量的条件下产生下一个状态预测，我们考虑作为条件生成过程的环境状态动作转换，这提供了更好地了解动态并在勘探中引发更好的性能。我们派生了环境过渡的负面日志可能性的上限，并使用这样一个上限作为勘探的内在奖励，这使得代理通过自我监督的探索来学习技能，而无需观察外在奖励。我们在基于图像的仿真任务和真正的机器人操纵任务中评估所提出的方法。我们的方法优于若干基于最先进的环境模型的勘探方法。

强化学习（RL）技术在许多具有挑战性的任务中引起了极大的关注，但是当应用于现实世界问题时，它们的性能急剧恶化。已经提出了各种方法，例如域随机化，以通过不同的环境设置下的培训代理来应对这种情况，因此在部署过程中可以将它们推广到不同的环境。但是，它们通常不包含与代理人正确相互作用的潜在环境因素信息，因此在面对周围环境变化时可能会过于保守。在本文中，我们首先将适应RL中的环境动态的任务形式化为使用上下文Markov决策过程（CMDP）的概括问题。然后，我们在上下文RL（AACC）中提出了不对称的参与者 - 作为处理此类概括任务的端到端参与者的方法。我们在一系列模拟环境中证明了AACC对现有基线的性能的基本改进。

强化学习（RL）通过与环境相互作用的试验过程解决顺序决策问题。尽管RL在玩复杂的视频游戏方面取得了巨大的成功，但在现实世界中，犯错误总是不希望的。为了提高样本效率并从而降低错误，据信基于模型的增强学习（MBRL）是一个有前途的方向，它建立了环境模型，在该模型中可以进行反复试验，而无需实际成本。在这项调查中，我们对MBRL进行了审查，重点是Deep RL的最新进展。对于非壮观环境，学到的环境模型与真实环境之间始终存在概括性错误。因此，非常重要的是分析环境模型中的政策培训与实际环境中的差异，这反过来又指导了更好的模型学习，模型使用和政策培训的算法设计。此外，我们还讨论了其他形式的RL，包括离线RL，目标条件RL，多代理RL和Meta-RL的最新进展。此外，我们讨论了MBRL在现实世界任务中的适用性和优势。最后，我们通过讨论MBRL未来发展的前景来结束这项调查。我们认为，MBRL在被忽略的现实应用程序中具有巨大的潜力和优势，我们希望这项调查能够吸引更多关于MBRL的研究。

Reinforcement learning (RL) gained considerable attention by creating decision-making agents that maximize rewards received from fully observable environments. However, many real-world problems are partially or noisily observable by nature, where agents do not receive the true and complete state of the environment. Such problems are formulated as partially observable Markov decision processes (POMDPs). Some studies applied RL to POMDPs by recalling previous decisions and observations or inferring the true state of the environment from received observations. Nevertheless, aggregating observations and decisions over time is impractical for environments with high-dimensional continuous state and action spaces. Moreover, so-called inference-based RL approaches require large number of samples to perform well since agents eschew uncertainty in the inferred state for the decision-making. Active inference is a framework that is naturally formulated in POMDPs and directs agents to select decisions by minimising expected free energy (EFE). This supplies reward-maximising (exploitative) behaviour in RL, with an information-seeking (exploratory) behaviour. Despite this exploratory behaviour of active inference, its usage is limited to discrete state and action spaces due to the computational difficulty of the EFE. We propose a unified principle for joint information-seeking and reward maximization that clarifies a theoretical connection between active inference and RL, unifies active inference and RL, and overcomes their aforementioned limitations. Our findings are supported by strong theoretical analysis. The proposed framework's superior exploration property is also validated by experimental results on partial observable tasks with high-dimensional continuous state and action spaces. Moreover, the results show that our model solves reward-free problems, making task reward design optional.

Adequately assigning credit to actions for future outcomes based on their contributions is a long-standing open challenge in Reinforcement Learning. The assumptions of the most commonly used credit assignment method are disadvantageous in tasks where the effects of decisions are not immediately evident. Furthermore, this method can only evaluate actions that have been selected by the agent, making it highly inefficient. Still, no alternative methods have been widely adopted in the field. Hindsight Credit Assignment is a promising, but still unexplored candidate, which aims to solve the problems of both long-term and counterfactual credit assignment. In this thesis, we empirically investigate Hindsight Credit Assignment to identify its main benefits, and key points to improve. Then, we apply it to factored state representations, and in particular to state representations based on the causal structure of the environment. In this setting, we propose a variant of Hindsight Credit Assignment that effectively exploits a given causal structure. We show that our modification greatly decreases the workload of Hindsight Credit Assignment, making it more efficient and enabling it to outperform the baseline credit assignment method on various tasks. This opens the way to other methods based on given or learned causal structures.

由于数据量增加，金融业的快速变化已经彻底改变了数据处理和数据分析的技术，并带来了新的理论和计算挑战。与古典随机控制理论和解决财务决策问题的其他分析方法相比，解决模型假设的财务决策问题，强化学习（RL）的新发展能够充分利用具有更少模型假设的大量财务数据并改善复杂的金融环境中的决策。该调查纸目的旨在审查最近的资金途径的发展和使用RL方法。我们介绍了马尔可夫决策过程，这是许多常用的RL方法的设置。然后引入各种算法，重点介绍不需要任何模型假设的基于价值和基于策略的方法。连接是用神经网络进行的，以扩展框架以包含深的RL算法。我们的调查通过讨论了这些RL算法在金融中各种决策问题中的应用，包括最佳执行，投资组合优化，期权定价和对冲，市场制作，智能订单路由和Robo-Awaring。

While reinforcement learning algorithms provide automated acquisition of optimal policies, practical application of such methods requires a number of design decisions, such as manually designing reward functions that not only define the task, but also provide sufficient shaping to accomplish it. In this paper, we view reinforcement learning as inferring policies that achieve desired outcomes, rather than as a problem of maximizing rewards. To solve this inference problem, we establish a novel variational inference formulation that allows us to derive a well-shaped reward function which can be learned directly from environment interactions. From the corresponding variational objective, we also derive a new probabilistic Bellman backup operator and use it to develop an off-policy algorithm to solve goal-directed tasks. We empirically demonstrate that this method eliminates the need to hand-craft reward functions for a suite of diverse manipulation and locomotion tasks and leads to effective goal-directed behaviors.

Policies produced by deep reinforcement learning are typically characterised by their learning curves, but they remain poorly understood in many other respects. ReLU-based policies result in a partitioning of the input space into piecewise linear regions. We seek to understand how observed region counts and their densities evolve during deep reinforcement learning using empirical results that span a range of continuous control tasks and policy network dimensions. Intuitively, we may expect that during training, the region density increases in the areas that are frequently visited by the policy, thereby affording fine-grained control. We use recent theoretical and empirical results for the linear regions induced by neural networks in supervised learning settings for grounding and comparison of our results. Empirically, we find that the region density increases only moderately throughout training, as measured along fixed trajectories coming from the final policy. However, the trajectories themselves also increase in length during training, and thus the region densities decrease as seen from the perspective of the current trajectory. Our findings suggest that the complexity of deep reinforcement learning policies does not principally emerge from a significant growth in the complexity of functions observed on-and-around trajectories of the policy.

Transformer, originally devised for natural language processing, has also attested significant success in computer vision. Thanks to its super expressive power, researchers are investigating ways to deploy transformers to reinforcement learning (RL) and the transformer-based models have manifested their potential in representative RL benchmarks. In this paper, we collect and dissect recent advances on transforming RL by transformer (transformer-based RL or TRL), in order to explore its development trajectory and future trend. We group existing developments in two categories: architecture enhancement and trajectory optimization, and examine the main applications of TRL in robotic manipulation, text-based games, navigation and autonomous driving. For architecture enhancement, these methods consider how to apply the powerful transformer structure to RL problems under the traditional RL framework, which model agents and environments much more precisely than deep RL methods, but they are still limited by the inherent defects of traditional RL algorithms, such as bootstrapping and "deadly triad". For trajectory optimization, these methods treat RL problems as sequence modeling and train a joint state-action model over entire trajectories under the behavior cloning framework, which are able to extract policies from static datasets and fully use the long-sequence modeling capability of the transformer. Given these advancements, extensions and challenges in TRL are reviewed and proposals about future direction are discussed. We hope that this survey can provide a detailed introduction to TRL and motivate future research in this rapidly developing field.

强化学习的主要方法是根据预期的回报将信贷分配给行动。但是，我们表明回报可能取决于政策，这可能会导致价值估计的过度差异和减慢学习的速度。取而代之的是，我们证明了优势函数可以解释为因果效应，并与因果关系共享相似的属性。基于此洞察力，我们提出了直接优势估计（DAE），这是一种可以对优势函数进行建模并直接从政策数据进行估算的新方法，同时同时最大程度地减少了返回的方差而无需（操作 - ）值函数。我们还通过显示如何无缝整合到DAE中来将我们的方法与时间差异方法联系起来。所提出的方法易于实施，并且可以通过现代参与者批评的方法很容易适应。我们对三个离散控制域进行经验评估DAE，并表明它可以超过广义优势估计（GAE），这是优势估计的强大基线，当将大多数环境应用于策略优化时。

在动态环境中，持续增强学习（CRL）的关键挑战是，随着环境在其生命周期的变化，同时最大程度地减少对学习的信息的灾难性忘记，随着环境在其一生中的变化而变化。为了应对这一挑战，在本文中，我们提出了Dacorl，即动态自动持续RL。 Dacorl使用渐进式上下文化学习了上下文条件条件的策略，该策略会逐步将动态环境中的一系列固定任务群集成一系列上下文，并选择一个可扩展的多头神经网络以近似策略。具体来说，我们定义了一组具有类似动力学的任务，并将上下文推理形式化为在线贝叶斯无限高斯混合物集群的过程，这些过程是在环境特征上，诉诸在线贝叶斯推断，以推断上下文的后端分布。在以前的中国餐厅流程的假设下，该技术可以将当前任务准确地分类为先前看到的上下文，或者根据需要实例化新的上下文，而无需依靠任何外部指标来提前向环境变化发出信号。此外，我们采用了可扩展的多头神经网络，其输出层与新实例化的上下文同步扩展，以及一个知识蒸馏正规化项来保留学习任务的性能。作为一个可以与各种深度RL算法结合使用的一般框架，Dacorl在稳定性，整体性能和概括能力方面具有一致的优势，而不是现有方法，这是通过对几种机器人导航和Mujoco Socomotion任务进行的广泛实验来验证的。

Meta强化学习（META-RL）旨在学习一项政策，同时并迅速适应新任务。它需要大量从培训任务中汲取的数据，以推断任务之间共享的共同结构。如果没有沉重的奖励工程，长期任务中的稀疏奖励加剧了元RL样品效率的问题。 Meta-RL中的另一个挑战是任务之间难度级别的差异，这可能会导致一个简单的任务主导共享策略的学习，从而排除政策适应新任务。这项工作介绍了一个新颖的目标功能，可以在培训任务中学习动作翻译。从理论上讲，我们可以验证带有操作转换器的传输策略的值可以接近源策略的值和我们的目标函数（大约）上限的值差。我们建议将动作转换器与基于上下文的元元算法相结合，以更好地收集数据，并在元训练期间更有效地探索。我们的方法从经验上提高了稀疏奖励任务上元RL算法的样本效率和性能。

对于许多强化学习（RL）应用程序，指定奖励是困难的。本文考虑了一个RL设置，其中代理仅通过查询可以询问可以的专家来获取有关奖励的信息，例如，评估单个状态或通过轨迹提供二进制偏好。从如此昂贵的反馈中，我们的目标是学习奖励的模型，允许标准RL算法实现高预期的回报，尽可能少的专家查询。为此，我们提出了信息定向奖励学习（IDRL），它使用奖励的贝叶斯模型，然后选择要最大化信息增益的查询，这些查询是有关合理的最佳策略之间的返回差异的差异。与针对特定类型查询设计的先前主动奖励学习方法相比，IDRL自然地适应不同的查询类型。此外，它通过将焦点转移降低奖励近似误差来实现类似或更好的性能，从而降低奖励近似误差，以改善奖励模型引起的策略。我们支持我们的调查结果，在多个环境中进行广泛的评估，并具有不同的查询类型。

人类是熟练的导航员：我们恰当地在新的地方进行了操纵，意识到我们回到以前见过的位置，甚至可以想到经历我们从未参观过的部分环境的捷径。另一方面，基于模型的强化学习中的当前方法与从训练分布中概括环境动态的努力。我们认为，两个原则可以帮助弥合这一差距：潜在的学习和简约的动态。人类倾向于以简单的术语来思考环境动态 - 我们认为轨迹不是指我们期望在路径上看到的东西，而是在抽象的潜在空间中，其中包含有关该位置的空间坐标的信息。此外，我们假设在环境的新颖部分中四处走动的工作方式与我们所熟悉的部分相同。这两个原则在串联中共同起作用：在潜在空间中，动态表现出了简约的特征。我们开发了一种学习这种简约动态的模型。使用一个变异目标，我们的模型经过培训，可以使用本地线性转换在潜在空间中重建经验丰富的过渡，同时鼓励尽可能少地调用不同的变换。使用我们的框架，我们演示了在一系列政策学习和计划任务中学习简化潜在动态模型的实用性。

随着我们日常环境中机器人的存在越来越多，提高社交技能至关重要。尽管如此，社会机器人技术仍然面临许多挑战。一种瓶颈是，由于社会规范的强烈取决于环境，因此需要经常适应机器人行为。例如，与办公室的工人相比，机器人应更仔细地在医院的患者周围进行仔细的导航。在这项工作中，我们将元强化学习（META-RL）作为潜在解决方案进行了研究。在这里，机器人行为是通过强化学习来学习的，需要选择奖励功能，以便机器人学习适合给定环境的行为。我们建议使用一种变异元过程，该过程迅速使机器人的行为适应新的奖励功能。结果，给定一个新的环境，可以快速评估不同的奖励功能，并选择适当的奖励功能。该过程学习奖励函数的矢量表示和可以在这种表示形式下进行条件的元政策。从新的奖励函数中进行观察，该过程确定了其表示形式，并条件元元素对其进行了条件。在研究程序的功能时，我们意识到它遭受了后塌陷的困扰，在表示表示中只有一个尺寸的子集编码有用的信息，从而导致性能降低。我们的第二个贡献是径向基函数（RBF）层，部分减轻了这种负面影响。 RBF层将表示形式提升到较高的维空间，这对于元容器更容易利用。我们证明了RBF层的兴趣以及在四个机器人模拟任务上对社会机器人技术的使用元素使用。

代理商学习广泛适用和通用策略具有重要意义，可以实现包括图像和文本描述在内的各种目标。考虑到这类感知的目标，深度加强学习研究的前沿是学习一个没有手工制作奖励的目标条件政策。要了解这种政策，最近的作品通常会像奖励到明确的嵌入空间中的给定目标的非参数距离。从不同的观点来看，我们提出了一种新的无监督学习方法，名为目标条件政策，具有内在动机（GPIM），共同学习抽象级别政策和目标条件的政策。摘要级别策略在潜在变量上被调节，以优化鉴别器，并发现进一步的不同状态，进一步呈现为目标条件策略的感知特定目标。学习鉴别者作为目标条件策略的内在奖励功能，以模仿抽象级别政策引起的轨迹。各种机器人任务的实验证明了我们所提出的GPIM方法的有效性和效率，其基本上优于现有技术。

离线RL算法必须说明其提供的数据集可能使环境的许多方面未知。应对这一挑战的最常见方法是采用悲观或保守的方法，避免行为与培训数据集中的行为过于不同。但是，仅依靠保守主义存在缺点：绩效对保守主义的确切程度很敏感，保守的目标可以恢复高度最佳的政策。在这项工作中，我们建议在不确定性的情况下，脱机RL方法应该是适应性的。我们表明，在贝叶斯的意义上，在离线RL中最佳作用涉及解决隐式POMDP。结果，离线RL的最佳策略必须是自适应的，这不仅取决于当前状态，而且还取决于迄今为止在评估期间看到的所有过渡。我们提出了一种无模型的算法，用于近似于此最佳自适应策略，并证明在离线RL基准测试中学习此类适应性政策。