One of the key challenges in deploying RL to real-world applications is to adapt to variations of unknown environment contexts, such as changing terrains in robotic tasks and fluctuated bandwidth in congestion control. Existing works on adaptation to unknown environment contexts either assume the contexts are the same for the whole episode or assume the context variables are Markovian. However, in many real-world applications, the environment context usually stays stable for a stochastic period and then changes in an abrupt and unpredictable manner within an episode, resulting in a segment structure, which existing works fail to address. To leverage the segment structure of piecewise stable context in real-world applications, in this paper, we propose a \textit{\textbf{Se}gmented \textbf{C}ontext \textbf{B}elief \textbf{A}ugmented \textbf{D}eep~(SeCBAD)} RL method. Our method can jointly infer the belief distribution over latent context with the posterior over segment length and perform more accurate belief context inference with observed data within the current context segment. The inferred belief context can be leveraged to augment the state, leading to a policy that can adapt to abrupt variations in context. We demonstrate empirically that SeCBAD can infer context segment length accurately and outperform existing methods on a toy grid world environment and Mujuco tasks with piecewise-stable context.

Reinforcement learning often suffer from the sparse reward issue in real-world robotics problems. Learning from demonstration (LfD) is an effective way to eliminate this problem, which leverages collected expert data to aid online learning. Prior works often assume that the learning agent and the expert aim to accomplish the same task, which requires collecting new data for every new task. In this paper, we consider the case where the target task is mismatched from but similar with that of the expert. Such setting can be challenging and we found existing LfD methods can not effectively guide learning in mismatched new tasks with sparse rewards. We propose conservative reward shaping from demonstration (CRSfD), which shapes the sparse rewards using estimated expert value function. To accelerate learning processes, CRSfD guides the agent to conservatively explore around demonstrations. Experimental results of robot manipulation tasks show that our approach outperforms baseline LfD methods when transferring demonstrations collected in a single task to other different but similar tasks.

Nonnegative Tucker Factorization (NTF) minimizes the euclidean distance or Kullback-Leibler divergence between the original data and its low-rank approximation which often suffers from grossly corruptions or outliers and the neglect of manifold structures of data. In particular, NTF suffers from rotational ambiguity, whose solutions with and without rotation transformations are equally in the sense of yielding the maximum likelihood. In this paper, we propose three Robust Manifold NTF algorithms to handle outliers by incorporating structural knowledge about the outliers. They first applies a half-quadratic optimization algorithm to transform the problem into a general weighted NTF where the weights are influenced by the outliers. Then, we introduce the correntropy induced metric, Huber function and Cauchy function for weights respectively, to handle the outliers. Finally, we introduce a manifold regularization to overcome the rotational ambiguity of NTF. We have compared the proposed method with a number of representative references covering major branches of NTF on a variety of real-world image databases. Experimental results illustrate the effectiveness of the proposed method under two evaluation metrics (accuracy and nmi).

End-to-end autonomous driving provides a feasible way to automatically maximize overall driving system performance by directly mapping the raw pixels from a front-facing camera to control signals. Recent advanced methods construct a latent world model to map the high dimensional observations into compact latent space. However, the latent states embedded by the world model proposed in previous works may contain a large amount of task-irrelevant information, resulting in low sampling efficiency and poor robustness to input perturbations. Meanwhile, the training data distribution is usually unbalanced, and the learned policy is hard to cope with the corner cases during the driving process. To solve the above challenges, we present a semantic masked recurrent world model (SEM2), which introduces a latent filter to extract key task-relevant features and reconstruct a semantic mask via the filtered features, and is trained with a multi-source data sampler, which aggregates common data and multiple corner case data in a single batch, to balance the data distribution. Extensive experiments on CARLA show that our method outperforms the state-of-the-art approaches in terms of sample efficiency and robustness to input permutations.

为设计控制器选择适当的参数集对于最终性能至关重要，但通常需要一个乏味而仔细的调整过程，这意味着强烈需要自动调整方法。但是，在现有方法中，无衍生物的可扩展性或效率低下，而基于梯度的方法可能由于可能是非差异的控制器结构而无法使用。为了解决问题，我们使用新颖的无衍生化强化学习（RL）框架来解决控制器调整问题，该框架在经验收集过程中在参数空间中执行时间段的扰动，并将无衍生策略更新集成到高级参与者 - 批判性RL中实现高多功能性和效率的体系结构。为了证明该框架的功效，我们在自动驾驶的两个具体示例上进行数值实验，即使用PID控制器和MPC控制器进行轨迹跟踪的自适应巡航控制。实验结果表明，所提出的方法的表现优于流行的基线，并突出了其强大的控制器调整潜力。

强化学习表现出巨大的潜力，可以解决复杂的接触率丰富的机器人操纵任务。但是，在现实世界中使用RL的安全是一个关键问题，因为在培训期间或看不见的情况下，RL政策是不完善的，可能会发生意外的危险碰撞。在本文中，我们提出了一个接触安全的增强增强学习框架，用于接触良好的机器人操纵，该框架在任务空间和关节空间中保持安全性。当RL政策导致机器人组与环境之间的意外冲突时，我们的框架能够立即检测到碰撞并确保接触力量很小。此外，最终效应器被强制执行，同时对外部干扰保持强大的态度。我们训练RL政策以模拟并将其转移到真正的机器人中。关于机器人擦拭任务的现实世界实验表明，即使在策略处于看不见的情况下，我们的方法也能够使接触在任务空间和关节空间中保持较小，同时拒绝对主要任务的干扰。

Transformer在学习视觉和语言表示方面取得了巨大的成功，这在各种下游任务中都是一般的。在视觉控制中，可以在不同控制任务之间转移的可转移状态表示对于减少训练样本量很重要。但是，将变压器移植到样品有效的视觉控制仍然是一个具有挑战性且未解决的问题。为此，我们提出了一种新颖的控制变压器（CTRLFORMER），具有先前艺术所没有的许多吸引人的好处。首先，CTRLFORMER共同学习视觉令牌和政策令牌之间的自我注意事项机制，在不同的控制任务之间可以学习和转移多任务表示无灾难性遗忘。其次，我们仔细设计了一种对比的增强学习范式来训练Ctrlformer，从而使其能够达到高样本效率，这在控制问题中很重要。例如，在DMControl基准测试中，与最近的高级方法不同，该方法在使用100K样品转移学习后通过在“ Cartpole”任务中产生零分数而失败，CTRLFORMER可以在维持100K样本的同时获得最先进的分数先前任务的性能。代码和模型已在我们的项目主页中发布。

Dynamic game arises as a powerful paradigm for multi-robot planning, for which safety constraint satisfaction is crucial. Constrained stochastic games are of particular interest, as real-world robots need to operate and satisfy constraints under uncertainty. Existing methods for solving stochastic games handle chance constraints using exponential penalties with hand-tuned weights. However, finding a suitable penalty weight is nontrivial and requires trial and error. In this paper, we propose the chance-constrained iterative linear-quadratic stochastic games (CCILQGames) algorithm. CCILQGames solves chance-constrained stochastic games using the augmented Lagrangian method. We evaluate our algorithm in three autonomous driving scenarios, including merge, intersection, and roundabout. Experimental results and Monte Carlo tests show that CCILQGames can generate safe and interactive strategies in stochastic environments.

最近基于进化的零级优化方法和基于策略梯度的一阶方法是解决加强学习（RL）问题的两个有希望的替代方案。前者的方法与任意政策一起工作，依赖状态依赖和时间扩展的探索，具有健壮性的属性，但遭受了较高的样本复杂性，而后者的方法更有效，但仅限于可区分的政策，并且学习的政策是不太强大。为了解决这些问题，我们提出了一种新颖的零级演员 - 批评算法（ZOAC），该算法将这两种方法统一为派对演员 - 批判性结构，以保留两者的优势。 ZOAC在参数空间，一阶策略评估（PEV）和零订单策略改进（PIM）的参数空间中进行了推出集合，每次迭代中都会进行推出。我们使用不同类型的策略在广泛的挑战连续控制基准上进行广泛评估我们的方法，其中ZOAC优于零阶和一阶基线算法。

传统的深度传感器产生准确的真实世界深度估计，即使仅在仿真域训练的最先进的学习方法也会超越。由于在模拟域中容易获得地面真理深度，但在真实域中很难获得，因此我们提出了一种利用两个世界的最佳方法的方法。在本文中，我们展示了一个新的框架，ActiveZero，这是一个混合域学习解决方案，适用于不需要真实世界深度注释的活动立体宽度系统。首先，我们通过使用混合域学习策略来证明我们的方法对分发外数据的可转换性。在仿真域中，我们在形状原语数据集上使用监督差异丢失和自我监督损失的组合。相比之下，在真实域中，我们只在数据集中使用自我监督损失，这些损失是从培训仿真数据或测试真实数据的分发。其次，我们的方法介绍了一种名为Temporal IR的自我监督损失，以增加我们在难以感知地区的重新注入的鲁棒性和准确性。最后，我们展示了如何训练该方法的端到端，并且每个模块对于获得最终结果很重要。关于真实数据的广泛定性和定量评估表明了甚至可以击败商业深度传感器的最新状态。