我们认为在情节环境中的强化学习（RL）中的遗憾最小化问题。在许多实际的RL环境中，状态和动作空间是连续的或非常大的。现有方法通过随机过渡模型的低维表示或$ q $ functions的近似值来确定遗憾的保证。但是，对国家价值函数的函数近似方案的理解基本上仍然缺失。在本文中，我们提出了一种基于在线模型的RL算法，即CME-RL，该算法将过渡分布的表示形式学习为嵌入在复制的内核希尔伯特领域中的嵌入，同时仔细平衡了利用探索 - 探索权衡取舍。我们通过证明频繁的（最糟糕的）遗憾结束了$ \ tilde {o} \ big（h \ gamma_n \ sqrt {n} \ big）$ \ footnote {$ footnote {$ tilde {$ o}（\ cdot）$仅隐藏绝对常数和poly-logarithmic因素。}，其中$ h $是情节长度，$ n $是时间步长的总数，$ \ gamma_n $是信息理论数量国家行动特征空间的有效维度。我们的方法绕过了估计过渡概率的需求，并适用于可以定义内核的任何域。它还为内核方法的一般理论带来了新的见解，以进行近似推断和RL遗憾的最小化。

Modern Reinforcement Learning (RL) is commonly applied to practical problems with an enormous number of states, where function approximation must be deployed to approximate either the value function or the policy. The introduction of function approximation raises a fundamental set of challenges involving computational and statistical efficiency, especially given the need to manage the exploration/exploitation tradeoff. As a result, a core RL question remains open: how can we design provably efficient RL algorithms that incorporate function approximation? This question persists even in a basic setting with linear dynamics and linear rewards, for which only linear function approximation is needed.This paper presents the first provable RL algorithm with both polynomial runtime and polynomial sample complexity in this linear setting, without requiring a "simulator" or additional assumptions. Concretely, we prove that an optimistic modification of Least-Squares Value Iteration (LSVI)-a classical algorithm frequently studied in the linear setting-achieves O( √ d 3 H 3 T ) regret, where d is the ambient dimension of feature space, H is the length of each episode, and T is the total number of steps. Importantly, such regret is independent of the number of states and actions.

我们考虑在具有非线性函数近似的两名玩家零和马尔可夫游戏中学习NASH平衡，其中动作值函数通过繁殖内核Hilbert Space（RKHS）中的函数近似。关键挑战是如何在高维函数空间中进行探索。我们提出了一种新颖的在线学习算法，以最大程度地减少双重性差距来找到NASH平衡。我们算法的核心是基于不确定性的乐观原理得出的上和下置信度界限。我们证明，在非常温和的假设上，我们的算法能够获得$ O（\ sqrt {t}）$遗憾，并在对奖励功能和马尔可夫游戏的基本动态下进行多项式计算复杂性。我们还提出了我们的算法的几个扩展，包括具有伯恩斯坦型奖励的算法，可以实现更严格的遗憾，以及用于模型错误指定的另一种算法，可以应用于神经功能近似。

我们解决了有限地平线的模型选择的问题，用于转换内核$ P ^ * $属于一个型号$ \ mathcal {p} ^ * $的offultic公制熵。在模型选择框架中，而不是$ \ mathcal {p} ^ * $，我们被给予了$ m $嵌套的转换内核rested interned内核$ \ cp_1 \ subset \ cp_2 \ subset \ ldots \ subset \ cp_m $。我们提出并分析了一种新颖的算法，即\ EMPH {自适应增强学习（常规）}（\ texttt {arl-gen}），它适应真正的转换内核$ p ^ * $谎言的最小这些家庭。 \ texttt {arl-gen}使用具有价值目标回归的上置信度强化学习（\ texttt {Ucrl}）算法作为Blackbox，并在每个时代的开头放置模型选择模块。在模型类上的温和可分离性假设下，我们显示\ texttt {arl-gen}获得$ \ tilde {\ mathcal {o}}的后悔（d _ {\ mathcal {e}} ^ * h ^ 2 + \ sqrt {d _ {\ mathcal {e}} ^ * \ mathbb {m} ^ * h ^ 2 t}）$，具有高概率，其中$ h $是地平线长度，$ t $是步骤总数， $ d _ {\ mathcal {e}} ^ * $是ecured维度和$ \ mathbb {m} ^ * $是与$ \ mathcal {p} ^ * $相对应的度量熵。请注意，这一遗憾缩放匹配Oracle的Oracle，它提前了解$ \ mathcal {p} ^ * $。我们表明，对于\ texttt {arl-gen}的模型选择成本是一个附加术语，遗憾是对$ t $的弱点。随后，我们删除可分离假设，并考虑线性混合MDP的设置，其中转换内核$ P ^ * $具有线性函数近似。通过这种低等级结构，我们提出了新颖的自适应算法，用于模型选择，并获得（令人令人令）与Oracle的遗憾相同，具有真正的模型类。

代表学习呈现在深入学习的经验成功的核心，以处理维度的诅咒。然而，由于i），表现力（RL）的钢筋学习（RL）尚未充分利用卓越的能力，表现力和易疏忽之间的权衡;二世），探索与代表学习之间的耦合。在本文中，我们首先揭示了在随机控制模型中的一些噪声假设下，我们可以免费获得其相应的马尔可夫过渡操作员的线性谱特征。基于该观察，我们提出了嵌入（Spede）的谱动力学嵌入（SPEDE），这将通过利用噪声结构来完成对代表学习的乐观探索。我们提供对Speded的严格理论分析，并展示了几种基准上现有最先进的实证算法的实际卓越性能。

We study time-inhomogeneous episodic reinforcement learning (RL) under general function approximation and sparse rewards. We design a new algorithm, Variance-weighted Optimistic $Q$-Learning (VO$Q$L), based on $Q$-learning and bound its regret assuming completeness and bounded Eluder dimension for the regression function class. As a special case, VO$Q$L achieves $\tilde{O}(d\sqrt{HT}+d^6H^{5})$ regret over $T$ episodes for a horizon $H$ MDP under ($d$-dimensional) linear function approximation, which is asymptotically optimal. Our algorithm incorporates weighted regression-based upper and lower bounds on the optimal value function to obtain this improved regret. The algorithm is computationally efficient given a regression oracle over the function class, making this the first computationally tractable and statistically optimal approach for linear MDPs.

我们提出了一种乐观的基于模型的算法，Dubbed SMRL，用于通过指数族分布指定的转换模型，以D $参数指定，奖励是有界和已知的。SMRL使用得分匹配，一种无通量的密度估计技术，可以通过RIDGE回归有效地估计模型参数。在标准规律性假设下，SMRL实现$ \ tilde o（d \ sqrt {h ^ 3t}）$在线遗憾，其中$ h $是每一集的长度，$ t $是互动的总数（忽略多项式依赖结构尺度参数）。

Many real-world reinforcement learning tasks require control of complex dynamical systems that involve both costly data acquisition processes and large state spaces. In cases where the transition dynamics can be readily evaluated at specified states (e.g., via a simulator), agents can operate in what is often referred to as planning with a \emph{generative model}. We propose the AE-LSVI algorithm for best-policy identification, a novel variant of the kernelized least-squares value iteration (LSVI) algorithm that combines optimism with pessimism for active exploration (AE). AE-LSVI provably identifies a near-optimal policy \emph{uniformly} over an entire state space and achieves polynomial sample complexity guarantees that are independent of the number of states. When specialized to the recently introduced offline contextual Bayesian optimization setting, our algorithm achieves improved sample complexity bounds. Experimentally, we demonstrate that AE-LSVI outperforms other RL algorithms in a variety of environments when robustness to the initial state is required.

我们研究了受限的强化学习问题，其中代理的目的是最大程度地提高预期的累积奖励，从而受到对实用程序函数的预期总价值的约束。与现有的基于模型的方法或无模型方法伴随着“模拟器”，我们旨在开发第一个无模型的无模拟算法，即使在大规模系统中，也能够实现sublinear遗憾和透明度的约束侵犯。为此，我们考虑具有线性函数近似的情节约束决策过程，其中过渡动力学和奖励函数可以表示为某些已知功能映射的线性函数。我们表明$ \ tilde {\ mathcal {o}}（\ sqrt {d^3h^3t}）$遗憾和$ \ tilde {\ tillcal {\ mathcal {o}}（\ sqrt {d^3h^3ht}）$约束$约束$约束可以实现违规范围，其中$ d $是功能映射的尺寸，$ h $是情节的长度，而$ t $是总数的总数。我们的界限是在没有明确估计未知过渡模型或需要模拟器的情况下达到的，并且仅通过特征映射的维度依赖于状态空间。因此，即使国家的数量进入无穷大，我们的界限也会存在。我们的主要结果是通过标准LSVI-UCB算法的新型适应来实现的。特别是，我们首先将原始二次优化引入LSVI-UCB算法中，以在遗憾和违反约束之间取得平衡。更重要的是，我们使用软马克斯政策取代了LSVI-UCB中的状态行动功能的标准贪婪选择。事实证明，这对于通过其近似平滑度的权衡来确定受约束案例的统一浓度是关键。我们还表明，一个人可以达到均匀的约束违规行为，同时仍然保持相同的订单相对于$ t $。

我们在非静止线性（AKA低级别）马尔可夫决策过程（MDP）中研究了集中加强学习，即奖励和转换内核都是关于给定特征映射的线性，并且被允许缓慢或突然演变时间。对于此问题设置，我们提出了一种基于加权最小二乘值的乐观模型算法的Opt-WLSVI，其使用指数权重来平滑地忘记过去远远的数据。我们表明我们的算法在每次竞争最佳政策时，实现了由$ \ widetilde {\ mathcal {o}}的上部界限的遗憾（d ^ {5/4} h ^ 2 \ delta ^ {1 / 4} k ^ {3/4}）$何地在$ d $是特征空间的尺寸，$ h $是规划地平线，$ k $是剧集的数量和$ \ delta $是一个合适的衡量标准MDP的非固定性。此外，我们指出了在忘记以前作品的非静止线性匪徒环境中忘记策略的技术差距，并提出了修复其遗憾分析。

We study model-based reinforcement learning (RL) for episodic Markov decision processes (MDP) whose transition probability is parametrized by an unknown transition core with features of state and action. Despite much recent progress in analyzing algorithms in the linear MDP setting, the understanding of more general transition models is very restrictive. In this paper, we establish a provably efficient RL algorithm for the MDP whose state transition is given by a multinomial logistic model. To balance the exploration-exploitation trade-off, we propose an upper confidence bound-based algorithm. We show that our proposed algorithm achieves $\tilde{\mathcal{O}}(d \sqrt{H^3 T})$ regret bound where $d$ is the dimension of the transition core, $H$ is the horizon, and $T$ is the total number of steps. To the best of our knowledge, this is the first model-based RL algorithm with multinomial logistic function approximation with provable guarantees. We also comprehensively evaluate our proposed algorithm numerically and show that it consistently outperforms the existing methods, hence achieving both provable efficiency and practical superior performance.

We consider a multi-agent episodic MDP setup where an agent (leader) takes action at each step of the episode followed by another agent (follower). The state evolution and rewards depend on the joint action pair of the leader and the follower. Such type of interactions can find applications in many domains such as smart grids, mechanism design, security, and policymaking. We are interested in how to learn policies for both the players with provable performance guarantee under a bandit feedback setting. We focus on a setup where both the leader and followers are {\em non-myopic}, i.e., they both seek to maximize their rewards over the entire episode and consider a linear MDP which can model continuous state-space which is very common in many RL applications. We propose a {\em model-free} RL algorithm and show that $\tilde{\mathcal{O}}(\sqrt{d^3H^3T})$ regret bounds can be achieved for both the leader and the follower, where $d$ is the dimension of the feature mapping, $H$ is the length of the episode, and $T$ is the total number of steps under the bandit feedback information setup. Thus, our result holds even when the number of states becomes infinite. The algorithm relies on {\em novel} adaptation of the LSVI-UCB algorithm. Specifically, we replace the standard greedy policy (as the best response) with the soft-max policy for both the leader and the follower. This turns out to be key in establishing uniform concentration bound for the value functions. To the best of our knowledge, this is the first sub-linear regret bound guarantee for the Markov games with non-myopic followers with function approximation.

我们研究了具有线性函数近似增强学习中的随机最短路径（SSP）问题，其中过渡内核表示为未知模型的线性混合物。我们将此类别的SSP问题称为线性混合物SSP。我们提出了一种具有Hoeffding-type置信度的新型算法，用于学习线性混合物SSP，可以获得$ \ tilde {\ Mathcal {o}}}}（d B _ {\ star}^{1.5} \ sqrt {k/c_ {k/c_ {k/c_ {k/c_ { \ min}}）$遗憾。这里$ k $是情节的数量，$ d $是混合模型中功能映射的维度，$ b _ {\ star} $限制了最佳策略的预期累积成本，$ c _ {\ min}>> 0 $是成本函数的下限。当$ c _ {\ min} = 0 $和$ \ tilde {\ mathcal {o}}}（k^{2/3}）$遗憾时，我们的算法也适用于情况。据我们所知，这是第一个具有sublrinear遗憾保证线性混合物SSP的算法。此外，我们设计了精致的伯恩斯坦型信心集并提出了改进的算法，该算法可实现$ \ tilde {\ Mathcal {o}}}（d b _ {\ star} \ sqrt {k/c/c/c {k/c _ {\ min}}） $遗憾。为了补充遗憾的上限，我们还证明了$ \ omega（db _ {\ star} \ sqrt {k}）$的下限。因此，我们的改进算法将下限匹配到$ 1/\ sqrt {c _ {\ min}} $ factor和poly-logarithmic因素，从而实现了近乎最佳的遗憾保证。

Deep latent variable models have achieved significant empirical successes in model-based reinforcement learning (RL) due to their expressiveness in modeling complex transition dynamics. On the other hand, it remains unclear theoretically and empirically how latent variable models may facilitate learning, planning, and exploration to improve the sample efficiency of RL. In this paper, we provide a representation view of the latent variable models for state-action value functions, which allows both tractable variational learning algorithm and effective implementation of the optimism/pessimism principle in the face of uncertainty for exploration. In particular, we propose a computationally efficient planning algorithm with UCB exploration by incorporating kernel embeddings of latent variable models. Theoretically, we establish the sample complexity of the proposed approach in the online and offline settings. Empirically, we demonstrate superior performance over current state-of-the-art algorithms across various benchmarks.

我们在适应性约束下研究了强化学习（RL），线性函数近似。我们考虑两个流行的有限适应性模型：批量学习模型和稀有策略交换机模型，并提出了两个有效的在线线性马尔可夫决策过程的在线RL算法，其中转换概率和奖励函数可以表示为一些线性函数已知的特征映射。具体而言，对于批量学习模型，我们提出的LSVI-UCB-批处理算法实现了$ \ tilde o（\ sqrt {d ^ 3h ^ 3t} + dht / b）$后悔，$ d $是尺寸特征映射，$ H $是剧集长度，$ t $是交互数量，$ b $是批次数。我们的结果表明，只使用$ \ sqrt {t / dh} $批量来获得$ \ tilde o（\ sqrt {d ^ 3h ^ 3t}）$后悔。对于稀有策略开关模型，我们提出的LSVI-UCB-RARESWICH算法享有$ \ TINDE O（\ SQRT {D ^ 3h ^ 3t [1 + T /（DH）] ^ {dh / b}）$遗憾，这意味着$ dh \ log t $策略交换机足以获得$ \ tilde o（\ sqrt {d ^ 3h ^ 3t}）$后悔。我们的算法达到与LSVI-UCB算法相同的遗憾（Jin等，2019），但具有大量较小的适应性。我们还为批量学习模式建立了较低的界限，这表明对我们遗憾的依赖于您的遗憾界限是紧张的。

尽管无奖励强化学习勘探阶段的主要目标（RF-RL）是减少具有最小轨迹数量的估计模型中的不确定性时间。目前尚不清楚这种安全的探索要求如何影响相应的样本复杂性，以实现所获得的计划中所需的最佳性。在这项工作中，我们首次尝试回答这个问题。特别是，我们考虑了事先知道安全基线政策的情况，并提出了一个统一的安全奖励探索（甜蜜）框架。然后，我们将甜蜜框架专门为表格和低级MDP设置，并分别开发出算法所构成的表格甜味和低级别甜味。两种算法都利用了新引入的截短值函数的凹度和连续性，并保证在探索过程中以高概率侵犯了零约束。此外，两种算法都可以在计划阶段的任何约束中找到近乎最佳的政策。值得注意的是，算法下的样本复杂性在无限制的对应物中匹配甚至超过最恒定因素的最新情况，这证明安全约束几乎不会增加RF-RL的样本复杂性。

我们研究了基于模型的无奖励加强学习，具有ePiSodic Markov决策过程的线性函数近似（MDP）。在此设置中，代理在两个阶段工作。在勘探阶段，代理商与环境相互作用并在没有奖励的情况下收集样品。在规划阶段，代理商给出了特定的奖励功能，并使用从勘探阶段收集的样品来学习良好的政策。我们提出了一种新的可直接有效的算法，称为UCRL-RFE在线性混合MDP假设，其中MDP的转换概率内核可以通过线性函数参数化，在状态，动作和下一个状态的三联体上定义的某些特征映射上参数化。我们展示了获得$ \ epsilon $-Optimal策略进行任意奖励函数，Ucrl-RFE需要以大多数$ \ tilde {\ mathcal {o}}来进行采样（h ^ 5d ^ 2 \ epsilon ^ { - 2}）勘探阶段期间的$派对。在这里，$ H $是集的长度，$ d $是特征映射的尺寸。我们还使用Bernstein型奖金提出了一种UCRL-RFE的变种，并表明它需要在大多数$ \ TINDE {\ MATHCAL {o}}（H ^ 4D（H + D）\ epsilon ^ { - 2}）进行样本$达到$ \ epsilon $ -optimal政策。通过构建特殊类的线性混合MDPS，我们还证明了对于任何无奖励算法，它需要至少为$ \ TINDE \ OMEGA（H ^ 2d \ epsilon ^ { - 2}）$剧集来获取$ \ epsilon $ -optimal政策。我们的上限与依赖于$ \ epsilon $的依赖性和$ d $ if $ h \ ge d $。

We study reinforcement learning (RL) with linear function approximation. For episodic time-inhomogeneous linear Markov decision processes (linear MDPs) whose transition dynamic can be parameterized as a linear function of a given feature mapping, we propose the first computationally efficient algorithm that achieves the nearly minimax optimal regret $\tilde O(d\sqrt{H^3K})$, where $d$ is the dimension of the feature mapping, $H$ is the planning horizon, and $K$ is the number of episodes. Our algorithm is based on a weighted linear regression scheme with a carefully designed weight, which depends on a new variance estimator that (1) directly estimates the variance of the \emph{optimal} value function, (2) monotonically decreases with respect to the number of episodes to ensure a better estimation accuracy, and (3) uses a rare-switching policy to update the value function estimator to control the complexity of the estimated value function class. Our work provides a complete answer to optimal RL with linear MDPs, and the developed algorithm and theoretical tools may be of independent interest.

Model-free reinforcement learning (RL) algorithms, such as Q-learning, directly parameterize and update value functions or policies without explicitly modeling the environment. They are typically simpler, more flexible to use, and thus more prevalent in modern deep RL than model-based approaches. However, empirical work has suggested that model-free algorithms may require more samples to learn [7,22]. The theoretical question of "whether model-free algorithms can be made sample efficient" is one of the most fundamental questions in RL, and remains unsolved even in the basic scenario with finitely many states and actions.We prove that, in an episodic MDP setting, Q-learning with UCB exploration achieves regret Õ( √ H 3 SAT ), where S and A are the numbers of states and actions, H is the number of steps per episode, and T is the total number of steps. This sample efficiency matches the optimal regret that can be achieved by any model-based approach, up to a single √ H factor. To the best of our knowledge, this is the first analysis in the model-free setting that establishes √ T regret without requiring access to a "simulator." * The first two authors contributed equally.

深度加强学习（RL）由Q函数的神经网络近似，具有巨大的经验成功。虽然RL的理论传统上专注于线性函数近似（或雕刻尺寸）方法，但是关于非线性RL的近似已知Q功能的神经网络近似。这是这项工作的重点，在那里我们研究了与双层神经网络的函数逼近（考虑到Relu和多项式激活功能）。我们的第一个结果是在两层神经网络的完整性下的生成模型设置中的计算上和统计学高效的算法。我们的第二个结果考虑了这个设置，而是通过神经网络函数类的可实现性。这里，假设确定性动态，样本复杂度在代数维度中线性缩放。在所有情况下，我们的结果显着改善了线性（或雕刻尺寸）方法可以获得的。