Via operator theoretic methods, we formalize the concentration phenomenon for a given observable `$r$' of a discrete time Markov chain with `$\mu_{\pi}$' as invariant ergodic measure, possibly having support on an unbounded state space. The main contribution of this paper is circumventing tedious probabilistic methods with a study of a composition of the Markov transition operator $P$ followed by a multiplication operator defined by $e^{r}$. It turns out that even if the observable/ reward function is unbounded, but for some for some $q>2$, $\|e^{r}\|_{q \rightarrow 2} \propto \exp\big(\mu_{\pi}(r) +\frac{2q}{q-2}\big) $ and $P$ is hyperbounded with norm control $\|P\|_{2 \rightarrow q }< e^{\frac{1}{2}[\frac{1}{2}-\frac{1}{q}]}$, sharp non-asymptotic concentration bounds follow. \emph{Transport-entropy} inequality ensures the aforementioned upper bound on multiplication operator for all $q>2$. The role of \emph{reversibility} in concentration phenomenon is demystified. These results are particularly useful for the reinforcement learning and controls communities as they allow for concentration inequalities w.r.t standard unbounded obersvables/reward functions where exact knowledge of the system is not available, let alone the reversibility of stationary measure.

We study the concentration phenomenon for discrete-time random dynamical systems with an unbounded state space. We develop a heuristic approach towards obtaining exponential concentration inequalities for dynamical systems using an entirely functional analytic framework. We also show that existence of exponential-type Lyapunov function, compared to the purely deterministic setting, not only implies stability but also exponential concentration inequalities for sampling from the stationary distribution, via \emph{transport-entropy inequality} (T-E). These results have significant impact in \emph{reinforcement learning} (RL) and \emph{controls}, leading to exponential concentration inequalities even for unbounded observables, while neither assuming reversibility nor exact knowledge of random dynamical system (assumptions at heart of concentration inequalities in statistical mechanics and Markov diffusion processes).

We consider autocovariance operators of a stationary stochastic process on a Polish space that is embedded into a reproducing kernel Hilbert space. We investigate how empirical estimates of these operators converge along realizations of the process under various conditions. In particular, we examine ergodic and strongly mixing processes and obtain several asymptotic results as well as finite sample error bounds. We provide applications of our theory in terms of consistency results for kernel PCA with dependent data and the conditional mean embedding of transition probabilities. Finally, we use our approach to examine the nonparametric estimation of Markov transition operators and highlight how our theory can give a consistency analysis for a large family of spectral analysis methods including kernel-based dynamic mode decomposition.

尽管U统计量在现代概率和统计学中存在着无处不在的，但其在依赖框架中的非反应分析可能被忽略了。在最近的一项工作中，已经证明了对统一的马尔可夫链的U级统计数据的新浓度不平等。在本文中，我们通过在三个不同的研究领域中进一步推动了当前知识状态，将这一理论突破付诸实践。首先，我们为使用MCMC方法估算痕量类积分运算符光谱的新指数不平等。新颖的是，这种结果适用于具有正征和负征值的内核，据我们所知，这是新的。此外，我们研究了使用成对损失函数和马尔可夫链样品的在线算法的概括性能。我们通过展示如何从任何在线学习者产生的假设序列中提取低风险假设来提供在线到批量转换结果。我们最终对马尔可夫链的不变度度量的密度进行了拟合优度测试的非反应分析。我们确定了一些类别的替代方案，基于$ L_2 $距离的测试具有规定的功率。

我们调查了一定类别的功能不等式，称为弱Poincar的不等式，以使Markov链的收敛性与均衡相结合。我们表明，这使得SubGoom测量收敛界的直接和透明的推导出用于独立的Metropolis - Hastings采样器和用于棘手似然性的伪边缘方法，后者在许多实际设置中是子表芯。这些结果依赖于马尔可夫链之间的新量化比较定理。相关证据比依赖于漂移/较小化条件的证据更简单，并且所开发的工具允许我们恢复并进一步延长特定情况的已知结果。我们能够为伪边缘算法的实际使用提供新的见解，分析平均近似贝叶斯计算（ABC）的效果以及独立平均值的产品，以及研究与之相关的逻辑重量的情况粒子边缘大都市 - 黑斯廷斯（PMMH）。

We study a class of dynamical systems modelled as Markov chains that admit an invariant distribution via the corresponding transfer, or Koopman, operator. While data-driven algorithms to reconstruct such operators are well known, their relationship with statistical learning is largely unexplored. We formalize a framework to learn the Koopman operator from finite data trajectories of the dynamical system. We consider the restriction of this operator to a reproducing kernel Hilbert space and introduce a notion of risk, from which different estimators naturally arise. We link the risk with the estimation of the spectral decomposition of the Koopman operator. These observations motivate a reduced-rank operator regression (RRR) estimator. We derive learning bounds for the proposed estimator, holding both in i.i.d. and non i.i.d. settings, the latter in terms of mixing coefficients. Our results suggest RRR might be beneficial over other widely used estimators as confirmed in numerical experiments both for forecasting and mode decomposition.

We study the problem of estimating the fixed point of a contractive operator defined on a separable Banach space. Focusing on a stochastic query model that provides noisy evaluations of the operator, we analyze a variance-reduced stochastic approximation scheme, and establish non-asymptotic bounds for both the operator defect and the estimation error, measured in an arbitrary semi-norm. In contrast to worst-case guarantees, our bounds are instance-dependent, and achieve the local asymptotic minimax risk non-asymptotically. For linear operators, contractivity can be relaxed to multi-step contractivity, so that the theory can be applied to problems like average reward policy evaluation problem in reinforcement learning. We illustrate the theory via applications to stochastic shortest path problems, two-player zero-sum Markov games, as well as policy evaluation and $Q$-learning for tabular Markov decision processes.

储层计算系统是使用驱动的动力系统构建的，在该系统中，外部输入可以改变系统的发展状态。这些范例用于信息处理，机器学习和计算。在此框架中需要解决的一个基本问题是输入与系统状态之间的统计关系。本文提供的条件可以保证驱动系统的渐近措施的存在和唯一性，并表明当输入和输出过程的集合赋予了Wasserstein距离时，它们对输入过程的依赖性是连续的。这些发展中的主要工具是将这些不变的度量表征为在这种情况下出现并在论文中进行了大量研究的自然定义的FOIA算子的固定点。这些固定点是通过在驱动系统中施加新引入的随机状态合同性来获得的，该系统在示例中很容易验证。可以通过非国家缩减的系统来满足随机状态的合同性，这通常是为了保证储层计算中的回声状态属性的需求。结果，即使不存在Echo State属性，也可能会得到满足。

我们证明了连续和离散时间添加功能的浓度不平等和相关的PAC界限，用于可能是多元，不可逆扩散过程的无界函数。我们的分析依赖于通过泊松方程的方法，使我们能够考虑一系列非常广泛的指数性千古过程。这些结果增加了现有的浓度不平等，用于扩散过程的加性功能，这些功能仅适用于有界函数或从明显较小的类别中的过程的无限函数。我们通过两个截然不同的区域的例子来证明这些指数不平等的力量。考虑到在稀疏性约束下可能具有高维参数非线性漂移模型，我们应用连续的时间浓度结果来验证套索估计的受限特征值条件，这对于甲骨文不平等的推导至关重要。离散添加功能的结果用于研究未经调整的Langevin MCMC算法，用于采样中等重尾密度$ \ pi $。特别是，我们为多项式增长功能$ f $的样品蒙特卡洛估计量$ \ pi（f）提供PAC边界，以量化足够的样本和阶梯尺寸，以在规定的边距内近似具有很高的可能性。

Q学习长期以来一直是最受欢迎的强化学习算法之一，几十年来，Q学习的理论分析一直是一个活跃的研究主题。尽管对Q-学习的渐近收敛分析的研究具有悠久的传统，但非肿瘤收敛性直到最近才受到积极研究。本文的主要目的是通过控制系统的观点研究马尔可夫观察模型下异步Q学习的新有限时间分析。特别是，我们引入了Q学习的离散时间变化的开关系统模型，并减少了分析的步骤尺寸，这显着改善了使用恒定步骤尺寸的开关系统分析的最新开发，并导致\（\（\）（\） Mathcal {o} \ left（\ sqrt {\ frac {\ log k} {k}}} \ right）\）\）\）\）\）\）\）\）与大多数艺术状态相当或更好。同时，新应用了使用类似转换的技术，以避免通过减小的步骤尺寸提出的分析中的难度。提出的分析带来了其他见解，涵盖了不同的方案，并提供了新的简化模板，以通过其独特的连接与离散时间切换系统的独特联系来加深我们对Q学习的理解。

我们在具有Martingale差异噪声的可实现的时间序列框架中学习正方形损失。我们的主要结果是一个快速率的多余风险结合，这表明每当轨迹超收缩条件成立时，依赖数据的最小二乘估计器的风险与燃烧时间后的IID速率订单匹配。相比之下，从依赖数据中学习的许多现有结果都具有有效的样本量，即使在燃烧时间之后，有效的样本量也被基础过程的混合时间降低。此外，我们的结果允许协变量过程表现出远距离相关性，这些相关性大大弱于几何牙齿。我们将这种现象学习称为几乎没有混合的方式，并为其示出了几个示例：$ l^2 $和$ l^{2+\ epsilon} $ norms的有界函数类是等效的，有限的有限态Markov链，各种参数模型，以及一个无限尺寸$ \ ell^2（\ mathbb {n}）$椭圆形的广阔家族。通过将我们的主要结果实例化，以使用广义线性模型过渡对非线性动力学的系统识别，我们仅在多项式燃烧时间后获得了几乎最小的最佳超量风险。

深度学习的概括分析通常假定训练会收敛到固定点。但是，最近的结果表明，实际上，用随机梯度下降优化的深神经网络的权重通常无限期振荡。为了减少理论和实践之间的这种差异，本文着重于神经网络的概括，其训练动力不一定会融合到固定点。我们的主要贡献是提出一个统计算法稳定性（SAS）的概念，该算法将经典算法稳定性扩展到非convergergent算法并研究其与泛化的联系。与传统的优化和学习理论观点相比，这种崇高的理论方法可导致新的见解。我们证明，学习算法的时间复杂行为的稳定性与其泛化有关，并在经验上证明了损失动力学如何为概括性能提供线索。我们的发现提供了证据表明，即使训练无限期继续并且权重也不会融合，即使训练持续进行训练，训练更好地概括”的网络也是如此。

我们研究了无限 - 马，连续状态和行动空间的政策梯度的全球融合以及熵登记的马尔可夫决策过程（MDPS）。我们考虑了在平均场状态下具有（单隐层）神经网络近似（一层）神经网络近似的策略。添加了相关的平均场概率度量中的其他熵正则化，并在2-Wasserstein度量中研究了相应的梯度流。我们表明，目标函数正在沿梯度流量增加。此外，我们证明，如果按平均场测量的正则化足够，则梯度流将成倍收敛到唯一的固定溶液，这是正则化MDP物镜的独特最大化器。最后，我们研究了相对于正则参数和初始条件，沿梯度流的值函数的灵敏度。我们的结果依赖于对非线性Fokker-Planck-Kolmogorov方程的仔细分析，并扩展了Mei等人的开拓性工作。 2020和Agarwal等。 2020年，量化表格环境中熵调控MDP的策略梯度的全局收敛速率。

我们解决了从单个观测轨迹估算马尔可夫链的混合时间的基本问题。与以前考虑了希尔伯特空间方法来估计光谱差距的作品相反，我们选择了基于收缩的总变异的方法。具体而言，我们根据Dobrushin定义并估算了广义收缩系数。我们表明，与光谱差距不同，该数量可以控制到强烈通用常数的混合时间，并且对于非可逆链仍然有效。我们在系数周围设计了完全依赖数据的置信区间，该系数既比其光谱对应物更易于计算和更薄。此外，我们通过展示如何利用有关过渡矩阵的其他信息来启动超越最坏情况的分析，以便获得有关其相对于诱导统一规范的实例依赖性速率以及其某些混合属性。

由于难以在具有不确定环境中处理高维空间中的函数近似的难度，因此对增强学习（RL）的大多数现有的理论分析仅限于表格设置或线性模型。这项工作通过在一般的再现内核希尔伯特空间（RKHS）中分析RL，提供了新的挑战。我们考虑一个Markov决策过程的家庭$ \ mathcal {m} $，其中奖励功能位于RKHS的单位球中，过渡概率在给定的任意集中。我们通过分发不匹配$ \ delta _ {\ mathcal {m}}（\ epsilon）$来描述可允许的状态动作分配空间的复杂性，以响应RKHS中的扰动，以规模$ \ epsilon的扰动来描述禁用的状态动作分配空间的复杂性。 $。我们展示$ \ delta _ {\ mathcal {m}}（\ epsilon）$给出所有可能算法的错误的下限和两个特定算法的上限（适合奖励和拟合Q-ereration）的RL问题。因此，$ \ delta_ \ mathcal {m}（\ epsilon）$关于$ \ epsilon $衡量$ \ mathcal {m} $的难度。我们进一步提供了一些具体的示例，并讨论了$ \ delta _ {\ mathcal {m}}（\ epsilon）$衰减在这些例子中。作为副产品，我们表明，当奖励功能在高维RKHS中时，即使接到概率是已知的并且动作空间是有限的，仍然可以遭受维度的诅咒。

In this paper we develop a theoretical analysis of the performance of sampling-based fitted value iteration (FVI) to solve infinite state-space, discounted-reward Markovian decision processes (MDPs) under the assumption that a generative model of the environment is available. Our main results come in the form of finite-time bounds on the performance of two versions of sampling-based FVI. The convergence rate results obtained allow us to show that both versions of FVI are well behaving in the sense that by using a sufficiently large number of samples for a large class of MDPs, arbitrary good performance can be achieved with high probability. An important feature of our proof technique is that it permits the study of weighted L p -norm performance bounds. As a result, our technique applies to a large class of function-approximation methods (e.g., neural networks, adaptive regression trees, kernel machines, locally weighted learning), and our bounds scale well with the effective horizon of the MDP. The bounds show a dependence on the stochastic stability properties of the MDP: they scale with the discounted-average concentrability of the future-state distributions. They also depend on a new measure of the approximation power of the function space, the inherent Bellman residual, which reflects how well the function space is "aligned" with the dynamics and rewards of the MDP. The conditions of the main result, as well as the concepts introduced in the analysis, are extensively discussed and compared to previous theoretical results. Numerical experiments are used to substantiate the theoretical findings.

We consider learning approximate Nash equilibria for discrete-time mean-field games with nonlinear stochastic state dynamics subject to both average and discounted costs. To this end, we introduce a mean-field equilibrium (MFE) operator, whose fixed point is a mean-field equilibrium (i.e. equilibrium in the infinite population limit). We first prove that this operator is a contraction, and propose a learning algorithm to compute an approximate mean-field equilibrium by approximating the MFE operator with a random one. Moreover, using the contraction property of the MFE operator, we establish the error analysis of the proposed learning algorithm. We then show that the learned mean-field equilibrium constitutes an approximate Nash equilibrium for finite-agent games.

我们研究了有限空间中值的静止随机过程的最佳运输。为了反映潜在流程的实向性，我们限制了对固定联轴器的关注，也称为联系。由此产生的最佳连接问题捕获感兴趣过程的长期平均行为的差异。我们介绍了最优联接的估算和最佳的加入成本，我们建立了温和条件下估算器的一致性。此外，在更强的混合假设下，我们为估计的最佳连接成本建立有限样本误差速率，其延伸了IID案件中的最佳已知结果。最后，我们将一致性和速率分析扩展到最佳加入问题的熵惩罚版本。

Offline policy evaluation is a fundamental statistical problem in reinforcement learning that involves estimating the value function of some decision-making policy given data collected by a potentially different policy. In order to tackle problems with complex, high-dimensional observations, there has been significant interest from theoreticians and practitioners alike in understanding the possibility of function approximation in reinforcement learning. Despite significant study, a sharp characterization of when we might expect offline policy evaluation to be tractable, even in the simplest setting of linear function approximation, has so far remained elusive, with a surprising number of strong negative results recently appearing in the literature. In this work, we identify simple control-theoretic and linear-algebraic conditions that are necessary and sufficient for classical methods, in particular Fitted Q-iteration (FQI) and least squares temporal difference learning (LSTD), to succeed at offline policy evaluation. Using this characterization, we establish a precise hierarchy of regimes under which these estimators succeed. We prove that LSTD works under strictly weaker conditions than FQI. Furthermore, we establish that if a problem is not solvable via LSTD, then it cannot be solved by a broad class of linear estimators, even in the limit of infinite data. Taken together, our results provide a complete picture of the behavior of linear estimators for offline policy evaluation, unify previously disparate analyses of canonical algorithms, and provide significantly sharper notions of the underlying statistical complexity of offline policy evaluation.

We study non-parametric estimation of the value function of an infinite-horizon $\gamma$-discounted Markov reward process (MRP) using observations from a single trajectory. We provide non-asymptotic guarantees for a general family of kernel-based multi-step temporal difference (TD) estimates, including canonical $K$-step look-ahead TD for $K = 1, 2, \ldots$ and the TD$(\lambda)$ family for $\lambda \in [0,1)$ as special cases. Our bounds capture its dependence on Bellman fluctuations, mixing time of the Markov chain, any mis-specification in the model, as well as the choice of weight function defining the estimator itself, and reveal some delicate interactions between mixing time and model mis-specification. For a given TD method applied to a well-specified model, its statistical error under trajectory data is similar to that of i.i.d. sample transition pairs, whereas under mis-specification, temporal dependence in data inflates the statistical error. However, any such deterioration can be mitigated by increased look-ahead. We complement our upper bounds by proving minimax lower bounds that establish optimality of TD-based methods with appropriately chosen look-ahead and weighting, and reveal some fundamental differences between value function estimation and ordinary non-parametric regression.