在强化学习中，蒙特卡洛算法通过平均偶发回报来更新Q功能。在Monte Carlo UCB（MC-UCB）算法中，在每个状态下采取的动作是最大化Q函数加上UCB勘探项的动作，该术语偏向于选择频率较低的动作的选择。尽管在为MC-UCB建立遗憾界限方面已经进行了重要的工作，但大多数工作都集中在该问题的有限培训版本上，每个情节都在不断数量的步骤后终止。对于此类有限的Horizon问题，最佳策略既取决于当前状态和情节中的时间。但是，对于许多自然的情节问题，例如GO，CHESS和机器人任务等游戏，该情节是随机的，最佳政策是静止的。对于此类环境，MC-UCB中的Q功能是否会收敛到最佳Q函数，这是一个空旷的问题。我们猜想，与Q学习不同，它并不是所有MDP的收敛。尽管如此，我们表明，对于大型MDP，其中包括二十一点和确定性MDP等随机MDP，例如GO，MC-UCB中的Q功能几乎可以肯定地收敛到最佳Q函数。该结果的直接推论是，它几乎肯定会为所有有限的Horizon MDP收敛。我们还提供了数值实验，为MC-UCB提供了进一步的见解。
translated by 谷歌翻译
2022-02-17
我们提出了VRL3，这是一个强大的数据驱动框架，其简单设计用于解决挑战性的视觉深度强化学习（DRL）任务。我们分析了采用数据驱动方法的许多主要障碍，并提出了一系列设计原理，新颖的发现以及有关数据驱动的视觉DRL的关键见解。我们的框架有三个阶段：在第1阶段，我们利用非RL数据集（例如ImageNet）学习任务无关的视觉表示；在第2阶段，我们使用离线RL数据（例如，专家演示数量有限）将任务不合时宜的表示转换为更强大的特定任务表示；在第3阶段，我们用在线RL微调了代理商。与以前的SOTA相比，在一系列具有稀疏奖励和现实视觉输入的具有挑战性的手动操纵任务上，VRL3平均达到了780％的样本效率。在最艰巨的任务上，VRL3的样本有效效率高1220％（使用较宽的编码器时2440％），仅使用计算的10％来解决任务。这些重要的结果清楚地表明了数据驱动的深度强化学习的巨大潜力。
translated by 谷歌翻译
最近，使用批评者分配表示截断的分量批评者（TQC），显示在Mujoco连续控制基准套件的所有环境中提供最先进的渐近培训表现。此外，使用高更新到数据比和目标随机化的随机集合双Q学习（REDQ）达到了具有基于最先进的模型的方法竞争的高样本效率。在本文中，我们提出了一种新的无模型算法，具有集合（AQE）的激进Q学习，这提高了REDQ的样品效率性能和TQC的渐近性能，从而提供了整体最先进的性能在培训的所有阶段。此外，AQE非常简单，要求批评者的分布表示也不是目标随机化。
translated by 谷歌翻译
一种简单自然的增强学习算法（RL）是蒙特卡洛探索开始（MCES），通过平均蒙特卡洛回报来估算Q功能，并通过选择最大化Q当前估计的行动来改进策略。 -功能。探索是通过“探索开始”来执行的，即每个情节以随机选择的状态和动作开始，然后遵循当前的策略到终端状态。在Sutton＆Barto（2018）的RL经典书中，据说建立MCES算法的收敛是RL中最重要的剩余理论问题之一。但是，MCE的收敛问题证明是非常细微的。 Bertsekas＆Tsitsiklis（1996）提供了一个反例，表明MCES算法不一定会收敛。 TSITSIKLIS（2002）进一步表明，如果修改了原始MCES算法，以使Q-功能估计值以所有状态行动对以相同的速率更新，并且折现因子严格少于一个，则MCES算法收敛。在本文中，我们通过Sutton＆Barto（1998）中给出的原始，更有效的MCES算法取得进展政策。这样的MDP包括大量的环境，例如所有确定性环境和所有具有时间步长的情节环境或作为状态的任何单调变化的值。与以前使用随机近似的证据不同，我们引入了一种新型的感应方法，该方法非常简单，仅利用大量的强规律。
translated by 谷歌翻译
Periocular refers to the region of the face that surrounds the eye socket. This is a feature-rich area that can be used by itself to determine the identity of an individual. It is especially useful when the iris or the face cannot be reliably acquired. This can be the case of unconstrained or uncooperative scenarios, where the face may appear partially occluded, or the subject-to-camera distance may be high. However, it has received revived attention during the pandemic due to masked faces, leaving the ocular region as the only visible facial area, even in controlled scenarios. This paper discusses the state-of-the-art of periocular biometrics, giving an overall framework of its most significant research aspects.
translated by 谷歌翻译
Detecting actions in untrimmed videos should not be limited to a small, closed set of classes. We present a simple, yet effective strategy for open-vocabulary temporal action detection utilizing pretrained image-text co-embeddings. Despite being trained on static images rather than videos, we show that image-text co-embeddings enable openvocabulary performance competitive with fully-supervised models. We show that the performance can be further improved by ensembling the image-text features with features encoding local motion, like optical flow based features, or other modalities, like audio. In addition, we propose a more reasonable open-vocabulary evaluation setting for the ActivityNet data set, where the category splits are based on similarity rather than random assignment.
translated by 谷歌翻译
We present Hybrid Infused Reranking for Passages Retrieval (HYRR), a framework for training rerankers based on a hybrid of BM25 and neural retrieval models. Retrievers based on hybrid models have been shown to outperform both BM25 and neural models alone. Our approach exploits this improved performance when training a reranker, leading to a robust reranking model. The reranker, a cross-attention neural model, is shown to be robust to different first-stage retrieval systems, achieving better performance than rerankers simply trained upon the first-stage retrievers in the multi-stage systems. We present evaluations on a supervised passage retrieval task using MS MARCO and zero-shot retrieval tasks using BEIR. The empirical results show strong performance on both evaluations.
translated by 谷歌翻译
Scholarly text is often laden with jargon, or specialized language that divides disciplines. We extend past work that characterizes science at the level of word types, by using BERT-based word sense induction to find additional words that are widespread but overloaded with different uses across fields. We define scholarly jargon as discipline-specific word types and senses, and estimate its prevalence across hundreds of fields using interpretable, information-theoretic metrics. We demonstrate the utility of our approach for science of science and computational sociolinguistics by highlighting two key social implications. First, we measure audience design, and find that most fields reduce jargon when publishing in general-purpose journals, but some do so more than others. Second, though jargon has varying correlation with articles' citation rates within fields, it nearly always impedes interdisciplinary impact. Broadly, our measurements can inform ways in which language could be revised to serve as a bridge rather than a barrier in science.
translated by 谷歌翻译
We present NusaCrowd, a collaborative initiative to collect and unite existing resources for Indonesian languages, including opening access to previously non-public resources. Through this initiative, we have has brought together 137 datasets and 117 standardized data loaders. The quality of the datasets has been assessed manually and automatically, and their effectiveness has been demonstrated in multiple experiments. NusaCrowd's data collection enables the creation of the first zero-shot benchmarks for natural language understanding and generation in Indonesian and its local languages. Furthermore, NusaCrowd brings the creation of the first multilingual automatic speech recognition benchmark in Indonesian and its local languages. Our work is intended to help advance natural language processing research in under-represented languages.
translated by 谷歌翻译
The recent advent of large language models - large neural networks trained on a simple predictive objective over a massive corpus of natural language - has reinvigorated debate over whether human cognitive capacities might emerge in such generic models given sufficient training data. Of particular interest is the ability of these models to reason about novel problems zero-shot, without any direct training on those problems. In human cognition, this capacity is closely tied to an ability to reason by analogy. Here, we performed a direct comparison between human reasoners and a large language model (GPT-3) on a range of analogical tasks, including a novel text-based matrix reasoning task closely modeled on Raven's Progressive Matrices. We found that GPT-3 displayed a surprisingly strong capacity for abstract pattern induction, matching or even surpassing human capabilities in most settings. Our results indicate that large language models such as GPT-3 have acquired an emergent ability to find zero-shot solutions to a broad range of analogy problems.
translated by 谷歌翻译