为了在结构因果模型(SCM)中执行反事实推理,需要了解因果机制,它提供条件分布的因子,并将噪声映射到样本的确定性函数。遗憾的是,因象无法通过观察和与世界互动收集的数据唯一确定的因果机制,因此仍然存在如何选择因果机制的问题。最近的工作中,Oberst&Sontag(2019)提出了Gumbel-Max SCM,它由于直观上吸引的反事实稳定性而导致Gumbel-Max Reparameterizations作为因果机制。在这项工作中,我们认为选择在估算反事实治疗效果时最小化的定量标准的因果机制,例如最小化方差。我们提出了一个参数化的因果机制,概括了Gumbel-Max。我们表明他们可以接受培训,以最大限度地减少对感兴趣查询的分布的反事实效果方差和其他损失,从而产生比固定替代方案的反复治疗效果的较低方差估计,也推广到在培训时间未见的查询。
translated by 谷歌翻译
强化学习中的信用作业是衡量行动对未来奖励的影响的问题。特别是,这需要从运气中分离技能,即解除外部因素和随后的行动对奖励行动的影响。为实现这一目标,我们将来自因果关系的反事件的概念调整为无模型RL设置。关键思想是通过学习从轨迹中提取相关信息来应对未来事件的价值函数。我们制定了一系列政策梯度算法,这些算法使用这些未来条件的价值函数作为基准或批评,并表明它们是可怕的差异。为避免对未来信息的调理潜在偏见,我们将后视信息限制为不包含有关代理程序行为的信息。我们展示了我们对许多说明性和具有挑战性问题的算法的功效和有效性。
translated by 谷歌翻译
反事实推断是一种强大的工具,能够解决备受瞩目的领域中具有挑战性的问题。要进行反事实推断,需要了解潜在的因果机制。但是,仅凭观察和干预措施就不能独特地确定因果机制。这就提出了一个问题,即如何选择因果机制,以便在给定领域中值得信赖。在具有二进制变量的因果模型中已经解决了这个问题,但是分类变量的情况仍未得到解答。我们通过为具有分类变量的因果模型引入反事实排序的概念来应对这一挑战。为了学习满足这些约束的因果机制,并对它们进行反事实推断,我们引入了深层双胞胎网络。这些是深层神经网络,在受过训练的情况下,可以进行双网络反事实推断 - 一种替代绑架,动作和预测方法的替代方法。我们从经验上测试了来自医学,流行病学和金融的多种现实世界和半合成数据的方法,并报告了反事实概率的准确估算,同时证明了反事实订购时不执行反事实的问题。
translated by 谷歌翻译
Adequately assigning credit to actions for future outcomes based on their contributions is a long-standing open challenge in Reinforcement Learning. The assumptions of the most commonly used credit assignment method are disadvantageous in tasks where the effects of decisions are not immediately evident. Furthermore, this method can only evaluate actions that have been selected by the agent, making it highly inefficient. Still, no alternative methods have been widely adopted in the field. Hindsight Credit Assignment is a promising, but still unexplored candidate, which aims to solve the problems of both long-term and counterfactual credit assignment. In this thesis, we empirically investigate Hindsight Credit Assignment to identify its main benefits, and key points to improve. Then, we apply it to factored state representations, and in particular to state representations based on the causal structure of the environment. In this setting, we propose a variant of Hindsight Credit Assignment that effectively exploits a given causal structure. We show that our modification greatly decreases the workload of Hindsight Credit Assignment, making it more efficient and enabling it to outperform the baseline credit assignment method on various tasks. This opens the way to other methods based on given or learned causal structures.
translated by 谷歌翻译
决策者需要在采用新的治疗政策之前预测结果的发展,该政策定义了何时以及如何连续地影响结果的治疗序列。通常,预测介入的未来结果轨迹的算法将未来治疗的固定顺序作为输入。这要么忽略了未来治疗对结果之前的结果的依赖性,要么隐含地假设已知治疗政策,因此排除了该政策未知或需要反事实分析的情况。为了应对这些局限性,我们开发了一种用于治疗和结果的联合模型,该模型允许估计处理策略和顺序治疗(OUT COMECTION数据)的影响。它可以回答有关治疗政策干预措施的介入和反事实查询,因为我们使用有关血糖进展的现实数据显示,并在此基础上进行了模拟研究。
translated by 谷歌翻译
We introduce a new probabilistic temporal logic for the verification of Markov Decision Processes (MDP). Our logic is the first to include operators for causal reasoning, allowing us to express interventional and counterfactual queries. Given a path formula $\phi$, an interventional property is concerned with the satisfaction probability of $\phi$ if we apply a particular change $I$ to the MDP (e.g., switching to a different policy); a counterfactual allows us to compute, given an observed MDP path $\tau$, what the outcome of $\phi$ would have been had we applied $I$ in the past. For its ability to reason about different configurations of the MDP, our approach represents a departure from existing probabilistic temporal logics that can only reason about a fixed system configuration. From a syntactic viewpoint, we introduce a generalized counterfactual operator that subsumes both interventional and counterfactual probabilities as well as the traditional probabilistic operator found in e.g., PCTL. From a semantics viewpoint, our logic is interpreted over a structural causal model (SCM) translation of the MDP, which gives us a representation amenable to counterfactual reasoning. We provide a proof-of-concept evaluation of our logic on a reach-avoid task in a grid-world model.
translated by 谷歌翻译
已经引入了生成流量网络(GFlowNETS)作为在主动学习背景下采样多样化候选的方法,具有培训目标,其使它们与给定奖励功能成比例地进行比例。在本文中,我们显示了许多额外的GFLOWN的理论特性。它们可用于估计联合概率分布和一些变量未指定的相应边际分布,并且特别感兴趣地,可以代表像集合和图形的复合对象的分布。 Gflownets摊销了通常通过计算昂贵的MCMC方法在单个但训练有素的生成通行证中进行的工作。它们还可用于估计分区功能和自由能量,给定子集(子图)的超标(超图)的条件概率,以及给定集合(图)的所有超标仪(超图)的边际分布。我们引入了熵和相互信息估计的变体,从帕累托前沿采样,与奖励最大化策略的连接,以及随机环境的扩展,连续动作和模块化能量功能。
translated by 谷歌翻译
基于AI和机器学习的决策系统已在各种现实世界中都使用,包括医疗保健,执法,教育和金融。不再是牵强的,即设想一个未来,自治系统将推动整个业务决策,并且更广泛地支持大规模决策基础设施以解决社会最具挑战性的问题。当人类做出决定时,不公平和歧视的问题普遍存在,并且当使用几乎没有透明度,问责制和公平性的机器做出决定时(或可能会放大)。在本文中,我们介绍了\ textit {Causal公平分析}的框架,目的是填补此差距,即理解,建模,并可能解决决策设置中的公平性问题。我们方法的主要见解是将观察到数据中存在的差异的量化与基本且通常是未观察到的因果机制收集的因果机制的收集,这些机制首先会产生差异,挑战我们称之为因果公平的基本问题分析(FPCFA)。为了解决FPCFA,我们研究了分解差异和公平性的经验度量的问题,将这种变化归因于结构机制和人群的不同单位。我们的努力最终达到了公平地图,这是组织和解释文献中不同标准之间关系的首次系统尝试。最后,我们研究了进行因果公平分析并提出一本公平食谱的最低因果假设,该假设使数据科学家能够评估不同影响和不同治疗的存在。
translated by 谷歌翻译
能够从专家推断出第二意见的自动决策支持系统可能有助于更有效地分配资源;他们可以帮助决定何时何地寻求第二意见。在本文中,我们从反事实推断的角度研究了这种类型的支持系统的设计。我们专注于多类分类设置,并首先表明,如果专家自行做出预测,那么产生其预测的基本因果机制就需要满足理想的设定不变属性。此外,我们表明,对于满足该特性的任何因果机制,存在一种等效机制,其中每个专家的预测是由由共同噪声控制的独立亚机制产生的。这激发了设定不变的gumbel-max结构因果模型的设计,其中管理模型的亚机制的噪声结构取决于专家之间相似性的直觉概念,可以从数据估算。合成数据和真实数据的实验表明,我们的模型可用于比其非伴侣对应物更准确地推断第二个意见。
translated by 谷歌翻译
Counterfactuals are often described as 'retrospective,' focusing on hypothetical alternatives to a realized past. This description relates to an often implicit assumption about the structure and stability of exogenous variables in the system being modeled -- an assumption that is reasonable in many settings where counterfactuals are used. In this work, we consider cases where we might reasonably make a different assumption about exogenous variables, namely, that the exogenous noise terms of each unit do exhibit some unit-specific structure and/or stability. This leads us to a different use of counterfactuals -- a 'forward-looking' rather than 'retrospective' counterfactual. We introduce "counterfactual treatment choice," a type of treatment choice problem that motivates using forward-looking counterfactuals. We then explore how mismatches between interventional versus forward-looking counterfactual approaches to treatment choice, consistent with different assumptions about exogenous noise, can lead to counterintuitive results.
translated by 谷歌翻译
This review presents empirical researchers with recent advances in causal inference, and stresses the paradigmatic shifts that must be undertaken in moving from traditional statistical analysis to causal analysis of multivariate data. Special emphasis is placed on the assumptions that underly all causal inferences, the languages used in formulating those assumptions, the conditional nature of all causal and counterfactual claims, and the methods that have been developed for the assessment of such claims. These advances are illustrated using a general theory of causation based on the Structural Causal Model (SCM) described in Pearl (2000a), which subsumes and unifies other approaches to causation, and provides a coherent mathematical foundation for the analysis of causes and counterfactuals. In particular, the paper surveys the development of mathematical tools for inferring (from a combination of data and assumptions) answers to three types of causal queries: (1) queries about the effects of potential interventions, (also called "causal effects" or "policy evaluation") (2) queries about probabilities of counterfactuals, (including assessment of "regret," "attribution" or "causes of effects") and (3) queries about direct and indirect effects (also known as "mediation"). Finally, the paper defines the formal and conceptual relationships between the structural and potential-outcome frameworks and presents tools for a symbiotic analysis that uses the strong features of both.
translated by 谷歌翻译
因果推断对于跨业务参与,医疗和政策制定等领域的数据驱动决策至关重要。然而,关于因果发现的研究已经与推理方法分开发展,从而阻止了两个领域方法的直接组合。在这项工作中,我们开发了深层端到端因果推理(DECI),这是一种基于流动的非线性添加噪声模型,该模型具有观察数据,并且可以执行因果发现和推理,包括有条件的平均治疗效果(CATE) )估计。我们提供了理论上的保证,即DECI可以根据标准因果发现假设恢复地面真实因果图。受应用影响的激励,我们将该模型扩展到具有缺失值的异质,混合型数据,从而允许连续和离散的治疗决策。我们的结果表明,与因果发现的相关基线相比,DECI的竞争性能和(c)在合成数据集和因果机器学习基准测试基准的一千多个实验中,跨数据类型和缺失水平进行了估计。
translated by 谷歌翻译
This work shows how to leverage causal inference to understand the behavior of complex learning systems interacting with their environment and predict the consequences of changes to the system. Such predictions allow both humans and algorithms to select the changes that would have improved the system performance. This work is illustrated by experiments on the ad placement system associated with the Bing search engine.
translated by 谷歌翻译
非政策评估(OPE)方法是评估高风险领域(例如医疗保健)中的政策的关键工具,在这些领域,直接部署通常是不可行的,不道德的或昂贵的。当期望部署环境发生变化(即数据集偏移)时,对于OPE方法,在此类更改中对策略进行强大的评估非常重要。现有的方法考虑对可以任意改变环境的任何可观察到的任何可观察到的属性的大量转变。这通常会导致对公用事业的高度悲观估计,从而使可能对部署有用的政策无效。在这项工作中,我们通过研究领域知识如何帮助提供对政策公用事业的更现实的估计来解决上述问题。我们利用人类的投入,在环境的哪些方面可能会发生变化,并适应OPE方法仅考虑这些方面的转变。具体而言,我们提出了一个新颖的框架,可靠的OPE(绳索),该框架认为基于用户输入的数据中的协变量子集,并估算了这些变化下最坏情况的效用。然后,我们为OPE开发了对OPE的计算有效算法,这些算法对上述强盗和马尔可夫决策过程的上述变化很强。我们还理论上分析了这些算法的样品复杂性。从医疗领域进行的合成和现实世界数据集进行了广泛的实验表明,我们的方法不仅可以捕获现实的数据集准确地转移,而且还会导致较少的悲观政策评估。
translated by 谷歌翻译
跨学科的一个重要问题是发现产生预期结果的干预措施。当可能的干预空间很大时,需要进行详尽的搜索,需要实验设计策略。在这种情况下,编码变量之间的因果关系以及因此对系统的影响,对于有效地确定理想的干预措施至关重要。我们开发了一种迭代因果方法来识别最佳干预措施,这是通过分布后平均值和所需目标平均值之间的差异来衡量的。我们制定了一种主动学习策略,该策略使用从不同干预措施中获得的样本来更新有关基本因果模型的信念,并确定对最佳干预措施最有用的样本,因此应在下一批中获得。该方法采用了因果模型的贝叶斯更新,并使用精心设计的,有因果关系的收购功能优先考虑干预措施。此采集函数以封闭形式进行评估,从而有效优化。理论上以信息理论界限和可证明的一致性结果在理论上基于理论上的算法。我们说明了综合数据和现实世界生物学数据的方法,即来自worturb-cite-seq实验的基因表达数据,以识别诱导特定细胞态过渡的最佳扰动;与几个基线相比,观察到所提出的因果方法可实现更好的样品效率。在这两种情况下,我们都认为因果知情的采集函数尤其优于现有标准,从而允许使用实验明显更少的最佳干预设计。
translated by 谷歌翻译
我们在无限地平线马尔可夫决策过程中考虑批量(离线)策略学习问题。通过移动健康应用程序的推动,我们专注于学习最大化长期平均奖励的政策。我们为平均奖励提出了一款双重强大估算器,并表明它实现了半导体效率。此外,我们开发了一种优化算法来计算参数化随机策略类中的最佳策略。估计政策的履行是通过政策阶级的最佳平均奖励与估计政策的平均奖励之间的差异来衡量,我们建立了有限样本的遗憾保证。通过模拟研究和促进体育活动的移动健康研究的分析来说明该方法的性能。
translated by 谷歌翻译
当他们更喜欢$ \ texit {exploit} $时,您如何激励自我兴趣的代理到$ \ texit {探索} $?我们考虑复杂的探索问题,其中每个代理面临相同(但未知)MDP。与传统的加固学习配方相比,代理商控制了政策的选择,而算法只能发出建议。然而,该算法控制信息流,并且可以通过信息不对称激励代理探索。我们设计一种算法,探讨MDP中的所有可达状态。我们达到了类似于先前研究的静态,无国籍探索问题中激励探索的保证担保。据我们所知,这是第一个考虑在有状态,强化学习环境中设计的工作。
translated by 谷歌翻译
因果表示学习是识别基本因果变量及其从高维观察(例如图像)中的关系的任务。最近的工作表明,可以从观测的时间序列中重建因果变量,假设它们之间没有瞬时因果关系。但是,在实际应用中,我们的测量或帧速率可能比许多因果效应要慢。这有效地产生了“瞬时”效果,并使以前的可识别性结果无效。为了解决这个问题,我们提出了ICITRI,这是一种因果表示学习方法,当具有已知干预目标的完美干预措施时,可以在时间序列中处理瞬时效应。 Icitris从时间观察中识别因果因素,同时使用可区分的因果发现方法来学习其因果图。在三个视频数据集的实验中,Icitris准确地识别了因果因素及其因果图。
translated by 谷歌翻译
This paper studies offline policy learning, which aims at utilizing observations collected a priori (from either fixed or adaptively evolving behavior policies) to learn an optimal individualized decision rule that achieves the best overall outcomes for a given population. Existing policy learning methods rely on a uniform overlap assumption, i.e., the propensities of exploring all actions for all individual characteristics are lower bounded in the offline dataset; put differently, the performance of the existing methods depends on the worst-case propensity in the offline dataset. As one has no control over the data collection process, this assumption can be unrealistic in many situations, especially when the behavior policies are allowed to evolve over time with diminishing propensities for certain actions. In this paper, we propose a new algorithm that optimizes lower confidence bounds (LCBs) -- instead of point estimates -- of the policy values. The LCBs are constructed using knowledge of the behavior policies for collecting the offline data. Without assuming any uniform overlap condition, we establish a data-dependent upper bound for the suboptimality of our algorithm, which only depends on (i) the overlap for the optimal policy, and (ii) the complexity of the policy class we optimize over. As an implication, for adaptively collected data, we ensure efficient policy learning as long as the propensities for optimal actions are lower bounded over time, while those for suboptimal ones are allowed to diminish arbitrarily fast. In our theoretical analysis, we develop a new self-normalized type concentration inequality for inverse-propensity-weighting estimators, generalizing the well-known empirical Bernstein's inequality to unbounded and non-i.i.d. data.
translated by 谷歌翻译
我们研究马尔可夫决策过程(MDP)框架中的离线数据驱动的顺序决策问题。为了提高学习政策的概括性和适应性,我们建议通过一套关于在政策诱导的固定分配所在的分发的一套平均奖励来评估每项政策。给定由某些行为策略生成的多个轨迹的预收集数据集,我们的目标是在预先指定的策略类中学习一个强大的策略,可以最大化此集的最小值。利用半参数统计的理论,我们开发了一种统计上有效的策略学习方法,用于估算DE NED强大的最佳政策。在数据集中的总决策点方面建立了达到对数因子的速率最佳遗憾。
translated by 谷歌翻译