我们表明,在随机林的培训过程下面,不仅存在众所周知的和几乎计算的释放速度超出袋点估计,而且还有一个路径来计算概念误差的置信区间要求再培训森林或任何形式的数据分裂。除了施工中涉及的低计算成本外,通过模拟显示这种置信区间,以在训练样本大小方面具有良好的覆盖率和适当的收缩速度。
translated by 谷歌翻译
无穷小夹刀是一种估计参数模型方差的通用方法,最近也用于某些集合方法。在本文中,我们扩展了无穷小折刀,以估计任意两种模型之间的协方差。这可用于量化模型组合的不确定性,或构建测试统计信息,以比较使用相同训练数据集拟合的模型的不同模型或组合。本文中的具体示例使用了随机森林和M估计剂等模型的增强组合。我们还研究了其在XGBOOST模型的神经网络和集合上的应用。我们通过广泛的模拟及其在北京住房数据中的应用来说明差异估计的疗效,并证明了无穷小折刀协方差估算的理论一致性。
translated by 谷歌翻译
Many scientific and engineering challenges-ranging from personalized medicine to customized marketing recommendations-require an understanding of treatment effect heterogeneity. In this paper, we develop a non-parametric causal forest for estimating heterogeneous treatment effects that extends Breiman's widely used random forest algorithm. In the potential outcomes framework with unconfoundedness, we show that causal forests are pointwise consistent for the true treatment effect, and have an asymptotically Gaussian and centered sampling distribution. We also discuss a practical method for constructing asymptotic confidence intervals for the true treatment effect that are centered at the causal forest estimates. Our theoretical results rely on a generic Gaussian theory for a large family of random forest algorithms. To our knowledge, this is the first set of results that allows any type of random forest, including classification and regression forests, to be used for provably valid statistical inference. In experiments, we find causal forests to be substantially more powerful than classical methods based on nearest-neighbor matching, especially in the presence of irrelevant covariates.
translated by 谷歌翻译
在过去几十年中,已经提出了各种方法,用于估计回归设置中的预测间隔,包括贝叶斯方法,集合方法,直接间隔估计方法和保形预测方法。重要问题是这些方法的校准:生成的预测间隔应该具有预定义的覆盖水平,而不会过于保守。在这项工作中,我们从概念和实验的角度审查上述四类方法。结果来自各个域的基准数据集突出显示从一个数据集中的性能的大波动。这些观察可能归因于违反某些类别的某些方法所固有的某些假设。我们说明了如何将共形预测用作提供不具有校准步骤的方法的方法的一般校准程序。
translated by 谷歌翻译
在这项工作中,我们对基本思想和新颖的发展进行了综述的综述,这是基于最小的假设的一种无创新的,无分配的,非参数预测的方法 - 能够以非常简单的方式预测集屈服在有限样本案例中,在统计意义上也有效。论文中提供的深入讨论涵盖了共形预测的理论基础,然后继续列出原始想法的更高级的发展和改编。
translated by 谷歌翻译
Random forests
分类:
Random forests are a combination of tree predictors such that each tree depends on the values of a random vector sampled independently and with the same distribution for all trees in the forest. The generalization error for forests converges a.s. to a limit as the number of trees in the forest becomes large. The generalization error of a forest of tree classifiers depends on the strength of the individual trees in the forest and the correlation between them. Using a random selection of features to split each node yields error rates that compare favorably to Adaboost (Y. Freund & R. Schapire, Machine Learning: Proceedings of the Thirteenth International conference, * * * , 148-156), but are more robust with respect to noise. Internal estimates monitor error, strength, and correlation and these are used to show the response to increasing the number of features used in the splitting. Internal estimates are also used to measure variable importance. These ideas are also applicable to regression.
translated by 谷歌翻译
We develop a general framework for distribution-free predictive inference in regression, using conformal inference. The proposed methodology allows for the construction of a prediction band for the response variable using any estimator of the regression function. The resulting prediction band preserves the consistency properties of the original estimator under standard assumptions, while guaranteeing finite-sample marginal coverage even when these assumptions do not hold. We analyze and compare, both empirically and theoretically, the two major variants of our conformal framework: full conformal inference and split conformal inference, along with a related jackknife method. These methods offer different tradeoffs between statistical accuracy (length of resulting prediction intervals) and computational efficiency. As extensions, we develop a method for constructing valid in-sample prediction intervals called rank-one-out conformal inference, which has essentially the same computational efficiency as split conformal inference. We also describe an extension of our procedures for producing prediction bands with locally varying length, in order to adapt to heteroskedascity in the data. Finally, we propose a model-free notion of variable importance, called leave-one-covariate-out or LOCO inference. Accompanying this paper is an R package conformalInference that implements all of the proposals we have introduced. In the spirit of reproducibility, all of our empirical results can also be easily (re)generated using this package.
translated by 谷歌翻译
我们举例说明数据生成模型的示例,其中Breiman的随机森林可能极慢地收敛到最佳预测器,甚至无法保持一致。为这些属性提供的证据是基于主要是直观的论点,类似于前面使用的那些具有更简单的示例以及数值实验。虽然可以始终选择随机森林表现得非常严重的模型,但我们表明基于“变量使用”和“变量重要性”统计的简单方法通常可用于构建基于“许多武装”的更好的预测因子通过强制初始拆分获得的随机森林,该变量是算法的默认版本倾向于忽略的变量。
translated by 谷歌翻译
我们提出\ textbf {jaws},这是一系列用于无分配的不确定性量化任务的包装方法,以协变量偏移为中心,以我们的核心方法\ textbf {jaw}为中心,\ textbf {ja} ckknife+ \ textbf {w}八 - 重量。下巴还包括使用高阶影响函数的JAW的计算有效\ TextBf {a} pproximations:\ textbf {jawa}。从理论上讲,我们表明JAW放宽了Jackknife+对数据交换性的假设,即使在协变量转移下,也可以实现相同的有限样本覆盖范围保证。 Jawa在轻度假设下进一步以样本量或影响函数顺序的限制接近JAW保证。此外,我们提出了一种通用方法,以重新利用任何无分配不确定性量化方法及其对风险评估的任务的保证:该任务产生了真正标签在用户指定间隔内的估计概率。然后,我们将\ textbf {Jaw-r}和\ textbf {Jawa-r}作为\ textbf {r} ISK评估的建议方法的重新定义版本。实际上,在各种有偏见的现实世界数据集中,下颌的最先进的预测推理基准都超出了间隔生成和风险评估审计任务的偏差。
translated by 谷歌翻译
随机森林仍然是最受欢迎的现成监督学习算法之一。尽管他们记录了良好的经验成功,但直到最近,很少有很少的理论结果来描述他们的表现和行为。在这项工作中,我们通过建立随机森林和其他受监督学习集合的融合率来推动最近的一致性和渐近正常的工作。我们培养了广义U形统计的概念,并显示在此框架内,随机森林预测可能对比以前建立的较大的子样本尺寸可能保持渐近正常。我们还提供Berry-esseen的界限,以量化这种收敛的速度,使得分列大小的角色和确定随机森林预测分布的树木的角色。
translated by 谷歌翻译
Precision Medicine根据患者的特征为患者提供定制的治疗方法,是提高治疗效率的一种有希望的方法。大规模的OMICS数据对于患者表征很有用,但是它们的测量经常会随着时间而变化,从而导致纵向数据。随机森林是用于构建预测模型的最先进的机器学习方法之一,并且可以在精密医学中发挥关键作用。在本文中,我们回顾了标准随机森林方法的扩展,以进行纵向数据分析。扩展方法根据其设计的数据结构进行分类。我们考虑单变量和多变量响应,并根据时间效应是否相关,进一步对重复测量进行分类。还提供了审查扩展程序的可用软件实现信息。最后,我们讨论了我们审查的局限性和一些未来的研究指示。
translated by 谷歌翻译
共形预测是一种简单而强大的工具,可以无需任何分布假设来量化不确定性。但是,现有方法只能提供平均覆盖范围保证,这与更强的条件覆盖范围保证相比并不理想。尽管实现确切的条件覆盖范围是不可能的,但近似条件覆盖范围仍然是一个重要的研究方向。在本文中,我们通过利用条件分布的局部近似来提出修改的不符合得分。修改后的分数继承了分裂保形方法的精神,与完整的保形方法相比,这是简单而有效的,但更好地近似条件覆盖范围保证。各种数据集的经验结果,包括图像上的高维年龄回归,表明我们的方法与现有方法相比提供了更紧密的间隔。
translated by 谷歌翻译
可变重要性措施是分析随机林的黑盒机制的主要工具。虽然平均值降低精度(MDA)被广泛接受作为随机森林最有效的可变重要性措施,但对其统计特性知之甚少。实际上,确切的MDA定义在主随机林软件上变化。在本文中,我们的目标是严格分析主要MDA实施的行为。因此,我们在数学上正式地形化各种实施的MDA算法,然后在样本量增加时建立限制。特别是,我们在三个组件中分解了这些限制:第一个与Sobol指数有关,这是对响应方差的协变度贡献的明确定义措施,广泛应用于敏感性分析领域,而不是TheThird术语,谁的价值随着协变量的依赖而增加。因此,我们理论上证明了MDA在协变者依赖时,MDA不会瞄准正确的数量,这是实验发现的事实。为了解决这个问题,我们为随机林,Sobol-MDA定义了一个新的重要性测量,它修复了原始MDA的缺陷。我们证明了Sobol-MDA的一致性,并表明Sobol-MDA在模拟和实际数据上经验胜过其竞争对手。 R和C ++中的开源实现可在线获取。
translated by 谷歌翻译
交叉验证是一种广泛使用的技术来估计预测误差,但其行为很复杂且不完全理解。理想情况下,人们想认为,交叉验证估计手头模型的预测错误,适合训练数据。我们证明,普通最小二乘拟合的线性模型并非如此。相反,它估计模型的平均预测误差适合于同一人群提取的其他看不见的训练集。我们进一步表明,这种现象发生在大多数流行的预测误差估计中,包括数据拆分,自举和锦葵的CP。接下来,从交叉验证得出的预测误差的标准置信区间可能的覆盖范围远低于所需水平。由于每个数据点都用于训练和测试,因此每个折叠的测量精度之间存在相关性,因此方差的通常估计值太小。我们引入了嵌套的交叉验证方案,以更准确地估计该方差,并从经验上表明,在传统的交叉验证间隔失败的许多示例中,这种修改导致间隔大致正确覆盖。
translated by 谷歌翻译
In many applications, heterogeneous treatment effects on a censored response variable are of primary interest, and it is natural to evaluate the effects at different quantiles (e.g., median). The large number of potential effect modifiers, the unknown structure of the treatment effects, and the presence of right censoring pose significant challenges. In this paper, we develop a hybrid forest approach called Hybrid Censored Quantile Regression Forest (HCQRF) to assess the heterogeneous effects varying with high-dimensional variables. The hybrid estimation approach takes advantage of the random forests and the censored quantile regression. We propose a doubly-weighted estimation procedure that consists of a redistribution-of-mass weight to handle censoring and an adaptive nearest neighbor weight derived from the forest to handle high-dimensional effect functions. We propose a variable importance decomposition to measure the impact of a variable on the treatment effect function. Extensive simulation studies demonstrate the efficacy and stability of HCQRF. The result of the simulation study also convinces us of the effectiveness of the variable importance decomposition. We apply HCQRF to a clinical trial of colorectal cancer. We achieve insightful estimations of the treatment effect and meaningful variable importance results. The result of the variable importance also confirms the necessity of the decomposition.
translated by 谷歌翻译
Random forests are some of the most widely used machine learning models today, especially in domains that necessitate interpretability. We present an algorithm that accelerates the training of random forests and other popular tree-based learning methods. At the core of our algorithm is a novel node-splitting subroutine, dubbed MABSplit, used to efficiently find split points when constructing decision trees. Our algorithm borrows techniques from the multi-armed bandit literature to judiciously determine how to allocate samples and computational power across candidate split points. We provide theoretical guarantees that MABSplit improves the sample complexity of each node split from linear to logarithmic in the number of data points. In some settings, MABSplit leads to 100x faster training (an 99% reduction in training time) without any decrease in generalization performance. We demonstrate similar speedups when MABSplit is used across a variety of forest-based variants, such as Extremely Random Forests and Random Patches. We also show our algorithm can be used in both classification and regression tasks. Finally, we show that MABSplit outperforms existing methods in generalization performance and feature importance calculations under a fixed computational budget. All of our experimental results are reproducible via a one-line script at https://github.com/ThrunGroup/FastForest.
translated by 谷歌翻译
捕获基于协变量的多变量响应载体之间的条件协方差或相关性对于包括神经科学,流行病学和生物医学在内的各个领域很重要。我们提出了一种新方法,称为随机森林(covregrf),以使用随机森林框架估算一个多变量响应的协方差矩阵。随机林木的建造具有专门设计的分裂规则,以最大化儿童节点的样本协方差矩阵估计值之间的差异。我们还提出了对协变量子集的部分效应的显着性检验。我们通过一项模拟研究评估了提出的方法和显着性测试的性能,该研究表明该方法提供了准确的协方差矩阵估计值,并且Type-1误差得到了很好的控制。我们还证明了该方法与甲状腺疾病数据集的应用。
translated by 谷歌翻译
We present a new distribution-free conformal prediction algorithm for sequential data (e.g., time series), called the \textit{sequential predictive conformal inference} (\texttt{SPCI}). We specifically account for the nature that the time series data are non-exchangeable, and thus many existing conformal prediction algorithms based on temporal residuals are not applicable. The main idea is to exploit the temporal dependence of conformity scores; thus, the past conformity scores contain information about future ones. Then we cast the problem of conformal prediction interval as predicting the quantile of a future residual, given a prediction algorithm. Theoretically, we establish asymptotic valid conditional coverage upon extending consistency analyses in quantile regression. Using simulation and real-data experiments, we demonstrate a significant reduction in interval width of \texttt{SPCI} compared to other existing methods under the desired empirical coverage.
translated by 谷歌翻译
本文开发了贝叶斯因果林的稀疏诱导版本,最近提出的非参数因果回归模型采用贝叶斯添加剂回归树,专门设计用于使用观察数据来估计异质治疗效果。我们介绍的稀疏诱导组件是通过实证研究的动机,其中不是所有可用的协变量相关的,导致在估计个体治疗效果的兴趣表面底层的不同程度。在这项工作中提供的扩展版本,我们命名贝叶斯因果森林,配备了一对允许模型通过树集合中的相应数量的分裂调节每个协变量的重量。这些前瞻改善了模型对稀疏数据产生过程的适应性,并且允许在治疗效果估计的框架中进行完全贝叶斯特征缩收,从而揭示推动异质性的调节因子。此外,该方法允许先前了解相关的混杂协变量和对模型中掺入结果的影响的相对幅度。我们说明了我们在模拟研究中的方法的表现,与贝叶斯因果林和其他最先进的模型相比,展示如何与越来越多的协变量以及其如何处理强烈混淆的情景。最后,我们还提供了使用真实数据的应用程序的示例。
translated by 谷歌翻译
在过去的十年中,出现了一些用于超参数搜索的新型框架,但大多数依赖于严格的,通常是正常的分布假设,从而限制了搜索模型的灵活性。本文提出了一个基于共形预测的新颖优化框架,假设仅交换性,并允许更大的搜索模型体系结构和方差估计器选择。在密集和卷积神经网络上探索和基准测试了几个这样的模型,并在最终的损失和成就时间中都表现出色。
translated by 谷歌翻译