This paper presents a novel probabilistic forecasting method called ensemble conformalized quantile regression (EnCQR). EnCQR constructs distribution-free and approximately marginally valid prediction intervals (PIs), which are suitable for nonstationary and heteroscedastic time series data. EnCQR can be applied on top of a generic forecasting model, including deep learning architectures. EnCQR exploits a bootstrap ensemble estimator, which enables the use of conformal predictors for time series by removing the requirement of data exchangeability. The ensemble learners are implemented as generic machine learning algorithms performing quantile regression, which allow the length of the PIs to adapt to local variability in the data. In the experiments, we predict time series characterized by a different amount of heteroscedasticity. The results demonstrate that EnCQR outperforms models based only on quantile regression or conformal prediction, and it provides sharper, more informative, and valid PIs.
translated by 谷歌翻译
在过去几十年中,已经提出了各种方法,用于估计回归设置中的预测间隔,包括贝叶斯方法,集合方法,直接间隔估计方法和保形预测方法。重要问题是这些方法的校准:生成的预测间隔应该具有预定义的覆盖水平,而不会过于保守。在这项工作中,我们从概念和实验的角度审查上述四类方法。结果来自各个域的基准数据集突出显示从一个数据集中的性能的大波动。这些观察可能归因于违反某些类别的某些方法所固有的某些假设。我们说明了如何将共形预测用作提供不具有校准步骤的方法的方法的一般校准程序。
translated by 谷歌翻译
The main objective of Prognostics and Health Management is to estimate the Remaining Useful Lifetime (RUL), namely, the time that a system or a piece of equipment is still in working order before starting to function incorrectly. In recent years, numerous machine learning algorithms have been proposed for RUL estimation, mainly focusing on providing more accurate RUL predictions. However, there are many sources of uncertainty in the problem, such as inherent randomness of systems failure, lack of knowledge regarding their future states, and inaccuracy of the underlying predictive models, making it infeasible to predict the RULs precisely. Hence, it is of utmost importance to quantify the uncertainty alongside the RUL predictions. In this work, we investigate the conformal prediction (CP) framework that represents uncertainty by predicting sets of possible values for the target variable (intervals in the case of RUL) instead of making point predictions. Under very mild technical assumptions, CP formally guarantees that the actual value (true RUL) is covered by the predicted set with a degree of certainty that can be prespecified. We study three CP algorithms to conformalize any single-point RUL predictor and turn it into a valid interval predictor. Finally, we conformalize two single-point RUL predictors, deep convolutional neural networks and gradient boosting, and illustrate their performance on the Commercial Modular Aero-Propulsion System Simulation (C-MAPSS) data sets.
translated by 谷歌翻译
We present a new distribution-free conformal prediction algorithm for sequential data (e.g., time series), called the \textit{sequential predictive conformal inference} (\texttt{SPCI}). We specifically account for the nature that the time series data are non-exchangeable, and thus many existing conformal prediction algorithms based on temporal residuals are not applicable. The main idea is to exploit the temporal dependence of conformity scores; thus, the past conformity scores contain information about future ones. Then we cast the problem of conformal prediction interval as predicting the quantile of a future residual, given a prediction algorithm. Theoretically, we establish asymptotic valid conditional coverage upon extending consistency analyses in quantile regression. Using simulation and real-data experiments, we demonstrate a significant reduction in interval width of \texttt{SPCI} compared to other existing methods under the desired empirical coverage.
translated by 谷歌翻译
机器学习(ML)的指数增长引起了极大的兴趣,以量化用户定义的信心水平的每个预测的不确定性。可靠的不确定性定量至关重要,是迈向增加对AI结果的信任的一步。在高风险决策中,它变得尤为重要,在这种决策中,真正的输出必须在置信度范围内具有很高的可能性。共形预测(CP)是一个无分布的不确定性定量框架,可适用于任何黑框模型,并产生预测间隔(PI),这些预测间隔(PIS)在轻度的交换性假设下有效。 CP型方法由于易于实施和计算便宜而变得越来越流行;但是,交换性假设立即排除时间序列预测。尽管最近的论文解决了协变量的转变,但对于一般时间序列预测生产H-Step提前有效PI的问题还不足。为了实现这样的目标,我们提出了一种称为AENBMIMOCQR的新方法(自适应集合批量多输入多输出保形的分数回归),该方法会产生渐近有效的PIS,适合异质驱动时间序列。我们将提出的方法与NN5预测竞争数据集中的最新竞争方法进行比较。所有用于复制实验的代码和数据都可以使用
translated by 谷歌翻译
机器学习方法越来越广泛地用于医疗保健,运输和金融等高危环境中。在这些环境中,重要的是,模型要产生校准的不确定性以反映其自信并避免失败。在本文中,我们调查了有关深度学习的不确定性定量(UQ)的最新著作,特别是针对其数学属性和广泛适用性的无分配保形方法。我们将涵盖共形方法的理论保证,引入在时空数据的背景下提高UQ的校准和效率的技术,并讨论UQ在安全决策中的作用。
translated by 谷歌翻译
有效的决策需要了解预测中固有的不确定性。在回归中,这种不确定性可以通过各种方法估算;然而,许多这些方法对调谐进行费力,产生过度自确性的不确定性间隔,或缺乏敏锐度(给予不精确的间隔)。我们通过提出一种通过定义具有两个不同损失功能的神经网络来捕获回归中的预测分布的新方法来解决这些挑战。具体地,一个网络近似于累积分布函数,第二网络近似于其逆。我们将此方法称为合作网络(CN)。理论分析表明,优化的固定点处于理想化的解决方案,并且该方法是渐近的与地面真理分布一致。凭经验,学习是简单且强大的。我们基准CN对两个合成和六个现实世界数据集的几种常见方法,包括预测来自电子健康记录的糖尿病患者的A1C值,其中不确定是至关重要的。在合成数据中,所提出的方法与基本上匹配地面真理。在真实世界数据集中,CN提高了许多性能度量的结果,包括对数似然估计,平均误差,覆盖估计和预测间隔宽度。
translated by 谷歌翻译
估计与机器学习预测(ML)模型相关的不确定性对于评估其稳健性和预测能力至关重要。在此提交中,我们介绍了Mapie(模型不可知的预测间隔估计器),这是一个开源Python库,可量化单输出回归和多类分类任务的ML模型的不确定性。Mapie实施了保形预测方法,使用户可以轻松地计算出在边际覆盖范围上具有强大理论保证的不确定性,并在模型或基础数据分布上进行了轻微的假设。Mapie托管在Scikit-Learn-Contrib上,完全“ Scikit-Learn兼容”。因此,它接受带有Scikit-Learn API的任何类型的回归器或分类器。该库可在以下网址获得:https://github.com/scikit-learn-contrib/mapie/。
translated by 谷歌翻译
我们开发了一个框架,用于在线环境中使用有效的覆盖范围保证构建不确定性集,其中基础数据分布可以急剧(甚至对手)随着时间的推移而发生巨大变化。我们提出的技术非常灵活,因为它可以与任何在线学习算法集成,需要最低限度的实施工作和计算成本。我们方法比现有替代方案的关键优势(也基于共形推断)是我们不需要将数据分为培训和保持校准集。这使我们能够以完全在线的方式拟合预测模型,并利用最新的观察结果来构建校准的不确定性集。因此,与现有技术相反,(i)我们构建的集合可以迅速适应分布的新变化; (ii)我们的过程不需要在每个时间步骤进行改装。使用合成和现实世界的基准数据集,我们证明了理论的有效性以及提案对现有技术的提高绩效。为了证明所提出的方法的更大灵活性,我们展示了如何为多出输出回归问题构造有效的间隔,而以前的顺序校准方法由于不切实际的计算和内存需求而无法处理。
translated by 谷歌翻译
我们提出了一种利用分布人工神经网络的概率电价预测(EPF)的新方法。EPF的新型网络结构基于包含概率层的正则分布多层感知器(DMLP)。使用TensorFlow概率框架,神经网络的输出被定义为一个分布,是正常或可能偏斜且重尾的Johnson的SU(JSU)。在预测研究中,将该方法与最新基准进行了比较。该研究包括预测,涉及德国市场的日常电价。结果显示了对电价建模时较高时刻的重要性的证据。
translated by 谷歌翻译
Accurate uncertainty measurement is a key step to building robust and reliable machine learning systems. Conformal prediction is a distribution-free uncertainty quantification algorithm popular for its ease of implementation, statistical coverage guarantees, and versatility for underlying forecasters. However, existing conformal prediction algorithms for time series are limited to single-step prediction without considering the temporal dependency. In this paper we propose a Copula Conformal Prediction algorithm for multivariate, multi-step Time Series forecasting, CopulaCPTS. On several synthetic and real-world multivariate time series datasets, we show that CopulaCPTS produces more calibrated and sharp confidence intervals for multi-step prediction tasks than existing techniques.
translated by 谷歌翻译
分位数回归是统计学习中的一个基本问题,这是由于需要量化预测中的不确定性或对多样化的人群建模而不过分减少的统计学习。例如,流行病学预测,成本估算和收入预测都可以准确地量化可能的值的范围。因此,在计量经济学,统计和机器学习的多年研究中,已经为这个问题开发了许多模型。而不是提出另一种(新的)算法用于分位数回归,而是采用元观点:我们研究用于汇总任意数量的有条件分位模型的方法,以提高准确性和鲁棒性。我们考虑加权合奏,其中权重不仅可能因单个模型,而且要多于分位数和特征值而变化。我们在本文中考虑的所有模型都可以使用现代深度学习工具包适合,因此可以广泛访问(从实现的角度)和可扩展。为了提高预测分位数的准确性(或等效地,预测间隔),我们开发了确保分位数保持单调排序的工具,并采用保形校准方法。可以使用这些,而无需对原始模型的原始库进行任何修改。我们还回顾了一些围绕分数聚集和相关评分规则的基本理论,并为该文献做出了一些新的结果(例如,在分类或等渗后回归只能提高加权间隔得分的事实)。最后,我们提供了来自两个不同基准存储库的34个数据集的广泛的经验比较套件。
translated by 谷歌翻译
In this paper, we propose a new short-term load forecasting (STLF) model based on contextually enhanced hybrid and hierarchical architecture combining exponential smoothing (ES) and a recurrent neural network (RNN). The model is composed of two simultaneously trained tracks: the context track and the main track. The context track introduces additional information to the main track. It is extracted from representative series and dynamically modulated to adjust to the individual series forecasted by the main track. The RNN architecture consists of multiple recurrent layers stacked with hierarchical dilations and equipped with recently proposed attentive dilated recurrent cells. These cells enable the model to capture short-term, long-term and seasonal dependencies across time series as well as to weight dynamically the input information. The model produces both point forecasts and predictive intervals. The experimental part of the work performed on 35 forecasting problems shows that the proposed model outperforms in terms of accuracy its predecessor as well as standard statistical models and state-of-the-art machine learning models.
translated by 谷歌翻译
近期不同尺度电力消耗的丰富数据开辟了新的挑战,并强调了新技术的需求,以利用更精细的尺度提供的信息,以便改善更广泛的尺度预测。在这项工作中,我们利用该分层预测问题与多尺度传输学习之间的相似性。我们分别开发了两种分层转移学习方法,分别基于广义添加剂模型和随机林的堆叠,以及专家聚合的使用。我们将这些方法应用于在第一种情况下使用智能仪表数据,以及第二种情况下的区域数据的智能仪表数据将这些方法应用于两种电力负荷预测。对于这两个useCases,我们将我们的方法的表现与基准算法的表演进行比较,我们使用可变重要性分析调查其行为。我们的结果表明了两种方法的兴趣,这导致预测的重大改善。
translated by 谷歌翻译
在回归设置中量化不确定性的许多方法中,指定完整量子函数具有吸引力,随着量级可用于解释和评估。预测每个输入的真实条件定量的模型,在所有量化水平上都具有潜在的不确定性的正确和有效的表示。为实现这一目标,许多基于当前的分位式的方法侧重于优化所谓的弹球损失。然而,这种损失限制了适用的回归模型的范围,限制了靶向许多所需特性的能力(例如校准,清晰度,中心间隔),并且可能产生差的条件量数。在这项工作中,我们开发了满足这些缺点的新分位式方法。特别是,我们提出了可以适用于任何类别的回归模型的方法,允许在校准和清晰度之间选择权衡,优化校准中心间隔,并产生更准确的条件定位。我们对我们的方法提供了彻底的实验评估,其中包括核融合中的高维不确定性量化任务。
translated by 谷歌翻译
杂交和集合学习技术是改善预测方法的预测能力的流行模型融合技术。通过有限的研究,将这两种有前途的方法结合在一起,本文着重于不同合奏的基础模型池中指数平滑的旋转神经网络(ES-RNN)的实用性。我们将某些最先进的结合技术和算术模型平均作为基准进行比较。我们对M4预测数据集进行了100,000个时间序列,结果表明,基于特征的预测模型平均(FFORFORA)平均是与ES-RNN的晚期数据融合的最佳技术。但是,考虑到M4的每日数据子集,堆叠是处理所有基本模型性能相似的情况下唯一成功的合奏。我们的实验结果表明,与N-Beats作为基准相比,我们达到了艺术的预测结果。我们得出的结论是,模型平均比模型选择和堆叠策略更强大。此外,结果表明,提高梯度对于实施合奏学习策略是优越的。
translated by 谷歌翻译
We consider the problem of predictive monitoring (PM), i.e., predicting at runtime the satisfaction of a desired property from the current system's state. Due to its relevance for runtime safety assurance and online control, PM methods need to be efficient to enable timely interventions against predicted violations, while providing correctness guarantees. We introduce \textit{quantitative predictive monitoring (QPM)}, the first PM method to support stochastic processes and rich specifications given in Signal Temporal Logic (STL). Unlike most of the existing PM techniques that predict whether or not some property $\phi$ is satisfied, QPM provides a quantitative measure of satisfaction by predicting the quantitative (aka robust) STL semantics of $\phi$. QPM derives prediction intervals that are highly efficient to compute and with probabilistic guarantees, in that the intervals cover with arbitrary probability the STL robustness values relative to the stochastic evolution of the system. To do so, we take a machine-learning approach and leverage recent advances in conformal inference for quantile regression, thereby avoiding expensive Monte-Carlo simulations at runtime to estimate the intervals. We also show how our monitors can be combined in a compositional manner to handle composite formulas, without retraining the predictors nor sacrificing the guarantees. We demonstrate the effectiveness and scalability of QPM over a benchmark of four discrete-time stochastic processes with varying degrees of complexity.
translated by 谷歌翻译
Accurate uncertainty quantification is necessary to enhance the reliability of deep learning models in real-world applications. In the case of regression tasks, prediction intervals (PIs) should be provided along with the deterministic predictions of deep learning models. Such PIs are useful or "high-quality'' as long as they are sufficiently narrow and capture most of the probability density. In this paper, we present a method to learn prediction intervals for regression-based neural networks automatically in addition to the conventional target predictions. In particular, we train two companion neural networks: one that uses one output, the target estimate, and another that uses two outputs, the upper and lower bounds of the corresponding PI. Our main contribution is the design of a loss function for the PI-generation network that takes into account the output of the target-estimation network and has two optimization objectives: minimizing the mean prediction interval width and ensuring the PI integrity using constraints that maximize the prediction interval probability coverage implicitly. Both objectives are balanced within the loss function using a self-adaptive coefficient. Furthermore, we apply a Monte Carlo-based approach that evaluates the model uncertainty in the learned PIs. Experiments using a synthetic dataset, six benchmark datasets, and a real-world crop yield prediction dataset showed that our method was able to maintain a nominal probability coverage and produce narrower PIs without detriment to its target estimation accuracy when compared to those PIs generated by three state-of-the-art neural-network-based methods.
translated by 谷歌翻译
When used in complex engineered systems, such as communication networks, artificial intelligence (AI) models should be not only as accurate as possible, but also well calibrated. A well-calibrated AI model is one that can reliably quantify the uncertainty of its decisions, assigning high confidence levels to decisions that are likely to be correct and low confidence levels to decisions that are likely to be erroneous. This paper investigates the application of conformal prediction as a general framework to obtain AI models that produce decisions with formal calibration guarantees. Conformal prediction transforms probabilistic predictors into set predictors that are guaranteed to contain the correct answer with a probability chosen by the designer. Such formal calibration guarantees hold irrespective of the true, unknown, distribution underlying the generation of the variables of interest, and can be defined in terms of ensemble or time-averaged probabilities. In this paper, conformal prediction is applied for the first time to the design of AI for communication systems in conjunction to both frequentist and Bayesian learning, focusing on demodulation, modulation classification, and channel prediction.
translated by 谷歌翻译
现在通常用于高风险设置,如医疗诊断,如医疗诊断,那么需要不确定量化,以避免后续模型失败。无分发的不确定性量化(无分布UQ)是用户友好的范式,用于为这种预测创建统计上严格的置信区间/集合。批判性地,间隔/集合有效而不进行分布假设或模型假设,即使具有最多许多DataPoints也具有显式保证。此外,它们适应输入的难度;当输入示例很困难时,不确定性间隔/集很大,信号传达模型可能是错误的。在没有多大的工作和没有再培训的情况下,可以在任何潜在的算法(例如神经网络)上使用无分​​发方法,以产生置信度集,以便包含用户指定概率,例如90%。实际上,这些方法易于理解和一般,应用于计算机视觉,自然语言处理,深度加强学习等领域出现的许多现代预测问题。这种实践介绍是针对对无需统计学家的免费UQ的实际实施感兴趣的读者。我们通过实际的理论和无分发UQ的应用领导读者,从保形预测开始,并使无关的任何风险的分布控制,如虚假发现率,假阳性分布检测,等等。我们将包括Python中的许多解释性插图,示例和代码样本,具有Pytorch语法。目标是提供读者对无分配UQ的工作理解,使它们能够将置信间隔放在算法上,其中包含一个自包含的文档。
translated by 谷歌翻译