Machine Unering是指删除培训数据子集的任务，从而删除其对训练有素的模型的贡献。近似学习是该任务的一类方法，避免了需要在保留数据上从头开始重新研究模型。贝叶斯的规则可用于将近似学习作为推理问题，其中目的是通过划分删除数据的可能性来获得更新后的后部。但是，这有自己的挑战集，因为人们通常无法访问模型参数的确切后验。在这项工作中，我们检查了拉普拉斯近似和变异推理的使用以获得更新的后验。通过对指导示例进行回归任务的神经网络培训，我们在实践场景中就贝叶斯学习的适用性进行了见解。

我们制定自然梯度变推理（VI），期望传播（EP），和后线性化（PL）作为牛顿法用于优化贝叶斯后验分布的参数扩展。这种观点明确地把数值优化框架下的推理算法。我们表明，通用近似牛顿法从优化文献，即高斯 - 牛顿和准牛顿方法（例如，该BFGS算法），仍然是这种“贝叶斯牛顿”框架下有效。这导致了一套这些都保证以产生半正定协方差矩阵，不像标准VI和EP新颖算法。我们统一的观点提供了新的见解各种推理方案之间的连接。所有提出的方法适用于具有高斯事先和非共轭的可能性，这是我们与（疏）高斯过程和状态空间模型展示任何模型。

用于估计模型不确定性的线性拉普拉斯方法在贝叶斯深度学习社区中引起了人们的重新关注。该方法提供了可靠的误差线，并接受模型证据的封闭式表达式，从而可以选择模型超参数。在这项工作中，我们检查了这种方法背后的假设，尤其是与模型选择结合在一起。我们表明，这些与一些深度学习的标准工具（构成近似方法和归一化层）相互作用，并为如何更好地适应这种经典方法对现代环境提出建议。我们为我们的建议提供理论支持，并在MLP，经典CNN，具有正常化层，生成性自动编码器和变压器的剩余网络上进行经验验证它们。

表示学习已成为一种实用的方法，可以在重建方面成功地建立大量高维数据的丰富参数编码。在考虑具有测试训练分布变化的无监督任务时，概率的观点有助于解决预测过度自信和不良校准。但是，由于多种原因，即维度或顽固性问题的诅咒，直接引入贝叶斯推断仍然是一个艰难的问题。 Laplace近似（LA）在这里提供了一个解决方案，因为可以通过二阶Taylor膨胀在参数空间的某些位置通过二阶Taylor膨胀来建立重量的高斯近似值。在这项工作中，我们为洛杉矶启发的无监督表示学习提供了贝叶斯自动编码器。我们的方法实现了迭代的拉普拉斯更新，以获得新型自动编码器证据的新变化下限。二阶部分衍生物的巨大计算负担是通过Hessian矩阵的近似来跳过的。从经验上讲，我们通过为分布外检测提供了良好的不确定性，用于差异几何形状的大地测量和缺失数据归思的方法来证明拉普拉斯自动编码器的可伸缩性和性能。

贝叶斯范式有可能解决深度神经网络的核心问题，如校准和数据效率低差。唉，缩放贝叶斯推理到大量的空间通常需要限制近似。在这项工作中，我们表明它足以通过模型权重的小子集进行推动，以便获得准确的预测后断。另一个权重被保存为点估计。该子网推断框架使我们能够在这些子集上使用表现力，否则难以相容的后近近似。特别是，我们将子网线性化LAPLACE作为一种简单，可扩展的贝叶斯深度学习方法：我们首先使用线性化的拉普拉斯近似来获得所有重量的地图估计，然后在子网上推断出全协方差高斯后面。我们提出了一个子网选择策略，旨在最大限度地保护模型的预测性不确定性。经验上，我们的方法对整个网络的集合和较少的表达后近似进行了比较。

Recent advances in coreset methods have shown that a selection of representative datapoints can replace massive volumes of data for Bayesian inference, preserving the relevant statistical information and significantly accelerating subsequent downstream tasks. Existing variational coreset constructions rely on either selecting subsets of the observed datapoints, or jointly performing approximate inference and optimizing pseudodata in the observed space akin to inducing points methods in Gaussian Processes. So far, both approaches are limited by complexities in evaluating their objectives for general purpose models, and require generating samples from a typically intractable posterior over the coreset throughout inference and testing. In this work, we present a black-box variational inference framework for coresets that overcomes these constraints and enables principled application of variational coresets to intractable models, such as Bayesian neural networks. We apply our techniques to supervised learning problems, and compare them with existing approaches in the literature for data summarization and inference.

通过强制了解输入中某些转换保留输出的知识，通常应用数据增强来提高深度学习的性能。当前，使用的数据扩大是通过人类的努力和昂贵的交叉验证来选择的，这使得应用于新数据集很麻烦。我们开发了一种基于梯度的方便方法，用于在没有验证数据的情况下和在深度神经网络的培训期间选择数据增强。我们的方法依赖于措辞增强作为先前分布的不变性，并使用贝叶斯模型选择学习，该模型已被证明在高斯过程中起作用，但尚未用于深神经网络。我们提出了一个可区分的Kronecker因拉普拉斯（Laplace）近似与边际可能性的近似，作为我们的目标，可以在没有人类监督或验证数据的情况下优化。我们表明，我们的方法可以成功地恢复数据中存在的不断增长，这提高了图像数据集的概括和数据效率。

已知生物制剂在他们的生活过程中学习许多不同的任务，并且能够重新审视以前的任务和行为，而没有表现不损失。相比之下，人工代理容易出于“灾难性遗忘”，在以前任务上的性能随着所获取的新的任务而恶化。最近使用该方法通过鼓励参数保持接近以前任务的方法来解决此缺点。这可以通过（i）使用特定的参数正常数来完成，该参数正常数是在参数空间中映射合适的目的地，或（ii）通过将渐变投影到不会干扰先前任务的子空间来指导优化旅程。然而，这些方法通常在前馈和经常性神经网络中表现出子分子表现，并且经常性网络对支持生物持续学习的神经动力学研究感兴趣。在这项工作中，我们提出了自然的持续学习（NCL），一种统一重量正则化和预测梯度下降的新方法。 NCL使用贝叶斯重量正常化来鼓励在收敛的所有任务上进行良好的性能，并将其与梯度投影结合使用先前的精度，这可以防止在优化期间陷入灾难性遗忘。当应用于前馈和经常性网络中的连续学习问题时，我们的方法占据了标准重量正则化技术和投影的方法。最后，训练有素的网络演变了特定于任务特定的动态，这些动态被认为是学习的新任务，类似于生物电路中的实验结果。

机器学习模型的预测失败通常来自训练数据中的缺陷，例如不正确的标签，离群值和选择偏见。但是，这些负责给定失败模式的数据点通常不知道先验，更不用说修复故障的机制了。这项工作借鉴了贝叶斯对持续学习的看法，并为两者开发了一个通用框架，确定了导致目标失败的培训示例，并通过删除有关它们的信息来修复模型。该框架自然允许将最近学习的最新进展解决这一新的模型维修问题，同时将现有的作品集成了影响功能和数据删除作为特定实例。在实验上，提出的方法优于基准，既可以识别有害训练数据，又要以可普遍的方式固定模型失败。

Large multilayer neural networks trained with backpropagation have recently achieved state-ofthe-art results in a wide range of problems. However, using backprop for neural net learning still has some disadvantages, e.g., having to tune a large number of hyperparameters to the data, lack of calibrated probabilistic predictions, and a tendency to overfit the training data. In principle, the Bayesian approach to learning neural networks does not have these problems. However, existing Bayesian techniques lack scalability to large dataset and network sizes. In this work we present a novel scalable method for learning Bayesian neural networks, called probabilistic backpropagation (PBP). Similar to classical backpropagation, PBP works by computing a forward propagation of probabilities through the network and then doing a backward computation of gradients. A series of experiments on ten real-world datasets show that PBP is significantly faster than other techniques, while offering competitive predictive abilities. Our experiments also show that PBP provides accurate estimates of the posterior variance on the network weights.

Existing deep-learning based tomographic image reconstruction methods do not provide accurate estimates of reconstruction uncertainty, hindering their real-world deployment. This paper develops a method, termed as the linearised deep image prior (DIP), to estimate the uncertainty associated with reconstructions produced by the DIP with total variation regularisation (TV). Specifically, we endow the DIP with conjugate Gaussian-linear model type error-bars computed from a local linearisation of the neural network around its optimised parameters. To preserve conjugacy, we approximate the TV regulariser with a Gaussian surrogate. This approach provides pixel-wise uncertainty estimates and a marginal likelihood objective for hyperparameter optimisation. We demonstrate the method on synthetic data and real-measured high-resolution 2D $\mu$CT data, and show that it provides superior calibration of uncertainty estimates relative to previous probabilistic formulations of the DIP. Our code is available at https://github.com/educating-dip/bayes_dip.

我们通过Pac-Bayes概括界的镜头研究冷后效应。我们认为，在非反应环境中，当训练样本的数量相对较小时，应考虑到冷后效应的讨论，即大概贝叶斯推理并不能容易地提供对样本外数据的性能的保证。取而代之的是，通过泛化结合更好地描述了样本外误差。在这种情况下，我们探讨了各种推理与PAC-Bayes目标的ELBO目标之间的联系。我们注意到，虽然Elbo和Pac-Bayes目标相似，但后一个目标自然包含温度参数$ \ lambda $，不限于$ \ lambda = 1 $。对于回归和分类任务，在各向同性拉普拉斯与后部的近似值的情况下，我们展示了这种对温度参数的PAC-bayesian解释如何捕获冷后效应。

在这项工作中，我们使用变分推论来量化无线电星系分类的深度学习模型预测的不确定性程度。我们表明，当标记无线电星系时，个体测试样本的模型后差水平与人类不确定性相关。我们探讨了各种不同重量前沿的模型性能和不确定性校准，并表明稀疏事先产生更良好的校准不确定性估计。使用单个重量的后部分布，我们表明我们可以通过从最低信噪比（SNR）中除去权重来修剪30％的完全连接的层权重，而无需显着损失性能。我们证明，可以使用基于Fisher信息的排名来实现更大程度的修剪，但我们注意到两种修剪方法都会影响Failaroff-Riley I型和II型无线电星系的不确定性校准。最后，我们表明，与此领域的其他工作相比，我们经历了冷的后效，因此后部必须缩小后加权以实现良好的预测性能。我们检查是否调整成本函数以适应模型拼盘可以弥补此效果，但发现它不会产生显着差异。我们还研究了原则数据增强的效果，并发现这改善了基线，而且还没有弥补观察到的效果。我们将其解释为寒冷的后效，因为我们的培训样本过于有效的策划导致可能性拼盘，并将其提高到未来无线电银行分类的潜在问题。

We introduce a new, efficient, principled and backpropagation-compatible algorithm for learning a probability distribution on the weights of a neural network, called Bayes by Backprop. It regularises the weights by minimising a compression cost, known as the variational free energy or the expected lower bound on the marginal likelihood. We show that this principled kind of regularisation yields comparable performance to dropout on MNIST classification. We then demonstrate how the learnt uncertainty in the weights can be used to improve generalisation in non-linear regression problems, and how this weight uncertainty can be used to drive the exploration-exploitation trade-off in reinforcement learning.

We introduce a new, efficient, principled and backpropagation-compatible algorithm for learning a probability distribution on the weights of a neural network, called Bayes by Backprop. It regularises the weights by minimising a compression cost, known as the variational free energy or the expected lower bound on the marginal likelihood. We show that this principled kind of regularisation yields comparable performance to dropout on MNIST classification. We then demonstrate how the learnt uncertainty in the weights can be used to improve generalisation in non-linear regression problems, and how this weight uncertainty can be used to drive the exploration-exploitation trade-off in reinforcement learning.

现代深度学习方法构成了令人难以置信的强大工具，以解决无数的挑战问题。然而，由于深度学习方法作为黑匣子运作，因此与其预测相关的不确定性往往是挑战量化。贝叶斯统计数据提供了一种形式主义来理解和量化与深度神经网络预测相关的不确定性。本教程概述了相关文献和完整的工具集，用于设计，实施，列车，使用和评估贝叶斯神经网络，即使用贝叶斯方法培训的随机人工神经网络。

Kullback-Leibler（KL）差异广泛用于贝叶斯神经网络（BNNS）的变异推理。然而，KL差异具有无限性和不对称性等局限性。我们检查了更通用，有限和对称的詹森 - 香农（JS）差异。我们根据几何JS差异为BNN制定新的损失函数，并表明基于KL差异的常规损失函数是其特殊情况。我们以封闭形式的高斯先验评估拟议损失函数的差异部分。对于任何其他一般的先验，都可以使用蒙特卡洛近似值。我们提供了实施这两种情况的算法。我们证明所提出的损失函数提供了一个可以调整的附加参数，以控制正则化程度。我们得出了所提出的损失函数在高斯先验和后代的基于KL差异的损失函数更好的条件。我们证明了基于嘈杂的CIFAR数据集和有偏见的组织病理学数据集的最新基于KL差异的BNN的性能提高。

One of the core problems of modern statistics is to approximate difficult-to-compute probability densities. This problem is especially important in Bayesian statistics, which frames all inference about unknown quantities as a calculation involving the posterior density. In this paper, we review variational inference (VI), a method from machine learning that approximates probability densities through optimization. VI has been used in many applications and tends to be faster than classical methods, such as Markov chain Monte Carlo sampling. The idea behind VI is to first posit a family of densities and then to find the member of that family which is close to the target. Closeness is measured by Kullback-Leibler divergence. We review the ideas behind mean-field variational inference, discuss the special case of VI applied to exponential family models, present a full example with a Bayesian mixture of Gaussians, and derive a variant that uses stochastic optimization to scale up to massive data. We discuss modern research in VI and highlight important open problems. VI is powerful, but it is not yet well understood. Our hope in writing this paper is to catalyze statistical research on this class of algorithms.

We develop an optimization algorithm suitable for Bayesian learning in complex models. Our approach relies on natural gradient updates within a general black-box framework for efficient training with limited model-specific derivations. It applies within the class of exponential-family variational posterior distributions, for which we extensively discuss the Gaussian case for which the updates have a rather simple form. Our Quasi Black-box Variational Inference (QBVI) framework is readily applicable to a wide class of Bayesian inference problems and is of simple implementation as the updates of the variational posterior do not involve gradients with respect to the model parameters, nor the prescription of the Fisher information matrix. We develop QBVI under different hypotheses for the posterior covariance matrix, discuss details about its robust and feasible implementation, and provide a number of real-world applications to demonstrate its effectiveness.

Scientists continue to develop increasingly complex mechanistic models to reflect their knowledge more realistically. Statistical inference using these models can be highly challenging, since the corresponding likelihood function is often intractable, and model simulation may be computationally burdensome or infeasible. Fortunately, in many of these situations, it is possible to adopt a surrogate model or approximate likelihood function. It may be convenient to base Bayesian inference directly on the surrogate, but this can result in bias and poor uncertainty quantification. In this paper we propose a new method for adjusting approximate posterior samples to reduce bias and produce more accurate uncertainty quantification. We do this by optimising a transform of the approximate posterior that minimises a scoring rule. Our approach requires only a (fixed) small number of complex model simulations and is numerically stable. We demonstrate good performance of the new method on several examples of increasing complexity.