自从首次引入多层感知器以来，连接主义者社区一直在不确定性的概念以及如何在这些类型的模型中代表这一概念。在过去的十年中，试图将概率建模的原则性方法与深度神经网络的可扩展性相结合在一起。尽管这种合并的理论利益很明显，但这些努力也有一些重要的实际方面。即强迫我们创建的模型来代表，学习和报告所做的每个预测中的不确定性。这些努力中有许多是基于扩展现有框架的其他结构。我们提出了Borch，这是一种可扩展的通用概率编程语言，它建在Pytorch之上。该代码可在我们的存储库中下载和使用https://gitlab.com/desupervise/borch。

现代深度学习方法构成了令人难以置信的强大工具，以解决无数的挑战问题。然而，由于深度学习方法作为黑匣子运作，因此与其预测相关的不确定性往往是挑战量化。贝叶斯统计数据提供了一种形式主义来理解和量化与深度神经网络预测相关的不确定性。本教程概述了相关文献和完整的工具集，用于设计，实施，列车，使用和评估贝叶斯神经网络，即使用贝叶斯方法培训的随机人工神经网络。

Pyro is a probabilistic programming language built on Python as a platform for developing advanced probabilistic models in AI research. To scale to large datasets and high-dimensional models, Pyro uses stochastic variational inference algorithms and probability distributions built on top of PyTorch, a modern GPU-accelerated deep learning framework. To accommodate complex or model-specific algorithmic behavior, Pyro leverages Poutine, a library of composable building blocks for modifying the behavior of probabilistic programs.

这本数字本书包含在物理模拟的背景下与深度学习相关的一切实际和全面的一切。尽可能多，所有主题都带有Jupyter笔记本的形式的动手代码示例，以便快速入门。除了标准的受监督学习的数据中，我们将看看物理丢失约束，更紧密耦合的学习算法，具有可微分的模拟，以及加强学习和不确定性建模。我们生活在令人兴奋的时期：这些方法具有从根本上改变计算机模拟可以实现的巨大潜力。

Deep learning frameworks have often focused on either usability or speed, but not both. PyTorch is a machine learning library that shows that these two goals are in fact compatible: it provides an imperative and Pythonic programming style that supports code as a model, makes debugging easy and is consistent with other popular scientific computing libraries, while remaining efficient and supporting hardware accelerators such as GPUs. In this paper, we detail the principles that drove the implementation of PyTorch and how they are reflected in its architecture. We emphasize that every aspect of PyTorch is a regular Python program under the full control of its user. We also explain how the careful and pragmatic implementation of the key components of its runtime enables them to work together to achieve compelling performance. We demonstrate the efficiency of individual subsystems, as well as the overall speed of PyTorch on several common benchmarks.

灵感来自HTTPS://Doi.org/10.1515/Jagi-2016-0001中呈现的“认知时间玻璃”模型，我们为开发旨在认知机器人的认知架构提出了一个新的框架。拟议框架的目的是通过鼓励和减轻合作和重复使用现有结果来缓解认知架构的发展。这是通过提出将认知架构的发展分成一系列层的框架来完成，该层可以部分地被认为是隔离的，其中一些可以与其他研究领域直接相关。最后，我们向拟议框架介绍和审查一些主题。

现代深度学习框架提供嵌入在Python中的必要的急切执行编程接口，以提供生产的开发体验。但是，深度学习从业者有时需要捕获和转换程序结构以进行性能优化，可视化，分析和硬件集成。我们研究了深度学习中使用的程序捕获和转型的不同设计。通过设计典型的深度学习用例而不是长尾部，可以为程序捕获和转换创建更简单的框架。我们在Torch.fx中应用了这一原理，是一个完全在Python写入的Pytorch的程序捕获和转换库，并通过ML从业者进行高开发人员生产力优化。我们存在案例研究，展示了Torch.fx如何实现先前在Pytorch生态系统中无法访问的工作流程。

There is a significant need for principled uncertainty reasoning in machine learning systems as they are increasingly deployed in safety-critical domains. A new approach with uncertainty-aware regression-based neural networks (NNs), based on learning evidential distributions for aleatoric and epistemic uncertainties, shows promise over traditional deterministic methods and typical Bayesian NNs, notably with the capabilities to disentangle aleatoric and epistemic uncertainties. Despite some empirical success of Deep Evidential Regression (DER), there are important gaps in the mathematical foundation that raise the question of why the proposed technique seemingly works. We detail the theoretical shortcomings and analyze the performance on synthetic and real-world data sets, showing that Deep Evidential Regression is a heuristic rather than an exact uncertainty quantification. We go on to propose corrections and redefinitions of how aleatoric and epistemic uncertainties should be extracted from NNs.

我们将反应性消息传递（RMP）作为框架，用于在概率模型的因子图表示中执行基于时间表，鲁棒和可扩展的消息通过的基于消息传递的推断。 RMP基于反应性编程风格，该样式仅描述因子图中的节点如何对连接节点中的更改作出反应。没有固定消息传递计划提高推理过程的稳健性，可伸缩性和执行时间。我们还存在ReactiveMp.jl，这是一个Julia包，用于通过最小化约束的自由能实现RMP。通过用户定义的本地表单和分解约束对变分后部分布的结构，ReastiveMp.jl执行混合消息传递算法，包括信仰传播，变分消息通过，期望传播和期望最大化更新规则。实验结果表明，与其他概率模型的贝叶斯推断的其他朱莉娅封装相比，基于Reactivemp的RMP的性能提高。特别是，我们表明RMP框架能够为大型概率状态空间模型运行贝叶斯人推断，并在标准膝上型计算机上具有数十万个随机变量。

随着我们远离数据，预测不确定性应该增加，因为各种各样的解释与鲜为人知的信息一致。我们引入了远距离感知的先验（DAP）校准，这是一种纠正训练域之外贝叶斯深度学习模型过度自信的方法。我们将DAPS定义为模型参数的先验分布，该模型参数取决于输入，通过其与训练集的距离度量。DAP校准对后推理方法不可知，可以作为后处理步骤进行。我们证明了其在各种分类和回归问题中对几个基线的有效性，包括旨在测试远离数据的预测分布质量的基准。

Derivatives, mostly in the form of gradients and Hessians, are ubiquitous in machine learning. Automatic differentiation (AD), also called algorithmic differentiation or simply "autodiff", is a family of techniques similar to but more general than backpropagation for efficiently and accurately evaluating derivatives of numeric functions expressed as computer programs. AD is a small but established field with applications in areas including computational fluid dynamics, atmospheric sciences, and engineering design optimization. Until very recently, the fields of machine learning and AD have largely been unaware of each other and, in some cases, have independently discovered each other's results. Despite its relevance, general-purpose AD has been missing from the machine learning toolbox, a situation slowly changing with its ongoing adoption under the names "dynamic computational graphs" and "differentiable programming". We survey the intersection of AD and machine learning, cover applications where AD has direct relevance, and address the main implementation techniques. By precisely defining the main differentiation techniques and their interrelationships, we aim to bring clarity to the usage of the terms "autodiff", "automatic differentiation", and "symbolic differentiation" as these are encountered more and more in machine learning settings.

Bayesian optimization provides sample-efficient global optimization for a broad range of applications, including automatic machine learning, engineering, physics, and experimental design. We introduce BOTORCH, a modern programming framework for Bayesian optimization that combines Monte-Carlo (MC) acquisition functions, a novel sample average approximation optimization approach, autodifferentiation, and variance reduction techniques. BOTORCH's modular design facilitates flexible specification and optimization of probabilistic models written in PyTorch, simplifying implementation of new acquisition functions. Our approach is backed by novel theoretical convergence results and made practical by a distinctive algorithmic foundation that leverages fast predictive distributions, hardware acceleration, and deterministic optimization. We also propose a novel "one-shot" formulation of the Knowledge Gradient, enabled by a combination of our theoretical and software contributions. In experiments, we demonstrate the improved sample efficiency of BOTORCH relative to other popular libraries.34th Conference on Neural Information Processing Systems (NeurIPS 2020),

随机过程提供了数学上优雅的方式模型复杂数据。从理论上讲，它们为可以编码广泛有趣的假设的功能类提供了灵活的先验。但是，实际上，难以通过优化或边缘化来有效推断，这一问题进一步加剧了大数据和高维输入空间。我们提出了一种新颖的变性自动编码器（VAE），称为先前的编码变量自动编码器（$ \ pi $ vae）。 $ \ pi $ vae是有限的交换且Kolmogorov一致的，因此是一个连续的随机过程。我们使用$ \ pi $ vae学习功能类的低维嵌入。我们表明，我们的框架可以准确地学习表达功能类，例如高斯流程，也可以学习函数的属性以启用统计推断（例如log高斯过程的积分）。对于流行的任务，例如空间插值，$ \ pi $ vae在准确性和计算效率方面都达到了最先进的性能。也许最有用的是，我们证明了所学的低维独立分布的潜在空间表示提供了一种优雅，可扩展的方法，可以在概率编程语言（例如Stan）中对随机过程进行贝叶斯推断。

在本文中，我们介绍Bayesldm，这是一个用于贝叶斯纵向数据建模的系统，该系统由高级建模语言组成，具有针对复杂的多变量时间序列数据建模的特定功能，并与编译器相结合，可以生成优化的概率程序代码，以在指定模型中执行指定的推理。 Bayesldm支持贝叶斯网络模型的建模，其特定关注动态贝叶斯网络（DBN）的高效，声明性规范。 Bayesldm编译器将模型规范与可用数据和输出代码相结合，用于执行贝叶斯推断，以同时处理丢失的数据，同时处理未知模型参数。这些功能有可能通过抽象产生计算有效的概率推断代码的过程来显着加速域中的迭代建模工作流，这些迭代建模工作流程涉及复杂纵向数据的分析。我们描述了Bayesldm系统组件，评估表示和推理优化的效率，并提供了该系统在分析异质和部分观察到的移动健康数据的应用示例。

背景信息：在过去几年中，机器学习（ML）一直是许多创新的核心。然而，包括在所谓的“安全关键”系统中，例如汽车或航空的系统已经被证明是非常具有挑战性的，因为ML的范式转变为ML带来完全改变传统认证方法。目的：本文旨在阐明与ML为基础的安全关键系统认证有关的挑战，以及文献中提出的解决方案，以解决它们，回答问题的问题如何证明基于机器学习的安全关键系统？'方法：我们开展2015年至2020年至2020年之间发布的研究论文的系统文献综述（SLR），涵盖了与ML系统认证有关的主题。总共确定了217篇论文涵盖了主题，被认为是ML认证的主要支柱：鲁棒性，不确定性，解释性，验证，安全强化学习和直接认证。我们分析了每个子场的主要趋势和问题，并提取了提取的论文的总结。结果：单反结果突出了社区对该主题的热情，以及在数据集和模型类型方面缺乏多样性。它还强调需要进一步发展学术界和行业之间的联系，以加深域名研究。最后，它还说明了必须在上面提到的主要支柱之间建立连接的必要性，这些主要柱主要主要研究。结论：我们强调了目前部署的努力，以实现ML基于ML的软件系统，并讨论了一些未来的研究方向。

Recent progress in artificial intelligence (AI) has renewed interest in building systems that learn and think like people. Many advances have come from using deep neural networks trained end-to-end in tasks such as object recognition, video games, and board games, achieving performance that equals or even beats humans in some respects. Despite their biological inspiration and performance achievements, these systems differ from human intelligence in crucial ways. We review progress in cognitive science suggesting that truly human-like learning and thinking machines will have to reach beyond current engineering trends in both what they learn, and how they learn it. Specifically, we argue that these machines should (a) build causal models of the world that support explanation and understanding, rather than merely solving pattern recognition problems; (b) ground learning in intuitive theories of physics and psychology, to support and enrich the knowledge that is learned; and (c) harness compositionality and learning-to-learn to rapidly acquire and generalize knowledge to new tasks and situations. We suggest concrete challenges and promising routes towards these goals that can combine the strengths of recent neural network advances with more structured cognitive models.

对于许多工程应用，例如实时模拟或控制，潜在的非线性问题的传统解决方案技术通常是过于计算的。在这项工作中，我们提出了一种高效的深度学习代理框架，能够预测负载下的超弹性体的响应。代理模型采用特殊的卷积神经网络架构，所谓的U-Net的形式，其具有用有限元方法获得的力 - 位移数据训练。我们提出了框架的确定性和概率版本，并研究了三个基准问题。特别是，我们检查最大可能性和变分贝叶斯推论配方的能力，以评估解决方案的置信区间。

近年来，机器学习（ML），深度学习（DL）和人工智能（AI）的普及已急剧上升。尽管受欢迎程度激增，但ML和DL算法的内部运作被认为是不透明的，并且它们与经典数据分析工具的关系仍然存在争议。通常认为ML和DL主要在做出预测方面出色。但是，最近，它们越来越多地用于传统上统计模型涵盖的经典分析任务。此外，最近对ML的评论专门针对DL，缺少综合具有不同优势和一般原则的ML算法财富。在这里，我们提供了ML和DL领域的全面概述，从其历史发展，现有算法家庭，与传统统计工具的差异以及通用ML原则的差异。然后，我们讨论为什么以及何时ML和DL模型在预测任务上表现出色，以及它们可以为推理提供传统统计方法的替代方法，从而突出了当前和新兴的生态问题应用程序。最后，我们总结了新兴趋势，例如科学和因果ML，可解释的AI以及负责的AI，这些AI可能会在未来显着影响生态数据分析。

The notion of uncertainty is of major importance in machine learning and constitutes a key element of machine learning methodology. In line with the statistical tradition, uncertainty has long been perceived as almost synonymous with standard probability and probabilistic predictions. Yet, due to the steadily increasing relevance of machine learning for practical applications and related issues such as safety requirements, new problems and challenges have recently been identified by machine learning scholars, and these problems may call for new methodological developments. In particular, this includes the importance of distinguishing between (at least) two different types of uncertainty, often referred to as aleatoric and epistemic. In this paper, we provide an introduction to the topic of uncertainty in machine learning as well as an overview of attempts so far at handling uncertainty in general and formalizing this distinction in particular.

There are two major types of uncertainty one can model. Aleatoric uncertainty captures noise inherent in the observations. On the other hand, epistemic uncertainty accounts for uncertainty in the model -uncertainty which can be explained away given enough data. Traditionally it has been difficult to model epistemic uncertainty in computer vision, but with new Bayesian deep learning tools this is now possible. We study the benefits of modeling epistemic vs. aleatoric uncertainty in Bayesian deep learning models for vision tasks. For this we present a Bayesian deep learning framework combining input-dependent aleatoric uncertainty together with epistemic uncertainty. We study models under the framework with per-pixel semantic segmentation and depth regression tasks. Further, our explicit uncertainty formulation leads to new loss functions for these tasks, which can be interpreted as learned attenuation. This makes the loss more robust to noisy data, also giving new state-of-the-art results on segmentation and depth regression benchmarks.