室外（OOD）检测是面向任务的对话框系统中的关键组件，旨在确定查询是否不在预定义的支持的意图集之外。事实证明，先前基于软磁性的检测算法对OOD样品被过度自信。在本文中，我们分析了过度自信的OOD来自由于训练和测试分布之间的不匹配而导致的分布不确定性，这使得该模型无法自信地做出预测，因此可能导致异常软磁得分。我们提出了一个贝叶斯OOD检测框架，以使用Monte-Carlo辍学来校准分布不确定性。我们的方法是灵活的，并且可以轻松地插入现有的基于软磁性的基线和增益33.33 \％OOD F1改进，而与MSP相比仅增加了0.41 \％的推理时间。进一步的分析表明，贝叶斯学习对OOD检测的有效性。

传统意图分类模型基于预定义的意图集，仅识别有限的内域（IND）意图类别。但是用户可以在实用的对话系统中输入室外（OOD）查询。这样的OOD查询可以提供未来改进的方向。在本文中，我们定义了一项新任务，广义意图发现（GID），旨在将IND意图分类器扩展到包括IND和OOD意图在内的开放世界意图集。我们希望在发现和识别新的未标记的OOD类型的同时，同时对一组标记的IND意图类进行分类。我们为不同的应用程序方案构建了三个公共数据集，并提出了两种框架，即基于管道的框架和端到端，以实现未来的工作。此外，我们进行详尽的实验和定性分析，以理解关键挑战，并为未来的GID研究提供新的指导。

在计算机视觉中探索的分销（OOD）检测良好的虽然，但在NLP分类的情况下已经开始较少尝试。在本文中，我们认为这些目前的尝试没有完全解决ood问题，并且可能遭受数据泄漏和所产生模型的校准差。我们呈现PNPOOD，通过使用最近提出的即插即用语言模型（Dathathri等，2020），通过域外样本生成进行数据增强技术来执行OOD检测。我们的方法产生靠近阶级边界的高质量辨别样本，从而在测试时间内进行准确的检测。我们展示了我们的模型优于预先样本检测的现有模型，并在20次新闻组文本和斯坦福情绪Teebank数据集上展示较低的校准错误（Lang，1995; Socheret al。，2013）。我们进一步突出显示了在EAC检测的先前尝试中使用的数据集进行了重要的数据泄露问题，并在新数据集中分享结果，以便无法遭受同样问题的检测。

Out-of-Domain (OOD) intent detection is important for practical dialog systems. To alleviate the issue of lacking OOD training samples, some works propose synthesizing pseudo OOD samples and directly assigning one-hot OOD labels to these pseudo samples. However, these one-hot labels introduce noises to the training process because some hard pseudo OOD samples may coincide with In-Domain (IND) intents. In this paper, we propose an adaptive soft pseudo labeling (ASoul) method that can estimate soft labels for pseudo OOD samples when training OOD detectors. Semantic connections between pseudo OOD samples and IND intents are captured using an embedding graph. A co-training framework is further introduced to produce resulting soft labels following the smoothness assumption, i.e., close samples are likely to have similar labels. Extensive experiments on three benchmark datasets show that ASoul consistently improves the OOD detection performance and outperforms various competitive baselines.

Accurate uncertainty quantification is a major challenge in deep learning, as neural networks can make overconfident errors and assign high confidence predictions to out-of-distribution (OOD) inputs. The most popular approaches to estimate predictive uncertainty in deep learning are methods that combine predictions from multiple neural networks, such as Bayesian neural networks (BNNs) and deep ensembles. However their practicality in real-time, industrial-scale applications are limited due to the high memory and computational cost. Furthermore, ensembles and BNNs do not necessarily fix all the issues with the underlying member networks. In this work, we study principled approaches to improve uncertainty property of a single network, based on a single, deterministic representation. By formalizing the uncertainty quantification as a minimax learning problem, we first identify distance awareness, i.e., the model's ability to quantify the distance of a testing example from the training data, as a necessary condition for a DNN to achieve high-quality (i.e., minimax optimal) uncertainty estimation. We then propose Spectral-normalized Neural Gaussian Process (SNGP), a simple method that improves the distance-awareness ability of modern DNNs with two simple changes: (1) applying spectral normalization to hidden weights to enforce bi-Lipschitz smoothness in representations and (2) replacing the last output layer with a Gaussian process layer. On a suite of vision and language understanding benchmarks, SNGP outperforms other single-model approaches in prediction, calibration and out-of-domain detection. Furthermore, SNGP provides complementary benefits to popular techniques such as deep ensembles and data augmentation, making it a simple and scalable building block for probabilistic deep learning. Code is open-sourced at https://github.com/google/uncertainty-baselines

Estimating how uncertain an AI system is in its predictions is important to improve the safety of such systems. Uncertainty in predictive can result from uncertainty in model parameters, irreducible data uncertainty and uncertainty due to distributional mismatch between the test and training data distributions. Different actions might be taken depending on the source of the uncertainty so it is important to be able to distinguish between them. Recently, baseline tasks and metrics have been defined and several practical methods to estimate uncertainty developed. These methods, however, attempt to model uncertainty due to distributional mismatch either implicitly through model uncertainty or as data uncertainty. This work proposes a new framework for modeling predictive uncertainty called Prior Networks (PNs) which explicitly models distributional uncertainty. PNs do this by parameterizing a prior distribution over predictive distributions. This work focuses on uncertainty for classification and evaluates PNs on the tasks of identifying out-of-distribution (OOD) samples and detecting misclassification on the MNIST and CIFAR-10 datasets, where they are found to outperform previous methods. Experiments on synthetic and MNIST data show that unlike previous non-Bayesian methods PNs are able to distinguish between data and distributional uncertainty.

It is known that neural networks have the problem of being over-confident when directly using the output label distribution to generate uncertainty measures. Existing methods mainly resolve this issue by retraining the entire model to impose the uncertainty quantification capability so that the learned model can achieve desired performance in accuracy and uncertainty prediction simultaneously. However, training the model from scratch is computationally expensive and may not be feasible in many situations. In this work, we consider a more practical post-hoc uncertainty learning setting, where a well-trained base model is given, and we focus on the uncertainty quantification task at the second stage of training. We propose a novel Bayesian meta-model to augment pre-trained models with better uncertainty quantification abilities, which is effective and computationally efficient. Our proposed method requires no additional training data and is flexible enough to quantify different uncertainties and easily adapt to different application settings, including out-of-domain data detection, misclassification detection, and trustworthy transfer learning. We demonstrate our proposed meta-model approach's flexibility and superior empirical performance on these applications over multiple representative image classification benchmarks.

用于图形分类的分布外检测的问题远未解决。现有模型往往对OOD示例过高自信，或者完全忽略检测任务。在这项工作中，我们从不确定性估计的角度考虑了这个问题，并进行了几种最近提出的方法的比较。在我们的实验中，我们发现没有通用的OOD检测方法，并且重要的是考虑图表和预测分类分布。

最近，深度学习中的不确定性估计已成为提高安全至关重要应用的可靠性和鲁棒性的关键领域。尽管有许多提出的方法要么关注距离感知模型的不确定性，要么是分布式检测的不确定性，要么是针对分布校准的输入依赖性标签不确定性，但这两种类型的不确定性通常都是必要的。在这项工作中，我们提出了用于共同建模模型和数据不确定性的HETSNGP方法。我们表明，我们提出的模型在这两种类型的不确定性之间提供了有利的组合，因此在包括CIFAR-100C，ImagEnet-C和Imagenet-A在内的一些具有挑战性的分发数据集上优于基线方法。此外，我们提出了HETSNGP Ensemble，这是我们方法的结合版本，该版本还对网络参数的不确定性进行建模，并优于其他集合基线。

对于在开放世界中部署的机器学习模型是必不可少的。最近，在训练期间（也称为离群暴露）在训练期间使用辅助外离群值数据集已显示出令人鼓舞的性能。由于潜在的OOD数据的样本空间可能是过大的，因此进行抽样信息的异常值至关重要。在这项工作中，我们提出了一种新型的基于后取样的离群矿井诗歌诗，该诗歌有助于有效利用异常数据，并促进了ID和OOD数据之间的紧凑决策边界，以改善检测。我们表明，诗在普通基准上建立了最先进的表现。与当前使用贪婪采样策略的最佳方法相比，诗在CIFAR-10和CIFAR-100上分别提高了相对性能的42.0％和24.2％（FPR95）。我们进一步提供了有关诗歌检测有效性的理论见解。

本文我们的目标是利用异质的温度缩放作为校准策略（OOD）检测。此处的异质性是指每个样品的最佳温度参数可能不同，而不是传统的方法对整个分布使用相同的值。为了实现这一目标，我们提出了一种称为锚定的新培训策略，可以估算每个样品的适当温度值，从而导致几个基准的最新OOD检测性能。使用NTK理论，我们表明该温度函数估计与分类器的认知不确定性紧密相关，这解释了其行为。与某些表现最佳的OOD检测方法相反，我们的方法不需要暴露于其他离群数据集，自定义校准目标或模型结合。通过具有不同OOD检测设置的经验研究 - 远处，OOD附近和语义相干OOD - 我们建立了一种高效的OOD检测方法。可以在此处访问代码和模型-https：//github.com/rushilanirudh/amp

检测到分布输入对于在现实世界中安全部署机器学习模型至关重要。然而，已知神经网络遭受过度自信的问题，在该问题中，它们对分布和分布的输入的信心异常高。在这项工作中，我们表明，可以通过在训练中实施恒定的向量规范来通过logit归一化（logitnorm）（logitnorm）来缓解此问题。我们的方法是通过分析的激励，即logit的规范在训练过程中不断增加，从而导致过度自信的产出。因此，LogitNorm背后的关键思想是将网络优化期间输出规范的影响解散。通过LogitNorm培训，神经网络在分布数据和分布数据之间产生高度可区分的置信度得分。广泛的实验证明了LogitNorm的优势，在公共基准上，平均FPR95最高为42.30％。

公开意图检测是自然语言理解中的一个重大问题，旨在以仅知道已知意图的先验知识来检测看不见的公开意图。当前方法在此任务中面临两个核心挑战。一方面，他们在学习友好表示方面有局限性来检测公开意图。另一方面，缺乏有效的方法来获得已知意图的特定和紧凑的决策边界。为了解决这些问题，本文介绍了一个原始框架DA-ADB，该框架连续学习了远距离感知的意图表示和自适应决策边界，以进行开放意图检测。具体而言，我们首先利用距离信息来增强意图表示的区别能力。然后，我们设计了一种新颖的损失函数，以通过平衡经验和开放空间风险来获得适当的决策界限。广泛的实验显示了距离了解和边界学习策略的有效性。与最先进的方法相比，我们的方法在三个基准数据集上实现了重大改进。它还具有不同比例的标记数据和已知类别的稳健性能。完整的数据和代码可在https://github.com/thuiar/textoir上获得

分布（OOD）检测是在开放世界中部署机器学习模型的关键任务。基于距离的方法已经证明了有望，如果测试样品离分布（ID）数据相对遥远，则将测试样品视为OOD。但是，先前的方法对基础特征空间施加了强有力的分布假设，这可能并不总是存在。在本文中，我们探讨了非参数最近邻居距离的疗效，以检测OOD，这在文献中很大程度上被忽略了。与先前的工作不同，我们的方法不会施加任何分布假设，因此提供了更强的灵活性和一般性。我们证明了在几个基准测试中基于邻元的OOD检测的有效性，并建立了卓越的性能。在对Imagenet-1K训练的同一模型下，我们的方法将假阳性率（FPR@tpr95）降低了24.77％，与强大的基线SSD+相比，使用参数方法Mahalanobis在检测中。可用代码：https：//github.com/deeplearning-wisc/knn-ood。

随着我们远离数据，预测不确定性应该增加，因为各种各样的解释与鲜为人知的信息一致。我们引入了远距离感知的先验（DAP）校准，这是一种纠正训练域之外贝叶斯深度学习模型过度自信的方法。我们将DAPS定义为模型参数的先验分布，该模型参数取决于输入，通过其与训练集的距离度量。DAP校准对后推理方法不可知，可以作为后处理步骤进行。我们证明了其在各种分类和回归问题中对几个基线的有效性，包括旨在测试远离数据的预测分布质量的基准。

不确定性的量化对于采用机器学习至关重要，尤其是拒绝分布（OOD）数据回到人类专家进行审查。然而，进步一直很慢，因为计算效率和不确定性估计质量之间必须达到平衡。因此，许多人使用神经网络或蒙特卡洛辍学的深层集合来进行相对最小的计算和记忆时合理的不确定性估计。出乎意料的是，当我们专注于$ \ leq 1 \％$ frese-falds正率（FPR）的现实世界中的约束时，先前的方法无法可靠地检测到OOD样本。值得注意的是，即使高斯随机噪声也无法触发这些流行的OOD技术。我们通过设计一种简单的对抗训练计划来帮助缓解这个问题，该计划结合了辍学合奏所预测的认知不确定性的攻击。我们证明了这种方法可以改善标准数据（即未经对抗制作）上的OOD检测性能，并将标准化的部分AUC从近乎随机的猜测性能提高到$ \ geq 0.75 $。

Modern machine learning methods including deep learning have achieved great success in predictive accuracy for supervised learning tasks, but may still fall short in giving useful estimates of their predictive uncertainty. Quantifying uncertainty is especially critical in real-world settings, which often involve input distributions that are shifted from the training distribution due to a variety of factors including sample bias and non-stationarity. In such settings, well calibrated uncertainty estimates convey information about when a model's output should (or should not) be trusted. Many probabilistic deep learning methods, including Bayesian-and non-Bayesian methods, have been proposed in the literature for quantifying predictive uncertainty, but to our knowledge there has not previously been a rigorous largescale empirical comparison of these methods under dataset shift. We present a largescale benchmark of existing state-of-the-art methods on classification problems and investigate the effect of dataset shift on accuracy and calibration. We find that traditional post-hoc calibration does indeed fall short, as do several other previous methods. However, some methods that marginalize over models give surprisingly strong results across a broad spectrum of tasks.

在深神经网络中量化预测性不确定性的流行方法通常涉及一组权重或模型，例如通过合并或蒙特卡罗辍学。这些技术通常必须产生开销，必须培训多种模型实例，或者不会产生非常多样化的预测。该调查旨在熟悉基于证据深度学习的概念的替代类模型的读者：对于不熟悉的数据，他们承认“他们不知道的内容”并返回到先前的信仰。此外，它们允许在单个模型中进行不确定性估计，并通过参数化分布分布来转发传递。该调查重新承认现有工作，重点是在分类设置中的实现。最后，我们调查了相同范例的应用到回归问题。我们还对现有的方法进行了反思，并与现有方法相比，并提供最大的核心理论成果，以便通知未来的研究。

Determining whether inputs are out-of-distribution (OOD) is an essential building block for safely deploying machine learning models in the open world. However, previous methods relying on the softmax confidence score suffer from overconfident posterior distributions for OOD data. We propose a unified framework for OOD detection that uses an energy score. We show that energy scores better distinguish in-and out-of-distribution samples than the traditional approach using the softmax scores. Unlike softmax confidence scores, energy scores are theoretically aligned with the probability density of the inputs and are less susceptible to the overconfidence issue. Within this framework, energy can be flexibly used as a scoring function for any pre-trained neural classifier as well as a trainable cost function to shape the energy surface explicitly for OOD detection. On a CIFAR-10 pre-trained WideResNet, using the energy score reduces the average FPR (at TPR 95%) by 18.03% compared to the softmax confidence score. With energy-based training, our method outperforms the state-of-the-art on common benchmarks.

变形自身偏移（VAES）是具有来自深神经网络架构和贝叶斯方法的丰富代表功能的有影响力的生成模型。然而，VAE模型具有比分布（ID）输入的分配方式分配更高的可能性较高的可能性。为了解决这个问题，认为可靠的不确定性估计是对对OOC投入的深入了解至关重要。在这项研究中，我们提出了一种改进的噪声对比之前（INCP），以便能够集成到VAE的编码器中，称为INCPVAE。INCP是可扩展，可培训和与VAE兼容的，它还采用了来自INCP的优点进行不确定性估计。各种数据集的实验表明，与标准VAE相比，我们的模型在OOD数据的不确定性估计方面是优越的，并且在异常检测任务中是强大的。INCPVAE模型获得了可靠的输入不确定性估算，并解决了VAE模型中的ood问题。