H-Measol是一个分类器性能测量，但考虑到应用程序的上下文，而不需要设置相对错误分类成本的刚性值。自2009年推出以来，它已被广泛采用。本文回答了用户提出的各种查询，包括关于其解释的问题，加权函数的选择，无论是严格正确的，还是与其他工作的措施相关。

本文介绍了分类器校准原理和实践的简介和详细概述。校准的分类器正确地量化了与其实例明智的预测相关的不确定性或信心水平。这对于关键应用，最佳决策，成本敏感的分类以及某些类型的上下文变化至关重要。校准研究具有丰富的历史，其中几十年来预测机器学习作为学术领域的诞生。然而，校准兴趣的最近增加导致了新的方法和从二进制到多种子体设置的扩展。需要考虑的选项和问题的空间很大，并导航它需要正确的概念和工具集。我们提供了主要概念和方法的介绍性材料和最新的技术细节，包括适当的评分规则和其他评估指标，可视化方法，全面陈述二进制和多字数分类的HOC校准方法，以及几个先进的话题。

Receiver operating characteristics (ROC) graphs are useful for organizing classifiers and visualizing their performance. ROC graphs are commonly used in medical decision making, and in recent years have been used increasingly in machine learning and data mining research. Although ROC graphs are apparently simple, there are some common misconceptions and pitfalls when using them in practice. The purpose of this article is to serve as an introduction to ROC graphs and as a guide for using them in research.

预测组合在预测社区中蓬勃发展，近年来，已经成为预测研究和活动主流的一部分。现在，由单个（目标）系列产生的多个预测组合通过整合来自不同来源收集的信息，从而提高准确性，从而减轻了识别单个“最佳”预测的风险。组合方案已从没有估计的简单组合方法演变为涉及时间变化的权重，非线性组合，组件之间的相关性和交叉学习的复杂方法。它们包括结合点预测和结合概率预测。本文提供了有关预测组合的广泛文献的最新评论，并参考可用的开源软件实施。我们讨论了各种方法的潜在和局限性，并突出了这些思想如何随着时间的推移而发展。还调查了有关预测组合实用性的一些重要问题。最后，我们以当前的研究差距和未来研究的潜在见解得出结论。

在这项工作中，我们对基本思想和新颖的发展进行了综述的综述，这是基于最小的假设的一种无创新的，无分配的，非参数预测的方法 - 能够以非常简单的方式预测集屈服在有限样本案例中，在统计意义上也有效。论文中提供的深入讨论涵盖了共形预测的理论基础，然后继续列出原始想法的更高级的发展和改编。

Virtually all machine learning tasks are characterized using some form of loss function, and "good performance" is typically stated in terms of a sufficiently small average loss, taken over the random draw of test data. While optimizing for performance on average is intuitive, convenient to analyze in theory, and easy to implement in practice, such a choice brings about trade-offs. In this work, we survey and introduce a wide variety of non-traditional criteria used to design and evaluate machine learning algorithms, place the classical paradigm within the proper historical context, and propose a view of learning problems which emphasizes the question of "what makes for a desirable loss distribution?" in place of tacit use of the expected loss.

The notion of uncertainty is of major importance in machine learning and constitutes a key element of machine learning methodology. In line with the statistical tradition, uncertainty has long been perceived as almost synonymous with standard probability and probabilistic predictions. Yet, due to the steadily increasing relevance of machine learning for practical applications and related issues such as safety requirements, new problems and challenges have recently been identified by machine learning scholars, and these problems may call for new methodological developments. In particular, this includes the importance of distinguishing between (at least) two different types of uncertainty, often referred to as aleatoric and epistemic. In this paper, we provide an introduction to the topic of uncertainty in machine learning as well as an overview of attempts so far at handling uncertainty in general and formalizing this distinction in particular.

This chapter is dedicated to the assessment and performance estimation of machine learning (ML) algorithms, a topic that is equally important to the construction of these algorithms, in particular in the context of cyberphysical security design. The literature is full of nonparametric methods to estimate a statistic from just one available dataset through resampling techniques, e.g., jackknife, bootstrap and cross validation (CV). Special statistics of great interest are the error rate and the area under the ROC curve (AUC) of a classification rule. The importance of these resampling methods stems from the fact that they require no knowledge about the probability distribution of the data or the construction details of the ML algorithm. This chapter provides a concise review of this literature to establish a coherent theoretical framework for these methods that can estimate both the error rate (a one-sample statistic) and the AUC (a two-sample statistic). The resampling methods are usually computationally expensive, because they rely on repeating the training and testing of a ML algorithm after each resampling iteration. Therefore, the practical applicability of some of these methods may be limited to the traditional ML algorithms rather than the very computationally demanding approaches of the recent deep neural networks (DNN). In the field of cyberphysical security, many applications generate structured (tabular) data, which can be fed to all traditional ML approaches. This is in contrast to the DNN approaches, which favor unstructured data, e.g., images, text, voice, etc.; hence, the relevance of this chapter to this field.%

Statistical learning is the process of estimating an unknown probabilistic input-output relationship of a system using a limited number of observations. A statistical learning machine (SLM) is the algorithm, function, model, or rule, that learns such a process; and machine learning (ML) is the conventional name of this field. ML and its applications are ubiquitous in the modern world. Systems such as Automatic target recognition (ATR) in military applications, computer aided diagnosis (CAD) in medical imaging, DNA microarrays in genomics, optical character recognition (OCR), speech recognition (SR), spam email filtering, stock market prediction, etc., are few examples and applications for ML; diverse fields but one theory. In particular, ML has gained a lot of attention in the field of cyberphysical security, especially in the last decade. It is of great importance to this field to design detection algorithms that have the capability of learning from security data to be able to hunt threats, achieve better monitoring, master the complexity of the threat intelligence feeds, and achieve timely remediation of security incidents. The field of ML can be decomposed into two basic subfields: \textit{construction} and \textit{assessment}. We mean by \textit{construction} designing or inventing an appropriate algorithm that learns from the input data and achieves a good performance according to some optimality criterion. We mean by \textit{assessment} attributing some performance measures to the constructed ML algorithm, along with their estimators, to objectively assess this algorithm. \textit{Construction} and \textit{assessment} of a ML algorithm require familiarity with different other fields: probability, statistics, matrix theory, optimization, algorithms, and programming, among others.f

当训练概率分类器和校准时，可以容易地忽略校准损耗的所谓分组损耗组件。分组损失是指观察信息与实际校准运动中的信息之间的差距。我们调查分组损失与充足概念之间的关系，将Conoonotonics识别为充足的有用标准。我们重新审视Langford＆Zadrozny（2005）的探测方法，发现它产生了减少分组损失的概率分类器的估计。最后，我们将Brier曲线讨论为支持培训的工具和“足够”概率分类器的校准。

Advocates of algorithmic techniques like data mining argue that these techniques eliminate human biases from the decision-making process. But an algorithm is only as good as the data it works with. Data is frequently imperfect in ways that allow these algorithms to inherit the prejudices of prior decision makers. In other cases, data may simply reflect the widespread biases that persist in society at large. In still others, data mining can discover surprisingly useful regularities that are really just preexisting patterns of exclusion and inequality. Unthinking reliance on data mining can deny historically disadvantaged and vulnerable groups full participation in society. Worse still, because the resulting discrimination is almost always an unintentional emergent property of the algorithm's use rather than a conscious choice by its programmers, it can be unusually hard to identify the source of the problem or to explain it to a court. This Essay examines these concerns through the lens of American antidiscrimination law-more particularly, through Title

接收器操作特性（ROC）曲线下的区域称为AUC，是监督学习域中众所周知的性能措施。由于其引人注目的功能，它已在许多研究中使用，以评估和比较不同分类器的性能。在这项工作中，我们在集群分析的背景下更具体地，我们将AUC作为无监督学习域中的性能措施。特别是，我们详细说明了AUC作为聚类质量的内部/相对测量，我们将其称为聚类曲线下的区域（AUCC）。我们表明给定候选聚类解决方案的AUCC在随机聚类解决方案的空模型下具有预期值，无论数据集的大小如何，更重要的是，无论在评估下的群集的数量或（IM）平衡如何。此外，我们详细阐述了在我们考虑的内部/相对聚类验证的背景下，AUCC实际上是Baker和Hubert（1975）的伽玛标准的线性转换，我们也正式得出了理论预期机会群集的价值。我们还讨论了这些标准的计算复杂性，并表明，对于集群分析的大多数真实应用，伽玛的普通实施可能是计算令人望而不容的，但对于大多数真实应用，其与AUCC的等价实际上推出了更有效的算法过程。我们的理论发现得到了实验结果的支持。这些结果表明，除了由AUCC提供的有效和稳健的定量评估之外，ROC曲线本身的目视检查对于进一步评估来自更广泛的，定性的透视的候选聚类解决方案也是有用的。

这项正在进行的工作旨在为统计学习提供统一的介绍，从诸如GMM和HMM等经典模型到现代神经网络（如VAE和扩散模型）缓慢地构建。如今，有许多互联网资源可以孤立地解释这一点或新的机器学习算法，但是它们并没有（也不能在如此简短的空间中）将这些算法彼此连接起来，或者与统计模型的经典文献相连现代算法出现了。同样明显缺乏的是一个单一的符号系统，尽管对那些已经熟悉材料的人（如这些帖子的作者）不满意，但对新手的入境造成了重大障碍。同样，我的目的是将各种模型（尽可能）吸收到一个用于推理和学习的框架上，表明（以及为什么）如何以最小的变化将一个模型更改为另一个模型（其中一些是新颖的，另一些是文献中的）。某些背景当然是必要的。我以为读者熟悉基本的多变量计算，概率和统计以及线性代数。这本书的目标当然不是​​完整性，而是从基本知识到过去十年中极强大的新模型的直线路径或多或少。然后，目标是补充而不是替换，诸如Bishop的\ emph {模式识别和机器学习}之类的综合文本，该文本现在已经15岁了。

Objectives: Discussions of fairness in criminal justice risk assessments typically lack conceptual precision. Rhetoric too often substitutes for careful analysis. In this paper, we seek to clarify the tradeoffs between different kinds of fairness and between fairness and accuracy.Methods: We draw on the existing literatures in criminology, computer science and statistics to provide an integrated examination of fairness and accuracy in criminal justice risk assessments. We also provide an empirical illustration using data from arraignments.Results: We show that there are at least six kinds of fairness, some of which are incompatible with one another and with accuracy.Conclusions: Except in trivial cases, it is impossible to maximize accuracy and fairness at the same time, and impossible simultaneously to satisfy all kinds of fairness. In practice, a major complication is different base rates across different legally protected groups. There is a need to consider challenging tradeoffs.

Label noise is an important issue in classification, with many potential negative consequences. For example, the accuracy of predictions may decrease, whereas the complexity of inferred models and the number of necessary training samples may increase. Many works in the literature have been devoted to the study of label noise and the development of techniques to deal with label noise. However, the field lacks a comprehensive survey on the different types of label noise, their consequences and the algorithms that consider label noise. This paper proposes to fill this gap. First, the definitions and sources of label noise are considered and a taxonomy of the types of label noise is proposed. Second, the potential consequences of label noise are discussed. Third, label noise-robust, label noise cleansing, and label noise-tolerant algorithms are reviewed. For each category of approaches, a short discussion is proposed to help the practitioner to choose the most suitable technique in its own particular field of application. Eventually, the design of experiments is also discussed, what may interest the researchers who would like to test their own algorithms. In this paper, label noise consists of mislabeled instances: no additional information is assumed to be available like e.g. confidences on labels.

保证案件旨在为其最高主张的真理提供合理的信心，这通常涉及安全或保障。那么一个自然的问题是，案件提供了“多少”信心？我们认为，置信度不能简化为单个属性或测量。取而代之的是，我们建议它应该基于以三种不同观点的属性为基础：正面，消极和残留疑问。积极的观点考虑了该案件的证据和总体论点结合起来的程度，以表明其主张的信念是正当的。我们为理由设置了一个高标准，要求它是不可行的。对此的主要积极度量是健全性，它将论点解释为逻辑证明。对证据的信心可以概率地表达，我们使用确认措施来确保证据的“权重”跨越了一定的阈值。此外，可以通过使用概率逻辑的参数步骤从证据中汇总概率，以产生我们所谓的索赔概率估值。负面观点记录了对案件的怀疑和挑战，通常表示为叛逆者及其探索和解决。保证开发商必须防止确认偏见，并应在制定案件时大力探索潜在的叛逆者，并应记录下来及其解决方案，以避免返工并帮助审阅者。残留疑问：世界不确定，因此并非所有潜在的叛逆者都可以解决。我们探索风险，可能认为它们是可以接受或不可避免的。但是，至关重要的是，这些判断是有意识的判断，并且在保证案例中记录下来。本报告详细介绍了这些观点，并指示了我们的保证2.0的原型工具集如何协助他们的评估。

机器学习通常以经典的概率理论为前提，这意味着聚集是基于期望的。现在有多种原因可以激励人们将经典概率理论作为机器学习的数学基础。我们系统地检查了一系列强大而丰富的此类替代品，即各种称为光谱风险度量，Choquet积分或Lorentz规范。我们提出了一系列的表征结果，并演示了使这个光谱家族如此特别的原因。在此过程中，我们证明了所有连贯的风险度量的自然分层，从它们通过利用重新安排不变性Banach空间理论的结果来诱导的上层概率。我们凭经验证明了这种新的不确定性方法如何有助于解决实用的机器学习问题。

While methods for comparing two learning algorithms on a single data set have been scrutinized for quite some time already, the issue of statistical tests for comparisons of more algorithms on multiple data sets, which is even more essential to typical machine learning studies, has been all but ignored. This article reviews the current practice and then theoretically and empirically examines several suitable tests. Based on that, we recommend a set of simple, yet safe and robust non-parametric tests for statistical comparisons of classifiers: the Wilcoxon signed ranks test for comparison of two classifiers and the Friedman test with the corresponding post-hoc tests for comparison of more classifiers over multiple data sets. Results of the latter can also be neatly presented with the newly introduced CD (critical difference) diagrams.

当可能的许多标签是可能的时，选择单个可以导致低精度。一个常见的替代方案，称为顶级k $分类，是选择一些数字$ k $（通常约5），并返回最高分数的$ k $标签。不幸的是，对于明确的案例，$ k> 1 $太多，对于非常暧昧的情况，$ k \ leq 5 $（例如）太小。另一种明智的策略是使用一种自适应方法，其中返回的标签数量随着计算的歧义而变化，但必须平均到所有样本的某些特定的$ k $。我们表示这种替代方案 - $ k $分类。本文在平均值的含量较低的误差率时，本文正式地表征了模糊性曲线，比固定的顶级k $分类更低。此外，它为固定尺寸和自适应分类器提供了自然估计程序，并证明了它们的一致性。最后，它报告了实际图像数据集的实验，揭示了平均值的效益 - 在实践中的价格超过高度k $分类。总的来说，当含糊不清的歧义时，平均值-$ k $永远不会比Top-$ K $更差，并且在我们的实验中，当估计时，这也持有。

Recent discussion in the public sphere about algorithmic classification has involved tension between competing notions of what it means for a probabilistic classification to be fair to different groups. We formalize three fairness conditions that lie at the heart of these debates, and we prove that except in highly constrained special cases, there is no method that can satisfy these three conditions simultaneously. Moreover, even satisfying all three conditions approximately requires that the data lie in an approximate version of one of the constrained special cases identified by our theorem. These results suggest some of the ways in which key notions of fairness are incompatible with each other, and hence provide a framework for thinking about the trade-offs between them.