The problem of comparing two different partitions of a finite set of objects reappears continually in the clustering literature. We begin by reviewing a well-known measure of partition correspondence often attributed to Rand (1971), discuss the issue of correcting this index for chance, and note that a recent normalization strategy developed by Morey and Agresti (1984) and adopted by others (e.g., Milligan and Cooper 1985) is based on an incorrect assumption. Then, the general problem of comparing partitions is approached indirectly by assessing the congruence of two proximity matrices using a simple cross-product measure. They are generated from cOrresponding partitions using various scoring rules. Special cases derivable include traditionally familiar statistics and/or ones tailored to weight certain object pairs differentially. Finally, we propose a measure based on the comparison of object triples having the advantage of a probabilistic interpretation in addition to being corrected for chance (i.e., assuming a constant value under a reasonable null hypothesis) and bounded between -4-I.

接收器操作特性（ROC）曲线下的区域称为AUC，是监督学习域中众所周知的性能措施。由于其引人注目的功能，它已在许多研究中使用，以评估和比较不同分类器的性能。在这项工作中，我们在集群分析的背景下更具体地，我们将AUC作为无监督学习域中的性能措施。特别是，我们详细说明了AUC作为聚类质量的内部/相对测量，我们将其称为聚类曲线下的区域（AUCC）。我们表明给定候选聚类解决方案的AUCC在随机聚类解决方案的空模型下具有预期值，无论数据集的大小如何，更重要的是，无论在评估下的群集的数量或（IM）平衡如何。此外，我们详细阐述了在我们考虑的内部/相对聚类验证的背景下，AUCC实际上是Baker和Hubert（1975）的伽玛标准的线性转换，我们也正式得出了理论预期机会群集的价值。我们还讨论了这些标准的计算复杂性，并表明，对于集群分析的大多数真实应用，伽玛的普通实施可能是计算令人望而不容的，但对于大多数真实应用，其与AUCC的等价实际上推出了更有效的算法过程。我们的理论发现得到了实验结果的支持。这些结果表明，除了由AUCC提供的有效和稳健的定量评估之外，ROC曲线本身的目视检查对于进一步评估来自更广泛的，定性的透视的候选聚类解决方案也是有用的。

统计推断中的主要范式取决于I.I.D.的结构。来自假设的无限人群的数据。尽管它取得了成功，但在复杂的数据结构下，即使在清楚无限人口所代表的内容的情况下，该框架在复杂的数据结构下仍然不灵活。在本文中，我们探讨了一个替代框架，在该框架中，推断只是对模型误差的不变性假设，例如交换性或符号对称性。作为解决这个不变推理问题的一般方法，我们提出了一个基于随机的过程。我们证明了该过程的渐近有效性的一般条件，并在许多数据结构中说明了，包括单向和双向布局中的群集误差。我们发现，通过残差随机化的不变推断具有三个吸引人的属性：（1）在弱且可解释的条件下是有效的，可以解决重型数据，有限聚类甚至一些高维设置的问题。 （2）它在有限样品中是可靠的，因为它不依赖经典渐近学所需的规律性条件。 （3）它以适应数据结构的统一方式解决了推断问题。另一方面，诸如OLS或Bootstrap之类的经典程序以I.I.D.为前提。结构，只要实际问题结构不同，就需要修改。经典框架中的这种不匹配导致了多种可靠的误差技术和自举变体，这些变体经常混淆应用研究。我们通过广泛的经验评估证实了这些发现。残留随机化对许多替代方案的表现有利，包括可靠的误差方法，自举变体和分层模型。

没有，也不会有单一的最佳聚类算法，但是我们仍然希望能够确定那些在某些任务类型上表现出色并过滤掉系统令人失望的人。传统上，使用内部或外部有效性度量评估聚类算法。内部度量量化了所获得的分区的不同方面，例如簇紧凑性或点可分离性的平均程度。然而，他们的有效性是值得怀疑的，因为他们促进的聚类有时可能毫无意义。另一方面，外部措施将算法的输出与专家提供的基础真相分组进行了比较。常规的经典分区相似性分数，例如归一化的互信息，福克斯 - 马洛或调整后的兰德指数，可能没有所有期望的特性，例如，它们无法正确识别病理边缘病例。此外，它们不能很好地解释：很难说出0.8的分数。它的行为也可能随着真实簇的数量的变化而有所不同。这使得在许多基准数据集中比较聚类算法变得困难。为了解决这个问题，我们提出并分析了一种新措施：最佳设置匹配精度的不对称版本。它可以纠正机会和集群大小的不平衡性。

我们提出了对学度校正随机块模型（DCSBM）的合适性测试。该测试基于调整后的卡方统计量，用于测量$ n $多项式分布的组之间的平等性，该分布具有$ d_1，\ dots，d_n $观测值。在网络模型的背景下，多项式的数量（$ n $）的数量比观测值数量（$ d_i $）快得多，与节点$ i $的度相对应，因此设置偏离了经典的渐近学。我们表明，只要$ \ {d_i \} $的谐波平均值生长到无穷大，就可以使统计量在NULL下分配。顺序应用时，该测试也可以用于确定社区数量。该测试在邻接矩阵的压缩版本上进行操作，因此在学位上有条件，因此对大型稀疏网络具有高度可扩展性。我们结合了一个新颖的想法，即在测试$ K $社区时根据$（k+1）$ - 社区分配来压缩行。这种方法在不牺牲计算效率的情况下增加了顺序应用中的力量，我们证明了它在恢复社区数量方面的一致性。由于测试统计量不依赖于特定的替代方案，因此其效用超出了顺序测试，可用于同时测试DCSBM家族以外的各种替代方案。特别是，我们证明该测试与具有社区结构的潜在可变性网络模型的一般家庭一致。

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact

Testing the significance of a variable or group of variables $X$ for predicting a response $Y$, given additional covariates $Z$, is a ubiquitous task in statistics. A simple but common approach is to specify a linear model, and then test whether the regression coefficient for $X$ is non-zero. However, when the model is misspecified, the test may have poor power, for example when $X$ is involved in complex interactions, or lead to many false rejections. In this work we study the problem of testing the model-free null of conditional mean independence, i.e. that the conditional mean of $Y$ given $X$ and $Z$ does not depend on $X$. We propose a simple and general framework that can leverage flexible nonparametric or machine learning methods, such as additive models or random forests, to yield both robust error control and high power. The procedure involves using these methods to perform regressions, first to estimate a form of projection of $Y$ on $X$ and $Z$ using one half of the data, and then to estimate the expected conditional covariance between this projection and $Y$ on the remaining half of the data. While the approach is general, we show that a version of our procedure using spline regression achieves what we show is the minimax optimal rate in this nonparametric testing problem. Numerical experiments demonstrate the effectiveness of our approach both in terms of maintaining Type I error control, and power, compared to several existing approaches.

Classical asymptotic theory for statistical inference usually involves calibrating a statistic by fixing the dimension $d$ while letting the sample size $n$ increase to infinity. Recently, much effort has been dedicated towards understanding how these methods behave in high-dimensional settings, where $d$ and $n$ both increase to infinity together. This often leads to different inference procedures, depending on the assumptions about the dimensionality, leaving the practitioner in a bind: given a dataset with 100 samples in 20 dimensions, should they calibrate by assuming $n \gg d$, or $d/n \approx 0.2$? This paper considers the goal of dimension-agnostic inference; developing methods whose validity does not depend on any assumption on $d$ versus $n$. We introduce an approach that uses variational representations of existing test statistics along with sample splitting and self-normalization to produce a new test statistic with a Gaussian limiting distribution, regardless of how $d$ scales with $n$. The resulting statistic can be viewed as a careful modification of degenerate U-statistics, dropping diagonal blocks and retaining off-diagonal blocks. We exemplify our technique for some classical problems including one-sample mean and covariance testing, and show that our tests have minimax rate-optimal power against appropriate local alternatives. In most settings, our cross U-statistic matches the high-dimensional power of the corresponding (degenerate) U-statistic up to a $\sqrt{2}$ factor.

Low-rank matrix approximations, such as the truncated singular value decomposition and the rank-revealing QR decomposition, play a central role in data analysis and scientific computing. This work surveys and extends recent research which demonstrates that randomization offers a powerful tool for performing low-rank matrix approximation. These techniques exploit modern computational architectures more fully than classical methods and open the possibility of dealing with truly massive data sets.This paper presents a modular framework for constructing randomized algorithms that compute partial matrix decompositions. These methods use random sampling to identify a subspace that captures most of the action of a matrix. The input matrix is then compressed-either explicitly or implicitly-to this subspace, and the reduced matrix is manipulated deterministically to obtain the desired low-rank factorization. In many cases, this approach beats its classical competitors in terms of accuracy, speed, and robustness. These claims are supported by extensive numerical experiments and a detailed error analysis.The specific benefits of randomized techniques depend on the computational environment. Consider the model problem of finding the k dominant components of the singular value decomposition of an m × n matrix. (i) For a dense input matrix, randomized algorithms require O(mn log(k)) floating-point operations (flops) in contrast with O(mnk) for classical algorithms. (ii) For a sparse input matrix, the flop count matches classical Krylov subspace methods, but the randomized approach is more robust and can easily be reorganized to exploit multi-processor architectures. (iii) For a matrix that is too large to fit in fast memory, the randomized techniques require only a constant number of passes over the data, as opposed to O(k) passes for classical algorithms. In fact, it is sometimes possible to perform matrix approximation with a single pass over the data.

由于其出色的经验表现，随机森林是过去十年中使用的机器学习方法之一。然而，由于其黑框的性质，在许多大数据应用中很难解释随机森林的结果。量化各个特征在随机森林中的实用性可以大大增强其解释性。现有的研究表明，一些普遍使用的特征对随机森林的重要性措施遭受了偏见问题。此外，对于大多数现有方法，缺乏全面的规模和功率分析。在本文中，我们通过假设检验解决了问题，并提出了一个自由化特征 - 弥散性相关测试（事实）的框架，以评估具有偏见性属性的随机森林模型中给定特征的重要性，我们零假设涉及该特征是否与所有其他特征有条件地独立于响应。关于高维随机森林一致性的一些最新发展，对随机森林推断的这种努力得到了赋予的能力。在存在功能依赖性的情况下，我们的事实测试的香草版可能会遇到偏见问题。我们利用偏置校正的不平衡和调节技术。我们通过增强功率的功能转换将合奏的想法进一步纳入事实统计范围。在相当普遍的具有依赖特征的高维非参数模型设置下，我们正式确定事实可以提供理论上合理的随机森林具有P值，并通过非催化分析享受吸引人的力量。新建议的方法的理论结果和有限样本优势通过几个模拟示例和与Covid-19的经济预测应用进行了说明。

We consider a problem of considerable practical interest: the recovery of a data matrix from a sampling of its entries. Suppose that we observe m entries selected uniformly at random from a matrix M . Can we complete the matrix and recover the entries that we have not seen?We show that one can perfectly recover most low-rank matrices from what appears to be an incomplete set of entries. We prove that if the number m of sampled entries obeys m ≥ C n 1.2 r log n for some positive numerical constant C, then with very high probability, most n × n matrices of rank r can be perfectly recovered by solving a simple convex optimization program. This program finds the matrix with minimum nuclear norm that fits the data. The condition above assumes that the rank is not too large. However, if one replaces the 1.2 exponent with 1.25, then the result holds for all values of the rank. Similar results hold for arbitrary rectangular matrices as well. Our results are connected with the recent literature on compressed sensing, and show that objects other than signals and images can be perfectly reconstructed from very limited information.

这项正在进行的工作旨在为统计学习提供统一的介绍，从诸如GMM和HMM等经典模型到现代神经网络（如VAE和扩散模型）缓慢地构建。如今，有许多互联网资源可以孤立地解释这一点或新的机器学习算法，但是它们并没有（也不能在如此简短的空间中）将这些算法彼此连接起来，或者与统计模型的经典文献相连现代算法出现了。同样明显缺乏的是一个单一的符号系统，尽管对那些已经熟悉材料的人（如这些帖子的作者）不满意，但对新手的入境造成了重大障碍。同样，我的目的是将各种模型（尽可能）吸收到一个用于推理和学习的框架上，表明（以及为什么）如何以最小的变化将一个模型更改为另一个模型（其中一些是新颖的，另一些是文献中的）。某些背景当然是必要的。我以为读者熟悉基本的多变量计算，概率和统计以及线性代数。这本书的目标当然不是​​完整性，而是从基本知识到过去十年中极强大的新模型的直线路径或多或少。然后，目标是补充而不是替换，诸如Bishop的\ emph {模式识别和机器学习}之类的综合文本，该文本现在已经15岁了。

尽管在机器学习的方法论核心中是一个问题，但如何比较分类器仍未达成一致的共识。每个比较框架都面临着（至少）三个基本挑战：质量标准的多样性，数据集的多样性以及选择数据集选择的随机性/任意性。在本文中，我们通过采用决策理论的最新发展，为生动的辩论增添了新的观点。我们最终的框架基于所谓的偏好系统，通过广义的随机优势概念对分类器进行排名，该概念强大地绕过了繁琐的，甚至通常是自相矛盾的，对聚合的依赖。此外，我们表明，可以通过解决易于手柄的线性程序和通过适应的两样本观察随机化测试进行统计测试来实现广泛的随机优势。这确实产生了一个有力的框架，可以同时相对于多个质量标准进行分类器的统计比较。我们在模拟研究和标准基准数据集中说明和研究我们的框架。

本文涉及使用多项式的有限样品的平滑，高维函数的近似。这项任务是计算科学和工程中许多应用的核心 - 尤其是由参数建模和不确定性量化引起的。通常在此类应用中使用蒙特卡洛（MC）采样，以免屈服于维度的诅咒。但是，众所周知，这种策略在理论上是最佳的。尺寸$ n $有许多多项式空间，样品复杂度尺度划分为$ n $。这种有据可查的现象导致了一致的努力，以设计改进的，实际上是近乎最佳的策略，其样本复杂性是线性的，甚至线性地缩小了$ n $。自相矛盾的是，在这项工作中，我们表明MC实际上是高维度中的一个非常好的策略。我们首先通过几个数值示例记录了这种现象。接下来，我们提出一个理论分析，该分析能够解决这种悖论，以实现无限多变量的全体形态功能。我们表明，基于$ M $ MC样本的最小二乘方案，其错误衰减为$ m/\ log（m）$，其速率与最佳$ n $ term的速率相同多项式近似。该结果是非构造性的，因为它假定了进行近似的合适多项式空间的知识。接下来，我们提出了一个基于压缩感应的方案，该方案达到了相同的速率，除了较大的聚类因子。该方案是实用的，并且在数值上，它的性能和比知名的自适应最小二乘方案的性能和更好。总体而言，我们的发现表明，当尺寸足够高时，MC采样非常适合平滑功能近似。因此，改进的采样策略的好处通常仅限于较低维度的设置。

群集分析需要许多决定：聚类方法和隐含的参考模型，群集数，通常，几个超参数和算法调整。在实践中，一个分区产生多个分区，基于验证或选择标准选择最终的分区。存在丰富的验证方法，即隐式或明确地假设某个聚类概念。此外，它们通常仅限于从特定方法获得的分区上操作。在本文中，我们专注于可以通过二次或线性边界分开的群体。参考集群概念通过二次判别符号函数和描述集群大小，中心和分散的参数定义。我们开发了两个名为二次分数的群集质量标准。我们表明这些标准与从一般类椭圆对称分布产生的组一致。对这种类型的组追求在应用程序中是常见的。研究了与混合模型和模型的聚类的似然理论的连接。基于Bootstrap重新采样的二次分数，我们提出了一个选择规则，允许在许多聚类解决方案中选择。所提出的方法具有独特的优点，即它可以比较不能与其他最先进的方法进行比较的分区。广泛的数值实验和实际数据的分析表明，即使某些竞争方法在某些设置中出现优越，所提出的方法也实现了更好的整体性能。

机器学习通常以经典的概率理论为前提，这意味着聚集是基于期望的。现在有多种原因可以激励人们将经典概率理论作为机器学习的数学基础。我们系统地检查了一系列强大而丰富的此类替代品，即各种称为光谱风险度量，Choquet积分或Lorentz规范。我们提出了一系列的表征结果，并演示了使这个光谱家族如此特别的原因。在此过程中，我们证明了所有连贯的风险度量的自然分层，从它们通过利用重新安排不变性Banach空间理论的结果来诱导的上层概率。我们凭经验证明了这种新的不确定性方法如何有助于解决实用的机器学习问题。

马尔可夫链是一类概率模型，在定量科学中已广泛应用。这部分是由于它们的多功能性，但是可以通过分析探测的便利性使其更加复杂。本教程为马尔可夫连锁店提供了深入的介绍，并探索了它们与图形和随机步行的联系。我们利用从线性代数和图形论的工具来描述不同类型的马尔可夫链的过渡矩阵，特别着眼于探索与这些矩阵相对应的特征值和特征向量的属性。提出的结果与机器学习和数据挖掘中的许多方法有关，我们在各个阶段描述了这些方法。本文并没有本身就成为一项新颖的学术研究，而是提出了一些已知结果的集合以及一些新概念。此外，该教程的重点是向读者提供直觉，而不是正式的理解，并且仅假定对线性代数和概率理论的概念的基本曝光。因此，来自各种学科的学生和研究人员可以访问它。

or if the system is intended to service several individual channels in multiplex; (e) Several functions of several variables --in color television the message consists of three functions f (x,y, t), g(x, y, t), h(x, y, t) defined in a three-dimensional continuum --we may also think of these three functions as components of a vector field defined in the region --similarly, several black and white television sources would produce "messages" consisting of a number of functions of three variables; (f) Various combinations also occur, for example in television with an associated audio channel.. A transmitter which operates on the message in some way to produce a signal suitable for transmission over the channel. In telephony this operation consists merely of changing sound pressure into a proportional electrical current. In telegraphy we have an encoding operation which produces a sequence of dots, dashes and spaces on the channel corresponding to the message. In a multiplex PCM system the different speech functions must be sampled, compressed, quantized and encoded, and finally interleaved properly to construct the signal. Vocoder systems, television and frequency modulation are other examples of complex operations applied to the message to obtain the signal.3. The channel is merely the medium used to transmit the signal from transmitter to receiver. It may be a pair of wires, a coaxial cable, a band of radio frequencies, a beam of light, etc. 4. The receiver ordinarily performs the inverse operation of that done by the transmitter, reconstructing the message from the signal.5. The destination is the person (or thing) for whom the message is intended.We wish to consider certain general problems involving communication systems. To do this it is first necessary to represent the various elements involved as mathematical entities, suitably idealized from their physical counterparts. We may roughly classify communication systems into three main categories: discrete, continuous and mixed. By a discrete system we will mean one in which both the message and the signal are a sequence of discrete symbols. A typical case is telegraphy where the message is a sequence of letters and the signal a sequence of dots, dashes and spaces. A continuous system is one in which the message and signal are both treated

In nonparametric independence testing, we observe i.i.d.\ data $\{(X_i,Y_i)\}_{i=1}^n$, where $X \in \mathcal{X}, Y \in \mathcal{Y}$ lie in any general spaces, and we wish to test the null that $X$ is independent of $Y$. Modern test statistics such as the kernel Hilbert-Schmidt Independence Criterion (HSIC) and Distance Covariance (dCov) have intractable null distributions due to the degeneracy of the underlying U-statistics. Thus, in practice, one often resorts to using permutation testing, which provides a nonasymptotic guarantee at the expense of recalculating the quadratic-time statistics (say) a few hundred times. This paper provides a simple but nontrivial modification of HSIC and dCov (called xHSIC and xdCov, pronounced ``cross'' HSIC/dCov) so that they have a limiting Gaussian distribution under the null, and thus do not require permutations. This requires building on the newly developed theory of cross U-statistics by Kim and Ramdas (2020), and in particular developing several nontrivial extensions of the theory in Shekhar et al. (2022), which developed an analogous permutation-free kernel two-sample test. We show that our new tests, like the originals, are consistent against fixed alternatives, and minimax rate optimal against smooth local alternatives. Numerical simulations demonstrate that compared to the full dCov or HSIC, our variants have the same power up to a $\sqrt 2$ factor, giving practitioners a new option for large problems or data-analysis pipelines where computation, not sample size, could be the bottleneck.

社区检测是网络科学中最重要的方法领域之一，在过去的几十年里引起了大量关注的方法之一。该区域处理网络的自动部门到基础构建块中，目的是提供其大规模结构的概要。尽管它的重要性和广泛的采用普及，所谓的最先进和实际在各种领域实际使用的方法之间存在明显的差距。在这里，我们试图通过根据是否具有“描述性”或“推论”目标来划分现有方法来解决这种差异。虽然描述性方法在基于社区结构的直观概念的网络中找到模式的模式，但是推理方法阐述了精确的生成模型，并尝试将其符合数据。通过这种方式，他们能够为网络形成机制提供见解，并以统计证据支持的方式与随机性的单独结构。我们审查如何使用推论目标采用描述性方法被陷入困境和误导性答案，因此应该一般而言。我们认为推理方法更通常与更清晰的科学问题一致，产生更强大的结果，并且应该是一般的首选。我们试图消除一些神话和半真半假在实践中使用社区检测时，努力改善这些方法的使用以及对结果的解释。