我们研究通过应用具有多个初始化的梯度上升方法来源的估计器的统计特性。我们派生了该估算器的目标的人口数量，并研究了从渐近正常性和自举方法构成的置信区间（CIS）的性质。特别是，我们通过有限数量的随机初始化来分析覆盖范围。我们还通过反转可能性比率测试，得分测试和WALD测试来调查CI，我们表明所得到的CIS可能非常不同。即使MLE是棘手的，我们也提出了一种两个样本测试程序。此外，我们在随机初始化下分析了EM算法的性能，并通过有限数量的初始化导出了CI的覆盖范围。

Classical asymptotic theory for statistical inference usually involves calibrating a statistic by fixing the dimension $d$ while letting the sample size $n$ increase to infinity. Recently, much effort has been dedicated towards understanding how these methods behave in high-dimensional settings, where $d$ and $n$ both increase to infinity together. This often leads to different inference procedures, depending on the assumptions about the dimensionality, leaving the practitioner in a bind: given a dataset with 100 samples in 20 dimensions, should they calibrate by assuming $n \gg d$, or $d/n \approx 0.2$? This paper considers the goal of dimension-agnostic inference; developing methods whose validity does not depend on any assumption on $d$ versus $n$. We introduce an approach that uses variational representations of existing test statistics along with sample splitting and self-normalization to produce a new test statistic with a Gaussian limiting distribution, regardless of how $d$ scales with $n$. The resulting statistic can be viewed as a careful modification of degenerate U-statistics, dropping diagonal blocks and retaining off-diagonal blocks. We exemplify our technique for some classical problems including one-sample mean and covariance testing, and show that our tests have minimax rate-optimal power against appropriate local alternatives. In most settings, our cross U-statistic matches the high-dimensional power of the corresponding (degenerate) U-statistic up to a $\sqrt{2}$ factor.

套索是一种高维回归的方法，当时，当协变量$ p $的订单数量或大于观测值$ n $时，通常使用它。由于两个基本原因，经典的渐近态性理论不适用于该模型：$（1）$正规风险是非平滑的； $（2）$估算器$ \ wideHat {\ boldsymbol {\ theta}} $与true参数vector $ \ boldsymbol {\ theta}^*$无法忽略。结果，标准的扰动论点是渐近正态性的传统基础。另一方面，套索估计器可以精确地以$ n $和$ p $大，$ n/p $的订单为一。这种表征首先是在使用I.I.D的高斯设计的情况下获得的。协变量：在这里，我们将其推广到具有非偏差协方差结构的高斯相关设计。这是根据更简单的``固定设计''模型表示的。我们在两个模型中各种数量的分布之间的距离上建立了非反应界限，它们在合适的稀疏类别中均匀地固定在信号上$ \ boldsymbol {\ theta}^*$。作为应用程序，我们研究了借助拉索的分布，并表明需要校正程度对于计算有效的置信区间是必要的。

This paper revisits a fundamental problem in statistical inference from a non-asymptotic theoretical viewpoint $\unicode{x2013}$ the construction of confidence sets. We establish a finite-sample bound for the estimator, characterizing its asymptotic behavior in a non-asymptotic fashion. An important feature of our bound is that its dimension dependency is captured by the effective dimension $\unicode{x2013}$ the trace of the limiting sandwich covariance $\unicode{x2013}$ which can be much smaller than the parameter dimension in some regimes. We then illustrate how the bound can be used to obtain a confidence set whose shape is adapted to the optimization landscape induced by the loss function. Unlike previous works that rely heavily on the strong convexity of the loss function, we only assume the Hessian is lower bounded at optimum and allow it to gradually becomes degenerate. This property is formalized by the notion of generalized self-concordance which originated from convex optimization. Moreover, we demonstrate how the effective dimension can be estimated from data and characterize its estimation accuracy. We apply our results to maximum likelihood estimation with generalized linear models, score matching with exponential families, and hypothesis testing with Rao's score test.

我们提出了一种基于优化的基于优化的框架，用于计算差异私有M估算器以及构建差分私立置信区的新方法。首先，我们表明稳健的统计数据可以与嘈杂的梯度下降或嘈杂的牛顿方法结合使用，以便分别获得具有全局线性或二次收敛的最佳私人估算。我们在局部强大的凸起和自我协调下建立当地和全球融合保障，表明我们的私人估算变为对非私人M估计的几乎最佳附近的高概率。其次，我们通过构建我们私有M估计的渐近方差的差异私有估算来解决参数化推断的问题。这自然导致近​​似枢轴统计，用于构建置信区并进行假设检测。我们展示了偏置校正的有效性，以提高模拟中的小样本实证性能。我们说明了我们在若干数值例子中的方法的好处。

我们解决了如何在没有严格缩放条件的情况下实现分布式分数回归中最佳推断的问题。由于分位数回归（QR）损失函数的非平滑性质，这是具有挑战性的，这使现有方法的使用无效。难度通过应用于本地（每个数据源）和全局目标函数的双光滑方法解决。尽管依赖局部和全球平滑参数的精致组合，但分位数回归模型是完全参数的，从而促进了解释。在低维度中，我们为顺序定义的分布式QR估计器建立了有限样本的理论框架。这揭示了通信成本和统计错误之间的权衡。我们进一步讨论并比较了基于WALD和得分型测试和重采样技术的反转的几种替代置信集结构，并详细介绍了对更极端分数系数有效的改进。在高维度中，采用了一个稀疏的框架，其中提出的双滑目标功能与$ \ ell_1 $ -penalty相辅相成。我们表明，相应的分布式QR估计器在近乎恒定的通信回合之后达到了全球收敛率。一项彻底的模拟研究进一步阐明了我们的发现。

变性推理（VI）为基于传统的采样方法提供了一种吸引人的替代方法，用于实施贝叶斯推断，因为其概念性的简单性，统计准确性和计算可扩展性。然而，常见的变分近似方案（例如平均场（MF）近似）需要某些共轭结构以促进有效的计算，这可能会增加不必要的限制对可行的先验分布家族，并对变异近似族对差异进行进一步的限制。在这项工作中，我们开发了一个通用计算框架，用于实施MF-VI VIA WASSERSTEIN梯度流（WGF），这是概率度量空间上的梯度流。当专门针对贝叶斯潜在变量模型时，我们将分析基于时间消化的WGF交替最小化方案的算法收敛，用于实现MF近似。特别是，所提出的算法类似于EM算法的分布版本，包括更新潜在变量变异分布的E step以及在参数的变异分布上进行最陡峭下降的m step。我们的理论分析依赖于概率度量空间中的最佳运输理论和细分微积分。我们证明了时间限制的WGF的指数收敛性，以最大程度地减少普通大地测量学严格的凸度的通用物镜功能。我们还提供了通过使用时间限制的WGF的固定点方程从MF近似获得的变异分布的指数收缩的新证明。我们将方法和理论应用于两个经典的贝叶斯潜在变量模型，即高斯混合模型和回归模型的混合物。还进行了数值实验，以补充这两个模型下的理论发现。

随机梯度算法在大规模学习和推理问题中广泛用于优化和采样。但是，实际上，调整这些算法通常是使用启发式和反复试验而不是严格的，可概括的理论来完成的。为了解决理论和实践之间的这一差距，我们通过表征具有固定步长的非常通用的预处理随机梯度算法的迭代术的大样本行为来对调整参数的效果进行新的见解。在优化设置中，我们的结果表明，具有较大固定步长的迭代平均值可能会导致（局部）M-静态器的统计效率近似。在抽样环境中，我们的结果表明，通过适当的调整参数选择，限制固定协方差可以与Bernstein匹配 - 后验的von Mises限制，对模型错误指定后验的调整或MLE的渐近分布；而幼稚的调整极限与这些都不相对应。此外，我们认为可以在数据集对固定数量的通行证后获得基本独立的样本。我们使用模拟和真实数据通过多个实验来验证渐近样结果。总体而言，我们证明具有恒定步长的正确调整的随机梯度算法为获得点估计或后部样品提供了计算上有效且统计上健壮的方法。

Testing the significance of a variable or group of variables $X$ for predicting a response $Y$, given additional covariates $Z$, is a ubiquitous task in statistics. A simple but common approach is to specify a linear model, and then test whether the regression coefficient for $X$ is non-zero. However, when the model is misspecified, the test may have poor power, for example when $X$ is involved in complex interactions, or lead to many false rejections. In this work we study the problem of testing the model-free null of conditional mean independence, i.e. that the conditional mean of $Y$ given $X$ and $Z$ does not depend on $X$. We propose a simple and general framework that can leverage flexible nonparametric or machine learning methods, such as additive models or random forests, to yield both robust error control and high power. The procedure involves using these methods to perform regressions, first to estimate a form of projection of $Y$ on $X$ and $Z$ using one half of the data, and then to estimate the expected conditional covariance between this projection and $Y$ on the remaining half of the data. While the approach is general, we show that a version of our procedure using spline regression achieves what we show is the minimax optimal rate in this nonparametric testing problem. Numerical experiments demonstrate the effectiveness of our approach both in terms of maintaining Type I error control, and power, compared to several existing approaches.

Integrative analysis of data from multiple sources is critical to making generalizable discoveries. Associations that are consistently observed across multiple source populations are more likely to be generalized to target populations with possible distributional shifts. In this paper, we model the heterogeneous multi-source data with multiple high-dimensional regressions and make inferences for the maximin effect (Meinshausen, B{\"u}hlmann, AoS, 43(4), 1801--1830). The maximin effect provides a measure of stable associations across multi-source data. A significant maximin effect indicates that a variable has commonly shared effects across multiple source populations, and these shared effects may be generalized to a broader set of target populations. There are challenges associated with inferring maximin effects because its point estimator can have a non-standard limiting distribution. We devise a novel sampling method to construct valid confidence intervals for maximin effects. The proposed confidence interval attains a parametric length. This sampling procedure and the related theoretical analysis are of independent interest for solving other non-standard inference problems. Using genetic data on yeast growth in multiple environments, we demonstrate that the genetic variants with significant maximin effects have generalizable effects under new environments.

生成对抗网络（GAN）在数据生成方面取得了巨大成功。但是，其统计特性尚未完全理解。在本文中，我们考虑了GAN的一般$ f $ divergence公式的统计行为，其中包括Kullback- Leibler Divergence与最大似然原理密切相关。我们表明，对于正确指定的参数生成模型，在适当的规律性条件下，所有具有相同歧视类别类别的$ f $ divergence gans均在渐近上等效。 Moreover, with an appropriately chosen local discriminator, they become equivalent to the maximum likelihood estimate asymptotically.对于被误解的生成模型，具有不同$ f $ -Divergences {收敛到不同估计器}的gan，因此无法直接比较。但是，结果表明，对于某些常用的$ f $ -Diverences，原始的$ f $ gan并不是最佳的，因为当更换原始$ f $ gan配方中的判别器培训时，可以实现较小的渐近方差通过逻辑回归。结果估计方法称为对抗梯度估计（年龄）。提供了实证研究来支持该理论，并证明了年龄的优势，而不是模型错误的原始$ f $ gans。

近年来目睹了采用灵活的机械学习模型进行乐器变量（IV）回归的兴趣，但仍然缺乏不确定性量化方法的发展。在这项工作中，我们为IV次数回归提出了一种新的Quasi-Bayesian程序，建立了最近开发的核化IV模型和IV回归的双/极小配方。我们通过在$ l_2 $和sobolev规范中建立最低限度的最佳收缩率，并讨论可信球的常见有效性来分析所提出的方法的频繁行为。我们进一步推出了一种可扩展的推理算法，可以扩展到与宽神经网络模型一起工作。实证评价表明，我们的方法对复杂的高维问题产生了丰富的不确定性估计。

Deep generative models parametrized up to a normalizing constant (e.g. energy-based models) are difficult to train by maximizing the likelihood of the data because the likelihood and/or gradients thereof cannot be explicitly or efficiently written down. Score matching is a training method, whereby instead of fitting the likelihood $\log p(x)$ for the training data, we instead fit the score function $\nabla_x \log p(x)$ -- obviating the need to evaluate the partition function. Though this estimator is known to be consistent, its unclear whether (and when) its statistical efficiency is comparable to that of maximum likelihood -- which is known to be (asymptotically) optimal. We initiate this line of inquiry in this paper, and show a tight connection between statistical efficiency of score matching and the isoperimetric properties of the distribution being estimated -- i.e. the Poincar\'e, log-Sobolev and isoperimetric constant -- quantities which govern the mixing time of Markov processes like Langevin dynamics. Roughly, we show that the score matching estimator is statistically comparable to the maximum likelihood when the distribution has a small isoperimetric constant. Conversely, if the distribution has a large isoperimetric constant -- even for simple families of distributions like exponential families with rich enough sufficient statistics -- score matching will be substantially less efficient than maximum likelihood. We suitably formalize these results both in the finite sample regime, and in the asymptotic regime. Finally, we identify a direct parallel in the discrete setting, where we connect the statistical properties of pseudolikelihood estimation with approximate tensorization of entropy and the Glauber dynamics.

预测一组结果 - 而不是独特的结果 - 是统计学习中不确定性定量的有前途的解决方案。尽管有关于构建具有统计保证的预测集的丰富文献，但适应未知的协变量转变（实践中普遍存在的问题）还是一个严重的未解决的挑战。在本文中，我们表明具有有限样本覆盖范围保证的预测集是非信息性的，并提出了一种新型的无灵活分配方法PredSet-1Step，以有效地构建了在未知协方差转移下具有渐近覆盖范围保证的预测集。我们正式表明我们的方法是\ textIt {渐近上可能是近似正确}，对大型样本的置信度有很好的覆盖误差。我们说明，在南非队列研究中，它在许多实验和有关HIV风险预测的数据集中实现了名义覆盖范围。我们的理论取决于基于一般渐近线性估计器的WALD置信区间覆盖范围的融合率的新结合。

bump狩猎与样本空间中的发现有意义的数据子集，称为颠簸。这些传统上被认为是基础密度函数图中的模态或凹区域。我们根据概率密度的曲率功能定义抽象的凸起构建体。然后，我们探讨了涉及衍生物最高到二阶的几种替代特征。特别是，在多元案例中提出了适当的善良和加斯金斯原始凹凸凹凸的实施。此外，我们将探索性数据分析概念（如平均曲率和拉普拉斯人）在应用域中产生良好结果。我们的方法可以通过插件内核密度估计器来解决曲率功能的近似。我们提供了理论上的结果，以确保在Hausdorff距离内的凸界边界的渐近一致性，并具有负担得起的收敛速度。我们还提出了渐近有效且一致的置信区域边界曲率凸起。该理论通过NBA，MLB和NFL的数据集的体育分析中的几种用例来说明。我们得出的结论是，不同的曲率实例有效地结合了以产生洞察力的可视化。

In the classical setting of self-selection, the goal is to learn $k$ models, simultaneously from observations $(x^{(i)}, y^{(i)})$ where $y^{(i)}$ is the output of one of $k$ underlying models on input $x^{(i)}$. In contrast to mixture models, where we observe the output of a randomly selected model, here the observed model depends on the outputs themselves, and is determined by some known selection criterion. For example, we might observe the highest output, the smallest output, or the median output of the $k$ models. In known-index self-selection, the identity of the observed model output is observable; in unknown-index self-selection, it is not. Self-selection has a long history in Econometrics and applications in various theoretical and applied fields, including treatment effect estimation, imitation learning, learning from strategically reported data, and learning from markets at disequilibrium. In this work, we present the first computationally and statistically efficient estimation algorithms for the most standard setting of this problem where the models are linear. In the known-index case, we require poly$(1/\varepsilon, k, d)$ sample and time complexity to estimate all model parameters to accuracy $\varepsilon$ in $d$ dimensions, and can accommodate quite general selection criteria. In the more challenging unknown-index case, even the identifiability of the linear models (from infinitely many samples) was not known. We show three results in this case for the commonly studied $\max$ self-selection criterion: (1) we show that the linear models are indeed identifiable, (2) for general $k$ we provide an algorithm with poly$(d) \exp(\text{poly}(k))$ sample and time complexity to estimate the regression parameters up to error $1/\text{poly}(k)$, and (3) for $k = 2$ we provide an algorithm for any error $\varepsilon$ and poly$(d, 1/\varepsilon)$ sample and time complexity.

我们研究一类弱识别的位置尺度混合模型，其中基于$ N $ i.d.d的最大似然估计。已知样品具有比经典$ N ^ { - \ frac {1} {2}} $错误的较低的精度。我们调查期望 - 最大化（EM）算法是否也会缓慢收敛这些模型。我们为EM提供了严格的表征，用于在一个单变量的环境中拟合弱识别的高斯混合物，其中我们证明EM算法以$ N ^ {\ FRAC {3} {4}} $步骤汇聚，并返回A处的估计欧几里德订单距离$ {n ^ { - \ frac {1} {8}}} $和$ {n ^ { - \ frac {1} {4}} {4}} {4}}分别从真实位置和比例参数。建立单变量环境中的缓慢速率需要具有两个阶段的新型本地化参数，每个阶段都涉及以人口水平应用于不同代理EM操作员的划分基于epoch的参数。我们展示了几种多元（$ d \ geq 2 $）的例子，表现出与单变量案件相同的缓慢。当拟合协方差受到限制为身份的倍数时，我们还在特殊情况下在特殊情况下以更高的尺寸证明了更高的统计率。

We develop a general framework for distribution-free predictive inference in regression, using conformal inference. The proposed methodology allows for the construction of a prediction band for the response variable using any estimator of the regression function. The resulting prediction band preserves the consistency properties of the original estimator under standard assumptions, while guaranteeing finite-sample marginal coverage even when these assumptions do not hold. We analyze and compare, both empirically and theoretically, the two major variants of our conformal framework: full conformal inference and split conformal inference, along with a related jackknife method. These methods offer different tradeoffs between statistical accuracy (length of resulting prediction intervals) and computational efficiency. As extensions, we develop a method for constructing valid in-sample prediction intervals called rank-one-out conformal inference, which has essentially the same computational efficiency as split conformal inference. We also describe an extension of our procedures for producing prediction bands with locally varying length, in order to adapt to heteroskedascity in the data. Finally, we propose a model-free notion of variable importance, called leave-one-covariate-out or LOCO inference. Accompanying this paper is an R package conformalInference that implements all of the proposals we have introduced. In the spirit of reproducibility, all of our empirical results can also be easily (re)generated using this package.

古典统计学习理论表示，拟合太多参数导致过度舒服和性能差。尽管大量参数矛盾，但是现代深度神经网络概括了这一发现，并构成了解释深度学习成功的主要未解决的问题。随机梯度下降（SGD）引起的隐式正规被认为是重要的，但其特定原则仍然是未知的。在这项工作中，我们研究了当地最小值周围的能量景观的局部几何学如何影响SGD的统计特性，具有高斯梯度噪声。我们争辩说，在合理的假设下，局部几何形状力强制SGD保持接近低维子空间，这会引起隐式正则化并导致深神经网络的泛化误差界定更严格的界限。为了获得神经网络的泛化误差界限，我们首先引入局部最小值周围的停滞迹象，并施加人口风险的局部基本凸性财产。在这些条件下，推导出SGD的下界，以保留在这些停滞套件中。如果发生停滞，我们会导出涉及权重矩阵的光谱规范的深神经网络的泛化误差的界限，但不是网络参数的数量。从技术上讲，我们的证据基于控制SGD中的参数值的变化以及基于局部最小值周围的合适邻域的熵迭代的参数值和局部均匀收敛。我们的工作试图通过统一收敛更好地连接非凸优化和泛化分析。

统计推断中的主要范式取决于I.I.D.的结构。来自假设的无限人群的数据。尽管它取得了成功，但在复杂的数据结构下，即使在清楚无限人口所代表的内容的情况下，该框架在复杂的数据结构下仍然不灵活。在本文中，我们探讨了一个替代框架，在该框架中，推断只是对模型误差的不变性假设，例如交换性或符号对称性。作为解决这个不变推理问题的一般方法，我们提出了一个基于随机的过程。我们证明了该过程的渐近有效性的一般条件，并在许多数据结构中说明了，包括单向和双向布局中的群集误差。我们发现，通过残差随机化的不变推断具有三个吸引人的属性：（1）在弱且可解释的条件下是有效的，可以解决重型数据，有限聚类甚至一些高维设置的问题。 （2）它在有限样品中是可靠的，因为它不依赖经典渐近学所需的规律性条件。 （3）它以适应数据结构的统一方式解决了推断问题。另一方面，诸如OLS或Bootstrap之类的经典程序以I.I.D.为前提。结构，只要实际问题结构不同，就需要修改。经典框架中的这种不匹配导致了多种可靠的误差技术和自举变体，这些变体经常混淆应用研究。我们通过广泛的经验评估证实了这些发现。残留随机化对许多替代方案的表现有利，包括可靠的误差方法，自举变体和分层模型。