We study fine-grained error bounds for differentially private algorithms for counting under continual observation. Our main insight is that the matrix mechanism when using lower-triangular matrices can be used in the continual observation model. More specifically, we give an explicit factorization for the counting matrix $M_\mathsf{count}$ and upper bound the error explicitly. We also give a fine-grained analysis, specifying the exact constant in the upper bound. Our analysis is based on upper and lower bounds of the {\em completely bounded norm} (cb-norm) of $M_\mathsf{count}$. Along the way, we improve the best-known bound of 28 years by Mathias (SIAM Journal on Matrix Analysis and Applications, 1993) on the cb-norm of $M_\mathsf{count}$ for a large range of the dimension of $M_\mathsf{count}$. Furthermore, we are the first to give concrete error bounds for various problems under continual observation such as binary counting, maintaining a histogram, releasing an approximately cut-preserving synthetic graph, many graph-based statistics, and substring and episode counting. Finally, we note that our result can be used to get a fine-grained error bound for non-interactive local learning {and the first lower bounds on the additive error for $(\epsilon,\delta)$-differentially-private counting under continual observation.} Subsequent to this work, Henzinger et al. (SODA2023) showed that our factorization also achieves fine-grained mean-squared error.

The first large-scale deployment of private federated learning uses differentially private counting in the continual release model as a subroutine (Google AI blog titled "Federated Learning with Formal Differential Privacy Guarantees"). In this case, a concrete bound on the error is very relevant to reduce the privacy parameter. The standard mechanism for continual counting is the binary mechanism. We present a novel mechanism and show that its mean squared error is both asymptotically optimal and a factor 10 smaller than the error of the binary mechanism. We also show that the constants in our analysis are almost tight by giving non-asymptotic lower and upper bounds that differ only in the constants of lower-order terms. Our algorithm is a matrix mechanism for the counting matrix and takes constant time per release. We also use our explicit factorization of the counting matrix to give an upper bound on the excess risk of the private learning algorithm of Denisov et al. (NeurIPS 2022). Our lower bound for any continual counting mechanism is the first tight lower bound on continual counting under approximate differential privacy. It is achieved using a new lower bound on a certain factorization norm, denoted by $\gamma_F(\cdot)$, in terms of the singular values of the matrix. In particular, we show that for any complex matrix, $A \in \mathbb{C}^{m \times n}$, \[ \gamma_F(A) \geq \frac{1}{\sqrt{m}}\|A\|_1, \] where $\|\cdot \|$ denotes the Schatten-1 norm. We believe this technique will be useful in proving lower bounds for a larger class of linear queries. To illustrate the power of this technique, we show the first lower bound on the mean squared error for answering parity queries.

在最新的应用中，我们需要在自适应流中进行差异隐私，我们研究了在这种情况下矩阵机制的最佳实例化问题。我们证明了矩阵因素化对自适应流的适用性的基本理论结果，并提供了用于计算最佳因素化的无参数固定点算法。我们就机器学习中自然出现的混凝土矩阵实例化了该框架，并通过用户级别的差异私密性来培训用户级别的差异私有模型，从而在联邦学习中产生了显着的问题。

我们介绍了一种基于约翰逊·林登斯特劳斯引理的统计查询的新方法，以释放具有差异隐私的统计查询的答案。关键的想法是随机投影查询答案，以较低的维空间，以便将可行的查询答案的任何两个向量之间的距离保留到添加性错误。然后，我们使用简单的噪声机制回答投影的查询，并将答案提升到原始维度。使用这种方法，我们首次给出了纯粹的私人机制，具有最佳情况下的最佳情况样本复杂性，在平均错误下，以回答$ n $ $ n $的宇宙的$ k $ Queries的工作量。作为其他应用，我们给出了具有最佳样品复杂性的第一个纯私人有效机制，用于计算有限的高维分布的协方差，并用于回答2向边缘查询。我们还表明，直到对错误的依赖性，我们机制的变体对于每个给定的查询工作负载几乎是最佳的。

Learning problems form an important category of computational tasks that generalizes many of the computations researchers apply to large real-life data sets. We ask: what concept classes can be learned privately, namely, by an algorithm whose output does not depend too heavily on any one input or specific training example? More precisely, we investigate learning algorithms that satisfy differential privacy, a notion that provides strong confidentiality guarantees in contexts where aggregate information is released about a database containing sensitive information about individuals.Our goal is a broad understanding of the resources required for private learning in terms of samples, computation time, and interaction. We demonstrate that, ignoring computational constraints, it is possible to privately agnostically learn any concept class using a sample size approximately logarithmic in the cardinality of the concept class. Therefore, almost anything learnable is learnable privately: specifically, if a concept class is learnable by a (non-private) algorithm with polynomial sample complexity and output size, then it can be learned privately using a polynomial number of samples. We also present a computationally efficient private PAC learner for the class of parity functions. This result dispels the similarity between learning with noise and private learning (both must be robust to small changes in inputs), since parity is thought to be very hard to learn given random classification noise.Local (or randomized response) algorithms are a practical class of private algorithms that have received extensive investigation. We provide a precise characterization of local private learning algorithms. We show that a concept class is learnable by a local algorithm if and only if it is learnable in the statistical query (SQ) model. Therefore, for local private learning algorithms, the similarity to learning with noise is stronger: local learning is equivalent to SQ learning, and SQ algorithms include most known noise-tolerant learning algorithms. Finally, we present a separation between the power of interactive and noninteractive local learning algorithms. Because of the equivalence to SQ learning, this result also separates adaptive and nonadaptive SQ learning.

差异隐私通常使用比理论更大的隐私参数应用于理想的理想。已经提出了宽大隐私参数的各种非正式理由。在这项工作中，我们考虑了部分差异隐私（DP），该隐私允许以每个属性为基础量化隐私保证。在此框架中，我们研究了几个基本数据分析和学习任务，并设计了其每个属性隐私参数的算法，其较小的人（即所有属性）的最佳隐私参数比最佳的隐私参数。

In this work, we give efficient algorithms for privately estimating a Gaussian distribution in both pure and approximate differential privacy (DP) models with optimal dependence on the dimension in the sample complexity. In the pure DP setting, we give an efficient algorithm that estimates an unknown $d$-dimensional Gaussian distribution up to an arbitrary tiny total variation error using $\widetilde{O}(d^2 \log \kappa)$ samples while tolerating a constant fraction of adversarial outliers. Here, $\kappa$ is the condition number of the target covariance matrix. The sample bound matches best non-private estimators in the dependence on the dimension (up to a polylogarithmic factor). We prove a new lower bound on differentially private covariance estimation to show that the dependence on the condition number $\kappa$ in the above sample bound is also tight. Prior to our work, only identifiability results (yielding inefficient super-polynomial time algorithms) were known for the problem. In the approximate DP setting, we give an efficient algorithm to estimate an unknown Gaussian distribution up to an arbitrarily tiny total variation error using $\widetilde{O}(d^2)$ samples while tolerating a constant fraction of adversarial outliers. Prior to our work, all efficient approximate DP algorithms incurred a super-quadratic sample cost or were not outlier-robust. For the special case of mean estimation, our algorithm achieves the optimal sample complexity of $\widetilde O(d)$, improving on a $\widetilde O(d^{1.5})$ bound from prior work. Our pure DP algorithm relies on a recursive private preconditioning subroutine that utilizes the recent work on private mean estimation [Hopkins et al., 2022]. Our approximate DP algorithms are based on a substantial upgrade of the method of stabilizing convex relaxations introduced in [Kothari et al., 2022].

We consider the problem of continually releasing an estimate of the population mean of a stream of samples that is user-level differentially private (DP). At each time instant, a user contributes a sample, and the users can arrive in arbitrary order. Until now these requirements of continual release and user-level privacy were considered in isolation. But, in practice, both these requirements come together as the users often contribute data repeatedly and multiple queries are made. We provide an algorithm that outputs a mean estimate at every time instant $t$ such that the overall release is user-level $\varepsilon$-DP and has the following error guarantee: Denoting by $M_t$ the maximum number of samples contributed by a user, as long as $\tilde{\Omega}(1/\varepsilon)$ users have $M_t/2$ samples each, the error at time $t$ is $\tilde{O}(1/\sqrt{t}+\sqrt{M}_t/t\varepsilon)$. This is a universal error guarantee which is valid for all arrival patterns of the users. Furthermore, it (almost) matches the existing lower bounds for the single-release setting at all time instants when users have contributed equal number of samples.

最大信息系数（MIC）是一个强大的统计量，可以识别变量之间的依赖性。但是，它可以应用于敏感数据，并且发布可能会泄漏私人信息。作为解决方案，我们提出算法以提供差异隐私的方式近似麦克风。我们表明，经典拉普拉斯机制的自然应用产生的精度不足。因此，我们介绍了MICT统计量，这是一种新的MIC近似值，与差异隐私更加兼容。我们证明MICS是麦克风的一致估计器，我们提供了两个差异性私有版本。我们对各种真实和合成数据集进行实验。结果表明，私人微统计数据极大地超过了拉普拉斯机制的直接应用。此外，对现实世界数据集的实验显示出准确性，当样本量至少适中时可用。

我们研究依靠敏感数据（例如医疗记录）的环境的顺序决策中，研究隐私的探索。特别是，我们专注于解决在线性MDP设置中受（联合）差异隐私的约束的增强学习问题（RL），在该设置中，动态和奖励均由线性函数给出。由于Luyo等人而引起的此问题的事先工作。 （2021）实现了$ o（k^{3/5}）$的依赖性的遗憾率。我们提供了一种私人算法，其遗憾率提高，最佳依赖性为$ o（\ sqrt {k}）$对情节数量。我们强烈遗憾保证的关键配方是策略更新时间表中的适应性，其中仅在检测到数据足够更改时才发生更新。结果，我们的算法受益于低切换成本，并且仅执行$ o（\ log（k））$更新，这大大降低了隐私噪声的量。最后，在最普遍的隐私制度中，隐私参数$ \ epsilon $是一个常数，我们的算法会造成可忽略不计的隐私成本 - 与现有的非私人遗憾界限相比，由于隐私而引起的额外遗憾在低阶中出现了术语。

我们考虑使用迷你批量梯度进行差异隐私（DP）的培训模型。现有的最先进的差异私有随机梯度下降（DP-SGD）需要通过采样或洗机来获得最佳隐私/准确性/计算权衡的隐私放大。不幸的是，在重要的实际情况下，精确采样和洗牌的精确要求可能很难获得，特别是联邦学习（FL）。我们设计和分析跟随 - 正规的领导者（DP-FTRL）的DP变体，其比较（理论上和经验地）与放大的DP-SGD相比，同时允许更灵活的数据访问模式。DP-FTRL不使用任何形式的隐私放大。该代码可在https://github.com/google-Research/federated/tree/master/dp_ftrl和https://github.com/google-reesearch/dp-ftrl处获得。

我们为其非私人对准减少$（\ varepsilon，\ delta）$差异私人（dp）统计估计，提供了一个相当一般的框架。作为本框架的主要应用，我们提供多项式时间和$（\ varepsilon，\ delta）$ - DP算法用于学习（不受限制的）高斯分布在$ \ mathbb {r} ^ d $。我们学习高斯的方法的样本复杂度高斯距离总变化距离$ \ alpha $是$ \ widetilde {o} \ left（\ frac {d ^ 2} {\ alpha ^ 2} + \ frac {d ^ 2 \ sqrt {\ ln {1 / \ delta}} {\ alpha \ varepsilon} \右）$，匹配（最多为对数因子）最佳已知的信息理论（非高效）样本复杂性上限的aden-ali， Ashtiani，Kamath〜（alt'21）。在一个独立的工作中，Kamath，Mouzakis，Singhal，Steinke和Ullman〜（Arxiv：2111.04609）使用不同的方法证明了类似的结果，并以$ O（d ^ {5/2}）$样本复杂性依赖于$ d $ 。作为我们的框架的另一个应用，我们提供了第一次多项式时间$（\ varepsilon，\ delta）$-dp算法，用于鲁棒学习（不受限制的）高斯。

Kernel matrices, as well as weighted graphs represented by them, are ubiquitous objects in machine learning, statistics and other related fields. The main drawback of using kernel methods (learning and inference using kernel matrices) is efficiency -- given $n$ input points, most kernel-based algorithms need to materialize the full $n \times n$ kernel matrix before performing any subsequent computation, thus incurring $\Omega(n^2)$ runtime. Breaking this quadratic barrier for various problems has therefore, been a subject of extensive research efforts. We break the quadratic barrier and obtain $\textit{subquadratic}$ time algorithms for several fundamental linear-algebraic and graph processing primitives, including approximating the top eigenvalue and eigenvector, spectral sparsification, solving linear systems, local clustering, low-rank approximation, arboricity estimation and counting weighted triangles. We build on the recent Kernel Density Estimation framework, which (after preprocessing in time subquadratic in $n$) can return estimates of row/column sums of the kernel matrix. In particular, we develop efficient reductions from $\textit{weighted vertex}$ and $\textit{weighted edge sampling}$ on kernel graphs, $\textit{simulating random walks}$ on kernel graphs, and $\textit{importance sampling}$ on matrices to Kernel Density Estimation and show that we can generate samples from these distributions in $\textit{sublinear}$ (in the support of the distribution) time. Our reductions are the central ingredient in each of our applications and we believe they may be of independent interest. We empirically demonstrate the efficacy of our algorithms on low-rank approximation (LRA) and spectral sparsification, where we observe a $\textbf{9x}$ decrease in the number of kernel evaluations over baselines for LRA and a $\textbf{41x}$ reduction in the graph size for spectral sparsification.

Differentially private algorithms for common metric aggregation tasks, such as clustering or averaging, often have limited practicality due to their complexity or to the large number of data points that is required for accurate results. We propose a simple and practical tool, $\mathsf{FriendlyCore}$, that takes a set of points ${\cal D}$ from an unrestricted (pseudo) metric space as input. When ${\cal D}$ has effective diameter $r$, $\mathsf{FriendlyCore}$ returns a "stable" subset ${\cal C} \subseteq {\cal D}$ that includes all points, except possibly few outliers, and is {\em certified} to have diameter $r$. $\mathsf{FriendlyCore}$ can be used to preprocess the input before privately aggregating it, potentially simplifying the aggregation or boosting its accuracy. Surprisingly, $\mathsf{FriendlyCore}$ is light-weight with no dependence on the dimension. We empirically demonstrate its advantages in boosting the accuracy of mean estimation and clustering tasks such as $k$-means and $k$-GMM, outperforming tailored methods.

作为标准本地模型和中央模型之间的中间信任模型，差异隐私的洗牌模型已引起了人们的极大兴趣[EFMRTT19；CSUZZ19]。该模型的关键结果是，随机洗牌本地随机数据放大了差异隐私保证。这种放大意味着对数据匿名贡献的系统提供了更大的隐私保证[BEMMRLRKTS17]。在这项工作中，我们通过在理论和数字上逐渐改造结果来改善最新隐私放大的状态。我们的第一个贡献是对LDP Randomizers洗牌输出的R \'enyi差异隐私参数的首次渐近最佳分析。我们的第二个贡献是通过改组对隐私放大的新分析。该分析改进了[FMT20]的技术，并导致所有参数设置中的数值范围更紧密。

We study the relationship between adversarial robustness and differential privacy in high-dimensional algorithmic statistics. We give the first black-box reduction from privacy to robustness which can produce private estimators with optimal tradeoffs among sample complexity, accuracy, and privacy for a wide range of fundamental high-dimensional parameter estimation problems, including mean and covariance estimation. We show that this reduction can be implemented in polynomial time in some important special cases. In particular, using nearly-optimal polynomial-time robust estimators for the mean and covariance of high-dimensional Gaussians which are based on the Sum-of-Squares method, we design the first polynomial-time private estimators for these problems with nearly-optimal samples-accuracy-privacy tradeoffs. Our algorithms are also robust to a constant fraction of adversarially-corrupted samples.

我们考虑对跨用户设备分发的私人数据培训模型。为了确保隐私，我们添加了设备的噪声并使用安全的聚合，以便仅向服务器揭示嘈杂的总和。我们提出了一个综合的端到端系统，该系统适当地离散数据并在执行安全聚合之前添加离散的高斯噪声。我们为离散高斯人的总和提供了新的隐私分析，并仔细分析了数据量化和模块化求和算术的影响。我们的理论保证突出了沟通，隐私和准确性之间的复杂张力。我们广泛的实验结果表明，我们的解决方案基本上能够将准确性与中央差分隐私相匹配，而每个值的精度少于16位。

我们展示了一个联合学习框架，旨在强大地提供具有异构数据的各个客户端的良好预测性能。所提出的方法对基于SuperQualile的学习目标铰接，捕获异构客户端的误差分布的尾统计。我们提出了一种随机训练算法，其与联合平均步骤交织差异私人客户重新重量步骤。该提出的算法支持有限时间收敛保证，保证覆盖凸和非凸面设置。关于联邦学习的基准数据集的实验结果表明，我们的方法在平均误差方面与古典误差竞争，并且在误差的尾统计方面优于它们。

We study the task of training regression models with the guarantee of label differential privacy (DP). Based on a global prior distribution on label values, which could be obtained privately, we derive a label DP randomization mechanism that is optimal under a given regression loss function. We prove that the optimal mechanism takes the form of a ``randomized response on bins'', and propose an efficient algorithm for finding the optimal bin values. We carry out a thorough experimental evaluation on several datasets demonstrating the efficacy of our algorithm.

我们考虑一个顺序设置，其中使用单个数据集用于执行自适应选择的分析，同时确保每个参与者的差别隐私丢失不超过预先指定的隐私预算。此问题的标准方法依赖于限制所有个人对所有个人的隐私损失的最坏情况估计，以及每个单一分析的所有可能的数据值。然而，在许多情况下，这种方法过于保守，特别是对于“典型”数据点，通过参与大部分分析产生很少的隐私损失。在这项工作中，我们基于每个分析中每个人的个性化隐私损失估计的价值，给出了更严格的隐私损失会计的方法。实现我们设计R \'enyi差异隐私的过滤器。过滤器是一种工具，可确保具有自适应选择的隐私参数的组合算法序列的隐私参数不超过预先预算。我们的过滤器比以往的$（\ epsilon，\ delta）$ - rogers等人的差别隐私更简单且更紧密。我们将结果应用于对嘈杂渐变下降的分析，并显示个性化会计可以实用，易于实施，并且只能使隐私式权衡更紧密。