Distributed training of massive machine learning models, in particular deep neural networks, via Stochastic Gradient Descent (SGD) is becoming commonplace. Several families of communication-reduction methods, such as quantization, largebatch methods, and gradient sparsification, have been proposed. To date, gradient sparsification methods-where each node sorts gradients by magnitude, and only communicates a subset of the components, accumulating the rest locally-are known to yield some of the largest practical gains. Such methods can reduce the amount of communication per step by up to three orders of magnitude, while preserving model accuracy. Yet, this family of methods currently has no theoretical justification. This is the question we address in this paper. We prove that, under analytic assumptions, sparsifying gradients by magnitude with local error correction provides convergence guarantees, for both convex and non-convex smooth objectives, for data-parallel SGD. The main insight is that sparsification methods implicitly maintain bounds on the maximum impact of stale updates, thanks to selection by magnitude. Our analysis and empirical validation also reveal that these methods do require analytical conditions to converge well, justifying existing heuristics.

Parallel implementations of stochastic gradient descent (SGD) have received significant research attention, thanks to its excellent scalability properties. A fundamental barrier when parallelizing SGD is the high bandwidth cost of communicating gradient updates between nodes; consequently, several lossy compresion heuristics have been proposed, by which nodes only communicate quantized gradients. Although effective in practice, these heuristics do not always converge. In this paper, we propose Quantized SGD (QSGD), a family of compression schemes with convergence guarantees and good practical performance. QSGD allows the user to smoothly trade off communication bandwidth and convergence time: nodes can adjust the number of bits sent per iteration, at the cost of possibly higher variance. We show that this trade-off is inherent, in the sense that improving it past some threshold would violate information-theoretic lower bounds. QSGD guarantees convergence for convex and non-convex objectives, under asynchrony, and can be extended to stochastic variance-reduced techniques. When applied to training deep neural networks for image classification and automated speech recognition, QSGD leads to significant reductions in end-to-end training time. For instance, on 16GPUs, we can train the ResNet-152 network to full accuracy on ImageNet 1.8× faster than the full-precision variant. time to the same target accuracy is 2.7×. Further, even computationally-heavy architectures such as Inception and ResNet can benefit from the reduction in communication: on 16GPUs, QSGD reduces the end-to-end convergence time of ResNet152 by approximately 2×. Networks trained with QSGD can converge to virtually the same accuracy as full-precision variants, and that gradient quantization may even slightly improve accuracy in some settings. Related Work. One line of related research studies the communication complexity of convex optimization. In particular, [40] studied two-processor convex minimization in the same model, provided a lower bound of Ω(n(log n + log(1/ ))) bits on the communication cost of n-dimensional convex problems, and proposed a non-stochastic algorithm for strongly convex problems, whose communication cost is within a log factor of the lower bound. By contrast, our focus is on stochastic gradient methods. Recent work [5] focused on round complexity lower bounds on the number of communication rounds necessary for convex learning.Buckwild! [10] was the first to consider the convergence guarantees of low-precision SGD. It gave upper bounds on the error probability of SGD, assuming unbiased stochastic quantization, convexity, and gradient sparsity, and showed significant speedup when solving convex problems on CPUs. QSGD refines these results by focusing on the trade-off between communication and convergence. We view quantization as an independent source of variance for SGD, which allows us to employ standard convergence results [7]. The main differences from Buckw

最近，随机梯度下降（SGD）及其变体已成为机器学习（ML）问题大规模优化的主要方法。已经提出了各种策略来调整步骤尺寸，从自适应步骤大小到启发式方法，以更改每次迭代中的步骤大小。此外，动力已被广泛用于ML任务以加速训练过程。然而，我们对它们的理论理解存在差距。在这项工作中，我们开始通过为一些启发式优化方法提供正式保证并提出改进的算法来缩小这一差距。首先，我们分析了凸面和非凸口设置的Adagrad（延迟Adagrad）步骤大小的广义版本，这表明这些步骤尺寸允许算法自动适应随机梯度的噪声水平。我们首次显示延迟Adagrad的足够条件，以确保梯度几乎融合到零。此外，我们对延迟的Adagrad及其在非凸面设置中的动量变体进行了高概率分析。其次，我们用指数级和余弦的步骤分析了SGD，在经验上取得了成功，但缺乏理论支持。我们在平滑和非凸的设置中为它们提供了最初的收敛保证，有或没有polyak-{\ l} ojasiewicz（pl）条件。我们还显示了它们在PL条件下适应噪声的良好特性。第三，我们研究动量方法的最后迭代。我们证明了SGD的最后一个迭代的凸设置中的第一个下限，并以恒定的动量。此外，我们研究了一类跟随基于领先的领导者的动量算法，并随着动量和收缩的更新而增加。我们表明，他们的最后一个迭代具有最佳的收敛性，用于无约束的凸随机优化问题。

Federated learning is a distributed framework according to which a model is trained over a set of devices, while keeping data localized. This framework faces several systemsoriented challenges which include (i) communication bottleneck since a large number of devices upload their local updates to a parameter server, and (ii) scalability as the federated network consists of millions of devices. Due to these systems challenges as well as issues related to statistical heterogeneity of data and privacy concerns, designing a provably efficient federated learning method is of significant importance yet it remains challenging. In this paper, we present FedPAQ, a communication-efficient Federated Learning method with Periodic Averaging and Quantization. FedPAQ relies on three key features: (1) periodic averaging where models are updated locally at devices and only periodically averaged at the server; (2) partial device participation where only a fraction of devices participate in each round of the training; and (3) quantized messagepassing where the edge nodes quantize their updates before uploading to the parameter server. These features address the communications and scalability challenges in federated learning. We also show that FedPAQ achieves near-optimal theoretical guarantees for strongly convex and non-convex loss functions and empirically demonstrate the communication-computation tradeoff provided by our method.

我们研究了在$ n $工人上的分布式培训的异步随机梯度下降算法，随着时间的推移，计算和通信频率变化。在此算法中，工人按照自己的步调并行计算随机梯度，并在没有任何同步的情况下将其返回服务器。该算法的现有收敛速率对于非凸平的光滑目标取决于最大梯度延迟$ \ tau _ {\ max} $，并表明$ \ epsilon $ stationary点在$ \ mathcal {o} \！\左后达到（\ sigma^2 \ epsilon^{ - 2}+ \ tau _ {\ max} \ epsilon^{ - 1} \ right）$ iterations，其中$ \ sigma $表示随机梯度的方差。在这项工作（i）中，我们获得了$ \ Mathcal {o} \！\ left（\ sigma^2 \ epsilon^{ - 2}+ sqrt {\ tau _ {\ max} \ max} \ tau_ {avg} {avg} } \ epsilon^{ - 1} \ right）$，没有任何更改的算法，其中$ \ tau_ {avg} $是平均延迟，可以大大小于$ \ tau _ {\ max} $。我们还提供（ii）一个简单的延迟自适应学习率方案，在该方案下，异步SGD的收敛速率为$ \ Mathcal {o} \！\ left（\ sigma^2 \ epsilon^{ - 2} { - 2}+ \ tau_ {-2 avg} \ epsilon^{ - 1} \ right）$，并且不需要任何额外的高参数调整或额外的通信。我们的结果首次显示异步SGD总是比迷你批次SGD快。此外，（iii）我们考虑了由联邦学习应用激发的异质功能的情况，并通过证明与先前的作品相比对最大延迟的依赖性较弱，并提高收敛率。特别是，我们表明，收敛率的异质性项仅受每个工人内平均延迟的影响。

我们考虑一个标准的分布式优化设置，其中$ n $ machines，每个持有$ d $ -dimension函数$ f_i $，旨在共同最大限度地减少函数$ \ sum_ {i = 1} ^ n f_i（x）$ 。该问题自然地出现在大规模分布式优化中，其中标准解决方案是施加（随机）梯度下降的变体。我们专注于这个问题的通信复杂性：我们的主要结果在$ N $ Machines中提供了需要发送和接收的比特总数的第一个完全无条件的界限，以便在点对点通信下解决这个问题给定的差错。具体来说，我们显示$ \ omega（ND \ log d / n \ varepsilon）$总比特在机器之间传达，找到一个添加剂$ \ epsilon $-xprupmation到$ \ sum_ {i = 1} ^ n f_i（x）$。结果适用于确定性和随机算法，并且重要的是，不需要对算法结构上的假设。在参数值的某些限制下，下限是紧张的，并且通过量化梯度下降的新变种在恒定因子中匹配，我们描述和分析。我们的结果带来了从通信复杂性到分布式优化的工具，这具有进一步应用的潜力。

联合学习（FL）是一种新兴的范式，可实现对机器学习模型的大规模分布培训，同时仍提供隐私保证。在这项工作中，我们在将联合优化扩展到大节点计数时共同解决了两个主要的实际挑战：中央权威和单个计算节点之间紧密同步的需求以及中央服务器和客户端之间的传输成本较大。具体而言，我们提出了经典联合平均（FedAvg）算法的新变体，该算法支持异步通信和通信压缩。我们提供了一种新的分析技术，该技术表明，尽管有这些系统放松，但在合理的参数设置下，我们的算法基本上与FedAvg的最著名界限相匹配。在实验方面，我们表明我们的算法确保标准联合任务的快速实用收敛。

Mini-batch stochastic gradient descent (SGD) is state of the art in large scale distributed training. The scheme can reach a linear speedup with respect to the number of workers, but this is rarely seen in practice as the scheme often suffers from large network delays and bandwidth limits. To overcome this communication bottleneck recent works propose to reduce the communication frequency. An algorithm of this type is local SGD that runs SGD independently in parallel on different workers and averages the sequences only once in a while. This scheme shows promising results in practice, but eluded thorough theoretical analysis.We prove concise convergence rates for local SGD on convex problems and show that it converges at the same rate as mini-batch SGD in terms of number of evaluated gradients, that is, the scheme achieves linear speedup in the number of workers and mini-batch size. The number of communication rounds can be reduced up to a factor of T 1/2 -where T denotes the number of total steps-compared to mini-batch SGD. This also holds for asynchronous implementations.Local SGD can also be used for large scale training of deep learning models. The results shown here aim serving as a guideline to further explore the theoretical and practical aspects of local SGD in these applications.

We initiate a formal study of reproducibility in optimization. We define a quantitative measure of reproducibility of optimization procedures in the face of noisy or error-prone operations such as inexact or stochastic gradient computations or inexact initialization. We then analyze several convex optimization settings of interest such as smooth, non-smooth, and strongly-convex objective functions and establish tight bounds on the limits of reproducibility in each setting. Our analysis reveals a fundamental trade-off between computation and reproducibility: more computation is necessary (and sufficient) for better reproducibility.

We show that parametric models trained by a stochastic gradient method (SGM) with few iterations have vanishing generalization error. We prove our results by arguing that SGM is algorithmically stable in the sense of Bousquet and Elisseeff. Our analysis only employs elementary tools from convex and continuous optimization. We derive stability bounds for both convex and non-convex optimization under standard Lipschitz and smoothness assumptions.Applying our results to the convex case, we provide new insights for why multiple epochs of stochastic gradient methods generalize well in practice. In the non-convex case, we give a new interpretation of common practices in neural networks, and formally show that popular techniques for training large deep models are indeed stability-promoting. Our findings conceptually underscore the importance of reducing training time beyond its obvious benefit.

在过去的几年中，各种通信压缩技术已经出现为一个不可或缺的工具，有助于缓解分布式学习中的通信瓶颈。然而，尽管{\ em偏见}压缩机经常在实践中显示出卓越的性能，但与更多的研究和理解的{\ EM无偏见}压缩机相比，非常少见。在这项工作中，我们研究了三类偏置压缩操作员，其中两个是新的，并且它们在施加到（随机）梯度下降和分布（随机）梯度下降时的性能。我们首次展示偏置压缩机可以在单个节点和分布式设置中导致线性收敛速率。我们证明了具有错误反馈机制的分布式压缩SGD方法，享受ergodic速率$ \ mathcal {o} \ left（\ delta l \ exp [ - \ frac {\ mu k} {\ delta l}] + \ frac {（c + \ delta d）} {k \ mu} \右）$，其中$ \ delta \ ge1 $是一个压缩参数，它在应用更多压缩时增长，$ l $和$ \ mu $是平滑性和强凸常数，$ C $捕获随机渐变噪声（如果在每个节点上计算完整渐变，则$ C = 0 $如果在每个节点上计算），则$ D $以最佳（$ d = 0 $ for over参数化模型）捕获渐变的方差）。此外，通过对若干合成和经验的通信梯度分布的理论研究，我们阐明了为什么和通过多少偏置压缩机优于其无偏的变体。最后，我们提出了几种具有有希望理论担保和实际表现的新型偏置压缩机。

Huge scale machine learning problems are nowadays tackled by distributed optimization algorithms, i.e. algorithms that leverage the compute power of many devices for training. The communication overhead is a key bottleneck that hinders perfect scalability. Various recent works proposed to use quantization or sparsification techniques to reduce the amount of data that needs to be communicated, for instance by only sending the most significant entries of the stochastic gradient (top-k sparsification). Whilst such schemes showed very promising performance in practice, they have eluded theoretical analysis so far. In this work we analyze Stochastic Gradient Descent (SGD) with k-sparsification or compression (for instance top-k or random-k) and show that this scheme converges at the same rate as vanilla SGD when equipped with error compensation (keeping track of accumulated errors in memory). That is, communication can be reduced by a factor of the dimension of the problem (sometimes even more) whilst still converging at the same rate. We present numerical experiments to illustrate the theoretical findings and the good scalability for distributed applications.

Training large neural networks requires distributing learning across multiple workers, where the cost of communicating gradients can be a significant bottleneck. SIGNSGD alleviates this problem by transmitting just the sign of each minibatch stochastic gradient. We prove that it can get the best of both worlds: compressed gradients and SGD-level convergence rate. The relative 1 / 2 geometry of gradients, noise and curvature informs whether SIGNSGD or SGD is theoretically better suited to a particular problem. On the practical side we find that the momentum counterpart of SIGNSGD is able to match the accuracy and convergence speed of ADAM on deep Imagenet models. We extend our theory to the distributed setting, where the parameter server uses majority vote to aggregate gradient signs from each worker enabling 1-bit compression of worker-server communication in both directions. Using a theorem by Gauss (1823) we prove that majority vote can achieve the same reduction in variance as full precision distributed SGD. Thus, there is great promise for sign-based optimisation schemes to achieve fast communication and fast convergence. Code to reproduce experiments is to be found at https://github.com/jxbz/signSGD.

我们研究了Adagrad-norm的收敛速率，作为自适应随机梯度方法（SGD）的典范，其中，基于观察到的随机梯度的步骤大小变化，以最大程度地减少非凸，平稳的目标。尽管它们很受欢迎，但在这种情况下，对自适应SGD的分析滞后于非自适应方法。具体而言，所有先前的作品都依赖以下假设的某个子集：（i）统一结合的梯度规范，（ii）均匀遇到的随机梯度方差（甚至噪声支持），（iii）步骤大小和随机性之间的有条件独立性坡度。在这项工作中，我们表明Adagrad-norm表现出$ \ Mathcal {O} \ left（\ frac {\ mathrm {poly} \ log（t）} {\ sqrt {\ sqrt {t}}} \ right）的订单最佳收敛率$在$ t $迭代之后，在与最佳调整的非自适应SGD（无界梯度规范和仿射噪声方差缩放）相同的假设下进行了$，而无需任何调整参数。因此，我们确定自适应梯度方法在比以前了解的更广泛的方案中表现出最佳的融合。

分散和联合学习的关键挑战之一是设计算法，这些算法有效地处理跨代理商的高度异构数据分布。在本文中，我们在数据异质性下重新审视分散的随机梯度下降算法（D-SGD）的分析。我们在D-SGD的收敛速率上展示了新数量的关键作用，称为\ emph {邻居异质性}。通过结合通信拓扑结构和异质性，我们的分析阐明了这两个分散学习中这两个概念之间的相互作用较低。然后，我们认为邻里的异质性提供了一种自然标准，可以学习数据依赖性拓扑结构，以减少（甚至可以消除）数据异质性对D-SGD收敛时间的有害影响。对于与标签偏度分类的重要情况，我们制定了学习这样一个良好拓扑的问题，例如我们使用Frank-Wolfe算法解决的可拖动优化问题。如一组模拟和现实世界实验所示，我们的方法提供了一种设计稀疏拓扑的方法，可以在数据异质性下平衡D-SGD的收敛速度和D-SGD的触电沟通成本。

我们介绍和分析结构化的随机零订单下降（S-SZD），这是一种有限的差异方法，该方法在一组$ l \ leq d $正交方向上近似于随机梯度，其中$ d $是环境空间的维度。这些方向是随机选择的，并且可能在每个步骤中发生变化。对于平滑的凸功能，我们几乎可以确保迭代的收敛性和对$ o（d/l k^{ - c}）$的功能值的收敛速率，每$ c <1/2 $，这是任意关闭的就迭代次数而言，是随机梯度下降（SGD）。我们的界限还显示了使用$ l $多个方向而不是一个方向的好处。对于满足polyak-{\ l} ojasiewicz条件的非convex函数，我们在这种假设下建立了随机Zeroth Order Order Order算法的第一个收敛速率。我们在数值模拟中证实了我们的理论发现，在数值模拟中，满足假设以及对超参数优化的现实世界问题，观察到S-SZD具有很好的实践性能。

我们调查随机镜面下降（SMD）的趋同相对光滑和平滑凸优化。在相对平滑的凸优化中，我们为SMD提供了新的收敛保证，并持续步骤。对于平滑的凸优化，我们提出了一种新的自适应步骤方案 - 镜子随机Polyak Spectize（MSP）。值得注意的是，我们的收敛导致两个设置都不会使有界渐变假设或有界方差假设，并且我们向邻域显示在插值下消失的邻居的融合。MSP概括了最近提出的随机Polyak Spectize（SPS）（Loizou等，2021）以镜子血液镜子，并且在继承镜子血清的好处的同时，现代机器学习应用仍然是实用和高效的。我们将我们的结果与各种监督的学习任务和SMD的不同实例相结合，展示了MSP的有效性。

Federated Averaging (FEDAVG) has emerged as the algorithm of choice for federated learning due to its simplicity and low communication cost. However, in spite of recent research efforts, its performance is not fully understood. We obtain tight convergence rates for FEDAVG and prove that it suffers from 'client-drift' when the data is heterogeneous (non-iid), resulting in unstable and slow convergence.As a solution, we propose a new algorithm (SCAFFOLD) which uses control variates (variance reduction) to correct for the 'client-drift' in its local updates. We prove that SCAFFOLD requires significantly fewer communication rounds and is not affected by data heterogeneity or client sampling. Further, we show that (for quadratics) SCAFFOLD can take advantage of similarity in the client's data yielding even faster convergence. The latter is the first result to quantify the usefulness of local-steps in distributed optimization.

近期在应用于培训深度神经网络和数据分析中的其他优化问题中的非凸优化的优化算法的兴趣增加，我们概述了最近对非凸优化优化算法的全球性能保证的理论结果。我们从古典参数开始，显示一般非凸面问题无法在合理的时间内有效地解决。然后，我们提供了一个问题列表，可以通过利用问题的结构来有效地找到全球最小化器，因为可能的问题。处理非凸性的另一种方法是放宽目标，从找到全局最小，以找到静止点或局部最小值。对于该设置，我们首先为确定性一阶方法的收敛速率提出了已知结果，然后是最佳随机和随机梯度方案的一般理论分析，以及随机第一阶方法的概述。之后，我们讨论了非常一般的非凸面问题，例如最小化$ \ alpha $ -weakly-are-convex功能和满足Polyak-lojasiewicz条件的功能，这仍然允许获得一阶的理论融合保证方法。然后，我们考虑更高阶和零序/衍生物的方法及其收敛速率，以获得非凸优化问题。

我们考虑随着延迟梯度的随机优化，在每次步骤$ $，该算法使用步骤$ t-d_t $的陈旧随机梯度进行更新，从而为某些任意延迟$ d_t $。此设置摘要异步分布式优化，其中中央服务器接收由工作人员计算的渐变更新。这些机器可以体验可能随时间变化而变化的计算和通信负载。在一般的非凸平滑优化设置中，我们提供了一种简单且高效的算法，需要$ o（\ sigma ^ 2 / \ epsilon ^ 4 + \ tau / epsilon ^ 2）$步骤查找$ \ epsilon $ - 静止点$ x $，其中$ \ tau $是\ emph {平均}延迟$ \ smash {\ frac {1} {t} \ sum_ {t = 1} ^ t d_t} $和$ \ sigma ^ 2 $是随机梯度的方差。这改善了以前的工作，这表明随机梯度体面可以实现相同的速率，而是相对于\ emph {maximal}延迟$ \ max_ {t} d_t $，这可以显着大于平均延迟，特别是在异构分布式系统中。我们的实验证明了我们算法在延迟分布歪斜或重尾的情况下的效力和稳健性。