我们考虑随着延迟梯度的随机优化，在每次步骤$ $，该算法使用步骤$ t-d_t $的陈旧随机梯度进行更新，从而为某些任意延迟$ d_t $。此设置摘要异步分布式优化，其中中央服务器接收由工作人员计算的渐变更新。这些机器可以体验可能随时间变化而变化的计算和通信负载。在一般的非凸平滑优化设置中，我们提供了一种简单且高效的算法，需要$ o（\ sigma ^ 2 / \ epsilon ^ 4 + \ tau / epsilon ^ 2）$步骤查找$ \ epsilon $ - 静止点$ x $，其中$ \ tau $是\ emph {平均}延迟$ \ smash {\ frac {1} {t} \ sum_ {t = 1} ^ t d_t} $和$ \ sigma ^ 2 $是随机梯度的方差。这改善了以前的工作，这表明随机梯度体面可以实现相同的速率，而是相对于\ emph {maximal}延迟$ \ max_ {t} d_t $，这可以显着大于平均延迟，特别是在异构分布式系统中。我们的实验证明了我们算法在延迟分布歪斜或重尾的情况下的效力和稳健性。

当任何延迟较大时，异步随机梯度下降（SGD）的现有分析显着降低，给人的印象是性能主要取决于延迟。相反，无论梯度中的延迟如何，我们都证明，我们可以更好地保证相同的异步SGD算法，而不是仅取决于用于实现算法的平行设备的数量。我们的保证严格比现有分析要好，我们还认为，异步SGD在我们考虑的设置中优于同步Minibatch SGD。为了进行分析，我们介绍了基于“虚拟迭代”和延迟自适应步骤的新颖递归，这使我们能够为凸面和非凸面目标得出最先进的保证。

我们研究了在$ n $工人上的分布式培训的异步随机梯度下降算法，随着时间的推移，计算和通信频率变化。在此算法中，工人按照自己的步调并行计算随机梯度，并在没有任何同步的情况下将其返回服务器。该算法的现有收敛速率对于非凸平的光滑目标取决于最大梯度延迟$ \ tau _ {\ max} $，并表明$ \ epsilon $ stationary点在$ \ mathcal {o} \！\左后达到（\ sigma^2 \ epsilon^{ - 2}+ \ tau _ {\ max} \ epsilon^{ - 1} \ right）$ iterations，其中$ \ sigma $表示随机梯度的方差。在这项工作（i）中，我们获得了$ \ Mathcal {o} \！\ left（\ sigma^2 \ epsilon^{ - 2}+ sqrt {\ tau _ {\ max} \ max} \ tau_ {avg} {avg} } \ epsilon^{ - 1} \ right）$，没有任何更改的算法，其中$ \ tau_ {avg} $是平均延迟，可以大大小于$ \ tau _ {\ max} $。我们还提供（ii）一个简单的延迟自适应学习率方案，在该方案下，异步SGD的收敛速率为$ \ Mathcal {o} \！\ left（\ sigma^2 \ epsilon^{ - 2} { - 2}+ \ tau_ {-2 avg} \ epsilon^{ - 1} \ right）$，并且不需要任何额外的高参数调整或额外的通信。我们的结果首次显示异步SGD总是比迷你批次SGD快。此外，（iii）我们考虑了由联邦学习应用激发的异质功能的情况，并通过证明与先前的作品相比对最大延迟的依赖性较弱，并提高收敛率。特别是，我们表明，收敛率的异质性项仅受每个工人内平均延迟的影响。

Mini-batch stochastic gradient descent (SGD) is state of the art in large scale distributed training. The scheme can reach a linear speedup with respect to the number of workers, but this is rarely seen in practice as the scheme often suffers from large network delays and bandwidth limits. To overcome this communication bottleneck recent works propose to reduce the communication frequency. An algorithm of this type is local SGD that runs SGD independently in parallel on different workers and averages the sequences only once in a while. This scheme shows promising results in practice, but eluded thorough theoretical analysis.We prove concise convergence rates for local SGD on convex problems and show that it converges at the same rate as mini-batch SGD in terms of number of evaluated gradients, that is, the scheme achieves linear speedup in the number of workers and mini-batch size. The number of communication rounds can be reduced up to a factor of T 1/2 -where T denotes the number of total steps-compared to mini-batch SGD. This also holds for asynchronous implementations.Local SGD can also be used for large scale training of deep learning models. The results shown here aim serving as a guideline to further explore the theoretical and practical aspects of local SGD in these applications.

Training large neural networks requires distributing learning across multiple workers, where the cost of communicating gradients can be a significant bottleneck. SIGNSGD alleviates this problem by transmitting just the sign of each minibatch stochastic gradient. We prove that it can get the best of both worlds: compressed gradients and SGD-level convergence rate. The relative 1 / 2 geometry of gradients, noise and curvature informs whether SIGNSGD or SGD is theoretically better suited to a particular problem. On the practical side we find that the momentum counterpart of SIGNSGD is able to match the accuracy and convergence speed of ADAM on deep Imagenet models. We extend our theory to the distributed setting, where the parameter server uses majority vote to aggregate gradient signs from each worker enabling 1-bit compression of worker-server communication in both directions. Using a theorem by Gauss (1823) we prove that majority vote can achieve the same reduction in variance as full precision distributed SGD. Thus, there is great promise for sign-based optimisation schemes to achieve fast communication and fast convergence. Code to reproduce experiments is to be found at https://github.com/jxbz/signSGD.

最近，随机梯度下降（SGD）及其变体已成为机器学习（ML）问题大规模优化的主要方法。已经提出了各种策略来调整步骤尺寸，从自适应步骤大小到启发式方法，以更改每次迭代中的步骤大小。此外，动力已被广泛用于ML任务以加速训练过程。然而，我们对它们的理论理解存在差距。在这项工作中，我们开始通过为一些启发式优化方法提供正式保证并提出改进的算法来缩小这一差距。首先，我们分析了凸面和非凸口设置的Adagrad（延迟Adagrad）步骤大小的广义版本，这表明这些步骤尺寸允许算法自动适应随机梯度的噪声水平。我们首次显示延迟Adagrad的足够条件，以确保梯度几乎融合到零。此外，我们对延迟的Adagrad及其在非凸面设置中的动量变体进行了高概率分析。其次，我们用指数级和余弦的步骤分析了SGD，在经验上取得了成功，但缺乏理论支持。我们在平滑和非凸的设置中为它们提供了最初的收敛保证，有或没有polyak-{\ l} ojasiewicz（pl）条件。我们还显示了它们在PL条件下适应噪声的良好特性。第三，我们研究动量方法的最后迭代。我们证明了SGD的最后一个迭代的凸设置中的第一个下限，并以恒定的动量。此外，我们研究了一类跟随基于领先的领导者的动量算法，并随着动量和收缩的更新而增加。我们表明，他们的最后一个迭代具有最佳的收敛性，用于无约束的凸随机优化问题。

我们研究随机梯度下降（SGD）在多大程度上被理解为“常规”学习规则，该规则通过获得良好的培训数据来实现概括性能。我们考虑基本的随机凸优化框架，其中（一通道，无需替代）SGD在经典上是众所周知的，可以最大程度地降低人口风险，以$ o（1/\ sqrt n）$ $ O（1/\ sqrt n）$，并且出人意料地证明，存在问题实例SGD解决方案既表现出$ \ omega（1）$的经验风险和概括差距。因此，事实证明，从任何意义上讲，SGD在算法上都不是稳定的，并且其概括能力不能通过均匀的收敛性或任何其他当前已知的概括性结合技术来解释（除了其经典分析外）。然后，我们继续分析与替代SGD密切相关的相关性，为此我们表明不会发生类似现象，并证明其人口风险实际上确实以最佳速度融合。最后，我们在没有替换SGD的背景下解释了我们的主要结果，用于有限的和凸优化问题，并得出多上类别制度的上限和下限，从而在先前已知的结果上有了显着改善。

我们介绍了一个框架 - Artemis-，以解决分布式或联合设置中的学习问题，并具有通信约束和设备部分参与。几位工人（随机抽样）使用中央服务器执行优化过程来汇总其计算。为了减轻通信成本，Artemis允许在两个方向上（从工人到服务器，相反）将发送的信息与内存机制相结合。它改进了仅考虑单向压缩（对服务器）的现有算法，或在压缩操作员上使用非常强大的假设，并且通常不考虑设备的部分参与。我们在非I.I.D中的随机梯度（仅在最佳点界定的噪声方差）提供了快速的收敛速率（线性最高到阈值）。设置，突出显示内存对单向和双向压缩的影响，分析Polyak-Ruppert平均。我们在分布中使用收敛性，以获得渐近方差的下限，该方差突出了实际的压缩极限。我们提出了两种方法，以解决设备部分参与的具有挑战性的案例，并提供实验结果以证明我们的分析有效性。

Parallel implementations of stochastic gradient descent (SGD) have received significant research attention, thanks to its excellent scalability properties. A fundamental barrier when parallelizing SGD is the high bandwidth cost of communicating gradient updates between nodes; consequently, several lossy compresion heuristics have been proposed, by which nodes only communicate quantized gradients. Although effective in practice, these heuristics do not always converge. In this paper, we propose Quantized SGD (QSGD), a family of compression schemes with convergence guarantees and good practical performance. QSGD allows the user to smoothly trade off communication bandwidth and convergence time: nodes can adjust the number of bits sent per iteration, at the cost of possibly higher variance. We show that this trade-off is inherent, in the sense that improving it past some threshold would violate information-theoretic lower bounds. QSGD guarantees convergence for convex and non-convex objectives, under asynchrony, and can be extended to stochastic variance-reduced techniques. When applied to training deep neural networks for image classification and automated speech recognition, QSGD leads to significant reductions in end-to-end training time. For instance, on 16GPUs, we can train the ResNet-152 network to full accuracy on ImageNet 1.8× faster than the full-precision variant. time to the same target accuracy is 2.7×. Further, even computationally-heavy architectures such as Inception and ResNet can benefit from the reduction in communication: on 16GPUs, QSGD reduces the end-to-end convergence time of ResNet152 by approximately 2×. Networks trained with QSGD can converge to virtually the same accuracy as full-precision variants, and that gradient quantization may even slightly improve accuracy in some settings. Related Work. One line of related research studies the communication complexity of convex optimization. In particular, [40] studied two-processor convex minimization in the same model, provided a lower bound of Ω(n(log n + log(1/ ))) bits on the communication cost of n-dimensional convex problems, and proposed a non-stochastic algorithm for strongly convex problems, whose communication cost is within a log factor of the lower bound. By contrast, our focus is on stochastic gradient methods. Recent work [5] focused on round complexity lower bounds on the number of communication rounds necessary for convex learning.Buckwild! [10] was the first to consider the convergence guarantees of low-precision SGD. It gave upper bounds on the error probability of SGD, assuming unbiased stochastic quantization, convexity, and gradient sparsity, and showed significant speedup when solving convex problems on CPUs. QSGD refines these results by focusing on the trade-off between communication and convergence. We view quantization as an independent source of variance for SGD, which allows us to employ standard convergence results [7]. The main differences from Buckw

我们开发了一种新方法来解决中央服务器中分布式学习问题中的通信约束。我们提出和分析了一种执行双向压缩的新算法，并仅使用uplink（从本地工人到中央服务器）压缩达到与算法相同的收敛速率。为了获得此改进，我们设计了MCM，一种算法，使下行链路压缩仅影响本地模型，而整体模型则保留。结果，与以前的工作相反，本地服务器上的梯度是在干扰模型上计算的。因此，融合证明更具挑战性，需要精确控制这种扰动。为了确保它，MCM还将模型压缩与存储机制相结合。该分析打开了新的门，例如纳入依赖工人的随机模型和部分参与。

许多深度学习领域都受益于使用越来越大的神经网络接受公共数据训练的培训，就像预先训练的NLP和计算机视觉模型一样。培训此类模型需要大量的计算资源（例如，HPC群集），而小型研究小组和独立研究人员则无法使用。解决问题的一种方法是，几个较小的小组将其计算资源汇总在一起并训练一种使所有参与者受益的模型。不幸的是，在这种情况下，任何参与者都可以通过故意或错误地发送错误的更新来危害整个培训。在此类同龄人的情况下进行培训需要具有拜占庭公差的专门分布式培训算法。这些算法通常通过引入冗余通信或通过受信任的服务器传递所有更新来牺牲效率，从而使它们无法应用于大规模深度学习，在该大规模深度学习中，模型可以具有数十亿个参数。在这项工作中，我们提出了一种新的协议，用于强调沟通效率的安全（容忍）分散培训。

分布式优化和学习的最新进展表明，沟通压缩是减少交流的最有效手段之一。尽管在通信压缩下的收敛速率有很多结果，但理论下限仍然缺失。通过通信压缩的算法的分析将收敛归因于两个抽象属性：无偏见的属性或承包属性。它们可以通过单向压缩（仅从工人到服务器的消息被压缩）或双向压缩来应用它们。在本文中，我们考虑了分布式随机算法，以最大程度地减少通信压缩下的平滑和非凸目标函数。我们为算法建立了收敛的下限，无论是在单向或双向中使用无偏压缩机还是使用承包压缩机。为了缩小下限和现有上限之间的差距，我们进一步提出了一种新石器时代的算法，该算法在轻度条件下几乎达到了我们的下限（达到对数因素）。我们的结果还表明，使用承包双向压缩可以产生迭代方法，该方法的收敛速度与使用无偏见的单向压缩的方法一样快。实验结果验证了我们的发现。

为了提高分布式学习的训练速度，近年来见证了人们对开发同步和异步分布式随机方差减少优化方法的极大兴趣。但是，所有现有的同步和异步分布式训练算法都遭受了收敛速度或实施复杂性的各种局限性。这激发了我们提出一种称为\ algname（\ ul {s} emi-as \ ul {yn}的算法} ent \ ul {s} earch），它利用方差减少框架的特殊结构来克服同步和异步分布式学习算法的局限性，同时保留其显着特征。我们考虑分布式和共享内存体系结构下的\ algname的两个实现。我们表明我们的\ algname算法具有\（o（\ sqrt {n} \ epsilon^{ - 2}（ - 2}（\ delta+1）+n）\）\）和\（o（\ sqrt {n} {n} 2}（\ delta+1）d+n）\）用于实现\（\ epsilon \）的计算复杂性 - 分布式和共享内存体系结构分别在非convex学习中的固定点，其中\（n \）表示培训样本的总数和\（\ delta \）表示工人的最大延迟。此外，我们通过建立二次强烈凸和非convex优化的算法稳定性界限来研究\ algname的概括性能。我们进一步进行广泛的数值实验来验证我们的理论发现

我们研究了Adagrad-norm的收敛速率，作为自适应随机梯度方法（SGD）的典范，其中，基于观察到的随机梯度的步骤大小变化，以最大程度地减少非凸，平稳的目标。尽管它们很受欢迎，但在这种情况下，对自适应SGD的分析滞后于非自适应方法。具体而言，所有先前的作品都依赖以下假设的某个子集：（i）统一结合的梯度规范，（ii）均匀遇到的随机梯度方差（甚至噪声支持），（iii）步骤大小和随机性之间的有条件独立性坡度。在这项工作中，我们表明Adagrad-norm表现出$ \ Mathcal {O} \ left（\ frac {\ mathrm {poly} \ log（t）} {\ sqrt {\ sqrt {t}}} \ right）的订单最佳收敛率$在$ t $迭代之后，在与最佳调整的非自适应SGD（无界梯度规范和仿射噪声方差缩放）相同的假设下进行了$，而无需任何调整参数。因此，我们确定自适应梯度方法在比以前了解的更广泛的方案中表现出最佳的融合。

We show that parametric models trained by a stochastic gradient method (SGM) with few iterations have vanishing generalization error. We prove our results by arguing that SGM is algorithmically stable in the sense of Bousquet and Elisseeff. Our analysis only employs elementary tools from convex and continuous optimization. We derive stability bounds for both convex and non-convex optimization under standard Lipschitz and smoothness assumptions.Applying our results to the convex case, we provide new insights for why multiple epochs of stochastic gradient methods generalize well in practice. In the non-convex case, we give a new interpretation of common practices in neural networks, and formally show that popular techniques for training large deep models are indeed stability-promoting. Our findings conceptually underscore the importance of reducing training time beyond its obvious benefit.

Distributed training of massive machine learning models, in particular deep neural networks, via Stochastic Gradient Descent (SGD) is becoming commonplace. Several families of communication-reduction methods, such as quantization, largebatch methods, and gradient sparsification, have been proposed. To date, gradient sparsification methods-where each node sorts gradients by magnitude, and only communicates a subset of the components, accumulating the rest locally-are known to yield some of the largest practical gains. Such methods can reduce the amount of communication per step by up to three orders of magnitude, while preserving model accuracy. Yet, this family of methods currently has no theoretical justification. This is the question we address in this paper. We prove that, under analytic assumptions, sparsifying gradients by magnitude with local error correction provides convergence guarantees, for both convex and non-convex smooth objectives, for data-parallel SGD. The main insight is that sparsification methods implicitly maintain bounds on the maximum impact of stale updates, thanks to selection by magnitude. Our analysis and empirical validation also reveal that these methods do require analytical conditions to converge well, justifying existing heuristics.

我们研究拜占庭的协作学习，其中$ N $节点寻求统称为彼此的本地数据。数据分发可能因一个节点而异。没有信任节点，$ f <n $节点可以行为任意。我们证明，协作学习相当于新的协议形式，我们称之为平均协议。在这个问题中，节点以初始向量启动每个初始向量，并寻求大致达成一个普通的向量，它接近诚实节点初始向量的平均值。我们为平均协议提供了两个异步解决方案，每个我们都证明了根据一些维度的最佳状态。首先，基于最小直径平均，需要$ n \ geq 6f + 1 $，但实现了渐近的最佳平均常量达到乘法常量。其次，基于可靠的广播和坐标 - 明智的均值，实现最佳的拜占庭恢复力，即$ N \ GEQ 3F + 1 $。这些算法中的每一个都会引发最佳的拜占庭协作学习协议。特别是，我们的等价会产生新的不可能性定理，就任何协作学习算法在对抗性和异构环境中实现的内容。

Federated Averaging (FEDAVG) has emerged as the algorithm of choice for federated learning due to its simplicity and low communication cost. However, in spite of recent research efforts, its performance is not fully understood. We obtain tight convergence rates for FEDAVG and prove that it suffers from 'client-drift' when the data is heterogeneous (non-iid), resulting in unstable and slow convergence.As a solution, we propose a new algorithm (SCAFFOLD) which uses control variates (variance reduction) to correct for the 'client-drift' in its local updates. We prove that SCAFFOLD requires significantly fewer communication rounds and is not affected by data heterogeneity or client sampling. Further, we show that (for quadratics) SCAFFOLD can take advantage of similarity in the client's data yielding even faster convergence. The latter is the first result to quantify the usefulness of local-steps in distributed optimization.

Federated learning is a distributed framework according to which a model is trained over a set of devices, while keeping data localized. This framework faces several systemsoriented challenges which include (i) communication bottleneck since a large number of devices upload their local updates to a parameter server, and (ii) scalability as the federated network consists of millions of devices. Due to these systems challenges as well as issues related to statistical heterogeneity of data and privacy concerns, designing a provably efficient federated learning method is of significant importance yet it remains challenging. In this paper, we present FedPAQ, a communication-efficient Federated Learning method with Periodic Averaging and Quantization. FedPAQ relies on three key features: (1) periodic averaging where models are updated locally at devices and only periodically averaged at the server; (2) partial device participation where only a fraction of devices participate in each round of the training; and (3) quantized messagepassing where the edge nodes quantize their updates before uploading to the parameter server. These features address the communications and scalability challenges in federated learning. We also show that FedPAQ achieves near-optimal theoretical guarantees for strongly convex and non-convex loss functions and empirically demonstrate the communication-computation tradeoff provided by our method.

大规模凸孔concave minimax问题在许多应用中出现，包括游戏理论，强大的培训和生成对抗网络的培训。尽管它们的适用性广泛，但使用现有的随机最小值方法在大量数据的情况下，有效，有效地解决此类问题是具有挑战性的。我们研究了一类随机最小值方法，并开发了一种沟通效率的分布式随机外算法Localadaseg，其自适应学习速率适合在参数 - 服务器模型中求解凸Conconcove minimax问题。 Localadaseg具有三个主要功能：（i）定期沟通策略，可降低工人与服务器之间的通信成本； （ii）在本地计算并允许无调实现的自适应学习率； （iii）从理论上讲，在随机梯度的估计中，相对于主要差异项的几乎线性加速在平滑和非平滑凸凸环设置中都证明了。 Localadaseg用于解决随机双线游戏，并训练生成的对抗网络。我们将localadaseg与几个用于最小问题的现有优化者进行了比较，并通过在均质和异质环境中的几个实验来证明其功效。