Federated Learning是一种机器学习培训范式，它使客户能够共同培训模型而无需共享自己的本地化数据。但是，实践中联合学习的实施仍然面临许多挑战，例如由于重复的服务器 - 客户同步以及基于SGD的模型更新缺乏适应性，大型通信开销。尽管已经提出了各种方法来通过梯度压缩或量化来降低通信成本，并且提出了联合版本的自适应优化器（例如FedAdam）来增加适应性，目前的联合学习框架仍然无法立即解决上述挑战。在本文中，我们提出了一种具有理论融合保证的新型沟通自适应联合学习方法（FedCAMS）。我们表明，在非convex随机优化设置中，我们提出的fedcams的收敛率与$ o（\ frac {1} {\ sqrt {tkm}}）$与其非压缩的对应物相同。各种基准的广泛实验验证了我们的理论分析。

由于培训数据集的大小爆炸，分布式学习近年来受到了日益增长的兴趣。其中一个主要瓶颈是中央服务器和本地工人之间的沟通成本。虽然已经证明错误反馈压缩以通过随机梯度下降（SGD）降低通信成本，但在培训大规模机器学习方面广泛用于培训的通信有效的适应性梯度方法楷模。在本文中，我们提出了一种新的通信 - 压缩AMSGRAD，用于分布式非透明的优化问题，可提供有效的效率。我们所提出的分布式学习框架具有有效的渐变压缩策略和工人侧模型更新设计。我们证明所提出的通信有效的分布式自适应梯度方法会聚到具有与随机非凸化优化设置中的未压缩的vanilla amsgrad相同的迭代复杂度的一阶静止点。关于各种基准备份我们理论的实验。

在联合学习（FL）的新兴范式中，大量客户端（例如移动设备）用于在各自的数据上训练可能的高维模型。由于移动设备的带宽低，分散的优化方法需要将计算负担从那些客户端转移到计算服务器，同时保留隐私和合理的通信成本。在本文中，我们专注于深度，如多层神经网络的培训，在FL设置下。我们提供了一种基于本地模型的层状和维度更新的新型联合学习方法，减轻了非凸起和手头优化任务的多层性质的新型联合学习方法。我们为Fed-Lamb提供了一种彻底的有限时间收敛性分析，表征其渐变减少的速度有多速度。我们在IID和非IID设置下提供实验结果，不仅可以证实我们的理论，而且与最先进的方法相比，我们的方法的速度更快。

数据异构联合学习（FL）系统遭受了两个重要的收敛误差来源：1）客户漂移错误是由于在客户端执行多个局部优化步骤而引起的，以及2）部分客户参与错误，这是一个事实，仅一小部分子集边缘客户参加每轮培训。我们发现其中，只有前者在文献中受到了极大的关注。为了解决这个问题，我们提出了FedVarp，这是在服务器上应用的一种新颖的差异算法，它消除了由于部分客户参与而导致的错误。为此，服务器只是将每个客户端的最新更新保持在内存中，并将其用作每回合中非参与客户的替代更新。此外，为了减轻服务器上的内存需求，我们提出了一种新颖的基于聚类的方差降低算法clusterfedvarp。与以前提出的方法不同，FedVarp和ClusterFedVarp均不需要在客户端上进行其他计算或其他优化参数的通信。通过广泛的实验，我们表明FedVarp优于最先进的方法，而ClusterFedVarp实现了与FedVarp相当的性能，并且记忆要求较少。

Federated learning is a distributed machine learning paradigm in which a large number of clients coordinate with a central server to learn a model without sharing their own training data. Standard federated optimization methods such as Federated Averaging (FEDAVG) are often difficult to tune and exhibit unfavorable convergence behavior. In non-federated settings, adaptive optimization methods have had notable success in combating such issues. In this work, we propose federated versions of adaptive optimizers, including ADAGRAD, ADAM, and YOGI, and analyze their convergence in the presence of heterogeneous data for general nonconvex settings. Our results highlight the interplay between client heterogeneity and communication efficiency. We also perform extensive experiments on these methods and show that the use of adaptive optimizers can significantly improve the performance of federated learning.

当今部署在边缘网络上的联合学习（FL）系统由大量在数据和/或计算能力中具有高度异质性的工人组成，这些工人要求在时间，努力，数据异质性等方面参加灵活的工作者参与为了满足灵活的工人参与的需求，我们考虑了一种新的FL范式，称为“无政府状态联邦学习”（AFL）（AFL）。与常规FL模型形成鲜明对比的是，AFL中的每个工人都可以自由选择i）何时参加FL，ii）根据当前情况（例如，电池，通信，电池级别，通信渠道，隐私问题）。但是，AFL中这种混乱的工人行为在算法设计中引发了许多新的开放问题。特别是，尚不清楚是否可以开发收敛的AFL训练算法，如果是的，则在什么条件下以及可实现的收敛速度的速度下。为此，我们提出了两种无政府状态的联合平均（AFA）算法，分别命名为AFA-CD和AFA-CS的跨设备和跨核心设置的双向学习率。令人惊讶的是，我们表明，在轻度的无政府状态假设下，这两种AFL算法都达到了最著名的收敛速率，作为常规FL的最新算法。此外，它们保留了新的AFL范式中的工人数量和本地步骤，保留了高度可取的{\ em线性加速效应}。我们通过对现实世界数据集进行广泛的实验来验证提出的算法。

To lower the communication complexity of federated min-max learning, a natural approach is to utilize the idea of infrequent communications (through multiple local updates) same as in conventional federated learning. However, due to the more complicated inter-outer problem structure in federated min-max learning, theoretical understandings of communication complexity for federated min-max learning with infrequent communications remain very limited in the literature. This is particularly true for settings with non-i.i.d. datasets and partial client participation. To address this challenge, in this paper, we propose a new algorithmic framework called stochastic sampling averaging gradient descent ascent (SAGDA), which i) assembles stochastic gradient estimators from randomly sampled clients as control variates and ii) leverages two learning rates on both server and client sides. We show that SAGDA achieves a linear speedup in terms of both the number of clients and local update steps, which yields an $\mathcal{O}(\epsilon^{-2})$ communication complexity that is orders of magnitude lower than the state of the art. Interestingly, by noting that the standard federated stochastic gradient descent ascent (FSGDA) is in fact a control-variate-free special version of SAGDA, we immediately arrive at an $\mathcal{O}(\epsilon^{-2})$ communication complexity result for FSGDA. Therefore, through the lens of SAGDA, we also advance the current understanding on communication complexity of the standard FSGDA method for federated min-max learning.

In federated optimization, heterogeneity in the clients' local datasets and computation speeds results in large variations in the number of local updates performed by each client in each communication round. Naive weighted aggregation of such models causes objective inconsistency, that is, the global model converges to a stationary point of a mismatched objective function which can be arbitrarily different from the true objective. This paper provides a general framework to analyze the convergence of federated heterogeneous optimization algorithms. It subsumes previously proposed methods such as FedAvg and FedProx and provides the first principled understanding of the solution bias and the convergence slowdown due to objective inconsistency. Using insights from this analysis, we propose Fed-Nova, a normalized averaging method that eliminates objective inconsistency while preserving fast error convergence.

Federated learning has attracted increasing attention with the emergence of distributed data. While extensive federated learning algorithms have been proposed for the non-convex distributed problem, the federated learning in practice still faces numerous challenges, such as the large training iterations to converge since the sizes of models and datasets keep increasing, and the lack of adaptivity by SGD-based model updates. Meanwhile, the study of adaptive methods in federated learning is scarce and existing works either lack a complete theoretical convergence guarantee or have slow sample complexity. In this paper, we propose an efficient adaptive algorithm (i.e., FAFED) based on the momentum-based variance reduced technique in cross-silo FL. We first explore how to design the adaptive algorithm in the FL setting. By providing a counter-example, we prove that a simple combination of FL and adaptive methods could lead to divergence. More importantly, we provide a convergence analysis for our method and prove that our algorithm is the first adaptive FL algorithm to reach the best-known samples $O(\epsilon^{-3})$ and $O(\epsilon^{-2})$ communication rounds to find an $\epsilon$-stationary point without large batches. The experimental results on the language modeling task and image classification task with heterogeneous data demonstrate the efficiency of our algorithms.

在这项工作中，我们提出了FedSSO，这是一种用于联合学习的服务器端二阶优化方法（FL）。与以前朝这个方向的工作相反，我们在准牛顿方法中采用了服务器端近似，而无需客户的任何培训数据。通过这种方式，我们不仅将计算负担从客户端转移到服务器，而且还消除了客户和服务器之间二阶更新的附加通信。我们为我们的新方法的收敛提供了理论保证，并从经验上证明了我们在凸面和非凸面设置中的快速收敛和沟通节省。

Federated learning (FL) is an effective technique to directly involve edge devices in machine learning training while preserving client privacy. However, the substantial communication overhead of FL makes training challenging when edge devices have limited network bandwidth. Existing work to optimize FL bandwidth overlooks downstream transmission and does not account for FL client sampling. In this paper we propose GlueFL, a framework that incorporates new client sampling and model compression algorithms to mitigate low download bandwidths of FL clients. GlueFL prioritizes recently used clients and bounds the number of changed positions in compression masks in each round. Across three popular FL datasets and three state-of-the-art strategies, GlueFL reduces downstream client bandwidth by 27% on average and reduces training time by 29% on average.

标准联合优化方法成功地适用于单层结构的随机问题。然而，许多当代的ML问题 - 包括对抗性鲁棒性，超参数调整和参与者 - 批判性 - 属于嵌套的双层编程，这些编程包含微型型和组成优化。在这项工作中，我们提出了\ fedblo：一种联合交替的随机梯度方法来解决一般的嵌套问题。我们在存在异质数据的情况下为\ fedblo建立了可证明的收敛速率，并引入了二聚体，最小值和组成优化的变化。\ fedblo引入了多种创新，包括联邦高级计算和降低方差，以解决内部级别的异质性。我们通过有关超参数\＆超代理学习和最小值优化的实验来补充我们的理论，以证明我们方法在实践中的好处。代码可在https://github.com/ucr-optml/fednest上找到。

In this work, we explore combining automatic hyperparameter tuning and optimization for federated learning (FL) in an online, one-shot procedure. We apply a principled approach on a method for adaptive client learning rate, number of local steps, and batch size. In our federated learning applications, our primary motivations are minimizing communication budget as well as local computational resources in the training pipeline. Conventionally, hyperparameter tuning methods involve at least some degree of trial-and-error, which is known to be sample inefficient. In order to address our motivations, we propose FATHOM (Federated AuTomatic Hyperparameter OptiMization) as a one-shot online procedure. We investigate the challenges and solutions of deriving analytical gradients with respect to the hyperparameters of interest. Our approach is inspired by the fact that, with the exception of local data, we have full knowledge of all components involved in our training process, and this fact can be exploited in our algorithm impactfully. We show that FATHOM is more communication efficient than Federated Averaging (FedAvg) with optimized, static valued hyperparameters, and is also more computationally efficient overall. As a communication efficient, one-shot online procedure, FATHOM solves the bottleneck of costly communication and limited local computation, by eliminating a potentially wasteful tuning process, and by optimizing the hyperparamters adaptively throughout the training procedure without trial-and-error. We show our numerical results through extensive empirical experiments with the Federated EMNIST-62 (FEMNIST) and Federated Stack Overflow (FSO) datasets, using FedJAX as our baseline framework.

Federated learning is a distributed framework according to which a model is trained over a set of devices, while keeping data localized. This framework faces several systemsoriented challenges which include (i) communication bottleneck since a large number of devices upload their local updates to a parameter server, and (ii) scalability as the federated network consists of millions of devices. Due to these systems challenges as well as issues related to statistical heterogeneity of data and privacy concerns, designing a provably efficient federated learning method is of significant importance yet it remains challenging. In this paper, we present FedPAQ, a communication-efficient Federated Learning method with Periodic Averaging and Quantization. FedPAQ relies on three key features: (1) periodic averaging where models are updated locally at devices and only periodically averaged at the server; (2) partial device participation where only a fraction of devices participate in each round of the training; and (3) quantized messagepassing where the edge nodes quantize their updates before uploading to the parameter server. These features address the communications and scalability challenges in federated learning. We also show that FedPAQ achieves near-optimal theoretical guarantees for strongly convex and non-convex loss functions and empirically demonstrate the communication-computation tradeoff provided by our method.

从经验上证明，在跨客户聚集之前应用多个本地更新的实践是克服联合学习（FL）中的通信瓶颈的成功方法。在这项工作中，我们提出了一种通用食谱，即FedShuffle，可以更好地利用FL中的本地更新，尤其是在异质性方面。与许多先前的作品不同，FedShuffle在每个设备的更新数量上没有任何统一性。我们的FedShuffle食谱包括四种简单的功能成分：1）数据的本地改组，2）调整本地学习率，3）更新加权，4）减少动量方差（Cutkosky and Orabona，2019年）。我们对FedShuffle进行了全面的理论分析，并表明从理论和经验上讲，我们的方法都不遭受FL方法中存在的目标功能不匹配的障碍，这些方法假设在异质FL设置中，例如FedAvg（McMahan等人，McMahan等， 2017）。此外，通过将上面的成分结合起来，FedShuffle在Fednova上改善（Wang等，2020），以前提议解决此不匹配。我们还表明，在Hessian相似性假设下，通过降低动量方差的FedShuffle可以改善非本地方法。最后，通过对合成和现实世界数据集的实验，我们说明了FedShuffle中使用的四种成分中的每种如何有助于改善FL中局部更新的使用。

由于分布式和联邦学习应用中的通信瓶颈，使用通信压缩的算法引起了显着的关注，并且广泛用于实践中。此外，由于异构客户端的总数通常非常大，并且服务器无法与每个通信中的所有客户端通信，存在联合学习的客户端 - 方差。在本文中，我们通过提出压缩和客户端 - 方差减少方法来解决这两个问题。具体地，我们介绍了COFIG和FRECON，成功享受了客户方差减少的通信压缩。 COFIG的总通信轮是$ O（\ FRAC {（1+ \ OMEGA）^ {3/2} \ sqrt {n}} {s \ epsilon ^ 2} + \ frac {（1+ \ omega）n ^ {2/3}} {s \ epsilon ^ 2}）$中的非核心设置，其中$ n $是客户的总数，$ s $是每轮的传达客户端的数量，$ \ epsilon $收敛误差和$ \ omega $是压缩运算符的参数。此外，我们的FRECON可以比非核心设置中的COFIG汇聚，它与$ O（\ FRAC {（1+ \ OMEGA）\ SQRT {n}）汇聚在一起。在凸设置中，COFIG在通信中收敛于通信循环$ O（\ FRAC {（1+ \ OMEGA）\ SQRT {n}} $，这也是不存在压缩方案的第一个收敛结果与每轮的所有客户通信。总之，COFIG和FRECON都不需要与所有客户端通信，并提供凸面和非谐波联合学习的第一/更快的融合结果，而以前的作用需要完整的客户端通信（因此不实用）或获得更糟糕的收敛结果。

众所周知，客户师沟通可能是联邦学习中的主要瓶颈。在这项工作中，我们通过一种新颖的客户端采样方案解决了这个问题，我们将允许的客户数量限制为将其更新传达给主节点的数量。在每个通信回合中，所有参与的客户都会计算他们的更新，但只有具有“重要”更新的客户可以与主人通信。我们表明，可以仅使用更新的规范来衡量重要性，并提供一个公式以最佳客户参与。此公式将所有客户参与的完整更新与我们有限的更新（参与客户数量受到限制）之间的距离最小化。此外，我们提供了一种简单的算法，该算法近似于客户参与的最佳公式，该公式仅需要安全的聚合，因此不会损害客户的隐私。我们在理论上和经验上都表明，对于分布式SGD（DSGD）和联合平均（FedAvg），我们的方法的性能可以接近完全参与，并且优于基线，在参与客户均匀地采样的基线。此外，我们的方法与现有的减少通信开销（例如本地方法和通信压缩方法）的现有方法兼容。

具有周期性模型的本地随机梯度下降（SGD）平均（FEDAVG）是联合学习中的基础算法。该算法在多个工人上独立运行SGD，并定期平均所有工人的模型。然而，当本地SGD与许多工人一起运行时，周期性平均导致跨越工人的重大模型差异，使全局损失缓慢收敛。虽然最近的高级优化方法解决了专注于非IID设置的问题，但由于底层定期模型平均而仍存在模型差异问题。我们提出了一个部分模型平均框架，这些框架减轻了联合学习中的模型差异问题。部分平均鼓励本地模型在参数空间上保持彼此接近，并且它可以更有效地最小化全局损失。鉴于固定数量的迭代和大量工人（128），验证精度高达2.2％的验证精度高于周期性的完整平均值。

虽然客户的采样是当前最先进的联邦学习（FL）方法的核心运营，但该程序对迄今为止的迄今为止迄今为止的收敛和速度的影响。在这项工作中，我们为FL的收敛介绍了一种新颖的分解定理，允许清楚地量化客户对全局模型更新的影响。与之前的收敛分析相反，我们的定理提供了给定的收敛步骤的精确分解，从而能够准确考虑客户端采样和异质性的作用。首先，我们为先前报告的结果提供了一种理论基础，从收敛性与聚集权重之间的关系之间的关系。其次，我们首次证明了FL收敛的质量也受到聚集重量之间产生的协方差的影响。第三，我们建立了聚集权重的总和是另一个减速的来源，应该等于1来提高流动速度。我们的理论是一般性的，这里申请了多项分布（MD）和统一采样，在FL中的两个默认客户端采样，并通过一系列非IID和不平衡情景进行了演示。我们的结果表明，MD采样应用作默认采样方案，因为在学习过程中的数据比变化的恢复，而统一的采样仅在客户端具有相同数量的数据时才是优越的。

Federated learning allows collaborative workers to solve a machine learning problem while preserving data privacy. Recent studies have tackled various challenges in federated learning, but the joint optimization of communication overhead, learning reliability, and deployment efficiency is still an open problem. To this end, we propose a new scheme named federated learning via plurality vote (FedVote). In each communication round of FedVote, workers transmit binary or ternary weights to the server with low communication overhead. The model parameters are aggregated via weighted voting to enhance the resilience against Byzantine attacks. When deployed for inference, the model with binary or ternary weights is resource-friendly to edge devices. We show that our proposed method can reduce quantization error and converges faster compared with the methods directly quantizing the model updates.