我们提供了一类用于强大的个性化联合学习的方法，称为FED+，该方法统一了许多联合学习算法。这类方法的主要优势是更好地适应联邦培训中发现的现实世界特征，例如各方缺乏IID数据，对异常值或散乱者的鲁棒性的需求以及在党派上表现良好的要求 - 特定数据集。我们通过问题公式实现这一目标，该问题使中央服务器能够采用可靠的方式来汇总本地模型，同时保持本地计算的结构完整。在各方跨各方的局部数据的异质性程度的情况下，我们为FED+提供了在不同（鲁棒）聚合方法下的convex和非凸损失函数的收敛保证。美联储+理论还可以处理包括散乱者在内的异质计算环境，没有其他假设；具体而言，融合结果涵盖了一般设置，在该设置中，各方跨各方的本地更新步骤的数量可能会有所不同。我们通过在标准基准数据集的广泛实验中证明了FED+的好处。

Federated Learning is a distributed learning paradigm with two key challenges that differentiate it from traditional distributed optimization: (1) significant variability in terms of the systems characteristics on each device in the network (systems heterogeneity), and (2) non-identically distributed data across the network (statistical heterogeneity). In this work, we introduce a framework, FedProx, to tackle heterogeneity in federated networks. FedProx can be viewed as a generalization and re-parametrization of FedAvg, the current state-of-the-art method for federated learning. While this re-parameterization makes only minor modifications to the method itself, these modifications have important ramifications both in theory and in practice. Theoretically, we provide convergence guarantees for our framework when learning over data from non-identical distributions (statistical heterogeneity), and while adhering to device-level systems constraints by allowing each participating device to perform a variable amount of work (systems heterogeneity). Practically, we demonstrate that FedProx allows for more robust convergence than FedAvg across a suite of realistic federated datasets. In particular, in highly heterogeneous settings, FedProx demonstrates significantly more stable and accurate convergence behavior relative to FedAvg-improving absolute test accuracy by 22% on average.

Federated learning (FL) is a decentralized and privacy-preserving machine learning technique in which a group of clients collaborate with a server to learn a global model without sharing clients' data. One challenge associated with FL is statistical diversity among clients, which restricts the global model from delivering good performance on each client's task. To address this, we propose an algorithm for personalized FL (pFedMe) using Moreau envelopes as clients' regularized loss functions, which help decouple personalized model optimization from the global model learning in a bi-level problem stylized for personalized FL. Theoretically, we show that pFedMe's convergence rate is state-of-the-art: achieving quadratic speedup for strongly convex and sublinear speedup of order 2/3 for smooth nonconvex objectives. Experimentally, we verify that pFedMe excels at empirical performance compared with the vanilla FedAvg and Per-FedAvg, a meta-learning based personalized FL algorithm.

FEDPROX算法是一种简单但功能强大的分布式近端优化方法，广泛用于联合学习（FL）而不是异质数据。尽管在实践中看到了它的知名度和杰出的成功，但对FEDPROX的理论理解在很大程度上是不足的：FedProx的吸引人的融合行为迄今在某些非标准和不切实际的地方功能的差异假设下的特征是，结果的优化仅限于优化的限制。问题。为了解决这些缺陷，我们通过算法稳定性的镜头开发了FedProx及其Minibatch随机扩展的新型局部差异不变理论。结果，我们有助于得出对FedProx的几个新的和更深入的见解，以实现联合优化的非凸面，包括：1）收敛确保独立于局部差异类型条件； 2）融合保证非平滑FL问题； 3）关于Minibatch的尺寸和采样设备的数量，线性加速。我们的理论首次揭示了局部差异和平稳性对于FedProx获得有利的复杂性界限并不是必备的。据报道，一系列基准FL数据集的初步实验结果证明了小型匹配以提高FEDPROX的样品效率的好处。

在联合学习（FL）中，通过跨设备的模型更新进行合作学习全球模型的目的倾向于通过本地信息反对个性化的目标。在这项工作中，我们通过基于多准则优化的框架以定量的方式校准了这一权衡，我们将其作为一个受约束的程序进行了：设备的目标是其本地目标，它试图最大程度地减少在满足非线性约束的同时，以使其满足非线性约束，这些目标是其本地目标。量化本地模型和全局模型之间的接近度。通过考虑该问题的拉格朗日放松，我们开发了一种算法，该算法允许每个节点通过查询到一阶梯度Oracle将其Lagrangian的本地组件最小化。然后，服务器执行Lagrange乘法器上升步骤，然后进行Lagrange乘法器加权步骤。我们称这种实例化的原始偶对方法是联合学习超出共识（$ \ texttt {fedBc} $）的实例。从理论上讲，我们确定$ \ texttt {fedBc} $以与最算好状态相匹配的速率收敛到一阶固定点，直到额外的错误项，取决于由于接近性约束而产生的公差参数。总体而言，该分析是针对非凸鞍点问题的原始偶对偶的方法的新颖表征。最后，我们证明了$ \ texttt {fedBc} $平衡了整个数据集（合成，MNIST，CIFAR-10，莎士比亚）的全球和本地模型测试精度指标，从而与艺术现状达到了竞争性能。

This study investigates clustered federated learning (FL), one of the formulations of FL with non-i.i.d. data, where the devices are partitioned into clusters and each cluster optimally fits its data with a localized model. We propose a novel clustered FL framework, which applies a nonconvex penalty to pairwise differences of parameters. This framework can automatically identify clusters without a priori knowledge of the number of clusters and the set of devices in each cluster. To implement the proposed framework, we develop a novel clustered FL method called FPFC. Advancing from the standard ADMM, our method is implemented in parallel, updates only a subset of devices at each communication round, and allows each participating device to perform a variable amount of work. This greatly reduces the communication cost while simultaneously preserving privacy, making it practical for FL. We also propose a new warmup strategy for hyperparameter tuning under FL settings and consider the asynchronous variant of FPFC (asyncFPFC). Theoretically, we provide convergence guarantees of FPFC for general nonconvex losses and establish the statistical convergence rate under a linear model with squared loss. Our extensive experiments demonstrate the advantages of FPFC over existing methods.

In Federated Learning, we aim to train models across multiple computing units (users), while users can only communicate with a common central server, without exchanging their data samples. This mechanism exploits the computational power of all users and allows users to obtain a richer model as their models are trained over a larger set of data points. However, this scheme only develops a common output for all the users, and, therefore, it does not adapt the model to each user. This is an important missing feature, especially given the heterogeneity of the underlying data distribution for various users. In this paper, we study a personalized variant of the federated learning in which our goal is to find an initial shared model that current or new users can easily adapt to their local dataset by performing one or a few steps of gradient descent with respect to their own data. This approach keeps all the benefits of the federated learning architecture, and, by structure, leads to a more personalized model for each user. We show this problem can be studied within the Model-Agnostic Meta-Learning (MAML) framework. Inspired by this connection, we study a personalized variant of the well-known Federated Averaging algorithm and evaluate its performance in terms of gradient norm for non-convex loss functions. Further, we characterize how this performance is affected by the closeness of underlying distributions of user data, measured in terms of distribution distances such as Total Variation and 1-Wasserstein metric.Recently, the idea of personalization in FL and its connections with MAML has gained a lot of attention. In particular, [32] considers a formulation and algorithm similar to our paper, and elaborates

我们展示了一个联合学习框架，旨在强大地提供具有异构数据的各个客户端的良好预测性能。所提出的方法对基于SuperQualile的学习目标铰接，捕获异构客户端的误差分布的尾统计。我们提出了一种随机训练算法，其与联合平均步骤交织差异私人客户重新重量步骤。该提出的算法支持有限时间收敛保证，保证覆盖凸和非凸面设置。关于联邦学习的基准数据集的实验结果表明，我们的方法在平均误差方面与古典误差竞争，并且在误差的尾统计方面优于它们。

Federated Learning有望在不访问数据的情况下与多个客户进行协作培训模型的能力，但是当客户的数据分布彼此差异时脆弱。这种差异进一步导致了困境：“我们是否应该优先考虑学习模型的通用性能（用于服务器的将来使用）或其个性化绩效（对于每个客户端）？”这两个看似竞争的目标使社区分裂了专注于一个或另一个，但在本文中，我们表明可以同时实现这两者。具体而言，我们提出了一个新颖的联邦学习框架，该框架将模型的双重职责与两个预测任务相结合。一方面，我们介绍了一个损失家族，这些损失家庭对非相同的班级分布，使客户能够培训一个通用的预测指标，并以一致的目标培训。另一方面，我们将个性化预测变量作为一种轻巧的自适应模块，以最大程度地减少每个客户在通用预测指标上的经验风险。借助我们将联合强大的脱钩（FED-ROD）命名的两个损失的两次挑战框架，学识渊博的模型可以同时实现最先进的通用和个性化的性能，从而实质上弥补了这两个任务。

The increasing size of data generated by smartphones and IoT devices motivated the development of Federated Learning (FL), a framework for on-device collaborative training of machine learning models. First efforts in FL focused on learning a single global model with good average performance across clients, but the global model may be arbitrarily bad for a given client, due to the inherent heterogeneity of local data distributions. Federated multi-task learning (MTL) approaches can learn personalized models by formulating an opportune penalized optimization problem. The penalization term can capture complex relations among personalized models, but eschews clear statistical assumptions about local data distributions. In this work, we propose to study federated MTL under the flexible assumption that each local data distribution is a mixture of unknown underlying distributions. This assumption encompasses most of the existing personalized FL approaches and leads to federated EM-like algorithms for both client-server and fully decentralized settings. Moreover, it provides a principled way to serve personalized models to clients not seen at training time. The algorithms' convergence is analyzed through a novel federated surrogate optimization framework, which can be of general interest. Experimental results on FL benchmarks show that our approach provides models with higher accuracy and fairness than state-of-the-art methods.

联邦学习（FL）试图在本地客户端分发模型培训，而无需在集中式数据中心收集数据，从而消除了数据私人关系问题。 FL的一个主要挑战是数据异质性（每个客户的数据分布可能会有所不同），因为它可能导致本地客户的权重差异并减慢全球融合。当前专为数据异质性设计的SOTA FL方法通常会施加正则化以限制非IID数据的影响，并且是状态算法，即它们随着时间的推移维持局部统计数据。尽管有效，但这些方法只能用于FL的特殊情况，仅涉及少数可靠的客户。对于fl的更典型应用，客户端数量很大（例如，边缘设备和移动应用程序），这些方法无法应用，激发了对任何可用于任何数量客户端使用的无状态方法的无状态方法的需求。我们得出了一阶梯度正则化，以惩罚由于本地数据异质性而导致的本地更新不一致。具体而言，为了减轻权重差异，我们将全局数据分布的一阶近似引入本地目标，该目标凭直觉地惩罚了与全局更新相反方向的更新。最终结果是一种无状态的FL算法，可实现1）在非IID数据分布下，比SOTA方法明显更快地收敛（即较少的通信回合）和2）总体融合性能更高。重要的是，我们的方法不会对客户大小施加不切实际的限制，从而可以从大多数FL应用程序中向大量客户学习。

Fairness and robustness are two important concerns for federated learning systems. In this work, we identify that robustness to data and model poisoning attacks and fairness, measured as the uniformity of performance across devices, are competing constraints in statistically heterogeneous networks. To address these constraints, we propose employing a simple, general framework for personalized federated learning, Ditto, that can inherently provide fairness and robustness benefits, and develop a scalable solver for it. Theoretically, we analyze the ability of Ditto to achieve fairness and robustness simultaneously on a class of linear problems. Empirically, across a suite of federated datasets, we show that Ditto not only achieves competitive performance relative to recent personalization methods, but also enables more accurate, robust, and fair models relative to state-of-the-art fair or robust baselines.

The statistical heterogeneity of the non-independent and identically distributed (non-IID) data in local clients significantly limits the performance of federated learning. Previous attempts like FedProx, SCAFFOLD, MOON, FedNova and FedDyn resort to an optimization perspective, which requires an auxiliary term or re-weights local updates to calibrate the learning bias or the objective inconsistency. However, in addition to previous explorations for improvement in federated averaging, our analysis shows that another critical bottleneck is the poorer optima of client models in more heterogeneous conditions. We thus introduce a data-driven approach called FedSkip to improve the client optima by periodically skipping federated averaging and scattering local models to the cross devices. We provide theoretical analysis of the possible benefit from FedSkip and conduct extensive experiments on a range of datasets to demonstrate that FedSkip achieves much higher accuracy, better aggregation efficiency and competing communication efficiency. Source code is available at: https://github.com/MediaBrain-SJTU/FedSkip.

我们提出了一种用于分布式培训神经网络模型的新型联合学习方法，其中服务器在每轮中随机选择的设备的子集之间编制协作。我们主要从通信角度查看联合学习问题，并允许更多设备级别计算来节省传输成本。我们指出了一个基本的困境，因为当地 - 设备水平的最低实证损失与全球经验损失的最小值不一致。与最近的事先有关的不同，尝试无所作用的最小化或利用用于并行化梯度计算的设备，我们为每轮的每个设备提出动态规范器，以便在极限中，全局和设备解决方案对齐。我们通过实证结果对真实的和合成数据以及我们的方案在凸和非凸面设置中导致有效培训的分析结果，同时对设备异质性完全不可知，以及大量设备，部分参与和不平衡的数据。

在联合学习（FL）的新兴范式中，大量客户端（例如移动设备）用于在各自的数据上训练可能的高维模型。由于移动设备的带宽低，分散的优化方法需要将计算负担从那些客户端转移到计算服务器，同时保留隐私和合理的通信成本。在本文中，我们专注于深度，如多层神经网络的培训，在FL设置下。我们提供了一种基于本地模型的层状和维度更新的新型联合学习方法，减轻了非凸起和手头优化任务的多层性质的新型联合学习方法。我们为Fed-Lamb提供了一种彻底的有限时间收敛性分析，表征其渐变减少的速度有多速度。我们在IID和非IID设置下提供实验结果，不仅可以证实我们的理论，而且与最先进的方法相比，我们的方法的速度更快。

可扩展性和隐私是交叉设备联合学习（FL）系统的两个关键问题。在这项工作中，我们确定了FL中的客户端更新的同步流动聚合不能高效地缩放到几百个并行培训之外。它导致ModelPerforce和训练速度的回报递减，Ampanysto大批量培训。另一方面，FL（即异步FL）中的客户端更新的异步聚合减轻了可扩展性问题。但是，聚合个性链子更新与安全聚合不兼容，这可能导致系统的不良隐私水平。为了解决这些问题，我们提出了一种新颖的缓冲异步聚合方法FedBuff，这是不可知的优化器的选择，并结合了同步和异步FL的最佳特性。我们经验证明FEDBuff比同步FL更有效，比异步FL效率更高3.3倍，同时兼容保留保护技术，如安全聚合和差异隐私。我们在平滑的非凸设置中提供理论融合保证。最后，我们显示在差异私有培训下，FedBuff可以在低隐私设置下占FEDAVGM并实现更高隐私设置的相同实用程序。

我们提出了一个新颖的框架，以研究异步联合学习优化，并在梯度更新中延迟。我们的理论框架通过引入随机聚合权重来表示客户更新时间的可变性，从而扩展了标准的FedAvg聚合方案，例如异质硬件功能。我们的形式主义适用于客户具有异质数据集并至少执行随机梯度下降（SGD）的一步。我们证明了这种方案的收敛性，并为相关最小值提供了足够的条件，使其成为联邦问题的最佳选择。我们表明，我们的一般框架适用于现有的优化方案，包括集中学习，FedAvg，异步FedAvg和FedBuff。这里提供的理论允许绘制有意义的指南，以设计在异质条件下的联合学习实验。特别是，我们在这项工作中开发了FedFix，这是FedAvg的新型扩展，从而实现了有效的异步联合训练，同时保留了同步聚合的收敛稳定性。我们在一系列实验上凭经验证明了我们的理论，表明异步FedAvg以稳定性为代价导致快速收敛，我们最终证明了FedFix比同步和异步FedAvg的改善。

随着对数据隐私和数据量迅速增加的越来越关注，联邦学习（FL）已成为重要的学习范式。但是，在FL环境中共同学习深层神经网络模型被证明是一项非平凡的任务，因为与神经网络相关的复杂性，例如跨客户的各种体系结构，神经元的置换不变性以及非线性的存在每一层的转换。这项工作介绍了一个新颖的联合异质神经网络（FEDHENN）框架，该框架允许每个客户构建个性化模型，而无需在跨客户范围内实施共同的架构。这使每个客户都可以优化本地数据并计算约束，同时仍能从其他（可能更强大）客户端的学习中受益。 Fedhenn的关键思想是使用从同行客户端获得的实例级表示，以指导每个客户的同时培训。广泛的实验结果表明，Fedhenn框架能够在跨客户的同质和异质体系结构的设置中学习更好地表现客户的模型。

知识共享和模型个性化是应对联邦学习（FL）中非IID挑战的重要组成部分。大多数现有的FL方法侧重于两个极端：1）学习共享模型，以使用非IID数据为所有客户提供服务，以及2）为每个客户（即个性化fl）学习个性化模型。有一个权衡解决方案，即群集或集群个性化的FL，旨在将相似的客户聚集到一个集群中，然后在集群中为所有客户学习共享模型。本文是通过将群集群集制定为可以统一现有方法的双层优化框架来重新审视群集的研究。我们提出了一个新的理论分析框架，以通过考虑客户之间的凝聚力来证明融合。此外，我们以一种称为加权聚类联合学习（WECFL）的算法体现了该框架。经验分析验证了理论结果，并证明了在拟议的集群非IID设置下提出的WECFL的有效性。

In federated optimization, heterogeneity in the clients' local datasets and computation speeds results in large variations in the number of local updates performed by each client in each communication round. Naive weighted aggregation of such models causes objective inconsistency, that is, the global model converges to a stationary point of a mismatched objective function which can be arbitrarily different from the true objective. This paper provides a general framework to analyze the convergence of federated heterogeneous optimization algorithms. It subsumes previously proposed methods such as FedAvg and FedProx and provides the first principled understanding of the solution bias and the convergence slowdown due to objective inconsistency. Using insights from this analysis, we propose Fed-Nova, a normalized averaging method that eliminates objective inconsistency while preserving fast error convergence.