Discriminative learning methods for classification perform well when training and test data are drawn from the same distribution. In many situations, though, we have labeled training data for a source domain, and we wish to learn a classifier which performs well on a target domain with a different distribution. Under what conditions can we adapt a classifier trained on the source domain for use in the target domain? Intuitively, a good feature representation is a crucial factor in the success of domain adaptation. We formalize this intuition theoretically with a generalization bound for domain adaption. Our theory illustrates the tradeoffs inherent in designing a representation for domain adaptation and gives a new justification for a recently proposed model. It also points toward a promising new model for domain adaptation: one which explicitly minimizes the difference between the source and target domains, while at the same time maximizing the margin of the training set. This situation arises often in machine learning. For example, we might want to adapt for a new user (the target domain) a spam filter trained on the email of a group of previous users (the source domain), under the assumption that users generally agree on what is spam and what is not. Then, the challenge is that the distributions of emails for the first set of users and for the new user are different. Intuitively, one might expect that the closer the two distributions are, the better the filter trained on the source domain will do on the target domain.

Discriminative learning methods for classification perform well when training and test data are drawn from the same distribution. Often, however, we have plentiful labeled training data from a source domain but wish to learn a classifier which performs well on a target domain with a different distribution and little or no labeled training data. In this work we investigate two questions. First, under what conditions can a classifier trained from source data be expected to perform well on target data? Second, given a small amount of labeled target data, how should we combine it during training with the large amount of labeled source data to achieve the lowest target error at test time?

Due to the ability of deep neural nets to learn rich representations, recent advances in unsupervised domain adaptation have focused on learning domain-invariant features that achieve a small error on the source domain. The hope is that the learnt representation, together with the hypothesis learnt from the source domain, can generalize to the target domain. In this paper, we first construct a simple counterexample showing that, contrary to common belief, the above conditions are not sufficient to guarantee successful domain adaptation. In particular, the counterexample exhibits conditional shift: the class-conditional distributions of input features change between source and target domains. To give a sufficient condition for domain adaptation, we propose a natural and interpretable generalization upper bound that explicitly takes into account the aforementioned shift. Moreover, we shed new light on the problem by proving an information-theoretic lower bound on the joint error of any domain adaptation method that attempts to learn invariant representations. Our result characterizes a fundamental tradeoff between learning invariant representations and achieving small joint error on both domains when the marginal label distributions differ from source to target. Finally, we conduct experiments on real-world datasets that corroborate our theoretical findings. We believe these insights are helpful in guiding the future design of domain adaptation and representation learning algorithms.

所有著名的机器学习算法构成了受监督和半监督的学习工作，只有在一个共同的假设下：培训和测试数据遵循相同的分布。当分布变化时，大多数统计模型必须从新收集的数据中重建，对于某些应用程序，这些数据可能是昂贵或无法获得的。因此，有必要开发方法，以减少在相关领域中可用的数据并在相似领域中进一步使用这些数据，从而减少需求和努力获得新的标签样品。这引起了一个新的机器学习框架，称为转移学习：一种受人类在跨任务中推断知识以更有效学习的知识能力的学习环境。尽管有大量不同的转移学习方案，但本调查的主要目的是在特定的，可以说是最受欢迎的转移学习中最受欢迎的次级领域，概述最先进的理论结果，称为域适应。在此子场中，假定数据分布在整个培训和测试数据中发生变化，而学习任务保持不变。我们提供了与域适应性问题有关的现有结果的首次最新描述，该结果涵盖了基于不同统计学习框架的学习界限。

This paper addresses the problem of unsupervised domain adaption from theoretical and algorithmic perspectives. Existing domain adaptation theories naturally imply minimax optimization algorithms, which connect well with the domain adaptation methods based on adversarial learning. However, several disconnections still exist and form the gap between theory and algorithm. We extend previous theories (Mansour et al., 2009c;Ben-David et al., 2010) to multiclass classification in domain adaptation, where classifiers based on the scoring functions and margin loss are standard choices in algorithm design. We introduce Margin Disparity Discrepancy, a novel measurement with rigorous generalization bounds, tailored to the distribution comparison with the asymmetric margin loss, and to the minimax optimization for easier training. Our theory can be seamlessly transformed into an adversarial learning algorithm for domain adaptation, successfully bridging the gap between theory and algorithm. A series of empirical studies show that our algorithm achieves the state of the art accuracies on challenging domain adaptation tasks.

We introduce a new representation learning approach for domain adaptation, in which data at training and test time come from similar but different distributions. Our approach is directly inspired by the theory on domain adaptation suggesting that, for effective domain transfer to be achieved, predictions must be made based on features that cannot discriminate between the training (source) and test (target) domains.The approach implements this idea in the context of neural network architectures that are trained on labeled data from the source domain and unlabeled data from the target domain (no labeled target-domain data is necessary). As the training progresses, the approach promotes the emergence of features that are (i) discriminative for the main learning task on the source domain and (ii) indiscriminate with respect to the shift between the domains. We show that this adaptation behaviour can be achieved in almost any feed-forward model by augmenting it with few standard layers and a new gradient reversal layer. The resulting augmented architecture can be trained using standard backpropagation and stochastic gradient descent, and can thus be implemented with little effort using any of the deep learning packages.We demonstrate the success of our approach for two distinct classification problems (document sentiment analysis and image classification), where state-of-the-art domain adaptation performance on standard benchmarks is achieved. We also validate the approach for descriptor learning task in the context of person re-identification application.

学习域不变的表示已成为域适应/概括的最受欢迎的方法之一。在本文中，我们表明不变的表示可能不足以保证良好的概括，在考虑标签函数转移的情况下。受到这一点的启发，我们首先在经验风险上获得了新的概括上限，该概括风险明确考虑了标签函数移动。然后，我们提出了特定领域的风险最小化（DRM），该风险最小化（DRM）可以分别对不同域的分布移动进行建模，并为目标域选择最合适的域。对四个流行的域概括数据集（CMNIST，PACS，VLCS和域）进行了广泛的实验，证明了所提出的DRM对域泛化的有效性，具有以下优点：1）它的表现明显超过了竞争性盆地的表现； 2）与香草经验风险最小化（ERM）相比，所有训练领域都可以在所有训练领域中具有可比性或优越的精度； 3）在培训期间，它仍然非常简单和高效，4）与不变的学习方法是互补的。

无监督域适应（UDA）的绝大多数现有算法都集中在以一次性的方式直接从标记的源域调整到未标记的目标域。另一方面，逐渐的域适应性（GDA）假设桥接源和目标的$（t-1）$未标记的中间域，并旨在通过利用中间的路径在目标域中提供更好的概括。在某些假设下，Kumar等人。 （2020）提出了一种简单的算法，逐渐自我训练，以及按$ e^{o（t）} \ left的顺序结合的概括（\ varepsilon_0+o \ of \ left（\ sqrt {log（log（log（t）/n log（t）/n） } \ right）\ right）$对于目标域错误，其中$ \ varepsilon_0 $是源域错误，$ n $是每个域的数据大小。由于指数因素，当$ t $仅适中时，该上限变得空虚。在这项工作中，我们在更一般和放松的假设下分析了逐步的自我训练，并证明概括为$ \ varepsilon_0 + o \ left（t \ delta + t/\ sqrt {n} {n} \ right） + \ widetilde { o} \ left（1/\ sqrt {nt} \ right）$，其中$ \ delta $是连续域之间的平均分配距离。与对$ t $作为乘法因素的指数依赖性的现有界限相比，我们的界限仅取决于$ t $线性和添加性。也许更有趣的是，我们的结果意味着存在最佳的$ t $的最佳选择，从而最大程度地减少了概括性错误，并且自然也暗示了一种构造中间域路径的最佳方法，以最大程度地减少累积路径长度$ t \ delta源和目标之间的$。为了证实我们理论的含义，我们检查了对多个半合成和真实数据集的逐步自我训练，这证实了我们的发现。我们相信我们的见解为未来GDA算法设计的途径提供了前进的途径。

转移学习或域适应性与机器学习问题有关，在这些问题中，培训和测试数据可能来自可能不同的概率分布。在这项工作中，我们在Russo和Xu发起的一系列工作之后，就通用错误和转移学习算法的过量风险进行了信息理论分析。我们的结果也许表明，也许正如预期的那样，kullback-leibler（kl）Divergence $ d（\ mu || \ mu'）$在$ \ mu $和$ \ mu'$表示分布的特征中起着重要作用。培训数据和测试测试。具体而言，我们为经验风险最小化（ERM）算法提供了概括误差上限，其中两个分布的数据在训练阶段都可用。我们进一步将分析应用于近似的ERM方法，例如Gibbs算法和随机梯度下降方法。然后，我们概括了与$ \ phi $ -Divergence和Wasserstein距离绑定的共同信息。这些概括导致更紧密的范围，并且在$ \ mu $相对于$ \ mu' $的情况下，可以处理案例。此外，我们应用了一套新的技术来获得替代的上限，该界限为某些学习问题提供了快速（最佳）的学习率。最后，受到派生界限的启发，我们提出了Infoboost算法，其中根据信息测量方法对源和目标数据的重要性权重进行了调整。经验结果表明了所提出的算法的有效性。

Top-performing deep architectures are trained on massive amounts of labeled data. In the absence of labeled data for a certain task, domain adaptation often provides an attractive option given that labeled data of similar nature but from a different domain (e.g. synthetic images) are available. Here, we propose a new approach to domain adaptation in deep architectures that can be trained on large amount of labeled data from the source domain and large amount of unlabeled data from the target domain (no labeled targetdomain data is necessary).As the training progresses, the approach promotes the emergence of "deep" features that are (i) discriminative for the main learning task on the source domain and (ii) invariant with respect to the shift between the domains. We show that this adaptation behaviour can be achieved in almost any feed-forward model by augmenting it with few standard layers and a simple new gradient reversal layer. The resulting augmented architecture can be trained using standard backpropagation.Overall, the approach can be implemented with little effort using any of the deep-learning packages. The method performs very well in a series of image classification experiments, achieving adaptation effect in the presence of big domain shifts and outperforming previous state-ofthe-art on Office datasets.

Recent work reported the label alignment property in a supervised learning setting: the vector of all labels in the dataset is mostly in the span of the top few singular vectors of the data matrix. Inspired by this observation, we derive a regularization method for unsupervised domain adaptation. Instead of regularizing representation learning as done by popular domain adaptation methods, we regularize the classifier so that the target domain predictions can to some extent ``align" with the top singular vectors of the unsupervised data matrix from the target domain. In a linear regression setting, we theoretically justify the label alignment property and characterize the optimality of the solution of our regularization by bounding its distance to the optimal solution. We conduct experiments to show that our method can work well on the label shift problems, where classic domain adaptation methods are known to fail. We also report mild improvement over domain adaptation baselines on a set of commonly seen MNIST-USPS domain adaptation tasks and on cross-lingual sentiment analysis tasks.

Domain adaptation aims at generalizing a high-performance learner on a target domain via utilizing the knowledge distilled from a source domain which has a different but related data distribution. One solution to domain adaptation is to learn domain invariant feature representations while the learned representations should also be discriminative in prediction. To learn such representations, domain adaptation frameworks usually include a domain invariant representation learning approach to measure and reduce the domain discrepancy, as well as a discriminator for classification. Inspired by Wasserstein GAN, in this paper we propose a novel approach to learn domain invariant feature representations, namely Wasserstein Distance Guided Representation Learning (WD-GRL). WDGRL utilizes a neural network, denoted by the domain critic, to estimate empirical Wasserstein distance between the source and target samples and optimizes the feature extractor network to minimize the estimated Wasserstein distance in an adversarial manner. The theoretical advantages of Wasserstein distance for domain adaptation lie in its gradient property and promising generalization bound. Empirical studies on common sentiment and image classification adaptation datasets demonstrate that our proposed WDGRL outperforms the state-of-the-art domain invariant representation learning approaches.

多类神经网络是现代无监督的领域适应性中的常见工具，但是在适应性文献中缺乏针对其非均匀样品复杂性的适当理论描述。为了填补这一空白，我们为多类学习者提出了第一个Pac-Bayesian适应范围。我们还提出了我们考虑的多类分布差异的第一个近似技术，从而促进了界限的实际使用。对于依赖Gibbs预测因子的分歧，我们提出了其他PAC-湾适应界限，以消除对蒙特卡洛效率低下的需求。从经验上讲，我们测试了我们提出的近似技术的功效以及一些新型的设计概念，我们在范围中包括。最后，我们应用界限来分析使用神经网络的常见适应算法。

Unlike human learning, machine learning often fails to handle changes between training (source) and test (target) input distributions. Such domain shifts, common in practical scenarios, severely damage the performance of conventional machine learning methods. Supervised domain adaptation methods have been proposed for the case when the target data have labels, including some that perform very well despite being "frustratingly easy" to implement. However, in practice, the target domain is often unlabeled, requiring unsupervised adaptation. We propose a simple, effective, and efficient method for unsupervised domain adaptation called CORrelation ALignment (CORAL). CORAL minimizes domain shift by aligning the second-order statistics of source and target distributions, without requiring any target labels. Even though it is extraordinarily simple-it can be implemented in four lines of Matlab code-CORAL performs remarkably well in extensive evaluations on standard benchmark datasets."Everything should be made as simple as possible, but not simpler."

A key learning scenario in large-scale applications is that of federated learning, where a centralized model is trained based on data originating from a large number of clients. We argue that, with the existing training and inference, federated models can be biased towards different clients. Instead, we propose a new framework of agnostic federated learning, where the centralized model is optimized for any target distribution formed by a mixture of the client distributions. We further show that this framework naturally yields a notion of fairness. We present data-dependent Rademacher complexity guarantees for learning with this objective, which guide the definition of an algorithm for agnostic federated learning. We also give a fast stochastic optimization algorithm for solving the corresponding optimization problem, for which we prove convergence bounds, assuming a convex loss function and hypothesis set. We further empirically demonstrate the benefits of our approach in several datasets. Beyond federated learning, our framework and algorithm can be of interest to other learning scenarios such as cloud computing, domain adaptation, drifting, and other contexts where the training and test distributions do not coincide. MotivationA key learning scenario in large-scale applications is that of federated learning. In that scenario, a centralized model is trained based on data originating from a large number of clients, which may be mobile phones, other mobile devices, or sensors (Konečnỳ, McMahan, Yu, Richtárik, Suresh, and Bacon, 2016b;Konečnỳ, McMahan, Ramage, and Richtárik, 2016a). The training data typically remains distributed over the clients, each with possibly unreliable or relatively slow network connections.Federated learning raises several types of issues and has been the topic of multiple research efforts. These include systems, networking and communication bottleneck problems due to frequent exchanges between the central server and the clients . To deal with such problems, suggested an averaging technique that consists of transmitting the central model to a subset of clients, training it with the data locally available, and averaging the local updates. Smith et al. (2017) proposed to further leverage the relationship between clients, assumed to be known, and cast

分发概括是将模型从实验室转移到现实世界时的关键挑战之一。现有努力主要侧重于源和目标域之间建立不变的功能。基于不变的功能，源域上的高性能分类可以在目标域上同样良好。换句话说，不变的功能是\ emph {transcorable}。然而，在实践中，没有完全可转换的功能，并且一些算法似乎学习比其他算法更学习“更可转移”的特征。我们如何理解和量化此类\ EMPH {可转录性}？在本文中，我们正式定义了一种可以量化和计算域泛化的可转换性。我们指出了与域之间的常见差异措施的差异和连接，例如总变化和Wassersein距离。然后，我们证明我们可以使用足够的样本估计我们的可转换性，并根据我们的可转移提供目标误差的新上限。经验上，我们评估现有算法学习的特征嵌入的可转换性，以获得域泛化。令人惊讶的是，我们发现许多算法并不完全学习可转让的功能，尽管很少有人仍然可以生存。鉴于此，我们提出了一种用于学习可转移功能的新算法，并在各种基准数据集中测试，包括RotationMnist，PACS，Office和Wilds-FMOW。实验结果表明，该算法在许多最先进的算法上实现了一致的改进，证实了我们的理论发现。

多集团不可知学习是一个正式的学习标准，涉及人口亚组内的预测因子的条件风险。标准解决了最近的实际问题，如亚组公平和隐藏分层。本文研究了对多组学习问题的解决方案的结构，为学习问题提供了简单和近最佳的算法。

域适应（DA）从严格的理论作品中获益，研究其富有识别特征和各个方面，例如学习领域 - 不变的表示及其权衡。然而，由于多个源域的参与和训练期间目标域的潜在不可用的域，因此似乎不是这种源DA和域泛化（DG）设置的情况非常复杂和复杂。在本文中，我们为目标一般损失开发了新的上限，吸引我们来定义两种域名不变的表示。我们进一步研究了利弊以及执行学习每个领域不变的表示的权衡。最后，我们进行实验检查这些陈述的权衡，以便在实践中提供有关如何使用它们的实践提示，并探索我们发达理论的其他有趣性质。

本文的目的是设计主动学习策略，从而在Lipschitz函数的假设下导致领域适应。以Mansour等人的先前作品为基础。（2009年）我们调整了源和目标分布之间的差异距离的概念，以将假设类别的最大化限制为在源域上执行准确标记的局部函数类别的最大化。我们根据Rademacher平均值和满足规律性条件的一般损失函数的局部差异来得出此类主动学习策略的概括误差界限。可以从理论界限推断出可以解决大数据集情况的实用k-媒体算法。我们的数值实验表明，在域适应性的背景下，所提出的算法与其他最先进的活跃学习技术具有竞争力，尤其是在大约十万张图像的大数据集上。

对抗性学习策略在处理单源域适应（DA）问题时表现出显着的性能，并且最近已应用于多源DA（MDA）问题。虽然大多数现有的MDA策略依赖于多个域歧视员设置，但其对潜伏空间表示的影响已经不知识。在这里，我们采用了一种信息 - 理论方法来识别和解决MDA上多个域鉴别器的潜在不利影响：域歧视信息的解体，有限的计算可扩展性以及培训期间损失梯度的大方差。我们在信息正规化的背景下通过情况进行对抗性DA来检查上述问题。这还提供了使用单一和统一域鉴别器的理论正当理由。基于这个想法，我们实施了一种名为多源信息正规化适应网络（MIAN）的新型神经结构。大规模实验表明，尽管其结构简洁，可靠，可显着优于其他最先进的方法。