Identifying Latent Causal Content for Multi-Source Domain Adaptation

Traditional machine learning requires the training and testing set to be independent and identically distributed distributions (vapnik1999overview). This strict condition may not be fulfilled in various potential real-world applications. For example, in medical applications, it is common to seek to train a model on patients from few hospitals, and generalize it to a new hospital (zech2018variable). In this case it is often reasonable to consider that the distributions of data from training hospitals are different from the new hospital (koh2021wilds). Domain adaptation (DA) is a promising research area to handle such problems. In general, DA can frequently be considered in multi-source settings (MSDA) where source domain data are collected from multiple domains. Formally, let $x$ denote the input, e.g. image, $y$ denote the labels in source and target domains, and $D$ denote the domain index. We observe labeled data pairs $(x^S,y^S)$ from the multiple joint distributions $p(x,y|D_1),p(x,y|D_2),...,p(x,y|D_M)$ in source domains, and unlabeled input data $x^T$ from the joint distribution $p(x,y|D_T)$ in target domain. The training phase of MSDA is to use $(x^S,y^S)$ and $x^T$, to train a predictor so that it can provide a satisfactory estimation for $y^T$ in the target domain. The key for MSDA is to understand how the joint distribution change across different domains, resulting in $p(x,y|D_1)\ne p(x,y|D_2)...\ne p(x,y|D_M)\ne p(x,y|D_T)$.

Most early methods assume the change of the joint distribution results from Covariate Shift (huang2006correcting; bickel2007discriminative; sugiyama2007direct; wen2014robust), e.g., $p(x,y|D_i)=p\left(y|x\right)p\left(x|D_i\right)$, as depicted by Figure 1(a). This setting assumes that $p\left(x\right)$ changes across domains, while the conditional distribution $p\left(y|x\right)$ is invariant across domain. Such assumption is too strong for some real applications, especially for image data. For example, the assumption of invariant $p\left(y|x\right)$ implies that $p\left(y\right)$ should change as the change of $p\left(x\right)$. However, we can easily change style information in image data that is irrelevant to the label (e.g., background, view), so that the distribution $p\left(x\right)$ changes while the distribution of $p\left(y\right)$ remains unchanged, which clearly violates the assumption.

In contrast to covariate shift above, most current works consider a more reasonable assumption, Conditional Shift as depicted by Figure 1(b). It assumes that $p\left(x|y\right)$ changes while the distribution $p\left(y\right)$ is invariant across domains (zhang2013domain; zhang2015multi; scholkopf2012causal; stojanov2021domain; peng2019moment). This setting motivates a popular class of methods focusing on learning invariant representations across different domains to approach the true $z$ in Figure 1(b) (ganin2016domain; zhao2018adversarial; saito2018maximum; mancini2018boosting; yang2020curriculum; wang2020learning; li2021t; stojanov2021domain). However, the label distribution $p\left(y\right)$ may change across domains in many real application scenarios (tachet2020domain; lipton2018detecting; zhang2013domain). In these scenarios, those methods based on learning invariant representations, may be resulting in degenerating performance. In fact, recent work has proven an upper bound on the performance of those methods when label distributions change across domains (zhao2019learning).

To relieve the problem above, this work proposes a new assumption, Latent Covariate Shift (LCS), as depicted by Figure 1(c). Unlike to conditional shift, LCS assumes that there is a latent content variable $z_c$, whose marginal distribution $p\left(z_c\right)$ is variant across domains, and the label distribution conditional on it $p\left(y|z_c\right)$ is invariant. Due to the change of $p\left(z_c\right)$ across domains, LCS allows label distribution $p\left(y\right)$ to change across domain, due to the invariant $p\left(y|z_c\right)$. To more deeply understand and handle LCS, we propose a latent causal graph to formulate data and label generating process, by using two latent variables, e.g., the latent content variable $z_c$ and the latent style variable $z_s$ as depicted in Figure 2. In the proposed latent causal graph, the domain variable $D$ causes the independent exogenous variables $n_c$ and $n_s$, which are corresponding to the latent content variable $z_c$ and the latent style variable $z_s$, respectively. The observed input data is caused by both $z_c$ and $z_s$, while label is only caused by $z_c$. We show that although it is often impossible to identify the latent style variable $z_s$ without strong assumptions, due to transitivity property in latent space, the latent content variable $z_c$ can be identifiable up to simple scaling by using the identifiability result from nonlinear ICA (khemakhem2020variational) and dependence between $n_c$ and $y$. This motivates us to propose a novel method to learn the invariant conditional distribution $p\left(y|z_c\right)$ for MSDA, instead of learning invariant representations conditional on label. Since $z_c$ is identifiable, the proposed method provides a principled way to guarantee that the learned predictor $p\left(y|z_c\right)$ can be generalized to the target domain. Empirical evaluation on synthetic and real data demonstrates the effectiveness of the proposed method, compared with many state-of-the-art methods.

Learning invariant representations. Due to the strong assumptions in covariate shift, most current works for domain adaptation consider conditional shift, which learns invariant representations across domains (ganin2016domain; zhao2018adversarial; saito2018maximum; mancini2018boosting; yang2020curriculum; wang2020learning; li2021t). Such representations can be obtained by applying suitable linear or nonlinear transformation on the input data. The key of these methods is how to enforce the invariance of the learned representations. For example, the invariance can be enforced by maximum classifier discrepancy (saito2018maximum), or by a domain discriminator for adversarial training (ganin2016domain; zhao2018adversarial), or by moment matching (peng2019moment), or by relation alignment loss (wang2020learning). However, all these methods assume label distribution to be invariant across domains. As a result, when label distribution is varying across domains, they may perform well only in the overlapping areas among the all label distributions in different domains, and face with challenges in the non-overlapping areas. To overcome this, some works propose to learn invariant representations conditional on the label across domains (gong2016domain; ghifary2016scatter; tachet2020domain). One of the challenges in these methods is that the labels in the target domain is unavailable. More importantly, these methods does not guarantee that the learnt representations to be consistent with the true relevant information for predicting the label in the target domain, thus there is no principled way to guarantee that the learned predictor can be generalized to the target domain.

Learning invariant conditional distribution $p\left(y|z_c\right)$. There exist few of works exploring the invariant conditional distribution $p\left(y|z_c\right)$ for domain adaptation (kull2014patterns; bouvier2019hidden). Differ from these two works, the proposed method provides the identifiability of $z_c$, so that the learned $p\left(y|z_c\right)$ in this work can be generalized to the target domain in a principled way. Besides, in the context of out-of-distribution generalization, some recent works explore learning invariant conditional distribution $p\left(y|z_c\right)$ (arjovsky2019invariant; sun2021recovering; liu2021learning; lu2021invariant). For example, arjovsky2019invariant imposes learn the optimal invariant predictor across domains from the viewpoint of an intimate link between invariance and causation, while the proposed method directly explores conditional invariance given the proposed latent causal graph. sun2021recovering mainly focus on single domain, while the proposed method consider multiple domains. The proposed method is also different from the work in liu2021learning in that the former assume the latent content variable caused by the style variable, while the latter depends on a confounder to model the causal relation between the latent content variable and the style variable. Unlike the work in lu2021invariant that the label is treated as a variable causing the other latent variables, the proposed method assumes that the label have no child nodes.

To more deeply understand and handle LCS, we introduce a latent causal graph as depicted by Figure 2. It introduces the observed domain variable $D$ to denote in which specific domain data are collected. Let $n_c$ and $n_s$ denote the noise (exogenous) variables According to the definition of structural causal model (Pearl00; Spirtes00), $n_c$ and $n_s$ should be mutually independent conditional on the observed $D$. Both $n_c$ and $n_s$ are corresponding to the latent content variable $z_c$ and the latent style variable $z_s$, respectively. Here $z_c$ and $z_s$ denote the latent causal content information and the latent style information, respectively. Generally speaking, $z_c$ and $z_s$ should be dependent given the domain variable $D$. Here we consider that $z_c$ causes $z_s$, to model the correlation between $z_s$ and $y$. In the proposed latent causal graph, $z_c$ change across domains while $p\left(y|z_c\right)$ is invariant across domains, which meets the basic assumption in latent covariate shift. In the following, we discuss two key causal relations, which also highlights the novelty of the proposed latent causal graph.

$z_c$ causes $y$: Previous works consider the causal relation between $x$ and $y$ as $y\to x$ (gong2016domain; stojanov2019data; li2018domain), while we employ $z_c\to y$. We argue that these two cases are not contradictory since the labels $y$ in these two cases represent two different physical meanings. To understand this point, let $\hat{y}$ replace $y$ in the first case (i.e., $\hat{y}\to x$) to distinguish from $y$ in the second case. For the first case, consider the generative process of images. A label should be first sampled, e.g., $\hat{y}$, then one may determine content information regarding to the label $\hat{y}$, and finally generate a image, which is a reasonable assumption in many real application scenarios. In the proposed latent causal graph, $n_c$ play a role to replace $\hat{y}$ and causes the content variable $z_c$. We then assume $z_c\to y$, which formulates the process that experts extract content information from given images and then provide reasonable labels according to their domain knowledge. This assumption has been made by some recent works (mahajan2021domain; liu2021learning; sun2021recovering). Particularly, these two different labels, $\hat{y}$ and $y$, has been considered in the work (mahajan2021domain). Here we provide a further detailed interpretation for the difference between $\hat{y}$ and $y$.

$z_c$ causes $z_s$: It is clear that there exists a spurious correlation between the label $y$ and the style variable $z_s$ in many real applications. We here employ $z_c$ as a confounding factor of both $y$ and $z_s$ to model the spurious correlation. The rationality of this assumption can be further verified from the viewpoint of the converse. In particular, if we assume that $z_s$ causes $z_c$, all high-level information in input data $x$, $z_s$ and $z_c$, would be causally related to the label $y$, which can not model the spurious correlation and is obviously unreasonable. Therefore, assuming $z_c\to z_s$ is more persuasive and consistent with previous works (gong2016domain; stojanov2019data; mahajan2021domain). One recent work in (sun2021recovering) leverages a additional variable as a confounding factor that causes both the content variable $z_c$ and the style variable $z_s$ to model their relation. However, the identifiability in the work (sun2021recovering) do not depend on the confounding factor. As a result, the confounding factor can be incorporated into the domain index, which is equivalent to the case where $z_c$ and $z_s$ are independent for a single domain. By contrast, the proposed latent graph assumes a more general setting where $z_c$ and $z_s$ are dependent for a single domain, as depicted by Figure 2. Experimental results will further highlight the advantages of the proposed latent causal graph, compared with the work (sun2021recovering).

In this section, we will show that $z_c$ can be identifiable up to simple scaling under some mild conditions. To this end, we first provide some identifiable results in nonlinear ICA (khemakhem2020variational), which shows that the exogenous variables, $n_c$ and $n_s$, can be identifiable up to simple permutation and scaling with some mild conditions. With this result, we then show that although the proposed latent causal graph as a whole is unidentifiable due to transitivity property, the part of the graph $z_c$ can also be identifiable up to simple scaling, by using the dependence between $n_c$ and $y$.

The identifiable $z_c$ provides a principled way to guarantee that we can learn the conditional distribution $p\left(y|z_c\right)$ to be generalized to the target domain. In this section we propose a novel method to show how to learn the invariant conditional distribution $p\left(y|z_c\right)$ for MSDA.

The key for domain adaptation is understanding how the joint distribution of input data and label changes across domains. Previous works assume the covariate shift or the conditional shift to interpret the change of the joint distribution, which may be severely restricted in real applications. This work relaxes these assumptions and proposes a new assumption, latent covariate shift. To handle it, we propose a latent causal graph to more precisely formulate the generative process of input data and label, by using a latent causal variable and a latent style variables. Built on the identifiability result of nonlinear ICA, we show that the latent causal variable can be identified up to scaling. This motivates us a new method to learn invariant label distribution conditional on the latent causal variable, with a principled way to guarantee generalization. Experiments show the advantages of our theoretical results and the performance of the the proposed method, compared with state-of-the-art methods across various dataset.