Uncertainty-Induced Transferability Representation for Source-Free Unsupervised Domain Adaptation

Recent years have witnessed great achievements in the vanilla UDA [ben2010theory, gretton2006kernel, long2017conditional, ganin2016domain, chhabra2021glocal, chhabra2021iterative, han2022learning, moon2022multi, deng2022dynamic, deng2021joint, xu2021neutral]. However, they assume that the source data is accessible during the adaptation, which is not always practical. SFUDA aims to adapt a source-trained model to an unlabeled target domain without access to source data [liang2020we, ye2021source, yang2021model]. Without the labeled source data, some methods propose to generate labeled data by generative adversarial net (GAN) [goodfellow2014generative]. Kurmi et al.[kurmi2021domain] generate source data using the source-trained classifier, so that the vanilla UDA methods can be applied. Li et al.[li2020model] leverage a conditional GAN to directly produce training samples in the target style. These methods use the source model as auxiliary supervisions to control the label of the generated data. Nevertheless, the source model is ineffective for the data generation process, due to the instability training of GAN [arjovsky2017towards]. Most existing SFUDA methods directly transfer all the source knowledge to the target model at the start of training, infer the semantic information of target data using the target model, and update the target model with the inferred semantic information. SHOT [liang2020we] and ISFDA [li2021imbalanced] predict the target category using the pseudo-labeling strategy. CPGA [qiu2021source] and BAIT [yang2020unsupervised] propose to align the samples with category-wise prototypes in a contrastive learning framework. NRC[yang2021exploiting] and LSC-SDA[yang2021generalized] aim at propagating the categorical semantics from the neighborhood/cluster structure to the feature space. Xia et al. [xia2021adaptive] focus on the disagreements between target data and the source model. They select partial target data with high agreements with the source model and apply the source model to these samples. However, without measuring the transferability of the source knowledge, these methods fail to control over discarding non-transferable knowledge and preserving transferable knowledge. Additionally, they fail to identify risks of applying the source model to infer target semantics. To our best knowledge, Wang et al. [wang2022exploring] is the most similar work as ours. They explored to transfer only partial source model parameters based on calculating the parameter variations after each adaptation procedure in the stochastic optimization. However, their measurement is susceptible to the quality of the adaptive procedures. In contrast, our method measures the transferability using only the source model and unlabeled target data, which is irrelevant to the adaptation procedure, so that can avoid the hazards of the unreliable adaptation.

Uncertainty is an important criterion to measure the robustness of a deep model[kwon2021repurposing, kendall2017uncertainties, gawlikowski2021survey, sensoy2018evidential]. Given an annotated sample $(x,y)$ and a model parameterized by $\theta$ trained on domain $D$, the uncertainty can be decomposed into the following equation:

where $\mu =\theta \left(x\right)$ is the predicted label distribution and the three probability density functions represent the data uncertainty, model uncertainty, and distributional uncertainty respectively [gawlikowski2021survey, nandy2020towards, gao2020reducing]. The data uncertainty is almost irreducible which arises from the natural complexity of the data, such as the class overlap, label noise, homoscedastic and heteroscedastic noise. The model uncertainty measures how well the model fits to its training distribution [malinin2018predictive, gawlikowski2021survey, nandy2020towards]. The distributional uncertainty measures the probability of an input instance that is sampled from a region that the model is “unfamiliar” with. Its characteristic has prompted its usage in the out-of-distribution detection [sedlmeier2019uncertainty, padhy2020revisiting, mcallister2019robustness] and also in vanilla UDA methods [gao2020reducing, liang2019exploring]. To our best knowledge, our work is the first to propose the use of uncertainty to explore transferability in SFUDA.

It is essential to asses and measure the model transferability and there are two mainstream methods in the deep learning community. Firstly, the transferability of a model is measured by how much it can bridge the domain discrepancy between the source and the target domain[chen2019transferability]. It can be calculated by domain discrepancy measurements such as Proxy $A$-distance [ben2010theory] and Maximum Mean Discrepancy (MMD) [gretton2006kernel]. In addition, Chen et al. [chen2019transferability] propose the Corresponding Angle to measure the transferability. However, these methods require the access to the source data which is not suitable for SFUDA. Secondly, some transfer learning methods investigated the transferability of pre-trained source representations to the target domain. Existing works in this line of research have been proposed, such as the NCE [tran2019transferability], LEEP [nguyen2020leep] and LogME [you2021logme]. Nevertheless, they need the target data annotations which are not applicable in SFUDA. In contrast, our proposed method can estimate the transferability in the absence of source data and target data labels that fits the challenging SFUDA setup.

Not all knowledge in the source model is transferable and discriminative to the target domain. Therefore, it brings risks if we do not measure and quantify the transferability of source knowledge but deploy it directly in the target domain. However, previous transferability measurements require some matched information, either both the source and target data, or data-annotation pairs of the target domain. These requirements are infeasible in SFUDA where only unmatched source model and target data are provided. The unmatched information makes it extremely challenging to measure the transferability by acquiring “known and certain” information as the supervision signal.

To this end, our work alternatively explores and exploits the uncertainty as a fundamental tool, and proposes an Uncertainty Distance (UD) to address these challenges. The UD is an implicit distance between the target instance $x_t$ and the source domain $D_s$. A low UD demonstrates that $x_t$ is “close” to $D_s$ given a source model parameterized by ${\theta }_s$, which reflects that it is efficient for the ${\theta }_s$ to reduce the domain discrepancy between the source and target domains. Therefore it suggests that the ${\theta }_s$ is transferable to the target domain and a high UD indicates the opposite.

Our consideration is shown in Fig. 2 (a)-(c). Given the source model parameterized by ${\theta }_s$, the model uncertainty characterizes the degree to which the fitted region of ${\theta }_s$ covers its training distribution (i.e. the source domain $D_s$). While given both the ${\theta }_s$ and the target instance $x_t$, the distributional uncertainty reveals how far the $x_t$ is from the fitted region of the ${\theta }_s$. Previous observations inspired us that the cooperation of the two uncertainties reveals the distance between the target instance $x_t$ and the source domain $D_s$. Such a distance implicitly reflects the contributions of the source model to reduce the domain discrepancy. It can also be used to probe and measure the transferability of the source model to the target instance for SFUDA.

By incorporating the distributional uncertainty and the model uncertainty, we first formulate the UD as:

where $M(\cdot )$ is the uncertainty measurement function such as the Sensitivity Analysis [nagy2007distributional], the Deep Ensembles[lakshminarayanan2016simple] and the MC dropout [gal2016dropout].

Although it requires the $D_s$ in Equation 2 to measure the model uncertainty, we argue that it is still feasible to estimate transferability using UD in the SFUDA. The reason is that the source model has been well-trained in the source domain so that the ${\theta }_s$ fits $D_s$ well. As shown in Fig. 2 (c), in this case, the model uncertainty is small enough to be ignored, and the UD in SFUDA can be approximated by the distributional uncertainty calculated by the target instance $x_t$ and source model parameters ${\theta }_s$:

The proposed UD in Equation 3 essentially measures the transferability of the whole source knowledge (i.e., the whole ${\theta }_s$) to the target instances. Nevertheless, as motivated in the introduction, only partial knowledge is useful for the target domain. Therefore we proposed to analyse the transferability of the knowledge in a finer-grained manner: to determine which part of the learned source parameters are transferable to the target domain. A straight-forward method is to measure the transferability of the partial and individual source parameters $\theta$ using $UD(x_t,\theta )$, where $\theta \subset {\theta }_s$. However, it is well-known that most deep neural networks belong to the end-to-end “black-box” system, where the knowledge is highly abstract and entangled. Individual parameters generally make no sense, let alone analyzing their transferability.

To tackle this challenge, we propose to estimate the transferability of different channels of the source encoder rather than different model parameters, as shown in Fig. 2 (d). In this way, the transferability of a particular channel natural represents the transferability of the ”partial” source knowledge (relevant parameter) that encodes the feature of this channel. More specifically, we propose the Uncertainty-induced Transferability Representation (UTR), a transferability spectrum, composed of the instance axis and the channel axis, which is formulated as:

where $UD(x_t,h_s^i)=M\left(P\right(z_i|x_t,h_s^i\left)\right)$, $x_t$ is the target instance, $z=h_s\left(x_t\right),z\in R^d$ denotes the $d$-channel target features produced by the source encoder $h_s$, and $z_i=\left(h_s\right(x_t){)}^i=h_s^i(x_t)$ is the target feature of the $i^{th}$ channel, $h_s^i$ is the potential source parameter to encode $z_i$. The instance axis of UTR denotes which target data is used to calculate the UD for the transferability estimating. The channel axis represents the transferability of different channels of the source encoder. The $i^{th}$ channel of the UTR (i.e., $UD(x_t,h_s^i)$) indicates the transferability of $z_i$ to the target domain. A low value of $UD(x_t,h_s^i)$ indicates that the target instance $x_t$ is close to the source one in the space of $z_i$ and suggests that the source knowledge to encode $z_i$ (i.e., the parameters $h_s^i$) is highly transferable across the two domains.

To calculate the UTR, we adopt the sensitivity analysis method [nagy2007distributional] as the uncertainty measurement $M(\cdot )$ for Equation 4. To be specific, the model parameters of $h_s$ are perturbed for $T$ times randomly as follows: $\{h_{s;T}=(1+r_t)\ast h_s{\}}_{t=1}^T$ are firstly calculated by inserting $T$ random perturbations $\{r_t{\}}_{t=1}^T$ to original parameter ${\theta }_{hs}$. Then the uncertainty is estimated by calculating the variance of the $T$ outputs of the $i^{th}$ dimension feature:

Given a source model parameterized by ${\theta }_s$ and a target instance $x_t$, the UTR (in Equation (4)) is able to quantify the fine-grained transferability of the instance-level target features. In order to tackle the limitations in the SFUDA community: 1) measuring the transferability of source knowledge to target domain to sufficiently exploit it; 2) measuring the risk of inferring semantic information of target instances using the source knowledge, we design two variants of the UTR on two levels: the domain-level UTR namely the $UT{R}_D$ and the instance-level UTR namely the $UT{R}_I$.

The $UT{R}_D$ describes the domain-level transferability estimation over the channel axis, which identifies how transferable each channel of the source encoder is to the target domain using the UD of the source model to all target instances. The $UT{R}_I$ characterizes the instance-level trasferability over all the target instances, which identifies the instance-level risk of inferring target semantic labels. The two are useful measurements proposed to fit in the later on adaptation framework for SFUDA problem.

Specifically, as shown in Fig. 2 (e), the $UT{R}_D$ is calculated by integrating the $UTR(x_t,h_s)$ of all $n_t$ target instances over the target domain $D_t$. Detailed formulation is as follows:

As for the instance-level transferability spectrum, the $UT{R}_I$ is calculated by integrating the $UTR(x_t,h_s)$ over all the $d$-channels of the source encoder $h_s$, as shown in Fig. 2 (f). The detailed formulation of the $UT{R}_I$ is as follows:

In this paper, we focus on the $K$-way visual object classification task. SFUDA provides a well-trained source model parameterized by ${\theta }_s$ to the target domain $D_t$, where ${\theta }_s=g_s\circ h_s$, $h_s$ is the parameter of the source encoder, and $g_s$ is the parameter of the source classifier. The target domain $D_t=\{x_t^i{\}}_{i=1}^{n_t}$ consists of $n_t$ unlabeled target instances. The SFUDA aims to learn a discriminative target model parameterized by ${\theta }_t=g_t\circ h_t$ using the ${\theta }_s$ and $D_t$.

Most existing SFUDA methods directly transfer all source knowledge to the target model at the start of training, infer the semantic information (target labels) of target data using the model, and directly update the model using the inferred semantic information. However, they are limited as follows: 1) the utilization of the source knowledge is limited. Directly transferring all source knowledge to the target model ignores discarding the less-transferable one. And updating the models using the inferred target semantics failed to preserve the discriminative knowledge in the source model. 2) the target semantic information inferred by the source model is risky due to the less-transferable source knowledge. Refining the model using the risky semantic information is unreliable.

To this end, we introduce the Calibrated Adaptation Framework (CAF). To tackle the first limitation, we propose to calibrate the source knowledge that transferred to the target model using our $UT{R}_D$. To tackle the second, we propose to calibrate the inferred semantic information of target instances based on our $UT{R}_I$. Finally, we adapt the model based on the calibrated source knowledge and target semantics. The overview of CAF is shown in Fig. 3. The pseudo-code of the whole algorithm is described in Algorithm 1.

Source knowledge calibration. To address the limitation 1, instead of directly inheriting all source knowledge, we design a transferability-controlled knowledge distillation loss $L_{kd}$, which used $UT{R}_D$ to control the knowledge distillation by quantifying different channels’ transferability and assigning more transferable channels larger weights. On the one hand, it prompts the target model to learn transferable source knowledge and discard less-transferable ones. On the other hand, it constrains the updated target model by unceasingly distilling the transferable source knowledge along the whole training process, rather than at the beginning only.

Target semantics calibration. The less-transferable knowledge is prone to lead to incorrect semantics (labels) inferred by the model. Considering that it is fundamental in SFUDA to update the target model based on the inferred semantics of target instances, calibrating the incorrect semantics is essential to learn a discriminative target model. Specifically, after inferring target semantics (using the source model at the beginning of training, and the target model later), we use the $UT{R}_I$ to select instances whose inferred semantics are unreliable. Then, a semantic calibration loss $L_{sc}$ is designed to calibrate their model predictions. Specifically, on the one hand, as the semantics inferred by the feature of less-transferable channels tend to be wrong, we proposed to use to “forget” the current semantics by minimizing a negative cross-entropy $L_c$. It implicitly guides the model to re-initialize the parameters representing the less-transferable knowledge. On the other hand, we minimize the entropy of the prediction probability distribution of these instances to force their predictions close to a new and appropriate class category. This procedure discovers the new and proper semantics of these instances.

Adaptation. With the above two steps, the target model “safely” integrates the source knowledge and target semantics. The adaptation step finally refines the target model using inferred semantics from the transferable knowledge, therefore summarizing a better-performing discriminative model.

Not all source knowledge is transferable and discriminative to the target domain. Directly transferring all source knowledge to the target model without dealing with the less-transferable parts of it is detrimental to the adaptation of the target domain. To this end, we instruct the target model to selectively learn the features of transferable channels of the source encoder, therefore, to inherit transferable knowledge from the source encoder. Given the source encoder $h_s$, the $UT{R}_D\left(h_s\right)$ is calculated following the Equation (6) to estimate the transferability of each channel in $h_s$, where a lower $UT{R}_D$ value suggests stronger transferability. Then, the target model learn the source knowledge based on the identified transferability. We proposed a novel transferability-controlled knowledge distillation loss as the objective:

where $Q\left(x\right)=sigmoid(-x)$ is a monotone minus function, $\odot$ is the Hadamard product. The $Q\left(UT{R}_D\right(h_s\left)\right)$ weights the mean squared error term $\parallel h_s\left(x_t\right)-h_t\left(x_t\right){\parallel }_2$ to distill knowledge within $h_s$ to $h_t$, aiming to assign large weights to features with low $UT{R}_D$ while small ones to those with high $UT{R}_D$, guiding the target model to learn more transferable knowledge from the source model and discard less-transferable ones in a well-controlled manner.

Refining a target model based on the inferred semantics (labels) of target instances is a fundamental and important step for the adaptation in SFUDA. Due to the less-transferable source knowledge, the predicted target semantics may be incorrect, which greatly hinders the adaptation to the target domain. To this end, we design the target semantics calibration module to calibrate the target semantics.

First, the inferred semantics of a target instance $x_t$ is $\hat{y}=\arg maxp\left(x_t\right)$ with probability $p^{\hat{y}}\left(x_t\right)$, where $p\left(x_t\right)=\sigma \left({\theta }_s\right(x_t\left)/{\theta }_t\right(x_t\left)\right)$ is the source/target model predicted probability distribution, $\sigma (.)$ is the softmax function. Note that we use the source model to infer target semantics at the first epoch, and turn to use the target model later since the target model will be more discriminative to the target domain after adaptation.

Second, we leverage $UT{R}_I$ to detect risk target instances whose semantic is prone to be incorrectly inferred that satisfies $\{x_t:UT{R}_I(x_t)>\tau \}$ as $X_{t;risk}$, where $\tau$ denotes the threshold. Following the first step, the feature encoder that calculates $UT{R}_I\left(x_t\right)$ (Equation 7) changes from $h_s$ to $h_t$ after the first epoch.

Third, based on the detected instances $X_{t;risk}$, we propose a semantics calibrated loss $L_{sc}$ to calibrate the semantics of these instances. Since their semantics is prone to be inaccurate, we train the target model to firstly forget these semantics by minimizing the negative cross-entropy loss. The forget objective $L_f$ is represented as follows:

As illustrated in Fig. 3 (b.3), optimizing this term decreases the prediction probability to the misclassified category $\hat{y}$.

On the other hand, we guide the target model to discover the true semantic by the following discover objective $L_d$:

where $p\left(x_t\right)=\sigma \left({\theta }_t\right(x_t\left)\right)$ is the target model predicted probability distribution. $L_d$ aims to minimize the entropy of the $p\left(x_t\right)$, thus guiding the model to assign the prediction to an appropriate class. Note that, instead of only minimizing the entropy on $X_{t;risk}$, we calculate $L_d$ on all target instances $X_t$. In this way, the semantic information of instances with low $UT{R}_I$, where the model tends to make the right predictions, is also introduced to help the semantic discovery of instances in $X_{t;risk}$.

Such a ”forget-discover” process implicitly guides the model to free itself from the shackles of less-transferable knowledge and facilitates the discovery of the true semantics of the target data, and the semantic calibration loss can be denoted as:

where $\gamma$ is the scale coefficient of the $L_f$.

With the above two steps to calibrate the source knowledge and target semantics, the target model then can safely adapt to the target model. In the adaptation step, we re-infer the target semantics by the model and use it to refine the target model, ultimately adapting the model to the target domain.

In this step, we adopt the pseudo-label strategy in [liang2020we] to re-infer the semantic $\hat{y}$ of the target instance $x_t$ consider its simplicity and effectiveness. Given $x_t$ and $\hat{y}$, we optimize the model with the cross-entropy loss and the objective of the adapt step can be formulated as:

In this subsection, we summarize the training steps of CAF framework. The two calibration steps are separate with the adapt step. Specifically, in the $i^{th}$ epoch, perform two calibration steps to calibrate transferable source knowledge and target semantics by:

where $\lambda$ and $\gamma$ is the scale coefficient.

And in the $i+1^{th}$ epoch, conduct the adaption step to adapt the target model to the target domain:

We evaluate our SFUDA method using the following three benchmarks: Office-31 [saenko2010adapting], the Office-Home[venkateswara2017deep] and the VisDA[peng2017visda]. Office-31[saenko2010adapting] contains 4,652 images in 31 categories from three domains: Amazon (A), Webcam (W) and DSLR (D). Office-Home[venkateswara2017deep] consists of four domains, i.e., Artistic images (Ar), Clip Art (Cl), Product images (Pr), and Real-World images (Rw), with 65 classes and a total of 15,500 images. VisDA[peng2017visda] is a more challenging dataset, whose source domain contains 152k synthetic images generated by rendering 3D models while the target domain has 55k real object images sampled from Microsoft COCO [lin2014microsoft].

For fair comparisons with existing methods, we adopt the backbone of ResNet-50 [he2016deep] for Office-31 and Office-Home and ResNet-101 for VisDA. Following the setups in [liang2020we, yang2021exploiting], along with the backbones, we used a fully-connected (fc) layer with the output channels of $256$ as the encoder. A fc layer with the weight normalization as the classifier. The source model is trained following the same strategy with [liang2020we, yang2021exploiting]. The pre-trained source model is used to adapt to the target domain but without using any labeled source data. In the optimization, we adopt SGD with momentum 0.9 and batch size of 64 on all datasets. For the Office-31 and office-Home datasets, the learning rates used to train the ResNet-50 backbone and the newly added layers are 1e-3 and 1e-2 respectively. The learning rate is 1e-4 for VisDA. We trained 40, 60 and 50 epochs for Office-31, Office-Home and VisDA respectively. Note that the mixup[zhang2017mixup] data augmentation is used in the adaptation step. The threshold of $UT{R}_I$, i.e. $\tau$, is set to be 3. The weight $\lambda$ of the transferability-controlled knowledge distillation loss is set to 10 at the beginning. As the training procedure progresses, the model is gradually adapted to the target domain, requiring less source knowledge. Therefore, after 10 epochs, we decrease $\lambda$ to zero. The weight $\gamma$ of the ”forget” loss is set to 0.9. For the uncertainty measurement (Equation 5), $T$ is set to $2$, and $r_t$ is randomly sampled from the uniform distribution $U(-0.05,0.05)$.

We report the results on Office-31, Office-Home, and VisDA, in Tables I, II, and III, respectively.

On Office-31 tasks, in terms of the average accuracy of 6 transfer tasks, our method outperforms the state-of-the-art work A2Net[xia2021adaptive] and DIPE[wang2022exploring] by 0.2%, improving from 90.1% to 90.3%. We also achieve the state-of-the-art results on W$\to$A and W$\to$D. For other transfer directions of Office-31, we achieved very competitive results. We hypothesize the reason of the results is that our method brings transferability risk quantification to SFUDA and integrates the “safe-to-transfer” source knowledge to the target domain for better adaptation. We also argue that our method is more useful and brings more improvements for challenging adaptation tasks, where the cross-domain transfer risk is high. Instead, the Office-31 transfer tasks are easy and bring less risks (considering that the average accuracy of the source-only model is 79.1%), so our method improvement is competitive and not that significant.

As expected, on the more challenging Office-Home tasks and VisDA tasks (the mean accuracy of the source models are 60.0% and 48.0%), our method brings larger improvement. In the Office-Home tasks, we achieve the state-of-the-art performance on 8 of 12 tasks, and also outperform the prior work in terms of the average accuracy of 0.6%, improving from 72.6% (by A2Net[xia2021adaptive]) to 73.2%. Particularly, we have achieved significant improvements in two difficult tasks Ar$\to$Cl and Re$\to$Cl and outperform the second best one by 1.2% and 0.7%, respectively. On the VisDA tasks, our method outperforms others among 10 out of 12 tasks and surpasses the SOTA method NRC by a large margin (2.4%) in the per-class accuracy.

Compared with the most related work DIPE[wang2022exploring], our method also obtains performance improvement in all three benchmarks, including 0.2% in Office-31, 0.7% in Office-Home and 4.2% in VisDA. The reported results clearly demonstrate the efficacy of our method.

In addition, using the $L_f$ only without the $L_d$ brings negative effects (see the comparison [v6 $\leftrightarrow$ Ours]). However, from comparisons [v3 $\leftrightarrow$ v4] and [v5 $\leftrightarrow$ Ours] in Table IV, we verify that $L_f$ is only effective when combined with $L_d$.

Moreover, we notice that the forget loss $L_f$ is more effective on challenging tasks, e.g. Ar$\to$Cl. We hypothesize that it is because the predictions on challenging tasks tend to be wrong and therefore forgetting model prediction completely brings more improvements.

To further understand the forget loss $L_f$ and prove the previous arguments, we visualize the correct/incorrect case of target prediction and the $UT{R}_I$ obtained by the source model, our CAF model without the forget loss and our CAF model in Fig. 4 (a), (b) and (c) respectively. It can be seen in (a) that the source model predicts many wrong semantics in the target domain (the red points) due to the less-transferable knowledge. By observing many red points on the upper right of the Fig. 4 (b), it seems that we can not calibrate the wrong semantics of samples with high prediction probability without $L_f$, since the model is confident in its inferred semantics and tends to maintain these semantics. Finally, from Fig. 4 (c), it can be seen that after adding the forget term $L_f$, the model forgets the wrong semantics and finally calibrates their semantics. The above experiments verify the effectiveness of the Target Semantics Calibration.

We analyse the sensitivity of the following hyperparameters: the $UT{R}_I$ threshold $\tau$ and the weights $\lambda$ and $\gamma$ of the losses $L_{KD}$ and $L_f$ respectively. The results in Table V demonstrate that our method is stable to the choices of hyperparameters in a wide range.

Without measuring the transferability, current SFUDA methods [yang2021exploiting, yang2021generalized, xia2021adaptive, yang2020unsupervised, liang2020we] directly perform adaptation but ignore calibration steps in our CAF framework. Our two calibration steps fill in the gap, and are flexible and “plug-and-play”. Therefore, we add our calibration modules on existing SFUDA works [yang2021exploiting, liang2020we] and report the experimental results in Table VI. It can be seen that adding our calibration modules on SHOT/NRC methods improves SHOT/NRC by 0.9/0.8%, 1.2/0.3%, and 2.2/2.1% on Cl$\to$Ar, Cl$\to$Pr and Cl$\to$Re respectively. It proves that our CAF method is “plug-and-play” and effective to different SFUDA baselines.

Implementation. The experiments are conducted on the Office-31, Office-Home, and VisDA tasks. For the Office-31 tasks and the Office-Home tasks, the backbone of ResNet-50 along with a fc layer is the source encoder, whose output channel $d=256$. A fc layer with weight normalization is the classifier. For the VisDA tasks, we replace the ResNet-50 with the ResNet-101 and keep the other settings the same. We follow [liang2020we, yang2021exploiting] to train the source model. For each task, we feedforward all target data to the pre-trained source model and finally calculate $UT{R}_D\left(h_s\right)$ according to Equation 6, where $T=2$, $r_t$ is randomly sampled from $U(-0.05,0.05)$.

Comparison Protocols. We evaluate our $UT{R}_D$ by comparing it with the existing transferability measurements, that are inapplicable in the SFUDA, including: MMD [gretton2006kernel], A-Distance [ben2010theory], Corresponding Angle [chen2019transferability], LogME [you2021logme], LEEP [nguyen2020leep] and NCE[tran2019transferability]. In addition, the performance (prediction accuracy) is also considered as an extra intuitive measurement. Considering that existing transferability measurements are not suitable for a single channel’s feature, we design the following comparison protocol. Specifically, we sort the 256 channels representations $z=h_s\left(x\right)$ and split them into two separate 128 channels vectors $Z_{low}$ and $Z_{high}$, representing the channels with the 128 smallest $UT{R}_D^i\left(h_s\right)$ and the 128 largest $UT{R}_D^i\left(h_s\right)$ respectively. In other words, the conclusion of $UT{R}_D\left(h_s\right)$ is that $Z_{low}$ is more transferable than $Z_{high}$. Then we calculate the existing transferability measurements on $Z_{low}$ and $Z_{high}$ and report the consistency of their conclusion with ours. Note that these source data and target annotation are given when using these measurements. The results are reported in Table VII and VIII.

First, we quantitatively measure the transferability of $Z_{low}$ and $Z_{high}$ with the two vanilla UDA methods, the MMD and the A-Distance, which requires the source data. These methods measure the ability to bridge the domain discrepancy between the source and target domain. The lower MMD/A-Distance, the more transferable the model is. From Table VII and VIII, it can be seen that in 15 out of 19 adaptation tasks, the MMD of $Z_{low}$ is lower than that of $Z_{high}$, and in 17 tasks, the A-Distance of $Z_{low}$ is lower than that of $Z_{high}$. The result indicates that $UT{R}_D$ is consistent with these domain discrepancy measurements in most case, which suggest features in channels with less $UT{R}_D$ are more effective to bridge the domain discrepancy, therefore; they are more transferable.

Second, we compared with the Corresponding Angle, which is proposed by Chen et al. [chen2019transferability] according to their observation that the eigenvectors with the largest singular values will dominate the feature transferability. We can observe that in 17 cases, the Corresponding Angle of $Z_{low}$ is larger than that of $Z_{high}$ in all experiments, demonstrating the consistency of $UT{R}_D$ with the Corresponding Angle.

Third, we compare the consistency of our method with the transferability measurements LogME, NCE, and LEEP that estimate the potential of the source model parameter in learning a well-performed target model by refining. In Table VII and VIII, it can be seen that $UT{R}_D$ is consistent with LogME in all tasks , and also consistent with NCE and LEEP in 18 and 17 cases, respectively. These results denote that the relevant source model parameters to encode $Z_{low}$ is more transferable than $Z_{high}$.

Finally, We also calculate the classification performances of $Z_{low}$ and $Z_{high}$ on the target domain. Using $Z_{low}$, for example, we set the features of channels which not belongs to $Z_{low}$ to zero, feedforward the modified feature into the classifier to get prediction and calculate the prediction accuracy.. $Z_{high}$ is evaluated in the same way. As shown in Table VII and VIII, the prediction using $Z_{low}$ is more accurate than $Z_{high}$ in target domain on all adaptation tasks. For example, the accuracy of $Z_{low}$ outperform $Z_{high}$ by 3.1%, 4.8% and 1.3% on Pr$\to$Ar, Pr$\to$Cl and Pr$\to$Re, respectively. Note that we did not extra train the source model but only split it into two part of channels $Z_{low}$ and $Z_{high}$ according to our $UT{R}_D$. The significant performance gap between the two parts indicates that $Z_{low}$ with less $UT{R}_D$ is more transferable to the target domain than $Z_{high}$. The experimental observations from the series of studies above illustrate that 1) the proposed $UT{R}_D$ is strongly consistent with current transferability measurements and can estimate the transferability, 2) Our method can effectively analyse the internal transferability of the source model, the channels of source encoder with lower $UT{R}_D$ is more transferable to the target domain, which allows us to leverage the source knowledge more efficiently and safely, which proves our motivation.