LANIT: Language-Driven Image-to-Image Translation for Unlabeled Data

We adopt an adversarial loss to guide the translated images $\hat{y}$ to arrive in the training distribution of the target domain. Following StarGAN2 and TUNIT, we also adopt the multi-domain discriminators, but we use the outputs of multi-domain discriminators weighted by a multi-hot domain label $d^y$:

where $D_n(\cdot )$ denotes $n$-th discriminator output and $d_n^y$ denotes $n$-th element of $d^y$. It should be noted that if we set $d^y$ as an one-hot domain label by ground-truth or pseudo-label, this loss function becomes the same adversarial loss in choi2020starganv2; baek2021rethinking.

We further present a domain-consistency loss that encourages the networks to generate an image $\hat{y}$ following the multi-hot domain label of input style image $y$ more precisely. Perhaps one of the simplest function is to use the dot product between $f^y$ and $f^{\hat{y}}$ such that $L=f^y\cdot f^{\hat{y}}$. Minimizing this, however, can induce erroneous solutions, e.g., a constant prompt at all domains, as the prompt is also used as input for the translation module. To overcome this, inspired by SimSiam chen2021exploring, our domain-consistency loss $L_{dc}$ employs the pseudo multi-hot domain labels as targets because they are detached:

where $H(\cdot ,\cdot )$ denotes a cross-entropy to handle multi-hot labels. In the loss, the first term enables training the prompt only, and the second term enables training the remaining translation modules with the current prompt fixed, helping stable convergence and local minima problem.

To make the translated image similar to its original image, which also regularizes the ill-posed translation problem, we adopt a cycle-consistency loss such that:

where $c^{\hat{y}}$ denotes the content from $\hat{y}$, and $a^x$ denotes the style vector of input $x$. By encouraging the generator $G$ to reconstruct the input image $x$ with estimated style vector $a^x$, $G$ learns to preserve the original characteristics of $x$.

While the generator is able to synthesize realistic images with the losses above, the synthetic results are not guaranteed to be style-consistent with $y$. In order to better learn style representation, we compute $l$-1 loss between style vector from the translated image and style image such that

Our full loss functions are as follows:

where ${\lambda }_{adv}$, ${\lambda }_{dc}$, ${\lambda }_{cyc}$, and ${\lambda }_{sty}$ are hyper-parameters.

We conduct the experiments on five standard datasets, including Animal Faces-10, Food-10, CelebA-HQ, FFHQ, and LHQ bossard2014food; liu2019few; liu2015deep; karras2019style; skorokhodov2021aligning. In particular, we consider 10 textual domain descriptions among Animal Faces-149 and Food-101, respectively, selected in TUNIT, to define initial prompt of our framework. For CelebA-HQ and FFHQ, we obtain 40 attributes from CelebA-HQ, and sample 10 attributes for three times to define initial prompt, and report the averaged results.

In our experiments, we adopt four quantitative evaluation metrics. First, the mean of class-wise Frénchet Inception Distance (mFID) heusel2017gans is used to evaluate how the translated image reflects the input style. In addition, we use Density and Coverage (D&C) naeem2020reliable, which measure the fidelity and diversity of the translated images, respectively. The lower mFID value, the better quality of the image. As fidelity and diversity get better, the D&C scores are larger or closer to 1. We also measure Acc., namely classification accuracy, between the domain labels with the highest probability and ground-truth domain labels.

We first report the quantitative comparison of our model with StarGAN2 (supervised), TUNIT and Kim et al. kim2022style (unsupervised). Note that StarGAN2 is trained using 10 classes of ground-truth per-sample domain labels. Tab. 1 shows the quantitative comparison in terms of mFID and D&C metrics. In this section, we mainly compare (IV), which is our final model, with other methods. Detailed analysis of the comparison from (I) to (III) will be provided in the ablation section. LANIT (IV) achieves the best mFID and D&C scores, proving that our model outperforms the state-of-the-art methods in terms of image quality, fidelity and diversity. We also provide the quantitative comparison on Animal Faces-10 and Food-10 datasets in Tab. 2 in terms of mFID, D&C, and Acc. metrics. We can observe that LANIT consistently outperforms TUNIT by a large margin on Animal Faces-10 and Food-10 datasets  on Acc. and mFID. Although our LANIT only utilizes unlabeled datasets with dataset-level text supervision, it shows competitive results against StarGAN2, which requires ground-truth per-sample domain labels. The classification accuracy demonstrates that our vision-language based clustering method has remarkably superior performance. Note that we do not provide quantitative results on FFHQ since it does not provide the ground-truth labels.

We show visual results for Tab. 1 in Fig. 5. We can observe that the quality differences between our LANIT and TUNIT or Kim et al. are significant. The images generated by the other methods tend to contain artifacts and show limited performance in style transferring from style images, while our method yields stable performance in terms of style representation and semantic consistency. Especially, we show that our model is capable of capturing multiple subtle attributes very well, such as “smiling” or “bang”, while TUNIT and Kim et al. often fail. Interestingly, our LANIT shows competitive results compared to StarGAN2 which is a supervised method, even with better style preservation. Moreover, in Fig. 6, we visualize the reference-guided image translation results of TUNIT, Kim et al and our LANIT on FFHQ and CelebA-HQ. The results of LANIT consistently show higher quality than TUNIT and Kim et al. on both datasets. We also show domain-to-domain diverse image synthesis of our LANIT in Fig. 7 guided by target domain labels. Thanks to our proposed prompt learning technique, the mapping network and style encoder can faithfully produce the style vectors reflecting the target multiple domain styles.

In Tab. 3, We validate the effects of the number of candidate domains $N$ compared to TUNIT. Our results show that a different number of $N$ brings minor changes to our model than TUNIT, which justifies the robustness of our framework. In addition, to analyze the effectiveness of each strategy on different $N$’s, we measured Acc. in Fig. 8. In our setting, the use of both $\varnothing$ domain and prompt learning technique improves the labeling accuracy, which is highlighted in $N=4,7,13,16$. In animal faces-10 and food-10, LANIT shows the best performance when there exist 10 candidate domains, whereas in CelebA-HQ, there is little difference in the FID score when $N$ is over 10. We observe a trade-off between our labeling accuracy and the number of domain prompts; the model with more domain labels has higher representation capacity with more diverse attributes, while it has high computational complexity.

We study the effectiveness of $\varnothing$ domain in Tab. 1 (III). We observe the model with $\varnothing$ domain highly outperforms the baseline model. This indicates that the unknown domain enables the model to mitigate the influence of ambiguous or noisy data. Our results show that LANIT with $\varnothing$ domain shows impressive performance improvement, compared to the model without prompt learning in terms of classification accuracy and mFID. Also in Fig. 8, we find that our $\varnothing$ domain brings large improvement in performance for every benchmark, especially in CelebA-HQ. In the results, we observe that our assumption that an image may have more than a single attribute is well demonstrated with datasets that inherits more diverse attributes; For CelebA-HQ, domain labels can be described with diverse attributes such as hair style or facial attributes, while Animal Faces-10 and Food-10 are visually distinct, leading to limited domain descriptions.

In order to evaluate the effectiveness of prompt learning, we compare the estimated pseudo-domain labels with and without the proposed prompt learning, where the latter can be regarded as an offline clustering. It shows that with prompt learning, the probability of image/text prompt pair is more accurately estimated. Since the prompt learning helps reduce a label noise by refining the prompt, the result without prompt learning shows rather limited clustering performance, which is verified in Tab. 1(IV).

In Fig. 8, prompt learning is more effective in Animal Faces-10 and Food-10, while $\varnothing$ domain brings large improvement in performance in CelebA-HQ. This highlights the importance of the use of both of our proposed strategies, helping to find optimal domain labels.

Additionally, we show the effect of selecting single or multiple activated Top-$K$ attributes in Tab. 1. We can see that the result of Top-3 (II) significantly surpasses Top-1 (I). This demonstrates that an image can be described more precisely when given with multiple keywords or text descriptions, while most prior works including TUNIT, Kim et al. and StarGAN2 assume to assign a single label for each image. However, we choose Top-1 domain label for Animal Faces-10 and Food-10, since animal species or kinds of food cannot be described as multi attributes . It shows the flexibility of our framework.

Finally, we conducted a user study to evaluate the image quality, content preservation, and style consistency of LANIT compared to StarGAN2, TUNIT and Kim et al. 204 users were involved in this study, asked to answer 60 questions, each of which is generated from randomly sampled 20 images from CelebA-HQ. The examples of questions are as follows: “Which results do you think have the highest quality/preserve content information such as pose/have similar style of reference image?" Fig. 9 summarizes the user study, where our LANIT achieves the top rank in all tasks.