Robust Sound-Guided Image Manipulation

As shown in Fig. 4, we train a set of encoders with three different modalities {audio, text, and image} to produce the matched representations in the embedding space from a video dataset. Specifically, given audio, text, and image inputs from the same video, i.e. $x_a$, $x_t$, and $x_v$, we use three different encoders to obtain a set of $d$ dimensional latent representations, i.e. $a$, $t$ and $v\in R^d$, respectively.

To learn these joint latent representations, we use a typical contrastive learning approach that follows the work by Radford et al. [radford2learning]: the latent representations for a positive triplet pair are mapped close together in the embedding space while representations of negative pair samples are mapped farther away from each other. Specifically, we use InfoNCE loss [alayrac2020self] to map positive audio-text pairs and audio-visual pairs close together in the CLIP-based joint embedding space, and we map negative pairs farther away from each other. Formally, given a minibatch of $N$ audio-text representation pairs $\{a_i,t_j\}$ for $i\in \{1,2,\dots ,N\}$, we first compute the $(i,j)$ component of the audio-text similarity matrix $M^{a\to t}\in R^{N\times N}$ between the $a_i$ and $t_j$ latent representation as:

where $a_i\cdot t_j$ represents vector dot product and $\tau$ is a temperature hyperparameter. We emphasize that $a_i$ and $t_j$ are normalized vectors, so that the magnitude of each vector is 1. This loss function is the log loss of an $N$-way classifier that seeks to predict the $i$-th diagonal component $\{a_i,t_i\}$ as the true representation pair in the similarity matrix $M^{a\to t}$.

As the loss function is asymmetric, we also define the following text-to-audio similarity matrix $M^{t\to a}$:

We maximize the diagonal component of the matrix $M^{t\to a}$ by minimizing the InfoNCE loss as:

Concretely, we minimize the following loss function $L_{\text{nce}}$ as a sum of the two losses $l_i^{(a\to t)}$ and $l_i^{(t\to a)}$ for all positive audio-text representation pairs in each minibatch of size $N$:

At the same time, we also calculate the diagonal component of the audio-visual similarity matrix $M^{a\to v}$ in the same way as in Eq. 4. We determine the following $L_{\text{nce}}^{(a\leftrightarrow v)}$:

where $l_i^{(v\to a)}$ and $l_i^{(a\to v)}$ denote the negative log-likelihood of $M_{(i,i)}^{v\to a}$ and $M_{(i,i)}^{a\to v}$ in each minibatch of size $N$.

Unlike text inputs, audio inputs can provide more diverse and find-grained semantic cues. For example, a ”thunderstorm” can be expressed in various ways: e.g., weak or severe thunderstorms, thunderstorms with heavy rain, winds, hail, or no precipitation. It may be challenging for texts to verbalize and differentiate these subtle differences, but it does not apply to sound. Thus, we apply self-supervised learning techniques between audio samples (e.g., weak vs. severe thunderstorms) to obtain an intensity-aware sound representation.

Existing self-supervised learning approaches depend on a contrastive loss that makes representations of the same-class different views close in the embedding space. In contrast, different-class views are pushed farther away. To this end, we regard the audio input of different augmented views as positive and different audio samples as negative. Note that we obtain a latent representation $\hat{a}\in R^d$ from an augmented audio input ${\hat{x}}_a$, which has been shown to be useful for improving the quality of the latent representation, as this is a common practice in self-supervised representation learning.

Formally, we define a similiarity matrix $M^{a\to \hat{a}}\in R^{N\times N}$ from embeddings obtained from audio and augmented audio. The $(i,j)$ component of the audio-audio similarity matrix $M^{a\to \hat{a}}$ is as follows:

The intensity-aware representation is obtained by minimizing the InfoNCE loss of $M_{(i,i)}^{a\to \hat{a}}$. We apply this technique to improve the quality of audio representations by minimizing $L_{\text{self}}^{(a\leftrightarrow \hat{a})}$ as follows:

where $l_i^{(a\to \hat{a})}$ and $l_i^{(\hat{a}\to a)}$ are defined in a similar way, as in Eq. 2 and 4. This loss function is useful for learning subtle differences over sound inputs, as it must maximize the mutual information between two different views of the same inputs but to minimize the mutual information between two views of the different inputs. For example, as shown in Fig. 5, an audio sample $a_i$ forms a negative pair with ${\hat{a}}_j$ for $i\ne j$, which induces a diffusive effect in the embedding space. For audio inputs, we apply the SpecAugment [park19e_interspeech], which visually augments Mel-spectrogram acoustic features by warping the features and masking blocks of frequency channels.

In addition, we propose weakly paired contrastive learning for audio-visual data to ensure that the representations of the input audio and different images with the same text labels are similar. The conventional contrastive loss only uses positive pairs which have audio and image from the same video. Therefore, audio and image from different videos are considered to be negative pairs although they may share visual features. Our new loss encourages these pairs to be closed to each other if they have the same text labels.

We first sample the new images ${\tilde{x}}_v$, which have the same text labels with given audio $x_a$. Then, we use the image encoder to obtain a new sampled image embedding $\tilde{v}\in R^d$. We apply the knowledge distillation technique [hinton2015distilling] to transfer the generalization ability of the CLIP to sound representation. The learning scheme allows the similarity matrix between audio-visual and visual-text to be equal in mini-batch (see Fig. 6).

The link knowledge between the audio and the new sampled image is transferred from CLIP to the diagonal component of a new audio-visual similarity matrix. After sampling a new triplet by sampling sound, text and new images of size $N$ in a mini-batch, we define the audio-visual similarity matrix from the $a_i$ and ${\tilde{v}}_j$ embeddings as:

where $\tau$ is a temperature hyperparameter to refine the softmax-probability distributions, as was done in [hinton2015distilling]. Specifically, we compute the pseudo cosine similarity matrix $M^{t\to \tilde{v}}$ between the latent representations of text and those of images. The audio-visual link knowledge is transferred from the fixed CLIP to $M_{(i,i)}^{a\to \tilde{v}}$. Then, we match the two types of logits in the sound-image and text-image similarity matrix instead of using hard labels such as $1$ for positive pairs. We minimize the following Kullback-Leibler divergence [hinton2015distilling] for the $i$-th diagonal component:

where KL denotes the Kullback-Leiber divergence. The KL can be rewritten as:

The KL divergence loss $L_{KL}$ for $N$ batches is as follows:

To summarize, we minimize the following loss function $L_{\text{total}}$:

Given user-provided audio input $x_a$, we employ the direct latent code optimization for sound-guided image manipulation by solving the following optimization problem:

where a given source latent vector $w_s\in W^{L\times D}$ (the intermediate latent space in StyleGAN), audio-driven latent code $w_a\in W^{L\times D}$ is optimized. $L$ denotes the number of style layers at different levels, and $D$ denotes the dimension. $L_{\text{reg}}$ is regularization loss. $L_{\text{ID}}$ and $G$ are the identity loss and StyleGAN-based generator, respectively. By minimizing the hinge loss $L_{\text{cos}}$, we find the sound-guided latent vector $w_a$ which generates an sound-guided image in the StyleGAN latent space. ${\lambda }_{\text{reg}}$ and ${\lambda }_{ID}$ are hyperparameters that control the strengths of the similarity loss term and the identity loss function $L_{\text{ID}}$. High values of ${\lambda }_{\text{reg}}$ and ${\lambda }_{ID}$ lead to maintenance of the content of the source image, whereas low values do not. The source latent code $w_s$ means the randomly generated latent code from $G$ or the latent code obtained from the existing input image through GAN inversion [richardson2021encoding, 10.1145/3450626.3459838].

With such an optimization scheme, the hinge loss $L_{\text{cos}}$ is calculated as:

In the CLIP embedding space, the distance from the source latent vector $w_s$ is increased and closer to the audio-driven latent vector $w_a$. We minimize cosine distance between the latent representation of the manipulated image $G\left(w_a\right)$ and audio latent representation $f_a\left(x_a\right)$. $f_a$ and $f_v$ denote the pre-trained audio and visual encoders.

Generally, $L_2$ regularization effectively keeps the image generated from the moved latent code from being different from the original [Patashnik_2021_ICCV]. In the StyleGAN latent space, we maintain the image similarity by minimizing the $l_2$- distance between $w_a$ and $w_s$ as follows:

However, StyleGAN’s latent code has different properties at each layer, so different weights should be applied to each layer if the user-provided attribute changes. We control style changes with adaptive layer masking. Layerwise masking keeps compact content information within style latent code as follows:

In StyleGAN2 [karras2020analyzing], the latent code is represented as $w\in W^{L\times D}$, where $L$ is the number of layers and $D$ is the latent code’s dimension. We declare a parameter vector $g\in R^L$. $g$ is a trainable vector that masks adaptively the specific style layer. $g$ ranks which of the $L$ layers is regulated by the softmax function. In the latent optimization step, $g$ and $w$ are multiplied according to the number of layers. $g$ is iteratively updated, which adaptively manipulates the latent code.

The similarity to the input image is also controlled by the identity loss function $L_{\text{ID}}$, which is defined as:

where $R$ is the pre-trained ArcFace [deng2019arcface] model for face recognition. Thus, this loss function minimizes the cosine distance $⟨R\left(G\right(w_s\left)\right),R\left(G\right(w_a\left)\right)⟩$ between its arguments in the latent space of the ArcFace network. This allows for manipulating facial expressions without changing the personal identity. Note that we disable the identity loss by setting ${\lambda }_{\text{ID}}=0$ for all other non-human image manipulations.

We first compare our sound-guided image manipulation results with existing text-guided manipulation approaches, including TediGAN [xia2021tedigan] and StyleCLIP [Patashnik_2021_ICCV]. We also compare our approach with the sound-guided image manipulation approach from our prior work [lee2022sound]. As can be observed in Fig. 8, image manipulation driven by sounds produces better manipulation quality (compare (b) and (c) vs. (d) and (e)). Furthermore, unlike text-guided methods, the audio-guided approach achieves natural image style transfer and is capable of representing multiple labels. For example, TediGAN emphasizes crying, whereas StyleCLIP focuses on the baby when the text “baby crying” is given. On the contrary, our proposed method can simultaneously handle “baby” and “crying”. Moreover, our proposed method produces more visually plausible manipulation results than does our prior work [lee2022sound], which often produced color shifts (e.g., skin tones).

We demonstrate that each audio sample has its own context, which makes the guidance in StyleGAN latent space richer than the text-based image manipulation method (see Fig. 7). Specifically, our sound representations provide guidance that correlates with the audio intensity in the StyleGAN latent space. If the magnitude of Thunder is altered or if a specific attribute like Rain is added to the audio, the style variation for the image is better correlated to the audio intensity than text-guided image manipulation. Likewise, Fire crackling, Gunfire, and Bomb sounds contain similar audio characteristics where something explodes, but our image manipulation results show the changes from the audio intensity. In particular, we would like to emphasize that no such explicit representation, such as Weak thunder in the text descriptions, is used for pre-training the multi-modal embedding space.

We visualize the direction vector with t-SNE [van2008visualizing] (see Fig. 7). By subtracting the vectors of the latent code guided by each modality and the source latent code, we show the distribution of the direction of manipulation. We select the attributes in VGG-Sound [chen2020vggsound] and randomly manipulate the audio and text prompts. Although we randomly sample the audio and text having the similar semantics, the sound-guided latent code shows a more significant transition than the text-guided latent code.

We compare our sound-guided image manipulation model with existing sound-based style-transfer models, including TräumerAI [jeong2021tr] and Crossing you in Style [lee2020crossing]. Fig. 9 compares image manipulation results in response to given audio inputs, e.g., audio of fire crackling and raining. We observe that our model produces a better quality of manipulated images. Existing models fail to capture semantic cues from the given audio input (See 2^nd and 3^rd columns). In contrast, our model manipulates images to be consistent with the semantic cues from the input sound.

Similar to our image-text-sound joint embedder, several works [wu2021wav2clip, guzhov2021audioclip, lee2022sound] have attempted to extend the CLIP model to jointly represent audio, text, and images. Here, we also compare those works with our learned embedding space to analyze their effectiveness for the image manipulation task. As shown in Fig. 10, we qualitatively compare image manipulation results based on different multi-modal joint embedding spaces learned by Wav2CLIP [wu2021wav2clip], AudioCLIP [guzhov2021audioclip], Lee et al. [lee2022sound], and our joint embedder. Given different sound inputs (e.g., bird song, bomb, weak thunder, etc.), our learned embedding space generally produces the best results compared with the other embedders (see 2nd, 3rd, and 4th rows vs. 5th row). This may confirm that our multi-modal joint embedding space is more effective for the image manipulation task than are existing approaches.

Weakly paired audio-visual contrastive learning makes the manipulation results robust. As demonstrated in Fig. 11, we demonstrate that using weak supervision is helpful for reasonable sound-guided image manipulation. For example, manipulated images maintain the audio-irrelevant attributes like glasses, hair styles and beards. Moreover, it is more robust for manipulating target attributes. It shows the manipulation results that satisfy the keywords ”baby” and ”laughing” at the same time.

In StyleGAN [jeong2021tr], it is necessary to adaptively regularize the style layer, because each layer of latent code has different style attributes. Each layer of latent code multiplies the trainable parameter that controls the diversity during regularization. The ablation study shows a qualitative comparison of the mechanism for applying adaptive layer masking to the style layer, as illustrated in Fig. 12. The adaptive masking rectifies the direction by changing the latent code based on the semantic cue. When the gate function is applied, sound-guided image manipulation is semantically reasonable. For example, a thunderstorm is a mixture of thunder and rain sound. Although thunder and lightning are not seen in the second row, lightning and rain appear in the last row.

We compare our model to the supervised method and the existing zero-shot audio classification method (see Table I). First, we compare audio embeddings trained by the supervised methods such as logistic regression and ResNet50 [hershey2017cnn] supervised by random initialization of weights as a baseline model. Our audio encoder shows better classification performance than the baseline. Secondly, we compare the zero-shot audio classification accuracy with Wav2CLIP [wu2021wav2clip] and AudioCLIP [guzhov2021audioclip]. Our proposed loss function learns three modalities in the CLIP embedding space and obtains a richer audio representation through contrastive loss. However, Wav2CLIP only understands the relationship between the audio and the visual context. Furthermore, Wav2CLIP learns from an audio-visual having a dataset with a relatively small data scale to distillate the CLIP representation to the audio encoder (the VGG-Sound dataset: 200K). AudioCLIP uses a dataset scaled up from contrastive learning between modalities from the audio, image, and text given as a triplet to obtain a more powerful sound representation. The AudioCLIP’s audio encoder was pre-trained on a large-scale audio dataset, namely AudioSet [gemmeke2017audio], which consists of over 2M sound clips.

As a result, Table I also illustrates that the power of sound representation improves when the scale of positive audio-visual data is increased with weakly paired contrastive learning than Lee et al. [lee2022sound]’s work. Moreover, our learned embedding space is more suitable for the image manipulation task.

We quantitatively analyze the effectiveness of our proposed audio-driven image manipulation approach. First, we measure performance on the semantic-level classification task. Given the audio embeddings from our pre-trained audio encoder, we train a linear classifier to recognize eight semantic labels, including giggling, sobbing, nose-blowing, fire crackling, wind noise, underwater bubbling, explosion, thunderstorm, gun, raining, and bird song. We use StyleGAN2 [karras2020analyzing] weights pre-trained from the FFHQ [karras2019style] dataset for the guiding with giggling, sobbing, and nose-blowing attributes to compare the semantic-level classification accuracy between text and audio. Also, for the guiding with fire crackling, wind noise, underwater bubbling, explosion, and thunderstorm attributes, the weights of StyleGAN2 pre-trained on the LSUN (church) [yu2015lsun] dataset are used. As shown in Fig. 16 (a), we generally outperform the existing text-driven manipulation approach with better semantically-rich latent representation.

We can see how much the latent code has changed by the cosine similarity between the source latent code and the manipulated latent code. We compare the cosine similarity between text-guided and sound-guided latent representations. We evaluate the mean and variance of the cosine similarity between $w_s$, a source latent code, $w_a$, an audio-driven latent code, and $w_t$, a text-driven latent code. The latent representations generally exhibit a high-level characteristic of the content (see Table II.). In the latent space of StyleGAN2, the sound-guided latent code moves more from the source latent code than does the text-guided latent code, and the image generated from the sound-guided latent code is more diverse and dramatic than for the text-guided method.