UniCon: Unidirectional Split Learning with Contrastive Loss for Visual Question Answering

Information in the real world usually comes in different modalities. Multimodal machine learning (MMML) (Alayrac et al., 2020; Radford et al., 2021; Rouditchenko et al., 2021; Ramesh et al., 2021, 2022) has been intensively studied with significant progress in cross-modal understanding and reasoning. In particular, Visual Question Answering (VQA) is the task to answer a natural language question according to the contents of a presented image. VQA is actively studied (Yang et al., 2016b; Anderson et al., 2018; Kim et al., 2018; Yu et al., 2020) with recent years’ progress in the attention mechanism (Vaswani et al., 2017a). Nevertheless, the vast majority of VQA studies so far are based on the modality network fusion methods (Yang et al., 2016b; Kim et al., 2017) where the VQA task is considered a multi-class classification. Such an assumption hinders the understanding of the semantic notions embedded in the natural language answers. Moreover, in practice, these studies do not touch on the privacy of VQA when using multi-modality data to train a model. In this work, we aim to bridge the gap between the large-scale training of VQA and the constraint of data sharing mainly due to confidentiality (Table 1).

The prerequisite of large-scale training tasks like VQA that requires a decent amount of labeled modality data hinders the efficacy of cross-modal learning since data collection and labeling processes are usually effortful. Most VQA research has focused on supervised learning where the one-hot vectors of answer classes are employed to perform a multi-class classification (Yang et al., 2016a; Anderson et al., 2018; Kim et al., 2018). Nevertheless, the learned correlation between the multi-modality input and the answer does not consider the semantic meaning of the answer.

In contrast, recent research in MMML has emerged in self-supervised learning (SSL)(Chopra et al., 2005; Chen et al., 2020b). In particular, contrastive learning is commonly used to learn a shared embedding space from unlabelled data, in which similar sample pairs stay close to each other while dissimilar ones from different pairs are far apart. For instance, Barlow Twins(Zbontar et al., 2021) learns a cross-correlation matrix by keeping the representations of different distorted versions of an input sample similar, while minimizing the redundancy between these representations. Moreover, Momentum Contrast (MoCo) (He et al., 2020) trains a visual representation encoder by matching an encoded query to a dictionary of encoded keys based on the Information Noise Contrastive Estimation (InfoNCE) loss (Oord et al., 2018a). Furthermore, contrastive learning was also reported usage in training multimodal models. For example, CLIP(Radford et al., 2021) computes a cosine similarity matrix between all possible candidates of images and texts within a batch. Then, the similarity between relevant pairs is maximized and the similarity between irrelevant ones is minimized. Furthermore, there are also studies such as MultiModal Versatile Networks (Alayrac et al., 2020) and AVLnet(Rouditchenko et al., 2021) demonstrating the model training on large collections of unlabelled video data with the contrastive loss.

The most recent work has also explored the implementation of SSL to VQA. Question-Image Correlation Estimation (QICE) (Zhu et al., 2020) trains on relevant and irrelevant image and question pairs in VQA datasets to alleviate the language prior problem (Kurakin et al., 2017; Agrawal et al., 2018; Goyal et al., 2019) improving the understanding of image contents. Kim et al. (Kim et al., 2021) studied Video Question Answering using SSL by measuring the similarity scores between the representations of questions and ground truth answers. These methods above adopt a two-step training strategy, i.e., the cross-modality pretraining with SSL and the fusion network fine-tuning. In contrast, we train a VQA model end-to-end based on the contrastive learning of different model components and determine a prediction result using the similarity measurement between modality representations. This work leverages several projection networks to align the cross-modal representations and embed the semantic meaning of answers.

Our method is inspired by Decentralized Machine Learning built upon distributed learning (Li et al., 2014; Smola and Narayanamurthy, 2010), such as Federated Learning (McMahan et al., 2017; Sun et al., 2020; Kairouz et al., 2021), Split Learning (Hancox and et al., 2020; Thapa et al., 2022), and Swarm Learning (Warnat et al., 2021). Though methods like Federated Learning have been intensively studied to tackle tasks of a single modality (Sun et al., 2021), the studies in multimodal models such as VQA are still insufficient. Notably, for the visual language grounding tasks, Liu et al. (Liu et al., 2020) proposed a Federated Learning-based VQA framework called the aimNet, which consists of an aligning module, an integrating module, and a mapping module. AimNet could acquire fine-grained representations from different clients for improved downstream tasks. However, aimNet is a supervised learning framework that trains on the hard labels for different answers. Moreover, though aimNet does not disclose the training data via Federated Learning, the model parameters are shared to obtain better-refined representations. Such shared model parameters can be used to mount adversarial attacks of model poisoning (Wang et al., 2020b; Fang et al., 2020; Zhou et al., 2021).

To overcome the above challenges, we devise a new split Learning-based VQA framework leveraging a simpler and faster unidirectional learning flow to improve classical split learning. Furthermore, neither the training data nor the complete model’s parameters are disclosed during the training. To the best of our knowledge, this is the first time any work has attempted to employ split learning to tackle the VQA task and consolidate split learning and the contrastive loss.

In VQA, the training data of images, questions, and answers are usually prepared beforehand for the model training. Nevertheless, previous efforts in the study of adversarial attacks on neural networks have shown that the use of large-scale training data can be correlated with the model’s venerability to poisoning attacks. The poisoning attacks can usually be divided into two categories depending on the prior knowledge of the adversary (Sun et al., 2021), i.e., data poisoning and model poisoning. Moreover, in the most recent study of attacks on state-of-the-art VQA models, an adversary exploited the complex fusion mechanisms to successfully embed effective and stealthy backdoors (Walmer et al., 2022). Such threats are enabled due to the adversary’s accessibility to either the training data or VQA models.

To this end, the line of work in decentralized machine learning (DML) approaches such as Federated Learning (FL) have been recently used to facilitate privacy-preserving learning of VQA models (Liu et al., 2020). FL trains a global model over the entire data distribution of different tasks thus attaining better-refined representations for downstream tasks. This is enabled by the sharing of local model parameters trained on different client tasks. Intuitively, FL can retain data confidentiality via model sharing, however, sharing the entire model renders the FL-based framework venerable to model poisoning attacks. For example, the adversary could reconstruct the input data from the shared model parameters (Hitaj et al., 2017). Our goal is to alleviate the threats of model poisoning while retaining the guarantee of data confidentiality based on an adapted method of Split Learning. The proposed method leverages contrastive learning to align knowledge between different components of a VQA model such that the model can be trained without disclosing the entire model architecture.

Visual Question Answering (VQA) is the task to answer questions according to given image contents. The VQA problem is usually considered a supervised learning task with a fixed list of $C$ possible answer options. In particular, let $f_{\text{small{VQA}}}$ be the VQA model that takes as the input the pair of an image $x_i\in R^V$ and a question $x_q\in R^Q$ and outputs an answer $\hat{y}\in \{y_1,y_2,...,y_C\}$ where $y_c\in R^A$. The goal of the VQA model is to predict the correct answer $y$ given the input pair $(x_i,x_q)\in D$ where $D$ is the dataset. $\hat{y}=\underset{y}{\text{arg max}}p(y|x_i,x_q;f_{\text{small{VQA}}})$ where $p(\cdot |\cdot )$ is the conditional probability.

Moreover, Split Learning (SL) splits a complete model into different parts and trains a global model via interactive representation and gradient sharing. There are mainly two types of SL for different tasks (Thapa et al., 2022). In particular, we consider Split Learning without label sharing that wraps the model on the server around the end layer and sends the layer output back to a client (Fig. 2.a). This architecture could guarantee the data confidentiality of a client since both the input data and labels are not shared for the training. Furthermore, a complete model is split into three different components for the training, i.e., a global component $f_g$ and two client components $\{f_{c,1},f_{c,2}\}$. We assume there are $K$ clients. The $k$th client has its own dataset $D^{\left(k\right)}:=\left\{\right(x_{i,j},x_{q,j},y_j){\}}_{j=1}^{N^{\left(k\right)}}$ where $N^{\left(k\right)}$ is the sample size of dataset $D^{\left(k\right)}$. Here, ${\cup }_{k=1}^K{D}^{\left(k\right)}=D$, $D^{\left(i\right)}\cap D^{\left(j\right)}=\varnothing \forall i\ne j$, and ${\sum }_{k=1}^K{N}^{\left(k\right)}=N$ where $N$ is the sample size of $D$. We suppose that the model components of different clients share the same architecture in SL and each client cannot share data mainly due to data confidentiality.

Then, SL proceeds by iterating the following steps: (1) each client $k$ computes the output of the component $f_{c,1}$ with $D^{\left(k\right)}$ and sends the output to the server, (2) the server further forward-propagates the client input with the global component $f_g$ and sends back the output, (3) the probability distribution is computed with the client component $f_{c,2}$ and the loss is computed with the ground-truths $\{y_j{\}}_{j=1}^{N^{\left(k\right)}}$, (4) the gradients of each client $k$ $({\delta }_k{\theta }_{c,1},{\delta }_k{\theta }_{c,2},{\delta }_k{\theta }_g)$ with respect to each component’s parameters $({\theta }_{c,1},{\theta }_{c,2},{\theta }_g)$ are computed via an inverse path $f_{c,2}\to f_g\to f_{c,1}$, (5) the gradients are averaging aggregated to update the different components, $\delta {\theta }_{c,1}=\frac{1}{K}\sum _{k\in K}{\delta }_k{\theta }_{c,1}$, $\delta {\theta }_{c,2}=\frac{1}{K}\sum _{k\in K}{\delta }_k{\theta }_{c,2}$, $\delta {\theta }_g=\frac{1}{K}\sum _{k\in K}{\delta }_k{\theta }_g$. The process above is repeated until a given training goal is achieved. We refer to the appendix for full details on the SL algorithm.

Previous work showed the plausible usages and efficacy of Contrastive Learning (Chopra et al., 2005) in a single modality (Chen et al., 2020b; He et al., 2020; Zbontar et al., 2021). In multimodal learning, Contrastive Learning has recently been used to learn refined cross-modal representations (Radford et al., 2021; Rouditchenko et al., 2021) by encouraging multimodal data from a relevant input to have more similar representations compared to data from irrelevant inputs. For the VQA tasks, one challenge is that most models are supervised where the one-hot vectors of answer classes are employed to perform a multi-class classification (Anderson et al., 2018; Kim et al., 2018). Therefore, the semantic notions of answers are usually not well correlated with the inputs reducing the generality of the trained model to unknown samples. To this end, we propose the use of the contrastive loss in Split Learning to correlate vision contents and language semantic notions such that each model component learns better-refined representations for the VQA tasks. Moreover, since the learning flow in classical SL is bidirectional between a client and the server, the waiting time for computing the next component is greatly increased (Fig. 2.a). By using the contrastive loss, we adapt SL to the unidirectional sharing architecture where the representations are shared by clients to the server and the gradients are shared by the server to clients. The unidirectional split learning with contrastive loss (UniCon) allows a client to compute the representations and gradients without waiting for the computation of the global component, which is more efficient and simpler compared to SL (Fig. 2.b).

The difficulty to measure the accuracy in UniCon is that we do not have a discriminative model to infer the class of the input. Inspired by (Radford et al., 2021), we evaluate the product similarity scores between the representations $v_{\text{small{NHA}}}$ of an image and question input pair $(x_i,x_q)$ from the hold-out validation dataset $D_{\text{small{val}}}$ of VQA and the representations $v_{\text{small{LTA}},c}$ of $C$ answer options $y_c\in \{y_1,y_2,\dots ,y_C\}$, where $v_{\text{small{LTA}},c}$ denotes the representation of answer option $y_c$. Then, we select the answer with the highest similarity with the input as the predicted answer$\hat{y}$. There also exist studies reporting the average accuracy of different batches, i.e., measuring the similarity scores within the same batch. However, we found that such a metric can easily produce a much higher accuracy compared to the evaluation metric based on all answer options. Therefore, we use as the metric the similarity scores between the representations of input and all answer options. We formulate the accuracy by the following

The proposed method is model agnostic and can be applied to different VQA models. We used PyTorch and the OpenVQA platform (Yu et al., 2019a) to implement the VQA models. We set the hyperparameters of different VQA models to their default author-recommended values. Due to the training time cost, we evaluated the model performance with three different seeds and reported the mean. We conducted experiments on four NVIDIA A100 Tensor Core GPU with 40GB memory each. The code will be made publicly available upon acceptance.

We employed the following architecture for the three model components in UniCon, respectively. For APN, we used the GloVe (Pennington et al., 2014) trained on Common Crawl to convert the pre-processed texts with a maximum word of eight into the dimension of $8\times 300$. Then, a fully-connected layer followed by the ReLU activation function was applied to project the representations into $8\times 512$. Finally, a Max Pooling layer was applied to the eight words with respect to each dimension producing a 512-dimension vector. Moreover, for LTA, we used a fully connected layer to project the representations received from each client to the shared space with a dimension of 256. For NHA, we used a two-layer architecture: (1) a fully connected layer to project the representations to a dimension of 512 followed by the ReLU, (2) and another fully connected layer to project the middle layer output to a dimension of 256 followed by the batch normalization. We refer to the appendix for full details on the model architecture.

Furthermore, we employed a batch size of 128, a total epoch of 20 (693400 steps), the Adam optimizer with parameters ${\beta }_1=0.9$, ${\beta }_2=0.999$, and $ϵ={10}^{-8}$, and an initial learning rate of 0.0001 with a linear warmup of 10K steps and a decay rate of 0.2 at the epoch 10 and 15. The hyperparameters were chosen using the grid search. We found that a smaller learning rate or a larger batch size would generally hinder the model learning and the model ceased learning in some cases with larger parameter sizes. In addition, for the InfoNCE loss, we adopted a temperature of 0.07 as in (Wu et al., 2018; Patrick et al., 2020; Alayrac et al., 2020).

We performed extensive experiments based on the five state-of-the-art VQA models above. In particular, for MMNas and MCAN, we further considered the effectiveness of different model complexities using MMNas-small (MMNas-s) and MMNas-large (MMNas-l), and MCAN-small (MCAN-s) and MCAN-large (MCAN-l). The detailed architecture designs of these models followed the settings in (Yu et al., 2020, 2019b). Moreover, we evaluated the proposed method’s performance based on Eq. 3. Notably, for each triplet in the validation set, we input the image and question pair to UniCon. Then, the output representation of the nonlinear head adapter is used to measure the similarity scores with the representations of the 3048 answers from the linear tail adapter. Finally, the prediction result is the answer text with the highest similarity score and the accuracy is computed based on the predictions and the ground truths of the validation set. Note that the label space of VQA-v2 is 3048-dimension which is much larger than the datasets considered in CLIP (Radford et al., 2021), hence, the VQA tasks are more challenging to perform the contrastive learning. The largest label space in CLIP is ImageNet (Deng et al., 2009) with only 1000 classes.

Furthermore, we studied the effectiveness of the contrastive learning-based VQA. The results are shown in Table 2. The empirical results demonstrate that the contrastive learning-based approach can be effectively applied to most VQA models. BAN showed the worst performance, particularly in counting the number. MMNas-l showed the best overall performance of 53.82% outperforming the other models for counting the number (Number) and answering the contents of an image (Other). Nevertheless, MCAN-l performed the best in the Yes/No questions.

To evaluate the efficacy of UniCon, we randomly divided the training set into two subsets. Then, we employed the two subsets as the local datasets of two clients to perform the split learning. The two clients share the same model component architecture and cannot share data mainly due to data confidentiality. The performance was evaluated on the global model after each round’s parameter aggregation. We show the numerical results in Table 3. Similarly, MMNas-l outperformed the other models, however, MCAN-l performed the best in the Yes/No questions. Moreover, there exists a trade-off between the model performance and using split learning for confidentiality. Compared to the overall accuracy of 53.82% of MMNas-l in the standalone training over the entire data set, the split learning-based approach obtained an accuracy of 49.89%. Nevertheless, when data sharing becomes an obstacle, the proposed approach can benefit the model training by leveraging other clients’ knowledge of different tasks. In addition, in case that data sharing is not allowed, a standalone model usually only has access to part of the data training over the partial distribution. UniCon allows clients to train over the entire data distribution thus improving the global model performance. We aim to study the efficacy of UniCon to improve the performance of standalone learning over the partial distribution in our future endeavors. We also presented the similarity measurement scores between the pairs of $v_{\text{small{NHA}}}$ and $v_{\text{small{LTA}}}$ within one training batch before and after training, respectively, in Fig. 4.