Few-Shot Learning Meets Transformer: Unified Query-Support Transformers for Few-Shot Classification

We introduce a novel Cross-scale Interactive Feature Extractor (CIFE) as backbone module, which aims to obtain the ego-context CNN feature representations for support and query samples.

As shown in Figure 3, taking the support image set $X^s=\{X_1^s,X_2^s,...,X_{n_s}^s\}$ as inputs, we first use the pre-trained ResNet-12 to generate the initial multi-scale feature representations $F_l^s\in R^{n_s\times c_l\times h_l\times w_l},l\in \{1,2,3,4\}$, where $n_s$ represents the number of support samples in each episode and $c_l$, $h_l$ and $w_l$ denote the channel, height and width of support feature map in the $l$-th level respectively. Then, we employ a Transformer architecture [39] consisting of multi-head self-attention (MSA), layer normalization (LN), feed-forward network (FFN) and residual connection to achieve the interaction of multi-scale features. Finally, we can obtain the spatial enhanced feature representations for support samples as ${\tilde{F}}^s=\{{\tilde{F}}_1^s,{\tilde{F}}_2^s,\cdots ,{\tilde{F}}_{n_s}^s\}\in R^{n_s\times c\times h\times w}$. Similarly, we obtain the spatial enhanced features for query samples as ${\tilde{F}}^q=\{{\tilde{F}}_1^q,{\tilde{F}}_2^q,\cdots ,{\tilde{F}}_{n_q}^q\}\in R^{n_q\times c\times h\times w}$. The parameters of CIFE are shared for support and query branches. In practice, we empirically set $c=640$ and $h=w=5$.

To achieve both image sample representation and metric learning of samples in a unified module, we design a novel query-support sample Transformer module, named sampleFormer. The proposed sampleFormer mainly consists of Encoder and Decoder, as shown in Figure 2.

Encoder. The purpose of the Encoder is to mine the relationships of samples in support set to obtain better support feature representations. To this end, based on the aforementioned support features ${\tilde{F}}^s\in R^{n_s\times c\times h\times w}$, we first introduce image tokenize, which utilizes a global average pooling and reshape operation to gain the token sequence $H^s=\{H_1^s,H_2^s,\cdots ,H_{n_s}^s\}\in R^{n_s\times c}$ of support samples, where each token $H_i^s$ denotes a support image sample. As shown in Figure 2, we can see that the main component of encoder is attention mechanism, whose inputs are Query $Q^s\in R^{n_s\times c}$, Key $K^s\in R^{n_s\times c}$, and Value $V^s\in R^{n_s\times c}$ obtained by conducting three linear projections on $H^s$ respectively. Next, it employs dot-product operation to obtain a correlation/affinity matrix $Att{n}_{s\to s}(Q^s,K^s)$ of different support samples as

where $c$ denotes the dimension of support features. It learns the representations for support samples by conducting the message passing operation as

where $LN(\cdot )$ refers to layer normalization. Besides, we add Feed-Forward Network (FFN) [6] and residual operation to obtain the final support sample representations as,

where ${\tilde{H}}^s=\{{\tilde{H}}_1^s,{\tilde{H}}_2^s\cdots ,{\tilde{H}}_{n_s}^s\}\in R^{n_s\times c}$. $n_s$ denotes the number of support samples and $c$ is the feature dimension. FFN consists of two fully-connection layers.

Decoder. The Decoder aims to explore the dependence of samples in query set to learn the representations for query samples and also mines the intrinsic metrics of samples in query and support sets. To be specific, it takes the aforementioned encoded support features ${\tilde{H}}^s\in R^{n_s\times c}$ and query feature embeddings ${\tilde{F}}^q\in R^{n_q\times c\times h\times w}$ as its inputs. The image tokenize is applied on ${\tilde{F}}^q$ to obtain the initial query token sequence $H^q=\{H_1^q,H_2^q,\cdots ,H_{n_q}^q\}\in R^{n_q\times c}$, where each token $H_j^q$ denotes a query image sample. Similar to the Encoder branch, we first leverage self-attention message passing mechanism to model the relationships among query samples and learn representations for query samples as

where $LN(\cdot )$ denotes layer normalization.

Afterward, based on the support features ${\tilde{H}}^s$ and query features ${\hat{H}}^q$, we employ a cross-attention mechanism to explore the relationships between support and query samples for query sample representations. Specifically, it first computes the cross-affinities between support and query samples as follows

Then, it learns query sample representations by aggregating the information from support samples as follows

where ${\tilde{H}}^q\in R^{n_q\times c}$ and $LN(\cdot )$ denotes layer normalization. $Q^q\in R^{n_q\times c}$ is computed by conducting a linear projection on ${\hat{H}}^q$. $K^s\in R^{n_s\times c}$ and $V^s\in R^{n_s\times c}$ are obtained by conducting two different linear projections on ${\tilde{H}}^s$, respectively.

Remark. The above cross-affinities $Att{n}_{q\to s}(Q^q,K^s)$ naturally reflect the similarities/affinities between support and query samples. In our work, we regard them as global metric $m_g$ for all support and query samples, i.e.,

where $m_g(X^s,X^q)$ contains the similarities for all query-support sample pairs in each episode. For convenience, in the following, we also use $m_g(X^s,X^q)$ to denote the metric between image $X^s$ and $X^q$, where $X^s\in X^s,X^q\in X^q$. We can utilize $m_g(X^s,X^q)$ for query sample classification, as discussed in the following Section Metric Learning and Few-shot Classification. Therefore, we can note that both query/support sample representation and metric learning in few-shot learning task are conducted simultaneously in our sampleFormer architecture. This is one main aspect of the proposed sampleFormer module.

As a complementary to the above sampleFormer branch, we also develop a query-support Patch Transformer Module (patchFormer) to capture the more visual content of each image sample for local metric. As shown in Figure 2, patchFormer mainly consists of multi-head self-attention (MSA) and residual connection. Here, we omit Feed-Forward Network used in regular Transformer [6] for simplicity consideration. The parameters of MSA are shared on both support and query branches.

Concretely, for each input support sample $X^s$ and query sample $X^q$, we first obtain their feature embedding ${\tilde{F}}^s\in R^{c\times h\times w}$ and ${\tilde{F}}^q\in R^{c\times h\times w}$ by using the above CIFE, followed by the patch tokenize [6] to obtain the initial patch token sequence for each support and query image, i.e., $P^s=\{p_1^s,p_2^s,\cdots ,p_{hw}^s\}\in R^{hw\times c}$ and $P^q=\{p_1^q,p_2^q,\cdots ,p_{hw}^q\}\in R^{hw\times c}$. Then, we employ multi-head self-attention (MSA) [39] with shared weights and residual operation to transform the support and query image patch features as

where $LN(\cdot )$ denotes layer normalization.

Based on the above patch representations ${\tilde{P}}^s=\{{\tilde{p}}_1^s,{\tilde{p}}_2^s,\cdots ,{\tilde{p}}_{hw}^s\}$ and ${\tilde{P}}^q=\{{\tilde{p}}_1^q,{\tilde{p}}_2^q,\cdots ,{\tilde{p}}_{hw}^q\}$, we then adopt the Earth Mover’s Distance (EMD) [12, 48] to compute their structural similarity. It first computes the distance between all patch pairs $({\tilde{p}}_i^s,{\tilde{p}}_j^q)$ and then acquires the optimal matching between patches of two images that have the minimum distance cost. Finally, it returns the image-level metric by aggregating the metrics of all matched patch pairs. In this paper, we denote this metric as local metric between support sample $X^s$ and query sample $X^q$, i.e.,

Given the support samples $(X^s,Y^s)\in X^s$ with known labels and input query sample $X^q\in X^q$, few-shot classification aims to determine the label of the query sample. To achieve this task, we first obtain the sample-based global metric $m_g(X^s,X^q)$ via Equ. (8) and patch-based local metric $m_l(X^s,X^q)$ via Equ. (10) respectively and combine them together to obtain the final metric/similarity between $X^s$ and $X^q$ as

where $\lambda \in (0,1)$ is a tradeoff parameter.

Then, we can conduct few-shot classification by using the nearest neighbor classification strategy, i.e., the label of query $X^q$ is determined by the label $Y^{s^{\ast }}$ of the support sample $X^{s^{\ast }}$ that is most similar with query $X^q$, as used in previous works [40, 48].

Loss Function. In the training phase, we employ two loss functions for the proposed QSFormer. First, for the sampleFormer module, we specifically introduce a contrastive loss as suggested in work [24, 20], which encourages the positive query-support sample pairs with same label (i.e., $Y^s=Y^q$) to be closing and the negative query-support sample pairs with different labels (i.e., $Y^s\ne Y^q$) are far away in each episode. This loss function can be written as follows,

where $m_g(X^s,X^q)$ is the global metric between query $X^q$ and support sample $X^s$. The whole network is trained in an end-to-end way by minimizing the Cross-Entropy (CE) loss function $L_{ce}$  [48]. Thus, the total loss function can be formulated as

where ${\hat{Y}}^q$ is the label prediction obtained by our method and $Y^q$ denotes the corresponding ground-truth label. $\alpha \in (0,1)$ is the balanced hyper-parameter.

Implementation Details. To achieve a fair comparison, the ResNet-12 [2, 48] with fully connected layers removed is adopted as the backbone module. It is firstly pre-trained from scratch and then use the episodic training based on meta-learning framework by following works [3, 48]. We empirically conduct the feature interaction of the last two levels in CIFE to obtain the enhanced sample features. We randomly sample 50/1000/5000 episodes from the training/validation/testing set on four public datasets. We compute the average accuracy and the corresponding 95$\%$ confidence interval to obtain the final performances of four datasets. Our proposed method is implemented by using Python on a server with a single 11G NVIDIA 2080Ti GPU. More hyper-parameter settings on four benchmarks for the proposed QSFormer are shown in Table VI.

To verify our proposed QSFormer, we conduct extensive experiments on four publicly popular datasets for few-shot classification task, including miniImageNet [40], tieredImageNet [32], Fewshot-CIFAR100 [25] and Caltech-UCSD Birds-200-2011 [41]. We also conduct cross-domain experiments to evaluate the domain transfer ability of the proposed model. The recognition accuracy is adopted as the evaluation metric for our experiments. More details of datasets description are as follow.

miniImageNet. This dataset is a sub-dataset of ImageNet [33]. It contains a total of 100 classes with 600 samples in each class. As suggested in work [30], we divide these classes into training set, validation set and testing set, which respectively contains 64, 16 and 20 classes.

tieredImageNet. It contains 608 classes from 34 super-classes, with a total of 779,165 samples. Following [32], we split 34 super-classes into 20 super-classes (351 classes) for meta-training, 6 super-classes (97 classes) for meta-validation and 8 super-classes (160 classes) for meta-testing.

FC100. Fewshot-CIFAR100 is built upon the CIFAR100 dataset for few-shot classification task. It’s named FC100 for short hereafter. It contains a total of 60,000 images from 100 classes. To reduce the information overlap, we group the 100 classes into 20 super-classes by following work [25]. Then, we divide these super-classes into training set, validation set and testing set, which contains 12, 4 and 4 super-classes respectively.

CUB. Caltech-UCSD Birds-200-2011 dataset is an extended vision of CUB-200 dataset. It’s termed CUB for short hereafter. CUB is originally presented in fine-grained bird classification task. It contains the total of 11,788 images from 200 classes. As suggested by [46], we divide 200 classes into 100 classes for meta-training, 50 classes for meta-validation and 50 classes for meta-testing.

miniImageNet $\to$ CUB. By following [2], we train a model on miniImageNet dataset and evaluate on the CUB dataset to verify the transfer ability of model. In this experimental setting, specifically, we use all 100 classes of miniImageNet, with 600 samples per class for meta-training and use the meta-testing set (50 classes) of CUB dataset for meta-testing.

As shown in Table I, we report our results and compare with other state-of-the-art (SOTA) approaches on miniImageNet [40] and tieredImageNet [32] datasets. From this Table, we can find that the proposed QSFormer beats many SOTA models on the miniImageNet dataset. For example, QSFormer exceeds the transformer-based HT [51] method by +11.14% and +11.46% in 1-shot and 5-shot tasks, respectively. For the attention mechanism based CAN [13], our model also outperforms it on the 1-shot/5-shot task by +1.39$\%$/+0.52$\%$. Compared with FETA [46] that is also developed based on ResNet12 and Transformer, the proposed QSFormer has better results.

From Table I, we can see that QSFormer achieves the best performance on the tieredImageNet dataset, i.e., 72.47$\pm$0.31 and 85.43$\pm$0.22 in 1-shot and 5-shot tasks. It exceeds the CAN [13] by +2.58 and +1.2 points in 1-shot and 5-shot tasks. Similar conclusions can also be drawn from the experimental results of Fewshot-CIFAR100 [25] and CUB [41] datasets, as illustrated in Table II and Table III. All in all, the proposed QSFormer attains SOTA performance on multiple FSL datasets, which fully demonstrates the effectiveness and advantages of our proposed QSFormer model.

To better understand the effectiveness of our proposed QSFormer, in this section, we conduct extensive ablation studies, including component analysis, similarity metric analysis, cross-domain analysis, etc.

Component Analysis. Our proposed QSFormer mainly contains three components: Cross-scale Interactive Feature Extractor (CIFE), Sample Transformer Module (sampleFormer) and Patch Transformer Module (patchFormer). The experimental results of ablation study are shown in Table V. We reproduce cosine classifier method [2] consisting of CNN network and cosine distance as the Baseline network for comparison. From Table V, we can observe: (1) By comparing #1 with #2, the performance of Baseline network can be significantly improved with the help of CIFE, which demonstrates the effectiveness of CIFE. (2) By comparing #2 with #3, we can find that sampleFormer significantly improves the performance of model based on #2, which indicates the effectiveness of sampleFormer module. (3) By adding patchFormer into #3, we further improve the performance of whole network, which shows the effectiveness of patchFormer module. All these experiments fully validate the effectiveness of each component in our proposed QSFormer framework.

Similarity Metric Analysis. To verify the effectiveness of the proposed QSFormer on metric learning, we visualize the similarity distribution of Baseline and QSFormer on the more challenging 5-way 1-shot task, as shown in Figure 4. For 5-way 1-shot task, each query sample generates the similarity results of one positive query-support sample pair (i.e., “Q-S pos”) and four negative query-support sample pairs (i.e., “Q-S neg”) during the metric learning process. To facilitate the comparison of the similarity results of “Q-S pos” and “Q-S neg”, we average the similarity values of four “Q-S neg” corresponding to each query sample. For this experiment, we perform 10 episodes, where each episode random selects $15\times 5=75$ query samples for classification, i.e., we can get the $75\times 10=750$ similarity values of “Q-S pos” and “Q-S neg”, respectively. Subsequently, we count the number of “Q-S pos” and “Q-S neg” within a certain range according to the normalized similarity values and thus produce the similarity distribution as shown in Figure 4. We can observe that: (1) the similarity values of “Q-S pos” obtained by the Baseline method are generally below 0.5, while “Q-S neg” are above 0.25. (2) In our proposed QSFormer, the similarity values of “Q-S pos” are mostly above 0.5, while “Q-S neg” are mostly below 0.25. Therefore, our proposed QSFormer can separate positive and negative query-support sample pairs more accurately.

In addition, we also compare our QSFormer with other metric learning algorithms, including cosine classifier [2], MatchNet [40], ProtoNet [35] and DeepEMD [48]. These compared methods are reproduced with the same settings and training schemes as ours for a more fair comparison. As shown in Table VII, we can observe that our proposed method obtains the best performance on four publicly popular datasets, which fully demonstrates the effectiveness and superiority of our proposed QSFormer. These experiments fully demonstrate the effectiveness of our proposed QSFormer for metric learning.

Cross-domain Analysis. To validate the transferable ability of our proposed QSFormer, we conduct a cross-domain experiment by following [2, 48]. The training and testing are implemented on miniImagenet dataset and CUB dataset, respectively. As shown in Table IV, our proposed QSFormer achieves the best performance on the 1-shot setting (55.04 $\pm$ 0.29) and the second-best results on the 5-shot, i.e., 77.12 $\pm$ 0.24. These results demonstrate that the proposed QSFormer learns the discriminative information across domains, and adaptively explores the correspondence of query-support samples.

Parameter Analysis. There are two important parameters in our model, including the balanced parameter $\lambda$ in Equ. (11) for local and global metric, and the number of sampleFormer layers $N$. In this section, we conduct experiments on the FC100 dataset on 5-way 1-shot task to check their influence. As shown in Figure 5, we can observe that the performance is relatively stable when we slightly adjust the balanced parameter $\lambda$ in the range of (0.2, 0.6). For the number $N$ of sampleFormer layers, we can find that our performance is increasing continuously when the $N$ is changing from 2 to 4. Therefore, we set $\lambda =0.4$ and $N=4$ for our experiments.