SIM-Trans: Structure Information Modeling Transformer for Fine-grained Visual Categorization

Recent approaches mainly focus on the discriminative regions discovery and feature extraction for fine-grained visual categorization (He et al., 2018, 2019; Ding et al., 2019; Wang et al., 2020b, a; Song et al., 2020). Ding et al. (Ding et al., 2019) propose S3N to utilize sparse attention to estimate informative regions and extract discriminative and complementary features for classification. He et al. (He et al., 2019) propose M2DRL to automatically determine the location and number of discriminative regions with reinforcement learning paradigm. Song et al. (Song et al., 2020) propose Bi-Modal PMA to capture discriminative parts stage-by-stage with the progressive mask attention model. Rao et al. (Rao et al., 2021) propose a counterfactual attention learning method to obtain more useful attention for fine-grained visual categorization. However, the above methods generally ignore object’s holistic structure information, which is greatly helpful for localizing the whole object extent. In our proposed method, the object structure information is introduced explicitly for highlighting the discriminative regions within object.

Recently, many vision transformer methods have been proposed for computer vision tasks. ViT  (Dosovitskiy et al., 2020) is the first work to introduce the pure transformer into image classification, which splits the image into a sequence of patch tokens as input of transformer and achieves promising performance. Zheng et al. (Zheng et al., 2021) propose SETR to utilize ViT as the encoder for image segmentation. He et al. (He et al., 2021) propose TransReID to introduce the jigsaw patch module and side information embeddings into transformer for object re-identification. Recently, RAMS-Trans (Hu et al., 2021) is proposed for fine-grained visual categorization, which designs the recurrent attention multi-scale transformer to find and amplify significant regions for learning multi-scale features. However, the above vision transformer works generally ignore the spatial relation of patches, which is important for significant patches discovery. Our proposed SIM-Trans incorporates the object structure information into transformer to enhance discriminative patch features for accurate fine-grained visual categorization.

The vision transformer backbone is composed of feature extractor and classifier head, which is shown in the middle part of Figure 1. Before being input to the feature extractor, the image $x\in R^{H\times W\times 3}$, where $H$ and $W$ denotes its height and width respectively, is generally split into $N$ patch tokens through non-overlapping splitting, denoted as $\{x_{patch}^1,x_{patch}^2,...,x_{patch}^N\}$. However, this splitting way may cause the incomplete neighboring information within the patch due to the hard separation. Thus, we adopt the sliding window splitting method following (He et al., 2021). Specifically, the patch size and the window’s sliding step are denoted as $P$ and $S$, the patch sequence length $N$ for the image $x$ can be calculated as follows:

where $\lfloor \cdot \rfloor$ denotes the floor operation, $N_H$ and $N_W$ denote the patch numbers in vertical and horizontal directions after splitting. In this way, the local neighboring information preservation problem is alleviated to some degree.

The patch token $\left\{x_{patch}^i\right\}$ is then projected into $D$-dimensional embedding through linear mapping $F(\cdot )$. For representing the whole image, a learnable $cls$ embedding token $x_{cls}$ is introduced and put at the beginning of the input sequence embeddings. To incorporate position information, learnable position embeddings $E_p\in R^{(N+1)\times D}$ are added to the input sequence embeddings to get $z_0$ as the input of the first transformer layer as follows:

The transformer feature extractor is composed of $L$ transformer layers, each of which consists of a multi-head self-attention (MSA) module and a feed forward neural network of two fully connected layers. The output of the $k_{th}$ transformer layer is calculated as follows:

where $LN(\cdot )$ denote the layer normalization (Ba et al., 2016). The output $cls$ tokens of the last three transformer layers are concatenated and regarded as image representation (more details in Sec. 3.3), which is forwarded into a classifier head to get the prediction vector $pred\left(I\right)$ for the input image $I$. The classification loss is as follows:

where $S\left(I\right)$ is the training set and $y$ is the one-hot label of image $I$.

The vision transformer backbone can achieve promising image classification results, which utilizes the self-attention mechanism to own a global receptive filed. However, the vision transformer framework generally ignores the spatial relation of patches, which is important for identifying discriminative patches in fine-grained visual categorization task. Thus, we propose the structure information learning (SIL) module to incorporate the object spatial context information into the vision transformer and the whole procedure is shown in the right part of Figure 1.

Localizing the object extent is the precondition for structure learning. In the transformer layer, the attention weight between the patch token and cls token depicts its importance for the final classification, which is highly correlated with whether the patch token contains the object information. Thus, object can be naturally localized with the help of the attention weights. Suppose the transformer layer has $H$ heads, $Q$ and $K$ are $D$-dimensional query vectors and key vectors of all tokens, then the attention weights can be calculated as follows:

where $At{t}_h\in R^{(N+1)\times (N+1)}$, $h=1,2,...,H$ and $N=N_H\times N_w$ is the number of patches. The attention weight between patch token and the cls token for each head is extracted and denoted as $At{t}_h^{cls}\in R^{N\times 1}$. Corresponding total attention weights are calculated as follows:

The attention weight between the patch token in $(x,y)$ position and the cls token is denoted as $A_{(x,y)}$. For filtering out insignificant patches, the mean value $\overline{A}$ is calculated as the threshold and the new attention weight is as follows:

Inspired by  (Zhou et al., 2020), polar coordinates are applied to measure the spatial relation between the most discriminative patch and other patch to mine object’s structure information. Specifically, the patch with the highest attention weight is regarded as the most discriminative patch, which is set as the reference patch. Given the reference patch $P_0=P_{x_0,y_0}$ and the horizontal reference direction, where $(x_0,y_0)$ are the indices in $N_H\times N_W$ plane, the polar coordinates of patch $P_{x,y}$ can be defined as:

where ${\rho }_{x,y}$ is the relative distance between $P_{x,y}$ and $P_0$ and ${\theta }_{x,y}$ is the normalized polar angle of $P_{x,y}$ relative to the horizontal direction. To introduce this object structure information, we design the graph convolutional neural network to obtain the object structure feature. We firstly construct the graph which contains two components: (i) the image patch node features $X$ which depict the spatial context correlation based on the calculation of the polar coordinates and (ii) the edge weights obtained based on attention weight calculation between cls token and image patch token in the vision transformer layer, which summarize the significance of image patch token. Specifically, the matrix $Adj=A^{new}\times (A^{new}{)}^T$ denotes the edge weights among nodes based on the $A^{new}$, where the edge weights related to insignificant patches are zeros to filter out their affect. The graph convolution is then adopted to further extract and incorporate the structure information into the vision transformer. The structure features $S$ are obtained by two-layer graph convolution as follows:

where $W^1$ and $W^2$ are learnable parameters and $\sigma (\cdot )$ is activation function. The feature of the reference patch node is regarded as the object structure feature, which is further added to the cls token feature for introducing structure information into the transformer backbone. Through the end-to-end training, the composition of the object can be modeled and the significant image patch can be highlighted, which improves the model’s classification performance.

Overall, the structure information learning module can incorporate the object structure information, i.e., its spatial composition of key discriminative patches, into one or more vision transformer layers. The transformer network can be empowered to learn both appearance and structure information for accurate fine-grained classification.

After obtaining image features by the transformer feature extractor, the feature of the cls token $z_L^{cls}$ of the last transformer layer is generally selected as the image representation for final classification in vision transformer methods, such as ViT. However, it generally ignores the complementary of different levels’ features and the utilization of the inherent intra-class and inter-class semantic relations. Thus, we propose the multi-level feature boosting (MFB) module for enhancing feature robustness, which is shown in the left part of the Figure 1.

Specifically, the features of cls tokens from the last three transformer layers are concatenated as the final image feature representation, which all introduce structure information through the structure learning module. In this simple but effective way, their complementary can be exploited to bring performance gains. To take full advantage of the semantic relations for feature enhancement, contrastive learning is adopted to enhance the feature similarity of the same category and weaken that of different categories. To mine the hard negative sample pairs to contribute to the model’s training, a hyper-parameter $\alpha$ is adopted in the following contrastive learning loss. The negative pairs with similarity which is $\alpha$ smaller than that of the positive pairs are filtered out. Formally, the contrastive learning loss on a batch of size $N$ is defined as follows:

where $(z_i,{z_j}^{+})$ denotes the positive image representation pair which have the same category label, i.e., $y\left(i\right)=y\left(j\right)$, $(z_i,{z_j}^{-})$ denotes the negative image representation pair which belongs to different categories, ${\Gamma }_{y_i=y_j,i\ne j}$ denotes the number of positive pairs, $sim(\cdot )$ denotes the cosine similarity calculation.

In summary, the vision transformer backbone, structure information learning module and feature boosting module of the proposed method are jointly trained end to end. The total objective is as follows:

Two standard fine-grained visual categorization benchmarks are adopted in the experiments:

• [leftmargin=30pt]

• CUB-200-2011 (Wah et al., 2011) is the most widely used dataset in the fine-grained visual categorization task. This dataset consists of 11788 images of 200 bird subcategories, where 5994 images are selected as training set and 5794 images are selected as testing set.

• iNaturalist 2017 (Van Horn et al., 2018) is one of the largest fine-grained visual categorization datasets. it contains more than 5,000 fine-grained categories and more than 95,000 test images. The detailed statistics are shown in table 1, which includes category number and the split of training data and testing data. The dataset is very challenging for its large-scale fine-grained category number and testing data, which provides a strong benchmark for fine-grained visual categorization performance contrast.

Accuracy is adopted as the evaluation metric, which is generally used in fine-grained visual categorization task. And the definition is as follows:

where $|I|$ denotes the number of images in the testing set and $|I_{right}|$ denotes the number of images that are correctly classified.

Structure information learning (SIL) module and multi-level feature boosting (MFB) module are proposed in our SIM-Trans approach. We conduct ablation experiments on CUB-200-2011 dataset and the effectiveness of each component is shown in Table 4 and Figure 4, we can observe that:

• [leftmargin=30pt]

• A classifier head is added to the ViT extractor as our baseline, which can achieve 90.6% classification accuracy on CUB-200-2011 dataset. Compared with our baseline, the structure information learning (SIL) module improves the classification accuracy by a margin of 0.5%, which is simply added to the last transformer layer as a plug-in. The SIL module boosts the model’s understanding ability for the object structure and highlights significant regions to make the feature representation more discriminative.

• Based on the SIL module, we conduct multi-layer structure information injection and the multi-level feature boosting (MFB) for robust feature learning, which can bring another 0.7% performance gain. We attempt to add the SIL module from the top layer to the bottom layer, and Figure 4 shows when the number of layers is set as three, the proposed model can achieve the best performance. It also demonstrates the combination of SIL’s expansion and MFB’s fusion contributes to each other for improving the classification performance.

• In MFB, the contrastive learning can improve the classification accuracy by a margin of 0.4%, which attends to hard negative samples and utilize the semantic relations to boost the model’s feature discrimination.

There is only one hyper-parameter in our SIM-Trans model designing. The $\alpha$ of MFB module in Eq. 12 controls the hardness degree of negative pairs in contrastive learning. We analyze the influence of $\alpha$ on the classification performance in Figure 5. When $\alpha$ is set to be 0.3, the proposed SIM-Trans model can achieve the best classification performance. Continuing to increase $\alpha$, the performance degrades because the over-high threshold cannot filter out easy negative pairs, which is harmful for contrastive learning.