Transformers in Remote Sensing: A Survey

Convolutional neural networks (CNNs) have dominated a variety of computer vision tasks, including image classification [krizhevsky2012imagenet] and object detection [ren2015faster]. CNNs are typically made up of series of two main parts: convolutional and pooling layers. The convolutional layer produces feature maps by convolving the local region in the input with a set of kernels. These features are subjected to a non-linear function with the same process repeated for each convolutional layer. In CNNs, the pooling layer carries out a downsampling operation (typically utilizing the max or mean operation) to feature maps. In different existing CNN architectures, the convolutional and pooling layers are followed by a set of fully connected layers, where the last fully connected layer is the softmax computing each object category score.
Popular CNN Backbones: Here, we briefly discuss different popular CNN backbone architectures in literature.
AlexNet: Krizhevsky et al. [krizhevsky2012imagenet] propose a CNN architecture, named AlexNet, for image classification task. AlexNet comprises five convolutional layers followed by three fully-connected layers. The proposed network architecture utilizes Rectified Linear Units (ReLU) for training efficiency. The network contains 60 million parameters and 500,000 neurons with network training performed on the large-scale ImageNet dataset [5206848]. Different data augmentation techniques are employed to increase the training set. In the ImageNet 2012 competition, AlexNet achieved competitive performance with top-1 and top-5 error rates of 39.7% and 18.9%, respectively.
VGGNet: Different from AlexNet, Simonyan and Zisserman [simonyan2014very] introduce an architecture, named VGGNet that comprises 16 layers in total. The network takes an input image of 224 $\times$ 224 size and has around 138 million parameters. It uses different data augmentation techniques, including scale jittering during network training. The VGGNet architecture comprises convolution layers of 3 $\times$ 3 filter, where the receptive fields are convolved at each pixel with a stride of one pixel. The VGGNet contains multiple pooling layers, performing spatial pooling over 2 $\times$ 2 windows with a stride of two pixels. Further, VGGNet contains two fully connected layers followed by a softmax for yielding output predictions. The VGG architecture achieved top classification accuracy on the 2014 ImageNet classification challenge.
ResNet: Different from AlexNet and VGGNet, He et al. [he2016deep] introduce residual neural networks (ResNet) that stacks residual blocks to build a network. ResNet provides a residual learning approach for training networks that are much deeper than their previously utilised counterparts. Instead of learning un-referenced functions, it explicitly reformulates the layers as learning residual functions with reference to the layer inputs. Extensive empirical evidence demonstrates that residual networks are easier to optimize with improved accuracy from higher depth.

The development of CNN-based architectures has led to rise of novel techniques, improved hardware (e.g. GPUs and TPUs), better optimization methods and many open-source libraries. Interested readers can go through the survey papers related to CNN methods for remote sensing [zhu2017deep, ma2019deep]. Previous works have analyzed that CNNs are able to capture image-specific inductive bias which increases their effectiveness in learning better feature representations. However, CNNs do not capture long-range dependencies that aids to enhance expressivity of the representations. Next, we briefly present vision transformers that are capable of modelling long-range dependencies in the images.

Recently, transformers-based models have achieved promising results across many computer vision and natural language processing (NLP) tasks. Vaswani et al.[vaswani2017attention] first introduce transformers as an attention-driven model for machine translation application. To capture the long-range dependencies, transformers use self-attention layers instead of traditional recurrent neural network that struggles to encode such dependencies between the elements of a sequence.

To effectively capture the long-range dependencies within an input image, the work of [dosovitskiy2021an] introduces vision transformers (ViTs) for image recognition task, as shown in Fig.3. ViTs [dosovitskiy2021an] interpret an image as a sequence of patches and process it via a conventional transformers encoder similar to those used in NLP tasks. The success of ViTs in generic visual data have sparked the interest not only in different areas of computer vision but also in the remote sensing community, where a number of ViT-based techniques have been explored in recent years for various tasks.

Next, we briefly describe the key component of self-attention within transformers.
Self-Attention: The self-attention mechanism has been an integral component of transformers as it captures the long-range dependencies and encodes the interaction between all of the sequences tokens (patch embedding). The key idea of self-attention is to learn self-alignment, that is to update the token by aggregating global knowledge from all the other tokens in the sequence[bahdanau2014neural]. Given a 2D image $x\in R^{H\times W\times C}$, the process starts with flattening the image into a series of 2D patches $x_{p}at\in R^{M\times \left(P^{2}C\right)}$, where $C$ represents number of channels, $H$ and $W$ represents height and width of the image, $P\times P$ is the dimension of each individual patch, and $M=HW/P^2$ represents the total number of patches. A learnable linear projection layer of $E$ dimension is used to project these flattened patches and can be showed as a matrix $X\in R^{N\times E}$. The aim of the self-attention is to apprehend interaction among all the $M$ embeddings, that is achieved by introducing the three learnable weight matrices to modify input $X$ into queries (as $W^Q\in R^{E\times E_q}$), keys (as $W^K\in R^{E\times E_k}$) and values (as $W^V\in R^{E\times E_v}$), where $E_q=E_k$. The sequence $X$ is first projected onto these weight matrices to obtain $K=X{W}^K$, $V=X{W}^V$ and $Q=X{W}^Q$. The relative attention matrix $A\in R^{M\times M}$ is

where $\circ$ represents the Hadamard product. In masked self-attention, the attention ratings of future entities are set to zero when predicting an entity in the sequence.
Multi-Head Attention: Multi-head attention (MHA) comprises multiple self-attention blocks concatenated simultaneously channel-wise, in order to capture different complex interactions between different sequence of embeddings. Each of the head of the multi-head self-attention has its own learnable weight matrices represented as $W^{Q_i}$, $W^{K_i}$ and $W^{V_i}$, where $i=0\cdot \cdot \cdot \cdot \cdot \cdot (h-1)$ were $h$ denotes the number head in multi-head self-attention. Hence, we can express,

where output of each head is concatenated to form single matrix $B\in R^{M\times h\cdot E_v}$, whereas $W^O\times R^{h.E_v\times M}$ computes the linear transformation of the heads.
Popular Transformers Backbones: Here, we briefly discuss some recent transformers-based backbones.
ViT: The work of [dosovitskiy2021an] introduces an architecture, where a pure transformer is utilized directly to a sequence of image patches for the task of image classification. The ViT architecture design does not employ image-specific inductive biases (e.g., translation equivariance and locality) and the pre-training is performed on large-scale ImageNet-21k or JFT-300M dataset.
Swin: Liu et al. [liu2021swin] improve over ViT design by introducing an architecture that produces hierarchical feature representation. Swin transformer has linear computational complexity with respect to input image size, where the efficiency is achieved by restricting the self-attention computation to non-overlapping local windows while enabling cross-window connection.
PVT: The work of [Wang2021PVT] introduces a pyramid vision transformer (PVT) architecture, to perform pixel-level dense prediction tasks. The PVT architecture utilizes a progressively shrinking pyramid and a spatial-reduction attention layer for producing high-resolution multi-scale feature maps. The PVT backbone has shown to achieve impressive performance on object detection and segmentation tasks, compared to its CNNs counterpart with similar number of parameters.

Transformers offer unique characteristics that are useful for different vision tasks. Compared to the convolution operation in CNNs where static filters are computed, filters in self-attention are dynamically calculated. Furthermore, permutations and changes in the number of input points have little effect on self-attention. Recent studies [MuzammalNeurIPS21, ParkICLR22] have explored different interesting properties of vision transformers and compare them with CNNs. For instance, the recent work of [MuzammalNeurIPS21] shows that vision transformers are more robust to severe occlusions, domain shifts and pertubations. Next, we present a review of transformers in remote sensing based on the taxonomy shown in Fig.4.

Remote sensing scene classification is a challenging problem, where the task is to automatically associate a semantic category label to a given high-resolution image comprising ground objects and different land cover types. Among the existing vision transformers-based VHR scene classification approaches, Bazi et al.[bazi2021vision] explore the impact of standard vision transformers architecture of [dosovitskiy2021an] (ViT) and investigate different data augmentation strategies for generating addition data. In addition, their work also evaluate the impact of compressing the network by pruning the layers while maintaining the classification accuracy. The work of [deng2021cnns] introduces a joint CNN-transformers framework, where there is one stream of CNNs and another stream of ViT, as shown in Fig.5. The features from the two streams are concatenated and the entire framework is trained using a joint loss function, comprising cross-entropy and center losses, to optimize the two-stream architecture. Zhang et al.[zhang2021trs] introduce a framework, called Remote Sensing Transformer (TRS), that strives to combine the merits of CNNs and transformers by replacing the spatial convolutions with multi-head self-attention. The resulting multi-head self attention bottleneck has fewer parameters and is shown to be effective compared to other bottlenecks. The work of [hao2022two] introduces a two-stream Swin transformers network (TSTNet), that comprises two streams: original and edge. The original stream extracts standard image features whereas the edge stream contains a differentiable edge Sobel operator module and provides edge information. Further, a weighted feature fusion module is introduced to effectively fuse the features from the two streams for boosting the classification performance. The work of [ma2022homo] introduces a transformers-based framework with a patch generation module designed to generate homogeneous and heterogeneous patches. The patch generation module generates the heterogeneous patches directly, whereas the homogeneous patches are obtained using a superpixel segmentation method.

Remote Sensing Pre-training: Different from the aforementioned approaches that either use only transformers or hybrid CNN-transformers designs with backbone networks pretrained on ImageNet datasets, the recent work of [Wang2022empirical] investigates training vision transformers backbones, such as Swin, from scratch on the large-scale MillionAID remote sensing dataset [MillionAID]. The resulting trained backbone models are then fine-tuned for different tasks, including scene classification. Fig.6 shows the response maps, obtained using Grad-CAM++ [chattopadhay2018grad], of different ImageNet (IMP) and remote sensing pre-trained (RSP) models. It can be observed that RSP models learn better semantic representations by paying more attention to the important targets, compared to IMP counterparts. Further, the transformers-based backbones, such as Swin-T better captures the contextual information due to the self-attention mechanism. Moreover, backbones such as, ViTAEv2-S that combines the merits of CNNs and transformers along with RSP can achieve better recognition performance.

Tab. I shows a comparison of aforementioned classification approaches on one of the most commonly used VHR classification benchmarks: AID [xia2017aid]. The AID dataset contains images acquired from multi-source sensors. The dataset possesses a high degree of intra-class variation, since the images are collected from different countries, under different time and seasons with variable imaging conditions. There are in total 10,000 images in the dataset and 30 categories. The performance is measured in terms of mean classification accuracy over all the categories. For more details on AID, we refer to [xia2017aid]. Other than RSP that performs an initial pre-training on Million-AID dataset, all approaches here utilize models pre-trained on ImageNet benchmark.

Localizing objects in VHR imaging is a challenging problem due to extreme scale variations and diversity of different object classes. Here, the task is to simultaneously recognize and localize (either rectangle or oriented bounding-boxes) all instances belonging to different object categories in an image. Most existing approaches employ a hybrid strategy by combining the merits of CNNs and transformers within existing two-stage and single-stage detectors. Other than the hybrid strategy, few recent works also explore DETR-based transformers object detection paradigm [carion2020end].

Hybrid CNN-Transformers based Methods: The work of [xu2021improved] introduces a local perception Swin transformer (LPSW) backbone to improve the standard transformers for detecting small-sized objects in VHR imagery. The proposed LPSW strives to combine the merits of transformers and CNNs to improve the local perception capabilities for better detection performance. The proposed approach is evaluated with different detectors such as, Mask RCNN [he2017mask]. The work of [li2022transformer] introduces a transformers-based detection architecture, where a pre-trained CNN is used to extract features and a transformer is adapted to process feature pyramid of a remote sensing image. Zhang et al.[zhang2022gansformer] introduce a detection framework, where an efficient transformer is utilized as a branch network to improve CNN’s ability to encode global features. Additionally, a generative model is employed to expand the input remote sensing aerial images ahead of the backbone network. The work of [ADTDet] proposes a detection framework based on RetinaNet, where a feature pyramid transformer (FPT) is utilized between the backbone network and the post-processing network to generate semantically meaningful features. The FPT enables the interaction among features at different levels across scale. The work of [Tangcvpr22] introduces a framework, where transformers are adopted to model the relationship of sampled features in order to group them appropriately. Consequently, better grouping and bounding box predictions are obtained without any post-processing operation. The proposed approach effectively eliminates the background information which helps in achieving improved detection performance.

Zhang et al.[Rodformer] introduce a hybrid architecture that combines the local characteristics of depth separable convolutions with the global (channel) characteristics of MLP. The work of [PointRCNN] introduces a two-stage angle-free detector, where both the RPN and regression are angle-free. Their work also evaluates the proposed detector with transformers-based backbone (Swin-Tiny). Liu et al.[HybridNetwork22] propose a hybrid network architecture, called TransConvNet, that aims at combining the advantages of CNNs and transformers by aggregating both global and local information to address the rotation invariability of CNNs with a better contextual attention. Furthermore, an adaptive feature fusion network is designed to capture information from multiple resolutions. The work of [li2021oriented] introduces a detection framework, called Oriented Rep-Points, that utilizes flexible adaptive points as a representation. The proposed anchor-free approach learns to select the point samples from classification, localization and orientation. Specifically, to learn geometric features for arbitrarily-oriented aerial objects, a quality assessment and sample assignment scheme is introduced that measures and identify high-quality sample points for training, as shown in Fig. 7. Further, their approach utilizes a spatial constraint for penalizing the sample points that are outside the oriented box for robust learning of the points.

DETR-based Detection Methods: Few recent approaches have investigated adapting the transformers-based DETR detection framework [carion2020end] for oriented object detection in VHR imaging. The work of [MaOrientedDetr] adapts the standard DETR for oriented object detection. In their approach, efficient encoder is designed for transformers by replacing the standard attention mechanism with a depthwise separable convolution. Dai et al.[AO2DETR] propose a transformers-based detector, called AO2-DETR, where an oriented proposal generation scheme is employed to explicitly produce oriented object proposals. Further, their approach comprises an adaptive oriented proposal refinement module that is designed to compute rotation-invariant features by eliminating the misalignment between region features and objects. Furthermore, a rotation-aware matching loss is utilized to perform a matching process for direct set prediction without the duplicated predictions.

Tab.II shows a comparison of the aforementioned detection approaches on the most commonly used VHR detection benchmark, DOTA [xia2018dota]. The dataset comprises 2,806 large aerial images of 15 different object categories plane, baseball diamond, basketball court, soccer-ball field, bridge, ground track field, small vehicle, Ship, large vehicle, tennis court, roundabout, swimming pool, harbor, storage tank and helicopter. The detection performance accuracy is measured in terms of mean average precision (mAP). For more details on DOTA, we refer to [xia2018dota]. The results show that most of these recent methods obtain similar detection accuracy, with a slight improvement in performance is obtained when using the Swin-T backbone.

In remote sensing, image change detection is an important task for detecting changes on the surface of the Earth with numerous applications in agriculture [muzein2006remote, haack1998remote], urban planning [bolorinos2020consumption], and map revision [metternicht1999change]. Here, the task is to generate change maps obtained by comparing the multi-temporal or bi-temporal images, with each pixel in the resulting binary change map having either zero or one value depending on whether the corresponding position has changed or not. Among the recent transformers-based change detection approaches, Chen et al.[ChenCDTGRS] propose a bi-temporal image transformer, encapsulated in a deep feature differencing-based framework that is designed to model the spatio-temporal contextual information. Within the proposed framework, the encoder is employed to capture context in token-based space-time. The resulting contextualized tokens are then fed to the decoder where the features are refined in the pixel-space. Guo et al.[guo2021deep] propose a deep multi-scale Siamese architecture, called MSPSNet, that utilizes a parallel convolutional structure (PCS) and self-attention. The proposed MSPSNet performs feature integration of different temporal images via PCS and then feature refinement based on self-attention to further enhance the multi-scale features. The work of [zhang2022swinsunet] introduces a Swin transformer-based network with a Siamese U-shaped structure, called SwinSUNet, for change detection. The proposed SwinSUNet comprises three modules: encoder, fusion, and decoder. The encoder transforms the input image into tokens and produces multi-scale features by employing a hierarchical Swin transformer. The resulting features are concatenated in the fusion having linear projection and Swin transformer blocks. The decoder contains upsampling and merging within Swin transformer blocks to progressively generate change predictions.

Wang et al.[Guanghui22RS] introduce an architecture, called UVACD, that combines CNNs and transformers for change detection. Within UVACD, the high-level semantic features are extracted via a CNN backbone, whereas transformers are utilized to generate better change features by capturing the temporal information interaction. The work of [li2022transunetcd] introduces a hybrid architecture, TransUNetCD, that strives to combine the merits of transformers and UNet. Here, the encoder takes features extracted from CNNs and enrich them with global contextual information. The corresponding features are then unsampled and combined with multi-scale features to obtain global-local features for localization. The work of [HybridTransCD] introduces a hybrid multi-scale transformer, called Hybrid-TransCD, that captures both fine-grained and large object features by utilizing heterogeneous tokens via multiple receptive fields.

Tab. III shows a comparison of aforementioned change detection approaches on the most commonly used benchmarks: WHU [ji2018fully] and LEVIR [chen2020spatial]. The WHU dataset comprises a single pair of high-resolution (0.075m) images. Here, the images are of size 32507 $\times$ 15354. The LEVIR dataset comprises 637 pairs of high-resolution (0.5m) images. The images are of size 1024 $\times$ 1024. The performance is measured in terms of F1 score with respect to the change category. Fig. 8 presents a qualitative comparison of different methods with SwinSUNet on example images from WHU-CD dataset.

In remote sensing, automatically segmenting an image into semantic categories by performing pixel-level classification is a challenging problem with a wide range of applications, including geological surveys, urban resources management, disaster management and monitoring. Most existing transformers-based remote sensing image segmentation approaches typically employ a hybrid design with an aim to combine the merits of CNNs and transformers. The work of [xu2021efficient] introduces a light-weight transformers-based framework, Efficient-T, that comprises an implicit edge enhancement technique. The proposed Efficient-T employs hierarchical Swin transformers along with MLP head. A coupled CNN-transformers framework, called CCTNet, is introduced in [wang2022cctnet] that aims at combining the local details such as, edges and texture captured by the CNNs along with the global contextual information obtained via transformers for crop segmentation in remote sensing images. Furthermore, different modules such as, test time augmentation and post-processing steps are introduced in order to remove holes and small objects at inference for restoring the complete segmented images. A CNN-transformers framework, named STransFuse, is introduced in [9573374] where both coarse-grained and fine-grained feature representations at multiple scales are extracted and later combined adaptively by utilizing self-attentive mechanism. The work of [9686732] proposes a hybrid architecture, where Swin transformer backbone that captures long-range dependencies is combined with a U-shaped decoder which employs an atrous spatial pyramid pooling block based on depth-wise separable convolution along with SE block to better preserve local details in an image. The work of [panboonyuen2021transformer] utilizes a pre-trained Swin Transformer backbone along with three decoder designs namely, U-Net, feature pyramid network and pyramid scene parsing network for semantic segmentation in aerial images.

We present in Tab. IV a quantitative comparison of aforementioned approaches on the two most commonly used semantic segmentation datasets: Potsdam[Potsdam] and Vaihingen[Vaihingen]. The Potsdam dataset comprises 38 patches, where each patch has a resolution of 6000 $\times$ 6000 pixels collected over the Potsdam City with a ground sampling distance of 5 cm. The dataset has six categories. The Vaihingen dataset comprises 33 samples, where each sample has a resolution from 1996 $\times$ 1995 to 3816 $\times$ 2550 pixels. Here, the ground sampling distance is 9 cm. This dataset contains same categories as Potsdam. The performance is measured in terms of overall accuracy (OA) computed using true positive, false positive, false negative and true negative. Fig.9 presents a qualitative comparison between Trans-CNN and other approaches on the Potsdam dataset.

Building Extraction: Transformers-based techniques have also been recently explored for the problem of building extraction, where the task is to automatically identify building and non-building pixels in a remote sensing image. A dual-pathway transformers framework is introduced in [chen2021building] that strives to learn long-range dependencies both in spatial and channel directions. The work of [xiao2022swin] proposes a transformers framework, STEB-UNet, comprising Swin transformer-based encoding booster that captures semantic information from multi-level features generated from different scales. The encoder booster is further integrated in a U-shaped network design that fuses local and large-scale semantic features. A transformers-based architectures, called BuildFormer, comprising a window-based linear attention, a convolutional MLP along with batch normalization is introduced in [wang2022buildformer]. The work of [qiu2022transferring] explores the problem of generalizability of building extraction models to different areas and propose a transfer learning approach to fine-tune models from one area to a subset of another unseen area.

Other than semantic image segmentation and building extraction with transformers, a recent work by [xu2021improved] explores the problem of instance segmentation where the task is to automatically classify each pixel into an object class within an image while also differentiating multiple object instances. Their approach aims at combining the advantages of CNNs and transformers by designing a local perception Swin transformer backbone to enhance both local and global feature information.

Apart from the problems discussed above, transformers-based techniques are also explored for other VHR remote sensing tasks such as, image captioning and super-resolution.

Image Captioning: Image captioning in remote sensing images is a challenging problem, where the task is to generate semantically natural description of a given image. Few recent works have explored using transformers for image captioning. The work of [shen2020remote] introduces a framework, where standard transformers are adapted for remote sensing image caption generation by integrating residual connections, dropout layers and fusing features adaptively. Moreover, a reinforcement learning technique is utilized to further improve the caption generation process. An encoder-decoder architecture is introduced in [liu2022remote], where the multi-scale features are first extracted from different layers of CNNs in the encoder and then a multi-layer aggregated transformer is utilized in the decoder to effectively exploit the multi-scale features for generating sentences. The work of [ren2022mask] introduces a topic token-based mask transformers framework, where a topic token is integrated into the encoder and serves as a prior in the decoder for capturing improved global semantic relationships.
Image Super Resolution Remote sensing image super-resolution is the task of recovering high-resolution images from their low-resolution counterparts. A few recent works have explored transformers for this task. A transformer-based multi-stage enhancement structure is introduced in [lei2021transformer] that leverages features from different stages. The proposed multi-stage structure can be combined with conventional super-resolution techniques in order to fuse multi-resolution low as well as high-dimension features. [ye2021super] proposes a CNN-transformer hybrid architecture to integrate both local and global feature information for super-resolution. The work of [an2022tr] explores the problem of multi-image super-resolution, where the task is to merge multiple low-resolution remote sensing images of the same scene into a high-resolution one. Here, a transformers-based approach is introduced comprising an encoder having residual blocks, a fusion module and a super-pixel convolution-based decoder.

To summarize the review of transformers in VHR imagery, we present a holistic overview of different techniques in literature in Tab.V.

Here, the task is to automatically classify and assign a category label to each pixel in an image acquired through hyperspectral sensors. Next, we first review recent works that are either based on the pure transformers design or utilize a hybrid CNN-transformers approach. Afterwards, we discuss few recent transformers-based approaches fusing different modalities for hyperspectral image classification.

Pure Transformers-based Methods: Among existing works, the approach of [he2019hsi] introduces a bi-directional encoder representation from transformers, called HSI-BERT, that strives to capture global dependencies. The proposed architecture is flexible and can be generalized from different regions with the need to perform pre-training. A transformers-based backbone, called SpectralFormer, is introduced in [hong2021spectralformer] that can take pixel-wise or patch-wise inputs and is designed to capture spectrally local sequence knowledge from nearby hyperspectral bands. SpectralFormer utilizes cross-layer skip connection to circulate information from shallow to deep layers by learning soft residuals across layers, thereby producing group-wise spectral embeddings. To circumvent the problem of fixed geometric structure of convolution kernels, a spectral–spatial transformer network is proposed in [zhong2021spectral] comprising a spatial attention and a spectral association module. While the spatial attention aims at connecting the local regions through aggregation of all input feature channels with spatial kernel weights, the spectral association is achieved through the integration of all spatial locations of the corresponding masked feature maps. Transformers are also explored in the spatial and spectral dimensions in [liu2022dss]. Here, a framework is introduced comprising spectral self-attention that learns to capture interactions along the spectral dimension and a spatial self-attention designed to pay attention to features along the spatial dimension. The resulting features from both spectral and spatial self-attention are then combined and input to the classifier.

Hybrid CNN-Transformers based Methods: Several works recently have explored combining the merits of CNNs and transformers to better capture both the local information as well as long-range dependencies for hyperspectral image classification. To this end, a convolutional transformer network, named CTN, is introduced in [zhao2022convolutional] that utilizes center position encoding to generate spatial position features by combining pixel positions with spectral features as well as convolutional transformer to further obtain local-global features, as shown in Fig.10. A hyperspectral image transformer (HiT) classification approach is proposed in [yang2022hyperspectral], where convolutions are embedded into transformers architecture to further integrate local spatial contextual information. The proposed approach comprises two main modules, where one module, called spectral-adaptive 3-D convolution projection, is designed to generate spatial-spectral local information via spectral adaptive 3D convolution layers from hyperspectral images. The other module, named Conv-Permutator, employs depthwise convolutions to capture spatial–spectral representations separately along the spectral, height and width dimensions. The work of [jia2022multiscale] introduces a multi-scale convolutional transformer that effectively captures spatial-spectral information which can be integrated with transformers network. Further, a self-supervised pre-task is defined that masks the token of the central pixel in the encoder, whereas remaining tokens are input to the decoder in order to reconstruct the spectral information corresponding to the central pixel. In [sun2022spectral], a spectral–spatial feature tokenization transformer, called SSFTT, is proposed that generates spectral-spatial and semantic features. The SSFTT comprises a feature extraction module that produces low-level spectral and spatial features by employing a 3D and a 2D convolution layer. Furthermore, a Gaussian weighted feature tokenizer is utilized in SSFTT for feature transformation which are then input to a transformer encoder for feature representation. Consequently, a linear layer is employed to generate the sample label. Zhao et al. [zhao2022convolutional] proposes a convolutional transformer network (CTN) that employs center position encoding to combine spectral features with pixel positions. The proposed architecture introduces convolutional transformer blocks that effectively integrates local and global features from hyperspectral image patches. Yang et al. [yang2022hyperspectral] introduces a hyperspectral image transformer (HiT) framework, where convolution operations are embedded within the transformers design for also integrating local spatial contextual information. The HiT framework comprises of a spectral-adaptive 3D convolution projection to capture local spatial-spectral information. Additionally, the HiT framework employs a conv-permutator module that uses the depthwise convolution for explicitly capturing the spatial-spectral information along different dimensions: height, width and spectral. The work of [sun2022spectral] introduces a spectral–spatial feature tokenization transformer, named SSFTT, that consists of a spectral-spatial feature extraction scheme for encoding shallow spectral-spatial features, a feature transformation module which produces transformed features used as input in the encoder.

Multi-modal Fusion Transformers based Methods: Few recent transformers-based works also explore fusing different modalities, such as hyperspectral, SAR, LiDAR for hyperspectral image classification. A Multi-modal fusion transformer, MFT, is introduced in [roy2022multimodal], that comprises a data fusion scheme to derive class tokens in the transformers from multi-modal data (e.g., LiDAR, SAR) along with the standard hyperspectral patch tokens. Further, the attention mechanism within MFT fuses information from tokens of hyperspectral and other modalities into a new token of integrated features. The work of [xue2022deep] introduces an approach, where a spectral sequence transformer is utilized to extract features from hyperspectral images along the spectral dimension and a spatial hierarchical transformer to generate spatial features in a hierarchical manner from both hyperspectral and LiDAR data.

Tab.VI shows a comparison of some representative CNN-based approaches with both pure transformers and hybrid CNN-transformers based methods on two popular hyperspectral image classification benchmarks: Indian Pines and Pavia. The Indian Pines dataset is acquired through airborne visible/infrared imaging spectrometer (AVIRIS) sensor in Northwestern Indiana, USA. Here, the images comprise 145 $\times$ 145 pixels in the spatial dimension, at a ground sampling distance (GSD) of 20m with 220 spectral bands that cover the wavelength range of 400–2500 nm. After the removal of noisy bands, 200 spectral brands are retained. The original dataset contains 16 class, where several methods discard the small classes. For the remaining categories, the number of training samples are 200 per class. The Pavia dataset comprises images acquired through the reflective optics system imaging spectrometer (ROSIS) sensor over Pavia, Italy. Here, the images consist of 610 $\times$ 340 pixels in the spatial dimension, at a GSD of 1.3m with 103 spectral bands covering from 430 to 860 nm. The dataset contains nine categories, where the number of training samples are 200 per class. Generally, three metrics are used to evaluate the performance of methods quantitatively: overall accuracy, average accuracy and kappa coefficient. The overall accuracy (OA) denotes to the proportion of correctly classified test samples, whereas average accuracy (AA) reflects the average recognition accuracy for each category. The kappa coefficient refers to the consistency between the generated classification maps from the model and the available ground-truth. Fig.11 presents a qualitative comparison between HSI-Bert [he2019hsi] and other existing CNN-based methods on the Pavia dataset.

In the hyperspectral pansharpening problem, the task is to enhance low-resolution hyperspectral image spatially using the spatial information from registered panchromatic image, while preserving the spectral information of the low-resolution image. Pansharpening plays an important role in a variety of tasks in remote sensing, including classification and change detection. Previously, CNN-based approaches have shown promising results for this task. Recently, transformers-based methods have performed favorably for this problem by also utilizing the useful global contextual information. A multi-scale spatial–spectral interaction transformer, MSIT, is proposed by [zhang2022multiscale] that comprises a convolution–transformer encoder to extract multi-scale local and global features from low-resolution and panchromatic images. The work of [li2022pan] introduces an architecture, where global features are constructed using transformers and local features are computed using a shallow CNN. These multi-scale features extracted in a pyramidal fashion are learned simultaneously. The proposed approach further introduces a loss formulation with spatial and spectral loss simultaneously used for training using the real data. Liang et al. [liang2022pmacnet] propose a framework, named PMACNet, where both the region-of-interest from the low-resolution image and the residuals for regression to high-resolution image are learned in a parallel CNN structure. Afterwards, a pixel-wise attention module is utilized to adapt the residuals based on the learned region-of-interest.

A transformers-based regression network is introduced by [su2022transformer], where the feature extraction of spatial and spectral information is performed by utilizing a Swin transformer model. The work of [DBLP:conf/aaai/ZhouHFFL22] introduces a transformers-based approach, where multi-spectral and panchromatic features are formulated as keys and queries for enabling joint learning of features across the modalities. Further, this work employs an invertible neural module to perform effective fusion of the features for generating the pansharpened images. Bandara et al. [Bandaracvpr22] propose a framework comprising separate feature extractors for panchromatic and hyperspectral images, a soft attention mechanism and a spectral-spatial fusion module. The pansharpened image quality is improved by learning cross-feature space dependencies of the different features.

To summarize the review of transformers in hyperspectral imaging, we provide a holistic overview of the existing techniques in literature in Tab.VII.

Classification: Accurately classifying the target categories within SAR images is a challenging problem with numerous real-world applications. Recently, transformers have been explored for automatic interpretation and target recognition in SAR imagery. The work of [9658539] explores vision transformers for polarimetric SAR (PolSAR) image classification. In this framework, the pixel values of the image patches are considered as tokens and the self-attention mechanism is employed to capture long-range dependencies followed by multi-layer perceptron (MLP) and learnable class tokens to integrate features. A contrastive learning technique is utilized within the framework to reduce the redundancies and perform the classification task. Fig.12 shows the overview of the framework and a qualitative comparison in terms of supervised classification is presented in Fig.13. s

Other than the aforementioned pure transformers-based approach, hybrid methods utilizing both CNNs and transformers also exist in literature. The work of [9713848] introduces a global–local network structure (GLNS) framework that combines the merits of CNNs and transformers for SAR image classification. The proposed GLNS employs a lightweight CNN along with an efficient vision transformer to capture both local and global features which are later fused to perform the classification task. Other than standard fully-supervised learning, transformers are also explored in the limited supervision regime such as, few-shot SAR image classification. Cai et al. [cai2022st] introduce a few-shot SAR classification approach, named ST-PN, where a spatial transformer network is utilized for performing spatial alignment on CNN-based features.

Segmentation and Detection: Detection and segmentation in SAR imagery is vital for different applications such as, crop identification, target detection, and terrain mapping. In SAR imagery, segmentation can be challenging due to the appearance of speckles which is a type of multiplicative noise that increases with the back-scattering radar magnitude. Among recent transformers-based approaches, the work of [ke2022gcbanet] introduces a framework, named GCBANet, for SAR ship instance segmentation. Within the GCBANet framework, a global contextual block is employed to encode spatial holistic long-range dependencies. Furthermore, a boundary-aware box prediction technique is introduced to predict the boundaries of the ship. Xia et al. [xia2022crtranssar] introduce an approach, named CRTransSar, that combines the benefits of CNNs and transformers to capture both local and global information for SAR object detection. The proposed CRTransSar works by constructing a backbone with attention and convolutional blocks. A geospatial transformer framework is introduced in [chen2022geospatial], comprising the steps of image decomposition, multi-scale geo-spatial contextual attention and recomposition for detecting aircrafts in SAR imagery. A feature relation enhancement framework is proposed in [zhang2022sfre] for aircraft detection in SAR imagery. The proposed framework adopts a fusion pyramid structure to combine features of different levels and scales. Further, a context attention enhancement technique is employed to improve the positioning accuracy in complex backgrounds.

Other than ship and aircraft detection, the recent work of [ma2021end] introduces a transformers-based framework for 3D detection of oil tank targets in SAR imagery. In this framework, the incidence angle is input to the transformer as a prior token followed by a feature description operator that utilizes scattering centers for refining the predictions.

Apart from SAR image classification, detection and segmentation, few works exist exploring transformers for other SAR imaging problems such as, image despeckling.

SAR Image Despeckling: The aforementioned interpretation of SAR imaging is made challenging due to the degradation of images caused by a multiplicative noise known as speckle. Recently, transformers have been explored for SAR image despeckling. The work of [Perera22neural] introduces a transformers-based framework comprising an encoder that learns global dependencies among various SAR image regions. The transformers-based network is trained in an end-to-end fashion with synthetic speckled data by utilizing a composite loss function.

Change Detection in SAR Images: SAR images can be affected by imaging noise which presents challenges when detecting changes in high-resolution (HR) SAR data. Recently, a self-supervised contrastive representation learning technique has been proposed by [Huihui22neural], where hierarchical representations are constructed using a convolution-enhanced transformer to distinguish the changes from HR SAR images. A convolution-based module is introduced to enable interactions across windows when performing self-attention computations within local windows.

SAR Image Registration: Several applications such as, change detection involves joint analysis and processing of multiple SAR images that are likely acquired in different imaging conditions. Thus, accurate SAR image registration is desired where the reference and the sensed images are registered. The recent work of [Fan22RS] explores transformers for large-size SAR dense-matching registration. Here, a hybrid CNN-transformer is employed to register images under weak texture condition. First, coarse registration is performed via the down-sampled original SAR image. Then, cluster centers of registration points are selected from the previous coarse registration step. Afterwards, the registration of image pairs are performed using a CNN-transformer module. Lastly, the resulting point pair subsets are integrated to achieve the final global transformation through RANSAC.

In summary, we present a holistic overview of the existing transformers techniques in SAR imagery in Tab.VIII.