Video Mobile-Former: Video Recognition with Efficient
Global Spatial-temporal Modeling

In this paper, we introduce a lightweight video transformer by extending Mobile-Former architecture that is designed for image tasks. Mobile-Former is a parallel architecture bridging MobileNet mobilenets; mobilenetv2 and Transformer vit with bidirectional cross-attention. Mobile-Former consists of stacked Mobile-Former blocks. Each block includes a Mobile block, a Former block and two interaction bridges.

A Mobile block uses an inverted bottleneck block of MobileNet V2 mobilenetv2 to extract local features. The Mobile block of the $i^{th}$ layer takes the feature map $X_i\in R^{H_i\times W_i\times C_i}$ as inputs, which is processed by the $1\times 1$ pointwise convolution and the $3\times 3$ depthwise convolution later. A Former block is a transformer module aiming to compute global relations of input features, which includes a multi-head attention (MHA) and a feed-forward (FFN). To reduce the computational cost, Former block takes $M$ learnable global tokens $Z\in R^{M\times d}$ instead of dense patch tokens as input. The input of attention is split as $Z=\left[Z_s\right]$ for MHA with $H$ heads. The simplified self-attention of multiple global tokens is defined as:

where $W_s^Q$ is the projection matrix of query in the $s^{th}$ head, $W^O$ aims to combine the channel information of different heads. And $Attn$ denotes the standard attention:

The computational complexity of the Former block is $O(M^{2}d+M{d}^2)$.

To exchange information between the Mobile block and the Former block, cross-attention is applied between local features and global tokens. We denote the reshaped local feature map as $\tilde{X}\in R^{{\tilde{L}}_i\times C_i}$ where ${\tilde{L}}_i=H_i\times W_i$. The cross-attention from local features to global tokens follows:

Then the derived global tokens are sent to the Former block. After computing the global relations, the cross-attention from global tokens to local features is defined as:

where global information is injected to the output of the Mobile block, and $W_s^K$ and $W_s^V$ are the projection matrices for the key and the value. In these two cross-attention modules, two policies are applied to reduce the computational cost: 1) taking local features at the bottleneck of Mobile block as the input of cross-attention due to fewer channels; b) removing the projection matrices from the side of the local features where the length of the input sequence is long, but keeping them at the side of global tokens.

In Mobile-Former, since the number of global tokens $M$ is no more than six, the Former block and cross-attention modules are computational efficient while the Mobile block consumes the most computation. In our work, we focus on adapting Mobile-Former from the image domain to the video domain and below we discuss how to extend the Mobile CNN branch from 2D to 3D and how to modify the Former branch for efficient temporal modeling. While these extensions sound straightforward, they are non-trivial as the additional temporal axis brings not only more information to process but also more redundancy as well.

To capture temporal information, 3D convolution is widely used in previous literature of video recognition. One effective method of constructing CNN-based video models is to extend existing image architectures by replacing 2D convolutions with 3D convolutions. In 2D CNN models, the spatial dimension of the feature map is substantially downsampled. However, in most recent 3D CNN models, the temporal dimension of video features is not downsampled x3d; movinets, or is progressively downsampled c3d; i3d. When the number of input video frames is large, the computational cost of the video model to encode a video clip will be much larger than that of the image model to encode an image.

In Video Mobile-Former, to make an efficient adaption of Mobile block, we introduce large temporal downsampling that reduces the size of feature maps at the early stage of the video model. Specifically, for the input video clip $X\in R^{T\times H\times W\times 3}$, we use large temporal stride $s_t$ ($s_t\ge 4$) in the 3D convolution to perform temporal downsampling at the stem of Video Mobile-Former, which reduces the temporal length of the output feature map to $T/s_t$. Therefore, the computational cost of the following Mobile blocks and the cross-attention blocks is reduced by a factor of $s_t$. For Mobile block, we replace 2D $3\times 3$ depthwise convolutions and 2D $1\times 1$ pointwise convolutions with 3D $3\times 3\times 3$ depthwise convolutions and 3D $1\times 1\times 1$ pointwise convolutions respectively. The computational complexity of Mobile block is $O\left(THW{C}^2/s_t\right)$.

Most video Transformers are extended from the image Transformers in the temporal dimension, and perform global spatial-temporal modeling for the input video  fan2021multiscale; arnab2021vivit. The longer the input token sequence length is, the heavier the computational cost. For image Transformers, the token sequence with a length of $L_i=H_i\times W_i$, generated by reshaping the image feature map, is processed by MHSA. When extended to the video domain, the input of the Transformer blocks is converted from 2D image features to 3D video features, and MHSA needs to process the tokens sequence of length $T_i\times H_i\times W_i$ after the video features are reshaped. Since the computational complexity of SA grows quadratically with respect to the sequence length of the input, the extension of MHSA in the temporal dimension will incur a very large increase in computation.

Although some video transformers perform self-attention through pooling  li2022mvitv2 or within a local window liu2021video, for video data, the length of the input token sequence is very long because it is proportional to the number of video frames, thus the huge computational cost of transformers is still a significant problem and cannot meet the extremely low FLOPs requirement for devices.

To extend Former block and the interaction bridge from 2D to 3D, if we follow the previous paradigm of extending image models and use $M$ global tokens for each frame ($X_{i,t}\in R^{H_i\times W_i\times C_i}$ at the $i^{th}$ layer) in the interaction bridge, we would obtain $T\times M$ global tokens for the input of Former block. This strategy is also used in TokenLearner tokenlearner. However, we find this will not only incur non-negligible cost compared to the Mobile block but also is unnecessary.

On account of the frame redundancy in videos, in the Video Mobile-Former, we extract the global information of one whole video with a small constant number of tokens $Z\in R^{M\times d}$ ($M\le 6$ in this paper), which is not related to the video length. Therefore, in the interaction bridge between Mobile block and Former block, all the global tokens communicate with the feature map of all the frames by the cross-attention. The formulation of the cross-attention in Video Mobile-Former is similar to that in the Mobile-Former but the token length of the reshaped feature map $\tilde{X}$ is $T/s_t\times H\times W$. And the computational complexity of interaction bridges is $O(THWMC/s_t+MdC)$.

In the Former block, MHSA is performed on only $M$ global tokens to obtain global relationships of the whole video. The computational complexity of Former block is $O(M^{2}d+M{d}^2)$, which means the computational cost of Former block for a video is equal to that for an image.

In Mobile-Former, ReLU in the Mobile block is replaced by dynamic ReLU chen2020dynamicrelu, the parameters of which are generated by applying two MLP layers on the first global token output from the Former block. In the Video Mobile-Former, to introduce the temporal difference of each frame, we propose frame-level dynamic ReLU. Specifically, we concatenate the average-pooled feature of each frame (the frame-level feature) with the first global tokens, and the concatenated feature (the shape is $(T/s_t\times 1\times 1\times (C+d\left)\right)$) is taken as the input of two MLP layers to generate the parameters of dynamic ReLU. Therefore, the activation functions on different temporal positions are different in the Mobile block of Video Mobile-Former.

As shown in Table 1, we build four Video Mobile-Former variants with different computational costs from 560M to 5.61G FLOPs, and refer them by their FLOPs, e.g. Video-Mobile-Former-560M, Video-Mobile-Former-1G. We design different variants by changing the depth of the model (#Blocks), the base channel number (#Dim) and the expansion ratio (#Exp) of each Mobile block, while all blocks have six global tokens with the same dimension 128. All variants have four stages and we employ spatial downsampling (stride $=2$) at the beginning of each stage. Note that we set temporal stride $s_t=8$ at the stem of Video-Mobile-Former-560M and set $s_t=4$ at the stem of other variants. The spatial resolution of input video (#Res) is 172 for Video-Mobile-Former-560M and 224 for other variants.