ELSR: Extreme Low-Power Super Resolution Network For Mobile Devices

As one of the most popular topics in computer vision, Super-Resolution has drawn a lot of attention from researchers. With the development of deep learning [1], methods based on convolution network show great advantages over traditional algorithms such as bilinear and bicubic unsampling. The pioneer work was done by Dong et al. [2], who proposed SRCNN for image super-resolution and achieve outstanding performance compared to previous works. VDSR [3] introduces residual connection and increase network depth which with 20 layers, achieving significant improvement in restoration quality. ESPCN [4] proposed pixel shuffle operation, and added it at the end of network to directly upsample to the size of HR image from LR feature maps.

VSR can be viewed as a generalization of image super-resolution. Videos additionally provide temporal information among frames, which can be explored to improve restoration quality. One of the representative work is FRVSR [5] which first produces a super-resolved HR output by concatenating the SR output of network from previous LR input and the current LR frame, which obtains better result compared with single frame methods. Different from FRVSR that only utilize one previous frame to help reconstruct current LR frame, EDVR [6] adopts sliding windows to obtain more temporal information for improving restoration quality. EDVR obtains aligned feature by introducing coarse-to-fine deformable alignment and a new spatio-temporal attention fusion module. As an integrator of video super-resolution models, BasicVSR [7] and BasicVSR++ [8] proposed a baseline VSR model composed of propagation, alignment, aggregation and upsampling. By conducting bidirectional propagation and deformable convolution for feature-level alignment, BasicVSR series has achieved excellent performance in VSR task.

Besides the manual design of deep neural networks, there are a plethora of works that try to improve the effectiveness and efficiency of deep neural networks via network pruning, low-rank filter decomposition, net quantization, neural architecture search and knowledge distillation etc. Among those network compression works, a couple of them have been successfully applied to image SR. However, it is still a challenging topic for mobile device SR because of the limit of computing resources and restrictions on power consumption. IMDN [9] proposed a multi-branch distillation network structure to reduce model parameters and speed up running time. RFDN [10] further modified model structure, replace channel spilt with $1\times 1$ convolution and proposed ESA module to improve restoration quality by introducing attention mechanism. However, the above model still measures lantency and power on the desktop devices which is not suitable for mobile devices.

At the same time, some of the researchers try to deploy deep learning super-resolution model to real-time mobile devices. Edge-SR [11] is proposed to super-resolve 4K resolution video streams to 8K resolution video streams in television using only one convolution layer. Under the same parameters, Edge-SR acquires better visual quality, and have less parameters under the same subjective quality. Moreover, in MobileOne [12] is proposed as lightweight backbone designed for mobile devices, which utilize re-parameterize tricks to speed up inference time.

Inspired by the above works, we proposed a mobile-friendly VSR model which has very few parameters and FLOPs and can run in real time on the device with low power consumption.

The core idea of our method is to design a mobile friendly network which consumes as little energy as possible, so we discord some complex operations such as optical flow, multi-frame feature alignment, and start our experiment from single frame baselines.

Currently, the lightweight super-resolution models tend to have a multi-branch structure (e.g. IMDN, RFDN). However, as shown above Table 1, the multi-branch distillation structure can show significant increase in energy consumption while also get a slight increase in PSNR compared with the plain convolutional network of similar parameters. For this reason, we abandoned further exploration on multi-branch network architectures, and focused on plain convolutional SR networks.

Attention mechanism [13] is also an effective way to improve the performance of SR models. We also tested some commonly used attention modules such as ESA, CCA and PA [14]. Experimental results show that though attention modules bring performance improvement, the extra energy consumption introduced is still unacceptable. As shown in Table 1, ESA module can boost performance by about 0.09dB, but it also increase power consumption by about 0.15W resulting in a lower score. As a result, we use plain serial network structure without bells and whistles, and a skip connection transfers the input features directly to the latter conv layers to benefit the training and compensate the loss of information. Residuals are used in many lightweight models for super-resolution, but as the add operation of former and latter feature maps will bring more FLOPs, so we use only one residual pathway in the final model.

Finally, we obtain a simple yet effective network structure with single frame input (as shown in Fig.1) which only have 6 layers, of which only 5 have learnable parameters, including 4 Conv layers and a PReLU activation layer. Pixel-Shuffle operation (also known as $depth2space$) is used at last to upscale the size of output without introducing more calculation. The intermediate feature channels are all set to 6.

For the choice of activation function, we compared the commonly used ReLU, Leaky ReLU and Parametric ReLU (PReLU) activations (Tab.2), and the results imply that the PReLU can boost the performance by about 0.05 dB due to its higher flexibility (Fig.2). At the same time, PReLU op has nearly the same power consumption as ReLU / LeakyReLU, thus is suitable for our low-power model design.

We utilize L1 and L2 loss (i.e. MSE loss) function separately since we employ a multi-stage training approach. Firstly, we use L1 loss function for pre-training the model to achieve better convergence, and then the previous model is finetuned using L2 loss to reach higher PSNR score.

Our model is trained on REDS [15] dataset, which have a large diversity of contents and dynamic scenes. It is widely used in video super-resolution and video frame-interpolation task. REDS dataset consists of 300 video sequences with each 100 frames of 720 $\times$ 1280 resolution, and the organizer provides the corresponding low-resolution video sequences by dowmsampling scale $\times 4$. At the same time, We generated low-resolution video sequences which downsampling scale $\times 2$ by ourselves using matlab bicubic function for model pretraining. In this challenge, the dataset is divided into 240 sequences for training, 30 sequences for validation, and the left 30 sequences for testing.

We adopt REDS dataset including $\times 2$ and $\times 4$ low-resolution video sequences for training where $\times 2$ data is used for model pre-training. In order to accelerate the model training process, all training video frames are randomly cropped to patches.

Pre-training on $\times 2$ task with the same training dataset for $\times 4$ task is a commonly used strategy in super-resolution tasks. We adopt this strategy with some modification due to the extreme tiny structure of our network. For $\times 2$ pre-training, the last conv layer with weight of size $[48,8,3,3]$ will be replaced with $[12,8,3,3]$ for $\times 4$ finetuning. As a common case, after the $\times 2$ model training is finished, the weights are loaded to $\times 4$ model except for the last layer which is incompatible in size. However, in our extremely small model, the weight of last layer occupies more percentage in all parameters compared with normal or large SR models 3.

Considering the operation of pixel shuffle, we keep the weight from last layer in $\times 2$ model and repeat the weight 4 times along channel dimension, complying with the spatial position of the output after pixel shuffle (Fig.4). After this process, the output of $\times 4$ model loaded with $\times 2$ weights becomes the nearest interpolation of the $\times 2$ output, which accelerates the convergence in training $\times 4$ model.

The model is trained for six stages, and the detailed training configurations are as follows:

(I) Pre-train a $\times 2$ model from scratch as stated above, with batch size 64 and patch size (HR) 256 for 500k iterations. The model is optimized using L1 loss with Adam optimizer, with initial learning rate of $5e-4$, and the learning rate decreases by 0.5 in 200k, 400k iterations respectively.

(II) Train $\times 4$ model using the pre-trained weights from $\times 2$ model. Batch size is 64 and patch size is 256, total iteration number is 500k. The model is optimized using L1 loss with Adam optimizer, with initial learning rate of $5e-5$, and the learning rate decreases by 0.5 in 100k, 300k and 450k iterations respectively.

(III) Continue to finetune the model from Stage (II) using L1 loss, batch size 64 and patch size 256, and optimizer type stay the same. The model is finetuned for 300k iterations, with initial learning rate of $2e-4$ which decreases by 0.5 in 200k iterations.

(IV) Finetune the model from Stage (III) using MSE loss, with patch size 256 and batch size 64. This stage takes 1000k iterations with learning rate is $2e-4$, and the learning rate decreases by 0.5 in 300k, 600k and 900k iterations respectively.

(V) Continue to finetune the model from Stage (IV) using MSE loss, with patch size 512 and batch size 64. This stage takes 500k iterations with learning rate is $2e-4$, and the learning rate decreases by 0.5 in 100k, 200k, 300k and 400k iterations respectively.

(VI) Finally, finetune the model from Stage (V) using MSE loss, with patch size 640 and batch size 64. This stage takes 50k iterations with learning rate is $2e-5$.

In this challenge, REDS test-set is used to evaluate the quality of the reconstructed results, and the power consumption of the model on the actual MediaTek 9000 platform is tested by organizers. The top three results are shown in Table 3. As shown in the leaderboard, our solution achieves a competitive result of 90.9 on challenge test set where the PSNR is 27.34 dB and the power consumption is 0.09W/30FPS, ranking 1st place in the Mobile AI & AIM 2022 real-time VSR challenge. Notably, our method has the lowest power consumption among all participants. Fig 5 shows the comparison of our results with bicubic. The result shows our method has more clear edges and details compared with traditional bicubic interpolation algorithm while is power efficient.