Sliding Window Recurrent Network for Efficient Video Super-Resolution

Single Image Super-Resolution (SISR) is the task of trying to reconstruct a high-resolution image from its degraded low-resolution low-quality counterpart. In the past few years, numerous works based on deep learning and deep convolution neural networks have achieved tremendous performance gains over traditional super-resolution approaches [dong2014learning, dong2015image, hui2018fast, kim2016accurate, ledig2017photo, luo2022deep, wang2018esrgan, zhang2018residual, lim2017enhanced]. SRCNN [dong2014learning] is the first work that uses CNN in super-resolution. Later, most subsequent works focus on optimizing the network architectures [dong2015image, lai2017deep, zhang2018residual, lim2017enhanced] and loss functions [johnson2016perceptual, ledig2017photo, wang2018esrgan, lugmayr2020srflow].

Starting from the pioneer network SRCNN [dong2015image], deep convolution neural network based methods have brought significant achievement in both image and video super-resolution tasks [kim2016accurate, lim2017enhanced, zhang2018image, ahn2018fast, wang2019edvr, liang2021swinir, bhat2021deep, luo2021ebsr, luo2022bsrt, lian2021kernel, chan2021basicvsr]. Particularly, in video super-resolution, where the most important parts are frame alignment, many advanced techniques have been developed to improve the accuracy. For example, VESPCN [caballero2017real] and TOFlow [xue2019video] propose to use optical flow to align frames. TDAN [tian2020tdan] and EDVR [wang2019edvr] point out that estimating an accurate flow for occlusion and large motion frames is difficult and they choose to align frames using deformable convolution [dai2017deformable, zhu2019deformable]. Especially, EDVR enjoys the merits of implicit alignment and its PCD module uses a pyramid and cascading architecture to handle occlusion and large motions. Another line of VSR methods also incorporates recurrent networks in video process [sajjadi2018frame, isobe2020video, haris2019recurrent, isobe2020revisiting], they usually use the hidden state to record the important temporal information. For the frame reconstruction part, residual blocks [he2016deep] and attention mechanism [wang2018non] are widely used to improve the performance. And recent transformer-based methods [liang2021swinir, liang2022vrt] also have shown attractive in image/video restoration.

Although most CNN-based approaches can obtain remarkable results in video super-resolution, their performance gains are often up to the network capacities [ignatov2021real, li2022ntire]. It means that they usually require huge memories and computational resources, which makes it difficult to apply these state-of-the-art models on real-world smartphones that have constrained resources and inference times. Thus the network that is deployed on mobile devices should take care of the particularities of mobile NPUs and DSPs [ignatov2021real, ignatov2021real2]. To deal with it, AI Benchmark application [ignatov2018ai, ignatov2019ai] is designed to allow researchers to run neural networks on the mobile AI acceleration hardware. Based on the AI Benchmark application, Mobile AI 2021 [ignatov2021real] and Mobile AI & AIM 2022 Real-Time Video Super-Resolution Challenge is held to promote the development of real-time mobile VSR networks. In this paper, we’d like to follow the setting of these challenges and design an efficient yet mobile-compatible network for smartphones.

The core idea of our method is to grasp and aggregate complementary information from neighbouring frames to improve the performance of video reconstruction. As illustrated in Figure 2, given the low-resolution frame sequence $\{x_i{\}}_{i=1}^N,x_i\in R^{h\times w\times c}$, at each time step $t$, our network accepts three basic inputs (including previous frame $x_{i-1}$, current frame $x_i$ and future video frame $x_{i+1}$) and output a high-quality frame $y_i$, which seems like a sliding window multi-frame super-resolution algorithm [wang2019edvr, luo2022bsrt], given by

where $f$ is the VSR network and $\theta$ represents its learnable parameters. Inspired by [isobe2020revisiting], we take the advantage of recurrent hidden states to preserve previous and future information. Specifically, the initial hidden states (forward and backward) are set to 0 and will be updated when the window slides to the next frames. Then we can reformulate Equation (1) as follows:

where $h_i^{+}$ and $h_i^{-}$ are $i$-th frame’s forward hidden state and backward hidden state, respectively. Note here previous frame $x_{t-1}$, current frame $x_t$ and forward hidden state are concatenated as a forward group, then future frame $x_{t+1}$, current frame $x_t$ and backward hidden state compose the backward group. Deep features for each group are separately extracted and concatenated to aggregate multi-frame information to reconstruct the HR frame as:

where $f_1$ and $f_2$ are the NN extractors that learn to obtain forward and backward features, respectively. Then $f_3$ is the aggregation function that merges all information to get final upscaled frames. Meanwhile, the extracted features of forward and backward groups will update the corresponding forward and backward hidden states by using two simple convolution layers. Then these hidden states can be used for the next frames.

In training, we use the robust $L_1$ Charbonnier loss [lai2017deep] to achieve high-quality video reconstruction, which can be formulated as follows:

where $N$ is the number of training samples, and $ϵ$ is fixed to $1\times {10}^{-6}$.

There are totally of 14 convolution layers as shown in Figure 2. To save the inference time on the mobile device, all layers of our network only consist of a single $3\times 3$ convolution layer with the ReLU activation function. The number of channels for all convolution layers is set to 16. And as described in Section 3.1, three additional concatenation layers are used to combine information of frames and hidden states. Moreover, we take the bilinear upsampled current frame as a low frequency residual connection to improve the restoration accuracy.

Our model is trained on the high-quality (720p) REDS [nah2019ntire] dataset, which is proposed in the NTIRE 2019 Competition [nah2019ntire] and is widely used in recent VSR researches. In the Mobile AI & AIM 2022 Real-Time Video Super-Resolution Challenge [ignatov2022vsr], the REDS dataset contains 240 video clips for training, 30 video clips for validation and 30 video clips for testing (each clip has 100 consecutive frames). All low-resolution videos are produced by bicubic downsampling with a scale factor of 4.

For training, the batch size is 16 and all training LR images are randomly cropped to $64\times 64$ patches. The total training iterations are set to 250000, and the number of video frames in the training phase is set to 10 and changes to 100 in the testing phase. We use Adam [kingma2014adam] optimizer with an initial learning rate of $1\times {10}^{-3}$, and decrease it by half every 50000 iterations. Our network is implemented using TensorFlow2.6 and Keras framework with a single Titan Xp GPU.

To evaluate the proposed method on Video Super-Resolution, we compare SWRN with FSRCNN [kim2016accurate] and mobile RRN [isobe2020revisiting]. The former is a pioneer lightweight CNN-based network and consists of 7 convolution layers (two 1$\times$1 layers for feature shrinking and expanding) and a transpose convolution layer for upscaling. The latter is a lite version of Revisiting Temporal Modeling (RRN) which is a recurrent network for video super-resolution to run on mobile. Both our SWRN and FSRCNN are quantized with INT8 mode. We use the Peak Signal-to-Noise Ratio (PSNR) as the evaluation metric, and we will also consider the number of parameters and runtime on different mobile accelerators.

The inference times and quantitative results are reported in Table 1. One can see that although the proposed SWRN has more parameters and performs slower on the CPU device, SWRN is more compatible with the mobile TFLite GPU delegate accelerator and has lower latency than FSRCNN. In addition, our method achieves a higher performance in terms of PSNR, which surpass FSRCNN by 1.2dB and surpass mobile RNN by 0.4dB. The qualitative results are illustrated in Figure 3 and Figure 4. One can seen that video sequences produced by our method are much sharper and clear, and are visually pleasant. The result demonstrates our method satisfies the realistic runtime requirement while preserving a high PSNR performance.

In this section, We conduct an ablation study to analyze the impact of the main components of the proposed SWRN framework: sliding-window strategy and (forward and backward) hidden states. For a fair comparison, we use a plain network as our baseline method that only receives the current frame and output a super-resolved image. Moreover, the baseline model uses 3$\times$3 convolution layers with the ReLU activation function, and the number of channels is the same as SWRN. As shown in Table 2, although the baseline model has fewer parameters and requires lower runtime, it can only achieve 27.05dB in terms of PSNR. By adding the sliding window strategy and hidden states, our method increases the PSNR by nearly 0.9dB, which further demonstrates the superiority of the proposed method. In addition, we also provide the analysis of the impact of the number of channels in Table 3. As we see, although the setting of 8 channels and 32 channels can achieve faster inference time and PSNR performance respectively, the 16-channel setting is more balance in runtime and fidelity metrics.