本文研究了动画视频的现实世界视频超分辨率（VSR）的问题，并揭示了实用动画VSR的三个关键改进。首先，最近的现实世界超分辨率方法通常依赖于使用基本运算符的降解模拟，而没有任何学习能力，例如模糊，噪声和压缩。在这项工作中，我们建议从真正的低质量动画视频中学习此类基本操作员，并将学习的操作员纳入降级生成管道中。这样的基于神经网络的基本操作员可以帮助更好地捕获实际降解的分布。其次，大规模的高质量动画视频数据集AVC构建，以促进动画VSR的全面培训和评估。第三，我们进一步研究了有效的多尺度网络结构。它利用单向复发网络的效率以及基于滑动窗口的方法的有效性。多亏了上述精致的设计，我们的方法Animesr能够有效，有效地恢复现实世界中的低质量动画视频，从而实现优于以前的最先进方法。

Video Super-Resolution (VSR) aims to restore high-resolution (HR) videos from low-resolution (LR) videos. Existing VSR techniques usually recover HR frames by extracting pertinent textures from nearby frames with known degradation processes. Despite significant progress, grand challenges are remained to effectively extract and transmit high-quality textures from high-degraded low-quality sequences, such as blur, additive noises, and compression artifacts. In this work, a novel Frequency-Transformer (FTVSR) is proposed for handling low-quality videos that carry out self-attention in a combined space-time-frequency domain. First, video frames are split into patches and each patch is transformed into spectral maps in which each channel represents a frequency band. It permits a fine-grained self-attention on each frequency band, so that real visual texture can be distinguished from artifacts. Second, a novel dual frequency attention (DFA) mechanism is proposed to capture the global frequency relations and local frequency relations, which can handle different complicated degradation processes in real-world scenarios. Third, we explore different self-attention schemes for video processing in the frequency domain and discover that a ``divided attention'' which conducts a joint space-frequency attention before applying temporal-frequency attention, leads to the best video enhancement quality. Extensive experiments on three widely-used VSR datasets show that FTVSR outperforms state-of-the-art methods on different low-quality videos with clear visual margins. Code and pre-trained models are available at https://github.com/researchmm/FTVSR.

在本文中，我们研究了实用的时空视频超分辨率（STVSR）问题，该问题旨在从低型低分辨率的低分辨率模糊视频中生成高富含高分辨率的夏普视频。当使用低填充和低分辨率摄像头记录快速动态事件时，通常会发生这种问题，而被捕获的视频将遭受三个典型问题：i）运动模糊发生是由于曝光时间内的对象/摄像机运动而发生的； ii）当事件时间频率超过时间采样的奈奎斯特极限时，运动异叠是不可避免的； iii）由于空间采样率低，因此丢失了高频细节。这些问题可以通过三个单独的子任务的级联来缓解，包括视频脱张，框架插值和超分辨率，但是，这些问题将无法捕获视频序列之间的空间和时间相关性。为了解决这个问题，我们通过利用基于模型的方法和基于学习的方法来提出一个可解释的STVSR框架。具体而言，我们将STVSR作为联合视频脱张，框架插值和超分辨率问题，并以另一种方式将其作为两个子问题解决。对于第一个子问题，我们得出了可解释的分析解决方案，并将其用作傅立叶数据变换层。然后，我们为第二个子问题提出了一个反复的视频增强层，以进一步恢复高频细节。广泛的实验证明了我们方法在定量指标和视觉质量方面的优势。

The recurrent structure is a prevalent framework for the task of video super-resolution, which models the temporal dependency between frames via hidden states. When applied to real-world scenarios with unknown and complex degradations, hidden states tend to contain unpleasant artifacts and propagate them to restored frames. In this circumstance, our analyses show that such artifacts can be largely alleviated when the hidden state is replaced with a cleaner counterpart. Based on the observations, we propose a Hidden State Attention (HSA) module to mitigate artifacts in real-world video super-resolution. Specifically, we first adopt various cheap filters to produce a hidden state pool. For example, Gaussian blur filters are for smoothing artifacts while sharpening filters are for enhancing details. To aggregate a new hidden state that contains fewer artifacts from the hidden state pool, we devise a Selective Cross Attention (SCA) module, in which the attention between input features and each hidden state is calculated. Equipped with HSA, our proposed method, namely FastRealVSR, is able to achieve 2x speedup while obtaining better performance than Real-BasicVSR. Codes will be available at https://github.com/TencentARC/FastRealVSR

Video super-resolution (VSR) aiming to reconstruct a high-resolution (HR) video from its low-resolution (LR) counterpart has made tremendous progress in recent years. However, it remains challenging to deploy existing VSR methods to real-world data with complex degradations. On the one hand, there are few well-aligned real-world VSR datasets, especially with large super-resolution scale factors, which limits the development of real-world VSR tasks. On the other hand, alignment algorithms in existing VSR methods perform poorly for real-world videos, leading to unsatisfactory results. As an attempt to address the aforementioned issues, we build a real-world 4 VSR dataset, namely MVSR4$\times$, where low- and high-resolution videos are captured with different focal length lenses of a smartphone, respectively. Moreover, we propose an effective alignment method for real-world VSR, namely EAVSR. EAVSR takes the proposed multi-layer adaptive spatial transform network (MultiAdaSTN) to refine the offsets provided by the pre-trained optical flow estimation network. Experimental results on RealVSR and MVSR4$\times$ datasets show the effectiveness and practicality of our method, and we achieve state-of-the-art performance in real-world VSR task. The dataset and code will be publicly available.

现有的视频denoising方法通常假设嘈杂的视频通过添加高斯噪声从干净的视频中降低。但是，经过这种降解假设训练的深层模型将不可避免地导致由于退化不匹配而导致的真实视频的性能差。尽管一些研究试图在摄像机捕获的嘈杂和无噪声视频对上训练深层模型，但此类模型只能对特定的相机很好地工作，并且对其他视频的推广不佳。在本文中，我们建议提高此限制，并专注于一般真实视频的问题，目的是在看不见的现实世界视频上概括。我们首先调查视频噪音的共同行为来解决这个问题，并观察两个重要特征：1）缩减有助于降低空间空间中的噪声水平； 2）来自相邻框架的信息有助于消除时间上的当前框架的噪声空间。在这两个观察结果的推动下，我们通过充分利用上述两个特征提出了多尺度的复发架构。其次，我们通过随机调整不同的噪声类型来训练Denoising模型来提出合成真实的噪声降解模型。借助合成和丰富的降解空间，我们的退化模型可以帮助弥合训练数据和现实世界数据之间的分布差距。广泛的实验表明，与现有方法相比，我们所提出的方法实现了最先进的性能和更好的概括能力，而在合成高斯denoising和实用的真实视频denoisising方面都具有现有方法。

Video restoration tasks, including super-resolution, deblurring, etc, are drawing increasing attention in the computer vision community. A challenging benchmark named REDS is released in the NTIRE19 Challenge. This new benchmark challenges existing methods from two aspects:(1) how to align multiple frames given large motions, and (2) how to effectively fuse different frames with diverse motion and blur. In this work, we propose a novel Video Restoration framework with Enhanced Deformable convolutions, termed EDVR, to address these challenges. First, to handle large motions, we devise a Pyramid, Cascading and Deformable (PCD) alignment module, in which frame alignment is done at the feature level using deformable convolutions in a coarse-to-fine manner. Second, we propose a Temporal and Spatial Attention (TSA) fusion module, in which attention is applied both temporally and spatially, so as to emphasize important features for subsequent restoration. Thanks to these modules, our EDVR wins the champions and outperforms the second place by a large margin in all four tracks in the NTIRE19 video restoration and enhancement challenges. EDVR also demonstrates superior performance to state-of-the-art published methods on video super-resolution and deblurring. The code is available at https://github.com/xinntao/EDVR.

现实世界图像超分辨率（SR）的关键挑战是在低分辨率（LR）图像中恢复具有复杂未知降解（例如，下采样，噪声和压缩）的缺失细节。大多数以前的作品还原图像空间中的此类缺失细节。为了应对自然图像的高度多样性，他们要么依靠难以训练和容易训练和伪影的不稳定的甘体，要么诉诸于通常不可用的高分辨率（HR）图像中的明确参考。在这项工作中，我们提出了匹配SR（FEMASR）的功能，该功能在更紧凑的特征空间中恢复了现实的HR图像。与图像空间方法不同，我们的FEMASR通过将扭曲的LR图像{\ IT特征}与我们预读的HR先验中的无失真性HR对应物匹配来恢复HR图像，并解码匹配的功能以获得现实的HR图像。具体而言，我们的人力资源先验包含一个离散的特征代码簿及其相关的解码器，它们在使用量化的生成对抗网络（VQGAN）的HR图像上预估计。值得注意的是，我们在VQGAN中结合了一种新型的语义正则化，以提高重建图像的质量。对于功能匹配，我们首先提取由LR编码器组成的LR编码器的LR功能，然后遵循简单的最近邻居策略，将其与预读的代码簿匹配。特别是，我们为LR编码器配备了与解码器的残留快捷方式连接，这对于优化功能匹配损耗至关重要，还有助于补充可能的功能匹配错误。实验结果表明，我们的方法比以前的方法产生更现实的HR图像。代码以\ url {https://github.com/chaofengc/femasr}发布。

Convolutional Neural Network (CNN)-based image super-resolution (SR) has exhibited impressive success on known degraded low-resolution (LR) images. However, this type of approach is hard to hold its performance in practical scenarios when the degradation process is unknown. Despite existing blind SR methods proposed to solve this problem using blur kernel estimation, the perceptual quality and reconstruction accuracy are still unsatisfactory. In this paper, we analyze the degradation of a high-resolution (HR) image from image intrinsic components according to a degradation-based formulation model. We propose a components decomposition and co-optimization network (CDCN) for blind SR. Firstly, CDCN decomposes the input LR image into structure and detail components in feature space. Then, the mutual collaboration block (MCB) is presented to exploit the relationship between both two components. In this way, the detail component can provide informative features to enrich the structural context and the structure component can carry structural context for better detail revealing via a mutual complementary manner. After that, we present a degradation-driven learning strategy to jointly supervise the HR image detail and structure restoration process. Finally, a multi-scale fusion module followed by an upsampling layer is designed to fuse the structure and detail features and perform SR reconstruction. Empowered by such degradation-based components decomposition, collaboration, and mutual optimization, we can bridge the correlation between component learning and degradation modelling for blind SR, thereby producing SR results with more accurate textures. Extensive experiments on both synthetic SR datasets and real-world images show that the proposed method achieves the state-of-the-art performance compared to existing methods.

现实世界视频超分辨率（VSR）降解的多样性和复杂性构成了推理和培训中的非琐碎挑战。首先，虽然长期繁殖导致在轻度降解的情况下提高性能，但通过传播可以夸大严重的野外降解，损害输出质量。为了平衡细节合成和工件抑制之间的权衡，我们发现在传播之前减少噪声和伪像的图像预清洁阶段。我们的RealBasicVSR配备了精心设计的清洁模块，以质量和效率为现有的现有方法。其次，现实世界的VSR模型通常受到多样化的培训，以提高普遍性，需要增加批量尺寸以产生稳定的梯度。不可避免地，增加的计算负担导致各种问题，包括1）速度 - 性能权衡和2）批量长期权衡。为了缓解第一个权衡，我们提出了一种随机退化计划，可在不牺牲性能的情况下减少高达40％的培训时间。然后，我们分析了不同的训练环境，并建议在训练期间使用更长的序列而不是较大的批次，允许更有效的时间信息使用，导致推论期间的性能更稳定。为了便于公平的比较，我们提出了新的Videolq数据集，其中包含含有丰富纹理和模式的大量现实较低的视频序列。我们的数据集可以作为基准测试的共同点。代码，模型和数据集将公开可用。

由于大气湍流的扭曲而恢复图像是一个长期存在的问题，这是由于变形的空间变化，图像形成过程的非线性以及训练和测试数据的稀缺性。现有方法通常在失真模型上具有强大的统计假设，在许多情况下，由于没有概括，因此在现实世界中的性能有限。为了克服挑战，本文提出了一种端到端物理驱动的方法，该方法有效，可以推广到现实世界的湍流。在数据合成方面，我们通过通过宽sense式的平稳性近似随机场来显着增加SOTA湍流模拟器可以处理的图像分辨率。新的数据合成过程使大规模的多级湍流和训练的地面真相对产生。在网络设计方面，我们提出了湍流缓解变压器（TMT），这是一个两级U-NET形状的多帧恢复网络，该网络具有Noval有效的自发机制，称为暂时通道关节关注（TCJA）。我们还引入了一种新的培训方案，该方案由新的模拟器启用，并设计新的变压器单元以减少内存消耗。在静态场景和动态场景上的实验结果是有希望的，包括各种真实的湍流场景。

Real-world image super-resolution (RISR) has received increased focus for improving the quality of SR images under unknown complex degradation. Existing methods rely on the heavy SR models to enhance low-resolution (LR) images of different degradation levels, which significantly restricts their practical deployments on resource-limited devices. In this paper, we propose a novel Dynamic Channel Splitting scheme for efficient Real-world Image Super-Resolution, termed DCS-RISR. Specifically, we first introduce the light degradation prediction network to regress the degradation vector to simulate the real-world degradations, upon which the channel splitting vector is generated as the input for an efficient SR model. Then, a learnable octave convolution block is proposed to adaptively decide the channel splitting scale for low- and high-frequency features at each block, reducing computation overhead and memory cost by offering the large scale to low-frequency features and the small scale to the high ones. To further improve the RISR performance, Non-local regularization is employed to supplement the knowledge of patches from LR and HR subspace with free-computation inference. Extensive experiments demonstrate the effectiveness of DCS-RISR on different benchmark datasets. Our DCS-RISR not only achieves the best trade-off between computation/parameter and PSNR/SSIM metric, and also effectively handles real-world images with different degradation levels.

尽管目前基于深度学习的方法在盲目的单图像超分辨率（SISR）任务中已获得了有希望的表现，但其中大多数主要集中在启发式上构建多样化的网络体系结构，并更少强调对Blur之间的物理发电机制的明确嵌入内核和高分辨率（HR）图像。为了减轻这个问题，我们提出了一个模型驱动的深神经网络，称为blind SISR。具体而言，为了解决经典的SISR模型，我们提出了一种简单的效果迭代算法。然后，通过将所涉及的迭代步骤展开到相应的网络模块中，我们自然构建了KXNET。所提出的KXNET的主要特异性是整个学习过程与此SISR任务的固有物理机制完全合理地集成在一起。因此，学习的模糊内核具有清晰的物理模式，并且模糊内核和HR图像之间的相互迭代过程可以很好地指导KXNET沿正确的方向发展。关于合成和真实数据的广泛实验很好地证明了我们方法的卓越准确性和一般性超出了当前代表性的最先进的盲目SISR方法。代码可在：\ url {https://github.com/jiahong-fu/kxnet}中获得。

Existing convolutional neural networks (CNN) based image super-resolution (SR) methods have achieved impressive performance on bicubic kernel, which is not valid to handle unknown degradations in real-world applications. Recent blind SR methods suggest to reconstruct SR images relying on blur kernel estimation. However, their results still remain visible artifacts and detail distortion due to the estimation errors. To alleviate these problems, in this paper, we propose an effective and kernel-free network, namely DSSR, which enables recurrent detail-structure alternative optimization without blur kernel prior incorporation for blind SR. Specifically, in our DSSR, a detail-structure modulation module (DSMM) is built to exploit the interaction and collaboration of image details and structures. The DSMM consists of two components: a detail restoration unit (DRU) and a structure modulation unit (SMU). The former aims at regressing the intermediate HR detail reconstruction from LR structural contexts, and the latter performs structural contexts modulation conditioned on the learned detail maps at both HR and LR spaces. Besides, we use the output of DSMM as the hidden state and design our DSSR architecture from a recurrent convolutional neural network (RCNN) view. In this way, the network can alternatively optimize the image details and structural contexts, achieving co-optimization across time. Moreover, equipped with the recurrent connection, our DSSR allows low- and high-level feature representations complementary by observing previous HR details and contexts at every unrolling time. Extensive experiments on synthetic datasets and real-world images demonstrate that our method achieves the state-of-the-art against existing methods. The source code can be found at https://github.com/Arcananana/DSSR.

压缩视频超分辨率（VSR）旨在从压缩的低分辨率对应物中恢复高分辨率帧。最近的VSR方法通常通过借用相邻视频帧的相关纹理来增强输入框架。尽管已经取得了一些进展，但是从压缩视频中有效提取和转移高质量纹理的巨大挑战，这些视频通常会高度退化。在本文中，我们提出了一种用于压缩视频超分辨率（FTVSR）的新型频率转换器，该频率在联合时空频域中进行自我注意。首先，我们将视频框架分为斑块，然后将每个贴片转换为DCT光谱图，每个通道代表频带。这样的设计使每个频带都可以进行细粒度的自我注意力，因此可以将真实的视觉纹理与伪影区分开，并进一步用于视频框架修复。其次，我们研究了不同的自我发场方案，并发现在对每个频带上应用暂时关注之前，会引起关节空间的注意力，从而带来最佳的视频增强质量。两个广泛使用的视频超分辨率基准的实验结果表明，FTVSR在未压缩和压缩视频的最先进的方法中都具有清晰的视觉边距。代码可在https://github.com/researchmm/ftvsr上找到。

近年来，压缩图像超分辨率已引起了极大的关注，其中图像被压缩伪像和低分辨率伪影降解。由于复杂的杂化扭曲变形，因此很难通过简单的超分辨率和压缩伪像消除掉的简单合作来恢复扭曲的图像。在本文中，我们向前迈出了一步，提出了层次的SWIN变压器（HST）网络，以恢复低分辨率压缩图像，该图像共同捕获分层特征表示并分别用SWIN Transformer增强每个尺度表示。此外，我们发现具有超分辨率（SR）任务的预处理对于压缩图像超分辨率至关重要。为了探索不同的SR预审查的影响，我们将常用的SR任务（例如，比科比奇和不同的实际超分辨率仿真）作为我们的预处理任务，并揭示了SR在压缩的图像超分辨率中起不可替代的作用。随着HST和预训练的合作，我们的HST在AIM 2022挑战中获得了低质量压缩图像超分辨率轨道的第五名，PSNR为23.51db。广泛的实验和消融研究已经验证了我们提出的方法的有效性。

交互式图像恢复旨在通过调整几个控制系数来恢复图像，从而确定恢复强度。现有方法在学习已知降解类型和级别的监督下学习可控功能受到限制。当真正的降解与假设不同时，它们通常会遭受严重的性能下降。这样的限制是由于现实世界下降的复杂性，无法在培训期间对交互式调制提供明确的监督。但是，尚未研究如何实现现实世界中超级分辨率中的交互式调制。在这项工作中，我们提出了基于公制的实现现实世界超级分辨率（MM-REALSR）的交互式调制。具体而言，我们提出了一种无监督的退化估计策略，以估计现实情况下的降解水平。我们提出了一种度量学习策略，而不是将已知的降解水平作为对互动机制的明确监督，而是提出了一种度量策略，以将现实世界情景中的不可量化的降解水平映射到公制空间，该度量空间以不受监督的方式进行培训。此外，我们在度量学习过程中引入了锚点策略，以使度量空间的分布正常化。广泛的实验表明，所提出的MM-REALSR在现实世界中的超级分辨率中实现了出色的调制和恢复性能。代码可在https://github.com/tencentarc/mm-realsr上找到。

时空视频超分辨率（STVSR）旨在从相应的低帧速率，低分辨率视频序列构建高空时间分辨率视频序列。灵感来自最近的成功，考虑空间时间超级分辨率的空间信息，我们在这项工作中的主要目标是在快速动态事件的视频序列中充分考虑空间和时间相关性。为此，我们提出了一种新颖的单级内存增强图注意网络（Megan），用于时空视频超分辨率。具体地，我们构建新颖的远程存储图聚合（LMGA）模块，以沿着特征映射的信道尺寸动态捕获相关性，并自适应地聚合信道特征以增强特征表示。我们介绍了一个非本地剩余块，其使每个通道明智的功能能够参加全局空间分层特征。此外，我们采用渐进式融合模块通过广泛利用来自多个帧的空间 - 时间相关性来进一步提高表示能力。实验结果表明，我们的方法与定量和视觉上的最先进的方法相比，实现了更好的结果。

Deep Convolutional Neural Networks (DCNNs) have exhibited impressive performance on image super-resolution tasks. However, these deep learning-based super-resolution methods perform poorly in real-world super-resolution tasks, where the paired high-resolution and low-resolution images are unavailable and the low-resolution images are degraded by complicated and unknown kernels. To break these limitations, we propose the Unsupervised Bi-directional Cycle Domain Transfer Learning-based Generative Adversarial Network (UBCDTL-GAN), which consists of an Unsupervised Bi-directional Cycle Domain Transfer Network (UBCDTN) and the Semantic Encoder guided Super Resolution Network (SESRN). First, the UBCDTN is able to produce an approximated real-like LR image through transferring the LR image from an artificially degraded domain to the real-world LR image domain. Second, the SESRN has the ability to super-resolve the approximated real-like LR image to a photo-realistic HR image. Extensive experiments on unpaired real-world image benchmark datasets demonstrate that the proposed method achieves superior performance compared to state-of-the-art methods.

突发超级分辨率（SR）提供了从低质量图像恢复丰富细节的可能性。然而，由于实际应用中的低分辨率（LR）图像具有多种复杂和未知的降级，所以现有的非盲（例如，双臂）设计的网络通常导致恢复高分辨率（HR）图像的严重性能下降。此外，处理多重未对准的嘈杂的原始输入也是具有挑战性的。在本文中，我们解决了从现代手持设备获取的原始突发序列重建HR图像的问题。中央观点是一个内核引导策略，可以用两个步骤解决突发SR：内核建模和HR恢复。前者估计来自原始输入的突发内核，而后者基于估计的内核预测超分辨图像。此外，我们引入了内核感知可变形对准模块，其可以通过考虑模糊的前沿而有效地对准原始图像。对综合和现实世界数据集的广泛实验表明，所提出的方法可以在爆发SR问题中对最先进的性能进行。