Reference-based Super-Resolution (Ref-SR) has recently emerged as a promising paradigm to enhance a low-resolution (LR) input image or video by introducing an additional high-resolution (HR) reference image. Existing Ref-SR methods mostly rely on implicit correspondence matching to borrow HR textures from reference images to compensate for the information loss in input images. However, performing local transfer is difficult because of two gaps between input and reference images: the transformation gap (e.g., scale and rotation) and the resolution gap (e.g., HR and LR). To tackle these challenges, we propose C2-Matching in this work, which performs explicit robust matching crossing transformation and resolution. 1) To bridge the transformation gap, we propose a contrastive correspondence network, which learns transformation-robust correspondences using augmented views of the input image. 2) To address the resolution gap, we adopt teacher-student correlation distillation, which distills knowledge from the easier HR-HR matching to guide the more ambiguous LR-HR matching. 3) Finally, we design a dynamic aggregation module to address the potential misalignment issue between input images and reference images. In addition, to faithfully evaluate the performance of Reference-based Image Super-Resolution under a realistic setting, we contribute the Webly-Referenced SR (WR-SR) dataset, mimicking the practical usage scenario. We also extend C2-Matching to Reference-based Video Super-Resolution task, where an image taken in a similar scene serves as the HR reference image. Extensive experiments demonstrate that our proposed C2-Matching significantly outperforms state of the arts on the standard CUFED5 benchmark and also boosts the performance of video SR by incorporating the C2-Matching component into Video SR pipelines.
translated by 谷歌翻译
基于参考的图像超分辨率(REFSR)旨在利用辅助参考(REF)图像为超溶解的低分辨率(LR)图像。最近,RefSR引起了极大的关注,因为它提供了超越单图SR的替代方法。但是,解决REFSR问题有两个关键的挑战:(i)当它们显着不同时,很难匹配LR和Ref图像之间的对应关系; (ii)如何将相关纹理从参考图像转移以补偿LR图像的细节非常具有挑战性。为了解决RefSR的这些问题,本文提出了一个可变形的注意变压器,即DATSR,具有多个尺度,每个尺度由纹理特征编码器(TFE)模块组成,基于参考的可变形注意(RDA)模块和残差功能聚合(RFA)模块。具体而言,TFE首先提取图像转换(例如,亮度)不敏感的LR和REF图像,RDA可以利用多个相关纹理来补偿更多的LR功能信息,而RFA最终汇总了LR功能和相关纹理,以获得更愉快的宜人的质地结果。广泛的实验表明,我们的DATSR在定量和质量上实现了基准数据集上的最新性能。
translated by 谷歌翻译
Reference-based image super-resolution (RefSR) is a promising SR branch and has shown great potential in overcoming the limitations of single image super-resolution. While previous state-of-the-art RefSR methods mainly focus on improving the efficacy and robustness of reference feature transfer, it is generally overlooked that a well reconstructed SR image should enable better SR reconstruction for its similar LR images when it is referred to as. Therefore, in this work, we propose a reciprocal learning framework that can appropriately leverage such a fact to reinforce the learning of a RefSR network. Besides, we deliberately design a progressive feature alignment and selection module for further improving the RefSR task. The newly proposed module aligns reference-input images at multi-scale feature spaces and performs reference-aware feature selection in a progressive manner, thus more precise reference features can be transferred into the input features and the network capability is enhanced. Our reciprocal learning paradigm is model-agnostic and it can be applied to arbitrary RefSR models. We empirically show that multiple recent state-of-the-art RefSR models can be consistently improved with our reciprocal learning paradigm. Furthermore, our proposed model together with the reciprocal learning strategy sets new state-of-the-art performances on multiple benchmarks.
translated by 谷歌翻译
基于参考的超分辨率(REFSR)在使用外部参考(REF)图像产生现实纹理方面取得了重大进展。然而,现有的REFSR方法可以获得与输入大小一起消耗二次计算资源的高质量对应匹配,限制其应用程序。此外,这些方法通常遭受低分辨率(LR)图像和REF图像之间的比例错位。在本文中,我们提出了一种加速的多尺度聚合网络(AMSA),用于基于参考的超分辨率,包括粗略嵌入式斑块(CFE-PACKPMATCH)和多尺度动态聚合(MSDA)模块。为了提高匹配效率,我们设计一种具有随机样本传播的新型嵌入式PACKMTH方案,其涉及具有渐近线性计算成本的端到端训练到输入大小。为了进一步降低计算成本和加速会聚,我们在构成CFE-PACKMATCH的嵌入式PACKMACTH上应用了粗略策略。为了完全利用跨多个尺度的参考信息并增强稳定性的稳定性,我们开发由动态聚合和多尺度聚合组成的MSDA模块。动态聚合通过动态聚合特征来纠正轻微比例的错位,并且多尺度聚合通过融合多尺度信息来为大规模错位带来鲁棒性。实验结果表明,该拟议的AMSA对定量和定性评估的最先进方法实现了卓越的性能。
translated by 谷歌翻译
We study on image super-resolution (SR), which aims to recover realistic textures from a low-resolution (LR) image. Recent progress has been made by taking high-resolution images as references (Ref), so that relevant textures can be transferred to LR images. However, existing SR approaches neglect to use attention mechanisms to transfer high-resolution (HR) textures from Ref images, which limits these approaches in challenging cases. In this paper, we propose a novel Texture Transformer Network for Image Super-Resolution (TTSR), in which the LR and Ref images are formulated as queries and keys in a transformer, respectively. TTSR consists of four closely-related modules optimized for image generation tasks, including a learnable texture extractor by DNN, a relevance embedding module, a hard-attention module for texture transfer, and a softattention module for texture synthesis. Such a design encourages joint feature learning across LR and Ref images, in which deep feature correspondences can be discovered by attention, and thus accurate texture features can be transferred. The proposed texture transformer can be further stacked in a cross-scale way, which enables texture recovery from different levels (e.g., from 1× to 4× magnification). Extensive experiments show that TTSR achieves significant improvements over state-of-the-art approaches on both quantitative and qualitative evaluations. The source code can be downloaded at https://github.com/ researchmm/TTSR.
translated by 谷歌翻译
在本文中,我们考虑了基于参考的超分辨率(REFSR)中的两个具有挑战性的问题,(i)如何选择适当的参考图像,以及(ii)如何以一种自我监督的方式学习真实世界RefSR。特别是,我们从双摄像头Zooms(SelfDZSR)观察到现实世界图像SR的新颖的自我监督学习方法。考虑到多台相机在现代智能手机中的普及,可以自然利用越来越多的缩放(远摄)图像作为指导较小的变焦(短对焦)图像的SR。此外,SelfDZSR学习了一个深层网络,以获得短对焦图像的SR结果,以具有与远摄图像相同的分辨率。为此,我们将远摄图像而不是其他高分辨率图像作为监督信息,然后从中选择中心贴片作为对相应的短对焦图像补丁的引用。为了减轻短对焦低分辨率(LR)图像和远摄地面真相(GT)图像之间未对准的影响,我们设计了辅助LR发电机,并将GT映射到辅助LR,同时保持空间位置不变。 。然后,可以利用辅助-LR通过建议的自适应空间变压器网络(ADASTN)将LR特征变形,并将REF特征与GT匹配。在测试过程中,可以直接部署SelfDZSR,以使用远摄映像的引用来超级解决整个短对焦图像。实验表明,我们的方法可以针对最先进的方法实现更好的定量和定性性能。代码可在https://github.com/cszhilu1998/selfdzsr上找到。
translated by 谷歌翻译
最新的多视图多媒体应用程序在高分辨率(HR)视觉体验与存储或带宽约束之间挣扎。因此,本文提出了一个多视图图像超分辨率(MVISR)任务。它旨在增加从同一场景捕获的多视图图像的分辨率。一种解决方案是将图像或视频超分辨率(SR)方法应用于低分辨率(LR)输入视图结果。但是,这些方法无法处理视图之间的大角度转换,并利用所有多视图图像中的信息。为了解决这些问题,我们提出了MVSRNET,该MVSRNET使用几何信息从所有LR多视图中提取尖锐的细节,以支持LR输入视图的SR。具体而言,MVSRNET中提出的几何感知参考合成模块使用几何信息和所有多视图LR图像来合成像素对齐的HR参考图像。然后,提出的动态高频搜索网络完全利用了SR参考图像中的高频纹理细节。关于几个基准测试的广泛实验表明,我们的方法在最新方法上有了显着改善。
translated by 谷歌翻译
Video restoration tasks, including super-resolution, deblurring, etc, are drawing increasing attention in the computer vision community. A challenging benchmark named REDS is released in the NTIRE19 Challenge. This new benchmark challenges existing methods from two aspects:(1) how to align multiple frames given large motions, and (2) how to effectively fuse different frames with diverse motion and blur. In this work, we propose a novel Video Restoration framework with Enhanced Deformable convolutions, termed EDVR, to address these challenges. First, to handle large motions, we devise a Pyramid, Cascading and Deformable (PCD) alignment module, in which frame alignment is done at the feature level using deformable convolutions in a coarse-to-fine manner. Second, we propose a Temporal and Spatial Attention (TSA) fusion module, in which attention is applied both temporally and spatially, so as to emphasize important features for subsequent restoration. Thanks to these modules, our EDVR wins the champions and outperforms the second place by a large margin in all four tracks in the NTIRE19 video restoration and enhancement challenges. EDVR also demonstrates superior performance to state-of-the-art published methods on video super-resolution and deblurring. The code is available at https://github.com/xinntao/EDVR.
translated by 谷歌翻译
Video super-resolution (VSR) aiming to reconstruct a high-resolution (HR) video from its low-resolution (LR) counterpart has made tremendous progress in recent years. However, it remains challenging to deploy existing VSR methods to real-world data with complex degradations. On the one hand, there are few well-aligned real-world VSR datasets, especially with large super-resolution scale factors, which limits the development of real-world VSR tasks. On the other hand, alignment algorithms in existing VSR methods perform poorly for real-world videos, leading to unsatisfactory results. As an attempt to address the aforementioned issues, we build a real-world 4 VSR dataset, namely MVSR4$\times$, where low- and high-resolution videos are captured with different focal length lenses of a smartphone, respectively. Moreover, we propose an effective alignment method for real-world VSR, namely EAVSR. EAVSR takes the proposed multi-layer adaptive spatial transform network (MultiAdaSTN) to refine the offsets provided by the pre-trained optical flow estimation network. Experimental results on RealVSR and MVSR4$\times$ datasets show the effectiveness and practicality of our method, and we achieve state-of-the-art performance in real-world VSR task. The dataset and code will be publicly available.
translated by 谷歌翻译
时空视频超分辨率(STVSR)的目标是增加低分辨率(LR)和低帧速率(LFR)视频的空间分辨率。基于深度学习的最新方法已取得了重大改进,但是其中大多数仅使用两个相邻帧,即短期功能,可以合成缺失的框架嵌入,这无法完全探索连续输入LR帧的信息流。此外,现有的STVSR模型几乎无法明确利用时间上下文以帮助高分辨率(HR)框架重建。为了解决这些问题,在本文中,我们提出了一个称为STDAN的可变形注意网络。首先,我们设计了一个长短的术语特征插值(LSTFI)模块,该模块能够通过双向RNN结构从更相邻的输入帧中挖掘大量的内容,以进行插值。其次,我们提出了一个空间 - 周期性变形特征聚合(STDFA)模块,其中动态视频框架中的空间和时间上下文被自适应地捕获并汇总以增强SR重建。几个数据集的实验结果表明,我们的方法的表现优于最先进的STVSR方法。该代码可在https://github.com/littlewhitesea/stdan上找到。
translated by 谷歌翻译
Because of the necessity to obtain high-quality images with minimal radiation doses, such as in low-field magnetic resonance imaging, super-resolution reconstruction in medical imaging has become more popular (MRI). However, due to the complexity and high aesthetic requirements of medical imaging, image super-resolution reconstruction remains a difficult challenge. In this paper, we offer a deep learning-based strategy for reconstructing medical images from low resolutions utilizing Transformer and Generative Adversarial Networks (T-GAN). The integrated system can extract more precise texture information and focus more on important locations through global image matching after successfully inserting Transformer into the generative adversarial network for picture reconstruction. Furthermore, we weighted the combination of content loss, adversarial loss, and adversarial feature loss as the final multi-task loss function during the training of our proposed model T-GAN. In comparison to established measures like PSNR and SSIM, our suggested T-GAN achieves optimal performance and recovers more texture features in super-resolution reconstruction of MRI scanned images of the knees and belly.
translated by 谷歌翻译
近年来,由于SR数据集的开发和相应的实际SR方法,真实的图像超分辨率(SR)已取得了令人鼓舞的结果。相比之下,真实视频SR领域落后,尤其是对于真实的原始视频。考虑到原始图像SR优于SRGB图像SR,我们构建了一个真实世界的原始视频SR(Real-Rawvsr)数据集,并提出了相应的SR方法。我们利用两个DSLR摄像机和一个梁切口来同时捕获具有2倍,3倍和4倍大型的高分辨率(LR)和高分辨率(HR)原始视频。我们的数据集中有450对视频对,场景从室内到室外各不相同,包括相机和对象运动在内的动作。据我们所知,这是第一个现实世界的RAW VSR数据集。由于原始视频的特征是拜耳模式,因此我们提出了一个两分支网络,该网络既涉及包装的RGGB序列和原始的拜耳模式序列,又涉及两个分支,并且两个分支相互互补。经过提出的共对象,相互作用,融合和重建模块后,我们生成了相应的HR SRGB序列。实验结果表明,所提出的方法优于原始或SRGB输入的基准实体和合成视频SR方法。我们的代码和数据集可在https://github.com/zmzhang1998/real-rawvsr上找到。
translated by 谷歌翻译
近年来,基于深度学习的模型在视频超分辨率(VSR)方面取得了显着性能,但是这些模型中的大多数不适用于在线视频应用程序。这些方法仅考虑失真质量,而忽略了在线应用程序的关键要求,例如低延迟和模型较低的复杂性。在本文中,我们专注于在线视频传输,其中需要VSR算法来实时生成高分辨率的视频序列。为了应对此类挑战,我们提出了一种基于一种新的内核知识转移方法,称为卷积核旁路移植物(CKBG)。首先,我们设计了一个轻巧的网络结构,该结构不需要将来的帧作为输入,并节省了缓存这些帧的额外时间成本。然后,我们提出的CKBG方法通过用``核移植物)''绕过原始网络来增强这种轻巧的基础模型,这些网络是包含外部预验证图像SR模型的先验知识的额外卷积内核。在测试阶段,我们通过将其转换为简单的单路结构来进一步加速移植的多支球网络。实验结果表明,我们提出的方法可以处理高达110 fps的在线视频序列,并且模型复杂性非常低和竞争性SR性能。
translated by 谷歌翻译
远程时间对齐至关重要,但对视频恢复任务有挑战性。最近,一些作品试图将远程对齐分成几个子对齐并逐步处理它们。虽然该操作有助于建模遥控对应关系,但由于传播机制,误差累积是不可避免的。在这项工作中,我们提出了一种新颖的通用迭代对准模块,其采用逐渐改进方案进行子对准,产生更准确的运动补偿。为了进一步提高对准精度和时间一致性,我们开发了一种非参数重新加权方法,其中每个相邻帧的重要性以用于聚合的空间方式自适应地评估。凭借拟议的策略,我们的模型在一系列视频恢复任务中实现了多个基准测试的最先进的性能,包括视频超分辨率,去噪和去束性。我们的项目可用于\ url {https:/github.com/redrock303/revisiting-temporal-alignment-for-video-Restion.git}。
translated by 谷歌翻译
Reference-based Super-resolution (RefSR) approaches have recently been proposed to overcome the ill-posed problem of image super-resolution by providing additional information from a high-resolution image. Multi-reference super-resolution extends this approach by allowing more information to be incorporated. This paper proposes a 2-step-weighting posterior fusion approach to combine the outputs of RefSR models with multiple references. Extensive experiments on the CUFED5 dataset demonstrate that the proposed methods can be applied to various state-of-the-art RefSR models to get a consistent improvement in image quality.
translated by 谷歌翻译
高动态范围(HDR)成像是一种允许广泛的动态曝光范围的技术,这在图像处理,计算机图形和计算机视觉中很重要。近年来,使用深度学习(DL),HDR成像有重大进展。本研究对深层HDR成像方法的最新发展进行了综合和富有洞察力的调查和分析。在分层和结构上,将现有的深层HDR成像方法基于(1)输入曝光的数量/域,(2)学习任务数,(3)新传感器数据,(4)新的学习策略,(5)应用程序。重要的是,我们对关于其潜在和挑战的每个类别提供建设性的讨论。此外,我们审查了深度HDR成像的一些关键方面,例如数据集和评估指标。最后,我们突出了一些打开的问题,并指出了未来的研究方向。
translated by 谷歌翻译
突发超级分辨率(SR)提供了从低质量图像恢复丰富细节的可能性。然而,由于实际应用中的低分辨率(LR)图像具有多种复杂和未知的降级,所以现有的非盲(例如,双臂)设计的网络通常导致恢复高分辨率(HR)图像的严重性能下降。此外,处理多重未对准的嘈杂的原始输入也是具有挑战性的。在本文中,我们解决了从现代手持设备获取的原始突发序列重建HR图像的问题。中央观点是一个内核引导策略,可以用两个步骤解决突发SR:内核建模和HR恢复。前者估计来自原始输入的突发内核,而后者基于估计的内核预测超分辨图像。此外,我们引入了内核感知可变形对准模块,其可以通过考虑模糊的前沿而有效地对准原始图像。对综合和现实世界数据集的广泛实验表明,所提出的方法可以在爆发SR问题中对最先进的性能进行。
translated by 谷歌翻译
时空视频超分辨率(STVSR)旨在从相应的低帧速率,低分辨率视频序列构建高空时间分辨率视频序列。灵感来自最近的成功,考虑空间时间超级分辨率的空间信息,我们在这项工作中的主要目标是在快速动态事件的视频序列中充分考虑空间和时间相关性。为此,我们提出了一种新颖的单级内存增强图注意网络(Megan),用于时空视频超分辨率。具体地,我们构建新颖的远程存储图聚合(LMGA)模块,以沿着特征映射的信道尺寸动态捕获相关性,并自适应地聚合信道特征以增强特征表示。我们介绍了一个非本地剩余块,其使每个通道明智的功能能够参加全局空间分层特征。此外,我们采用渐进式融合模块通过广泛利用来自多个帧的空间 - 时间相关性来进一步提高表示能力。实验结果表明,我们的方法与定量和视觉上的最先进的方法相比,实现了更好的结果。
translated by 谷歌翻译
时空视频超分辨率(STVSR)的目标是提高帧速率(也称为时间分辨率)和给定视频的空间分辨率。最近的方法通过端到端的深神经网络解决了STVSR。一个流行的解决方案是首先提高视频的帧速率;然后在不同的框架功能之间执行特征改进;最后增加了这些功能的空间分辨率。在此过程中,仔细利用了不同帧的特征之间的时间相关性。然而,尚未强调不同(空间)分辨率的特征之间的空间相关性。在本文中,我们提出了一个时空特征交互网络,以通过在不同框架和空间分辨率的特征之间利用空间和时间相关来增强STVSR。具体而言,引入了空间 - 周期框架插值模块,以同时和互动性地插值低分辨率和高分辨率的中间框架特征。后来分别部署了空间 - 周期性的本地和全局细化模块,以利用不同特征之间的空间 - 周期相关性进行细化。最后,采用了新的运动一致性损失来增强重建帧之间的运动连续性。我们对三个标准基准测试,即VID4,Vimeo-90K和Adobe240进行实验,结果表明,我们的方法可以通过相当大的余量提高了最先进的方法。我们的代码将在https://github.com/yuezijie/stinet-pace time-video-super-resolution上找到。
translated by 谷歌翻译
现实世界图像超分辨率(SR)的关键挑战是在低分辨率(LR)图像中恢复具有复杂未知降解(例如,下采样,噪声和压缩)的缺失细节。大多数以前的作品还原图像空间中的此类缺失细节。为了应对自然图像的高度多样性,他们要么依靠难以训练和容易训练和伪影的不稳定的甘体,要么诉诸于通常不可用的高分辨率(HR)图像中的明确参考。在这项工作中,我们提出了匹配SR(FEMASR)的功能,该功能在更紧凑的特征空间中恢复了现实的HR图像。与图像空间方法不同,我们的FEMASR通过将扭曲的LR图像{\ IT特征}与我们预读的HR先验中的无失真性HR对应物匹配来恢复HR图像,并解码匹配的功能以获得现实的HR图像。具体而言,我们的人力资源先验包含一个离散的特征代码簿及其相关的解码器,它们在使用量化的生成对抗网络(VQGAN)的HR图像上预估计。值得注意的是,我们在VQGAN中结合了一种新型的语义正则化,以提高重建图像的质量。对于功能匹配,我们首先提取由LR编码器组成的LR编码器的LR功能,然后遵循简单的最近邻居策略,将其与预读的代码簿匹配。特别是,我们为LR编码器配备了与解码器的残留快捷方式连接,这对于优化功能匹配损耗至关重要,还有助于补充可能的功能匹配错误。实验结果表明,我们的方法比以前的方法产生更现实的HR图像。代码以\ url {https://github.com/chaofengc/femasr}发布。
translated by 谷歌翻译