在立体声设置下,可以通过利用第二视图提供的其他信息来进一步改善图像JPEG伪像删除的性能。但是,将此信息纳入立体声图像jpeg trifacts删除是一个巨大的挑战,因为现有的压缩工件使像素级视图对齐变得困难。在本文中,我们提出了一个新颖的视差变压器网络(PTNET),以整合来自立体图像对的立体图像对jpeg jpeg trifacts删除的信息。具体而言,提出了精心设计的对称性双向视差变压器模块,以匹配具有不同视图之间相似纹理的特征,而不是像素级视图对齐。由于遮挡和边界的问题,提出了一个基于置信的跨视图融合模块,以实现两种视图的更好的特征融合,其中跨视图特征通过置信图加权。尤其是,我们为跨视图的互动采用粗到最新的设计,从而提高性能。全面的实验结果表明,与其他测试最新方法相比,我们的PTNET可以有效地消除压缩伪像并获得更高的性能。
translated by 谷歌翻译
Stereo images, containing left and right view images with disparity, are utilized in solving low-vision tasks recently, e.g., rain removal and super-resolution. Stereo image restoration methods usually obtain better performance than monocular methods by learning the disparity between dual views either implicitly or explicitly. However, existing stereo rain removal methods still cannot make full use of the complementary information between two views, and we find it is because: 1) the rain streaks have more complex distributions in directions and densities, which severely damage the complementary information and pose greater challenges; 2) the disparity estimation is not accurate enough due to the imperfect fusion mechanism for the features between two views. To overcome such limitations, we propose a new \underline{Stereo} \underline{I}mage \underline{R}ain \underline{R}emoval method (StereoIRR) via sufficient interaction between two views, which incorporates: 1) a new Dual-view Mutual Attention (DMA) mechanism which generates mutual attention maps by taking left and right views as key information for each other to facilitate cross-view feature fusion; 2) a long-range and cross-view interaction, which is constructed with basic blocks and dual-view mutual attention, can alleviate the adverse effect of rain on complementary information to help the features of stereo images to get long-range and cross-view interaction and fusion. Notably, StereoIRR outperforms other related monocular and stereo image rain removal methods on several datasets. Our codes and datasets will be released.
translated by 谷歌翻译
Low-light stereo image enhancement (LLSIE) is a relatively new task to enhance the quality of visually unpleasant stereo images captured in dark conditions. So far, very few studies on deep LLSIE have been explored due to certain challenging issues, i.e., the task has not been well addressed, and current methods clearly suffer from two shortages: 1) insufficient cross-view interaction; 2) lacking long-range dependency for intra-view learning. In this paper, we therefore propose a novel LLSIE model, termed \underline{Suf}ficient C\underline{r}oss-View \underline{In}teraction Network (SufrinNet). To be specific, we present sufficient inter-view interaction module (SIIM) to enhance the information exchange across views. SIIM not only discovers the cross-view correlations at different scales, but also explores the cross-scale information interaction. Besides, we present a spatial-channel information mining block (SIMB) for intra-view feature extraction, and the benefits are twofold. One is the long-range dependency capture to build spatial long-range relationship, and the other is expanded channel information refinement that enhances information flow in channel dimension. Extensive experiments on Flickr1024, KITTI 2012, KITTI 2015 and Middlebury datasets show that our method obtains better illumination adjustment and detail recovery, and achieves SOTA performance compared to other related methods. Our codes, datasets and models will be publicly available.
translated by 谷歌翻译
压缩在通过限制系统(例如流媒体服务,虚拟现实或视频游戏)等系统的有效传输和存储图像和视频中起着重要作用。但是,不可避免地会导致伪影和原始信息的丢失,这可能会严重降低视觉质量。由于这些原因,压缩图像的质量增强已成为流行的研究主题。尽管大多数最先进的图像恢复方法基于卷积神经网络,但基于Swinir等其他基于变压器的方法在这些任务上表现出令人印象深刻的性能。在本文中,我们探索了新型的Swin Transformer V2,以改善图像超分辨率的Swinir,尤其是压缩输入方案。使用这种方法,我们可以解决训练变压器视觉模型中的主要问题,例如训练不稳定性,预训练和微调之间的分辨率差距以及数据饥饿。我们对三个代表性任务进行实验:JPEG压缩伪像去除,图像超分辨率(经典和轻巧)以及压缩的图像超分辨率。实验结果表明,我们的方法SWIN2SR可以改善SWINIR的训练收敛性和性能,并且是“ AIM 2022挑战压缩图像和视频的超分辨率”的前5个解决方案。
translated by 谷歌翻译
Informative features play a crucial role in the single image super-resolution task. Channel attention has been demonstrated to be effective for preserving information-rich features in each layer. However, channel attention treats each convolution layer as a separate process that misses the correlation among different layers. To address this problem, we propose a new holistic attention network (HAN), which consists of a layer attention module (LAM) and a channel-spatial attention module (CSAM), to model the holistic interdependencies among layers, channels, and positions. Specifically, the proposed LAM adaptively emphasizes hierarchical features by considering correlations among layers. Meanwhile, CSAM learns the confidence at all the positions of each channel to selectively capture more informative features. Extensive experiments demonstrate that the proposed HAN performs favorably against the state-ofthe-art single image super-resolution approaches.
translated by 谷歌翻译
使用注意机制的深度卷积神经网络(CNN)在动态场景中取得了巨大的成功。在大多数这些网络中,只能通过注意图精炼的功能传递到下一层,并且不同层的注意力图彼此分开,这并不能充分利用来自CNN中不同层的注意信息。为了解决这个问题,我们引入了一种新的连续跨层注意传播(CCLAT)机制,该机制可以利用所有卷积层的分层注意信息。基于CCLAT机制,我们使用非常简单的注意模块来构建一个新型残留的密集注意融合块(RDAFB)。在RDAFB中,从上述RDAFB的输出中推断出的注意图和每一层直接连接到后续的映射,从而导致CRLAT机制。以RDAFB为基础,我们为动态场景Deblurring设计了一个名为RDAFNET的有效体系结构。基准数据集上的实验表明,所提出的模型的表现优于最先进的脱毛方法,并证明了CCLAT机制的有效性。源代码可在以下网址提供:https://github.com/xjmz6/rdafnet。
translated by 谷歌翻译
Face Restoration (FR) aims to restore High-Quality (HQ) faces from Low-Quality (LQ) input images, which is a domain-specific image restoration problem in the low-level computer vision area. The early face restoration methods mainly use statistic priors and degradation models, which are difficult to meet the requirements of real-world applications in practice. In recent years, face restoration has witnessed great progress after stepping into the deep learning era. However, there are few works to study deep learning-based face restoration methods systematically. Thus, this paper comprehensively surveys recent advances in deep learning techniques for face restoration. Specifically, we first summarize different problem formulations and analyze the characteristic of the face image. Second, we discuss the challenges of face restoration. Concerning these challenges, we present a comprehensive review of existing FR methods, including prior based methods and deep learning-based methods. Then, we explore developed techniques in the task of FR covering network architectures, loss functions, and benchmark datasets. We also conduct a systematic benchmark evaluation on representative methods. Finally, we discuss future directions, including network designs, metrics, benchmark datasets, applications,etc. We also provide an open-source repository for all the discussed methods, which is available at https://github.com/TaoWangzj/Awesome-Face-Restoration.
translated by 谷歌翻译
光场(LF)摄像机记录了光线的强度和方向,并将3D场景编码为4D LF图像。最近,为各种LF图像处理任务提出了许多卷积神经网络(CNN)。但是,CNN有效地处理LF图像是一项挑战,因为空间和角度信息与不同的差异高度缠绕。在本文中,我们提出了一种通用机制,以将这些耦合信息解开以进行LF图像处理。具体而言,我们首先设计了一类特定领域的卷积,以将LFS与不同的维度解开,然后通过设计特定于任务的模块来利用这些分离的功能。我们的解开机制可以在事先之前很好地纳入LF结构,并有效处理4D LF数据。基于提出的机制,我们开发了三个网络(即distgssr,distgasr和Distgdisp),用于空间超分辨率,角度超分辨率和差异估计。实验结果表明,我们的网络在所有这三个任务上都实现了最先进的性能,这表明了我们解散机制的有效性,效率和一般性。项目页面:https://yingqianwang.github.io/distglf/。
translated by 谷歌翻译
从一组多曝光图像中重建无精神的高动态范围(HDR)图像是一项具有挑战性的任务,尤其是在大型对象运动和闭塞的情况下,使用现有方法导致可见的伪影。为了解决这个问题,我们提出了一个深层网络,该网络试图学习以正规损失为指导的多尺度特征流。它首先提取多尺度功能,然后对非参考图像的特征对齐。对齐后,我们使用残留的通道注意块将不同图像的特征合并。广泛的定性和定量比较表明,我们的方法可实现最新的性能,并在颜色伪像和几何变形大大减少的情况下产生出色的结果。
translated by 谷歌翻译
近年来,压缩图像超分辨率已引起了极大的关注,其中图像被压缩伪像和低分辨率伪影降解。由于复杂的杂化扭曲变形,因此很难通过简单的超分辨率和压缩伪像消除掉的简单合作来恢复扭曲的图像。在本文中,我们向前迈出了一步,提出了层次的SWIN变压器(HST)网络,以恢复低分辨率压缩图像,该图像共同捕获分层特征表示并分别用SWIN Transformer增强每个尺度表示。此外,我们发现具有超分辨率(SR)任务的预处理对于压缩图像超分辨率至关重要。为了探索不同的SR预审查的影响,我们将常用的SR任务(例如,比科比奇和不同的实际超分辨率仿真)作为我们的预处理任务,并揭示了SR在压缩的图像超分辨率中起不可替代的作用。随着HST和预训练的合作,我们的HST在AIM 2022挑战中获得了低质量压缩图像超分辨率轨道的第五名,PSNR为23.51db。广泛的实验和消融研究已经验证了我们提出的方法的有效性。
translated by 谷歌翻译
Image restoration under hazy weather condition, which is called single image dehazing, has been of significant interest for various computer vision applications. In recent years, deep learning-based methods have achieved success. However, existing image dehazing methods typically neglect the hierarchy of features in the neural network and fail to exploit their relationships fully. To this end, we propose an effective image dehazing method named Hierarchical Contrastive Dehazing (HCD), which is based on feature fusion and contrastive learning strategies. HCD consists of a hierarchical dehazing network (HDN) and a novel hierarchical contrastive loss (HCL). Specifically, the core design in the HDN is a Hierarchical Interaction Module, which utilizes multi-scale activation to revise the feature responses hierarchically. To cooperate with the training of HDN, we propose HCL which performs contrastive learning on hierarchically paired exemplars, facilitating haze removal. Extensive experiments on public datasets, RESIDE, HazeRD, and DENSE-HAZE, demonstrate that HCD quantitatively outperforms the state-of-the-art methods in terms of PSNR, SSIM and achieves better visual quality.
translated by 谷歌翻译
This paper explores the problem of reconstructing high-resolution light field (LF) images from hybrid lenses, including a high-resolution camera surrounded by multiple low-resolution cameras. The performance of existing methods is still limited, as they produce either blurry results on plain textured areas or distortions around depth discontinuous boundaries. To tackle this challenge, we propose a novel end-to-end learning-based approach, which can comprehensively utilize the specific characteristics of the input from two complementary and parallel perspectives. Specifically, one module regresses a spatially consistent intermediate estimation by learning a deep multidimensional and cross-domain feature representation, while the other module warps another intermediate estimation, which maintains the high-frequency textures, by propagating the information of the high-resolution view. We finally leverage the advantages of the two intermediate estimations adaptively via the learned attention maps, leading to the final high-resolution LF image with satisfactory results on both plain textured areas and depth discontinuous boundaries. Besides, to promote the effectiveness of our method trained with simulated hybrid data on real hybrid data captured by a hybrid LF imaging system, we carefully design the network architecture and the training strategy. Extensive experiments on both real and simulated hybrid data demonstrate the significant superiority of our approach over state-of-the-art ones. To the best of our knowledge, this is the first end-to-end deep learning method for LF reconstruction from a real hybrid input. We believe our framework could potentially decrease the cost of high-resolution LF data acquisition and benefit LF data storage and transmission.
translated by 谷歌翻译
We study on image super-resolution (SR), which aims to recover realistic textures from a low-resolution (LR) image. Recent progress has been made by taking high-resolution images as references (Ref), so that relevant textures can be transferred to LR images. However, existing SR approaches neglect to use attention mechanisms to transfer high-resolution (HR) textures from Ref images, which limits these approaches in challenging cases. In this paper, we propose a novel Texture Transformer Network for Image Super-Resolution (TTSR), in which the LR and Ref images are formulated as queries and keys in a transformer, respectively. TTSR consists of four closely-related modules optimized for image generation tasks, including a learnable texture extractor by DNN, a relevance embedding module, a hard-attention module for texture transfer, and a softattention module for texture synthesis. Such a design encourages joint feature learning across LR and Ref images, in which deep feature correspondences can be discovered by attention, and thus accurate texture features can be transferred. The proposed texture transformer can be further stacked in a cross-scale way, which enables texture recovery from different levels (e.g., from 1× to 4× magnification). Extensive experiments show that TTSR achieves significant improvements over state-of-the-art approaches on both quantitative and qualitative evaluations. The source code can be downloaded at https://github.com/ researchmm/TTSR.
translated by 谷歌翻译
高动态范围(HDR)DEGHOSTING算法旨在生成具有现实细节的无幽灵HDR图像。受到接收场的局部性的限制,现有的基于CNN的方法通常容易产生大型运动和严重饱和的情况下产生鬼影和强度扭曲。在本文中,我们提出了一种新颖的背景感知视觉变压器(CA-VIT),用于无幽灵的高动态范围成像。 CA-VIT被设计为双分支结构,可以共同捕获全球和本地依赖性。具体而言,全球分支采用基于窗口的变压器编码器来建模远程对象运动和强度变化以解决hosting。对于本地分支,我们设计了局部上下文提取器(LCE)来捕获短范围的图像特征,并使用频道注意机制在提取的功能上选择信息丰富的本地详细信息,以补充全局分支。通过将CA-VIT作为基本组件纳入基本组件,我们进一步构建了HDR-Transformer,这是一个分层网络,以重建高质量的无幽灵HDR图像。在三个基准数据集上进行的广泛实验表明,我们的方法在定性和定量上优于最先进的方法,而计算预算大大降低。代码可从https://github.com/megvii-research/hdr-transformer获得
translated by 谷歌翻译
现实世界图像Denoising是一个实用的图像恢复问题,旨在从野外嘈杂的输入中获取干净的图像。最近,Vision Transformer(VIT)表现出强大的捕获远程依赖性的能力,许多研究人员试图将VIT应用于图像DeNosing任务。但是,现实世界的图像是一个孤立的框架,它使VIT构建了内部贴片的远程依赖性,该依赖性将图像分为贴片并混乱噪声模式和梯度连续性。在本文中,我们建议通过使用连续的小波滑动转换器来解决此问题,该小波滑动转换器在现实世界中构建频率对应关系,称为dnswin。具体而言,我们首先使用CNN编码器从嘈杂的输入图像中提取底部功能。 DNSWIN的关键是将高频和低频信息与功能和构建频率依赖性分开。为此,我们提出了小波滑动窗口变压器,该变压器利用离散的小波变换,自我注意力和逆离散小波变换来提取深度特征。最后,我们使用CNN解码器将深度特征重建为DeNo的图像。对现实世界的基准测试的定量和定性评估都表明,拟议的DNSWIN对最新方法的表现良好。
translated by 谷歌翻译
Recently, Transformer-based image restoration networks have achieved promising improvements over convolutional neural networks due to parameter-independent global interactions. To lower computational cost, existing works generally limit self-attention computation within non-overlapping windows. However, each group of tokens are always from a dense area of the image. This is considered as a dense attention strategy since the interactions of tokens are restrained in dense regions. Obviously, this strategy could result in restricted receptive fields. To address this issue, we propose Attention Retractable Transformer (ART) for image restoration, which presents both dense and sparse attention modules in the network. The sparse attention module allows tokens from sparse areas to interact and thus provides a wider receptive field. Furthermore, the alternating application of dense and sparse attention modules greatly enhances representation ability of Transformer while providing retractable attention on the input image.We conduct extensive experiments on image super-resolution, denoising, and JPEG compression artifact reduction tasks. Experimental results validate that our proposed ART outperforms state-of-the-art methods on various benchmark datasets both quantitatively and visually. We also provide code and models at the website https://github.com/gladzhang/ART.
translated by 谷歌翻译
深度映射记录场景中的视点和对象之间的距离,这在许多真实应用程序中起着关键作用。然而,消费者级RGB-D相机捕获的深度图遭受了低空间分辨率。引导深度地图超分辨率(DSR)是解决此问题的流行方法,该方法试图从输入的低分辨率(LR)深度及其耦合的HR RGB图像中恢复高分辨率(HR)深度映射和作为指引。引导DSR最具挑战性的问题是如何正确选择一致的结构并传播它们,并正确处理不一致的结构。在本文中,我们提出了一种用于引导DSR的新型关注的分层多模态融合(AHMF)网络。具体地,为了有效地提取和组合来自LR深度和HR引导的相关信息,我们提出了一种基于多模态注意力的融合(MMAF)策略,包括分层卷积层,包括特征增强块,以选择有价值的功能和特征重新校准块来统一不同外观特征的方式的相似性度量。此外,我们提出了一个双向分层特征协作(BHFC)模块,以完全利用多尺度特征之间的低级空间信息和高级结构信息。实验结果表明,在重建精度,运行速度和记忆效率方面,我们的方法优于最先进的方法。
translated by 谷歌翻译
立体类像素细分旨在通过左右视图将离散的像素分组为感知区域,以更加协作和高效地分组。现有的Superpixel分割算法主要利用颜色和空间特征作为输入,这可能会对空间信息施加强大的约束,同时利用立体声图像对的差异信息。为了减轻此问题,我们提出了一种立体声超级像素细分方法,并在本工作中具有空间信息的脱钩机制。为了解除立体视差信息和空间信息,在融合立体声图像对的特征之前,暂时删除空间信息,并提出了脱钩的立体声融合模块(DSFM),以处理立体声的特征特征特征对齐和遮挡问题。此外,由于空间信息对于超像素分割至关重要,因此我们进一步设计一个动态空间嵌入模块(DSEM)以重新添加空间信息,并且将通过DSEM中的DSEM进行自适应调整空间信息的权重(DF)用于实现更好的细分。全面的实验结果表明,我们的方法可以在KITTI2015和CityScapes数据集上实现最新性能,并且还可以在NJU2K数据集上的显着对象检测中验证效率。源代码将在接受纸张后公开提供。
translated by 谷歌翻译
Recently, great progress has been made in single-image super-resolution (SISR) based on deep learning technology. However, the existing methods usually require a large computational cost. Meanwhile, the activation function will cause some features of the intermediate layer to be lost. Therefore, it is a challenge to make the model lightweight while reducing the impact of intermediate feature loss on the reconstruction quality. In this paper, we propose a Feature Interaction Weighted Hybrid Network (FIWHN) to alleviate the above problem. Specifically, FIWHN consists of a series of novel Wide-residual Distillation Interaction Blocks (WDIB) as the backbone, where every third WDIBs form a Feature shuffle Weighted Group (FSWG) by mutual information mixing and fusion. In addition, to mitigate the adverse effects of intermediate feature loss on the reconstruction results, we introduced a well-designed Wide Convolutional Residual Weighting (WCRW) and Wide Identical Residual Weighting (WIRW) units in WDIB, and effectively cross-fused features of different finenesses through a Wide-residual Distillation Connection (WRDC) framework and a Self-Calibrating Fusion (SCF) unit. Finally, to complement the global features lacking in the CNN model, we introduced the Transformer into our model and explored a new way of combining the CNN and Transformer. Extensive quantitative and qualitative experiments on low-level and high-level tasks show that our proposed FIWHN can achieve a good balance between performance and efficiency, and is more conducive to downstream tasks to solve problems in low-pixel scenarios.
translated by 谷歌翻译
基于变压器的方法与基于CNN的方法相比,由于其对远程依赖性的模型,因此获得了令人印象深刻的图像恢复性能。但是,像Swinir这样的进步采用了基于窗口的和本地注意力的策略来平衡性能和计算开销,这限制了采用大型接收领域来捕获全球信息并在早期层中建立长期依赖性。为了进一步提高捕获全球信息的效率,在这项工作中,我们建议Swinfir通过更换具有整个图像范围的接收场的快速傅立叶卷积(FFC)组件来扩展Swinir。我们还重新访问其他先进技术,即数据增强,预训练和功能集合,以改善图像重建的效果。并且我们的功能合奏方法使模型的性能得以大大增强,而无需增加训练和测试时间。与现有方法相比,我们将算法应用于多个流行的大规模基准,并实现了最先进的性能。例如,我们的Swinfir在漫画109数据集上达到了32.83 dB的PSNR,该PSNR比最先进的Swinir方法高0.8 dB。
translated by 谷歌翻译