尽管许多远程成像系统旨在支持扩展视力应用,但由于大气湍流,其操作的自然障碍是退化。大气湍流通过引入模糊和几何变形而导致图像质量的显着降解。近年来,在文献中提出了各种基于深度学习的单图像缓解方法,包括基于CNN的基于CNN和基于GAN的反转方法,这些方法试图消除图像中的失真。但是,其中一些方法很难训练,并且通常无法重建面部特征并产生不切实际的结果,尤其是在高湍流的情况下。降级扩散概率模型(DDPM)最近由于其稳定的训练过程和产生高质量图像的能力而获得了一些吸引力。在本文中,我们提出了第一个基于DDPM的解决方案,用于缓解大气湍流问题。我们还提出了一种快速采样技术,用于减少条件DDPM的推理时间。对合成和现实世界数据进行了广泛的实验,以显示我们模型的重要性。为了促进进一步的研究,在审查过程之后,所有代码和验证的模型都将公开。
translated by 谷歌翻译
现代监视系统使用基于深度学习的面部验证网络执行人员认可。大多数最先进的面部验证系统都是使用可见光谱图像训练的。但是,在弱光和夜间条件的情况下,在可见光谱中获取图像是不切实际的,并且通常在诸如热红外域之类的替代域中捕获图像。在检索相应的可见域图像后,通常在热图像中进行面部验证。这是一个公认的问题,通常称为热能(T2V)图像翻译。在本文中,我们建议针对面部图像的T2V翻译基于Denoising扩散概率模型(DDPM)解决方案。在训练过程中,该模型通过扩散过程了解了它们相应的热图像,可见面部图像的条件分布。在推断过程中,可见的域图像是通过从高斯噪声开始并反复执行的。 DDPM的现有推理过程是随机且耗时的。因此,我们提出了一种新颖的推理策略,以加快DDPM的推理时间,特别是用于T2V图像翻译问题。我们在多个数据集上实现了最新结果。代码和验证的模型可在http://github.com/nithin-gk/t2v-ddpm上公开获得
translated by 谷歌翻译
Conditional diffusion probabilistic models can model the distribution of natural images and can generate diverse and realistic samples based on given conditions. However, oftentimes their results can be unrealistic with observable color shifts and textures. We believe that this issue results from the divergence between the probabilistic distribution learned by the model and the distribution of natural images. The delicate conditions gradually enlarge the divergence during each sampling timestep. To address this issue, we introduce a new method that brings the predicted samples to the training data manifold using a pretrained unconditional diffusion model. The unconditional model acts as a regularizer and reduces the divergence introduced by the conditional model at each sampling step. We perform comprehensive experiments to demonstrate the effectiveness of our approach on super-resolution, colorization, turbulence removal, and image-deraining tasks. The improvements obtained by our method suggest that the priors can be incorporated as a general plugin for improving conditional diffusion models.
translated by 谷歌翻译
While deep learning-based methods for blind face restoration have achieved unprecedented success, they still suffer from two major limitations. First, most of them deteriorate when facing complex degradations out of their training data. Second, these methods require multiple constraints, e.g., fidelity, perceptual, and adversarial losses, which require laborious hyper-parameter tuning to stabilize and balance their influences. In this work, we propose a novel method named DifFace that is capable of coping with unseen and complex degradations more gracefully without complicated loss designs. The key of our method is to establish a posterior distribution from the observed low-quality (LQ) image to its high-quality (HQ) counterpart. In particular, we design a transition distribution from the LQ image to the intermediate state of a pre-trained diffusion model and then gradually transmit from this intermediate state to the HQ target by recursively applying a pre-trained diffusion model. The transition distribution only relies on a restoration backbone that is trained with $L_2$ loss on some synthetic data, which favorably avoids the cumbersome training process in existing methods. Moreover, the transition distribution can contract the error of the restoration backbone and thus makes our method more robust to unknown degradations. Comprehensive experiments show that DifFace is superior to current state-of-the-art methods, especially in cases with severe degradations. Our code and model are available at https://github.com/zsyOAOA/DifFace.
translated by 谷歌翻译
大气湍流可以通过在大气折射索引中引起空间和时间随机的波动,从而显着降低远程成像系统获得的图像质量。折射率的变化导致捕获的图像几何扭曲和模糊。因此,重要的是要补偿由大气湍流引起的图像中的视觉降解。在本文中,我们提出了一种基于深度学习的方法,用于限制大气湍流降解的单个图像。我们利用基于蒙特卡洛辍学的认知不确定性来捕获网络很难恢复的图像中的区域。然后,使用估计的不确定性图来指导网络以获得还原图像。对合成图像和真实图像进行了广泛的实验,以显示拟议工作的重要性。代码可在以下网址找到:https://github.com/rajeevyasarla/at-net
translated by 谷歌翻译
图像deBlurring是一种对给定输入图像的多种合理的解决方案是一个不适的问题。然而,大多数现有方法产生了清洁图像的确定性估计,并且训练以最小化像素级失真。已知这些指标与人类感知差,并且通常导致不切实际的重建。我们基于条件扩散模型介绍了盲脱模的替代框架。与现有技术不同,我们训练一个随机采样器,它改进了确定性预测器的输出,并且能够为给定输入产生多样化的合理重建。这导致跨多个标准基准的现有最先进方法的感知质量的显着提高。与典型的扩散模型相比,我们的预测和精致方法也能实现更有效的采样。结合仔细调整的网络架构和推理过程,我们的方法在PSNR等失真度量方面具有竞争力。这些结果表明了我们基于扩散和挑战的扩散和挑战的策略的显着优势,生产单一确定性重建的广泛使用策略。
translated by 谷歌翻译
Face Restoration (FR) aims to restore High-Quality (HQ) faces from Low-Quality (LQ) input images, which is a domain-specific image restoration problem in the low-level computer vision area. The early face restoration methods mainly use statistic priors and degradation models, which are difficult to meet the requirements of real-world applications in practice. In recent years, face restoration has witnessed great progress after stepping into the deep learning era. However, there are few works to study deep learning-based face restoration methods systematically. Thus, this paper comprehensively surveys recent advances in deep learning techniques for face restoration. Specifically, we first summarize different problem formulations and analyze the characteristic of the face image. Second, we discuss the challenges of face restoration. Concerning these challenges, we present a comprehensive review of existing FR methods, including prior based methods and deep learning-based methods. Then, we explore developed techniques in the task of FR covering network architectures, loss functions, and benchmark datasets. We also conduct a systematic benchmark evaluation on representative methods. Finally, we discuss future directions, including network designs, metrics, benchmark datasets, applications,etc. We also provide an open-source repository for all the discussed methods, which is available at https://github.com/TaoWangzj/Awesome-Face-Restoration.
translated by 谷歌翻译
面部超分辨率(FSR),也称为面部幻觉,其旨在增强低分辨率(LR)面部图像以产生高分辨率(HR)面部图像的分辨率,是特定于域的图像超分辨率问题。最近,FSR获得了相当大的关注,并目睹了深度学习技术的发展炫目。迄今为止,有很少有基于深入学习的FSR的研究摘要。在本次调查中,我们以系统的方式对基于深度学习的FSR方法进行了全面审查。首先,我们总结了FSR的问题制定,并引入了流行的评估度量和损失功能。其次,我们详细说明了FSR中使用的面部特征和流行数据集。第三,我们根据面部特征的利用大致分类了现有方法。在每个类别中,我们从设计原则的一般描述开始,然后概述代表方法,然后讨论其中的利弊。第四,我们评估了一些最先进的方法的表现。第五,联合FSR和其他任务以及与FSR相关的申请大致介绍。最后,我们设想了这一领域进一步的技术进步的前景。在\ URL {https://github.com/junjun-jiang/face-hallucination-benchmark}上有一个策划的文件和资源的策划文件和资源清单
translated by 谷歌翻译
标准扩散模型涉及图像变换 - 添加高斯噪声 - 以及逆转此降解的图像恢复操作员。我们观察到,扩散模型的生成行为并不是很大程度上取决于图像降解的选择,实际上,可以通过改变这种选择来构建整个生成模型家族。即使使用完全确定性的降解(例如,模糊,掩蔽等),培训和测试时间更新规则是基于扩散模型的培训和测试时间更新规则,可以轻松地概括为创建生成模型。这些完全确定的模型的成功使社区对扩散模型的理解质疑,这依赖于梯度Langevin动力学或变异推理中的噪声,并为反转任意过程的广义扩散模型铺平了道路。我们的代码可从https://github.com/arpitbansal297/cold-diffusion-models获得
translated by 谷歌翻译
Generating photos satisfying multiple constraints find broad utility in the content creation industry. A key hurdle to accomplishing this task is the need for paired data consisting of all modalities (i.e., constraints) and their corresponding output. Moreover, existing methods need retraining using paired data across all modalities to introduce a new condition. This paper proposes a solution to this problem based on denoising diffusion probabilistic models (DDPMs). Our motivation for choosing diffusion models over other generative models comes from the flexible internal structure of diffusion models. Since each sampling step in the DDPM follows a Gaussian distribution, we show that there exists a closed-form solution for generating an image given various constraints. Our method can unite multiple diffusion models trained on multiple sub-tasks and conquer the combined task through our proposed sampling strategy. We also introduce a novel reliability parameter that allows using different off-the-shelf diffusion models trained across various datasets during sampling time alone to guide it to the desired outcome satisfying multiple constraints. We perform experiments on various standard multimodal tasks to demonstrate the effectiveness of our approach. More details can be found in https://nithin-gk.github.io/projectpages/Multidiff/index.html
translated by 谷歌翻译
扩散模型已显示出令人印象深刻的图像产生性能,并已用于各种计算机视觉任务。不幸的是,使用扩散模型的图像生成非常耗时,因为它需要数千个采样步骤。为了解决这个问题,我们在这里提出了一种新型的金字塔扩散模型,以使用训练有位置嵌入的单个分数函数从更粗的分辨率图像开始生成高分辨率图像。这使图像生成的时间效率抽样可以解决,并在资源有限的训练时也可以解决低批量的大小问题。此外,我们表明,使用单个分数函数可以有效地用于多尺度的超分辨率问题。
translated by 谷歌翻译
传播模型已被证明对各种应用程序有效,例如图像,音频和图形生成。其他重要的应用是图像超分辨率和逆问题的解决方案。最近,一些作品使用了随机微分方程(SDE)将扩散模型推广到连续时间。在这项工作中,我们介绍SDE来生成超分辨率的面部图像。据我们所知,这是SDE首次用于此类应用程序。所提出的方法比基于扩散模型的现有超级分辨率方法提供了改进的峰值信噪比(PSNR),结构相似性指数(SSIM)和一致性。特别是,我们还评估了该方法在面部识别任务中的潜在应用。通用面部特征提取器用于比较超分辨率图像与地面真相,并获得了与其他方法相比,获得了卓越的结果。我们的代码可在https://github.com/marcelowds/sr-sde上公开获取
translated by 谷歌翻译
在实践中,很难收集配对的培训数据,但是不合格的样本广泛存在。当前的方法旨在通过探索损坏的数据和清洁数据之间的关系来从未配对样本中生成合成的培训数据。这项工作提出了Lud-Vae,这是一种从边际分布中采样的数据中学习关节概率密度函数的深层生成方法。我们的方法基于一个经过精心设计的概率图形模型,在该模型中,干净和损坏的数据域在条件上是独立的。使用变异推断,我们最大化证据下限(ELBO)以估计关节概率密度函数。此外,我们表明在推理不变假设下没有配对样品的情况下,ELBO是可以计算的。该属性在未配对的环境中提供了我们方法的数学原理。最后,我们将我们的方法应用于现实世界图像denoising,超分辨率和低光图像增强任务,并使用Lud-vae生成的合成数据训练模型。实验结果验证了我们方法比其他方法的优势。
translated by 谷歌翻译
The introduction of high-quality image generation models, particularly the StyleGAN family, provides a powerful tool to synthesize and manipulate images. However, existing models are built upon high-quality (HQ) data as desired outputs, making them unfit for in-the-wild low-quality (LQ) images, which are common inputs for manipulation. In this work, we bridge this gap by proposing a novel GAN structure that allows for generating images with controllable quality. The network can synthesize various image degradation and restore the sharp image via a quality control code. Our proposed QC-StyleGAN can directly edit LQ images without altering their quality by applying GAN inversion and manipulation techniques. It also provides for free an image restoration solution that can handle various degradations, including noise, blur, compression artifacts, and their mixtures. Finally, we demonstrate numerous other applications such as image degradation synthesis, transfer, and interpolation.
translated by 谷歌翻译
在不利天气条件下的图像恢复对各种计算机视觉应用引起了重大兴趣。最近的成功方法取决于深度神经网络架构设计(例如,具有视觉变压器)的当前进展。由最新的条件生成模型取得的最新进展的动机,我们提出了一种基于贴片的图像恢复算法,基于脱氧扩散概率模型。我们的基于贴片的扩散建模方法可以通过使用指导的DeNoising过程进行尺寸 - 不足的图像恢复,并在推理过程中对重叠贴片进行平滑的噪声估计。我们在基准数据集上经验评估了我们的模型,以进行图像,混合的降低和飞行以及去除雨滴的去除。我们展示了我们在特定天气和多天气图像恢复上实现最先进的表演的方法,并在质量上表现出对现实世界测试图像的强烈概括。
translated by 谷歌翻译
盲人恢复通常会遇到各种规模的面孔输入,尤其是在现实世界中。但是,当前的大多数作品都支持特定的规模面,这限制了其在现实情况下的应用能力。在这项工作中,我们提出了一个新颖的尺度感知盲人面部修复框架,名为FaceFormer,该框架将面部特征恢复作为比例感知转换。所提出的面部特征上采样(FFUP)模块基于原始的比例比例动态生成UPSMPLING滤波器,这有助于我们的网络适应任意面部尺度。此外,我们进一步提出了面部特征嵌入(FFE)模块,该模块利用变压器来层次提取面部潜在的多样性和鲁棒性。因此,我们的脸部形式实现了富裕性和稳健性,恢复了面部的面孔,对面部成分具有现实和对称的细节。广泛的实验表明,我们提出的使用合成数据集训练的方法比当前的最新图像更好地推广到天然低质量的图像。
translated by 谷歌翻译
Diffusion Probabilistic Models (DPMs) have recently been employed for image deblurring. DPMs are trained via a stochastic denoising process that maps Gaussian noise to the high-quality image, conditioned on the concatenated blurry input. Despite their high-quality generated samples, image-conditioned Diffusion Probabilistic Models (icDPM) rely on synthetic pairwise training data (in-domain), with potentially unclear robustness towards real-world unseen images (out-of-domain). In this work, we investigate the generalization ability of icDPMs in deblurring, and propose a simple but effective guidance to significantly alleviate artifacts, and improve the out-of-distribution performance. Particularly, we propose to first extract a multiscale domain-generalizable representation from the input image that removes domain-specific information while preserving the underlying image structure. The representation is then added into the feature maps of the conditional diffusion model as an extra guidance that helps improving the generalization. To benchmark, we focus on out-of-distribution performance by applying a single-dataset trained model to three external and diverse test sets. The effectiveness of the proposed formulation is demonstrated by improvements over the standard icDPM, as well as state-of-the-art performance on perceptual quality and competitive distortion metrics compared to existing methods.
translated by 谷歌翻译
在许多远程成像的应用中,我们面临的情景,其中出现在捕获的图像中的人通常被大气湍流降级。然而,由于劣化使图像成为几何扭曲和模糊,因此恢复用于面部验证的这种降级的图像是困难的。为了减轻湍流效果,本文提出了第一种湍流缓解方法,该方法利用培训的GaN封装的视觉前沿。基于视觉前沿,我们建议学习在空间周期性上下文距离上保留恢复图像的身份。在考虑网络学习中的身份差异时,这种距离可以保持来自GaN的恢复图像的现实主义。另外,提出了通过在没有身份变化的情况下引入更多外观方差来促进身份保留学习的分层伪连接。广泛的实验表明,我们的方法在恢复结果的视觉质量和面部验证准确性中显着优于现有技术。
translated by 谷歌翻译
尽管基准数据集的成功,但大多数先进的面部超分辨率模型在真实情况下表现不佳,因为真实图像与合成训练对之间的显着域间隙。为了解决这个问题,我们提出了一种用于野外面部超分辨率的新型域 - 自适应降级网络。该降级网络预测流场以及中间低分辨率图像。然后,通过翘曲中间图像来生成降级的对应物。利用捕获运动模糊的偏好,这种模型在保护原始图像和劣化之间保持身份一致性更好地执行。我们进一步提出了超分辨率网络的自我调节块。该块将输入图像作为条件术语,以有效地利用面部结构信息,从而消除了对显式前沿的依赖性,例如,面部地标或边界。我们的模型在Celeba和真实世界的面部数据集上实现了最先进的性能。前者展示了我们所提出的建筑的强大生成能力,而后者展示了现实世界中的良好的身份一致性和感知品质。
translated by 谷歌翻译
The primary aim of single-image super-resolution is to construct a high-resolution (HR) image from a corresponding low-resolution (LR) input. In previous approaches, which have generally been supervised, the training objective typically measures a pixel-wise average distance between the super-resolved (SR) and HR images. Optimizing such metrics often leads to blurring, especially in high variance (detailed) regions. We propose an alternative formulation of the super-resolution problem based on creating realistic SR images that downscale correctly. We present a novel super-resolution algorithm addressing this problem, PULSE (Photo Upsampling via Latent Space Exploration), which generates high-resolution, realistic images at resolutions previously unseen in the literature. It accomplishes this in an entirely self-supervised fashion and is not confined to a specific degradation operator used during training, unlike previous methods (which require training on databases of LR-HR image pairs for supervised learning). Instead of starting with the LR image and slowly adding detail, PULSE traverses the high-resolution natural image manifold, searching for images that downscale to the original LR image. This is formalized through the "downscaling loss," which guides exploration through the latent space of a generative model. By leveraging properties of high-dimensional Gaussians, we restrict the search space to guarantee that our outputs are realistic. PULSE thereby generates super-resolved images that both are realistic and downscale correctly. We show extensive experimental results demonstrating the efficacy of our approach in the domain of face super-resolution (also known as face hallucination). We also present a discussion of the limitations and biases of the method as currently implemented with an accompanying model card with relevant metrics. Our method outperforms state-of-the-art methods in perceptual quality at higher resolutions and scale factors than previously pos-sible.
translated by 谷歌翻译