反问题包括从不完整的测量集重建信号，其性能高度取决于通过正则化编码的先验知识的质量。尽管传统方法着重于获得独特的解决方案，但新兴趋势考虑了探索多种临时解决方案。在本文中，我们提出了一种生成多个重建的方法，该重建既适合测量值，又是由生成对抗网络学到的数据驱动的先验。特别是，我们表明，从初始解决方案开始，可以在生成模型的潜在空间中找到对远期操作员无效的方向，从而与测量值保持一致，同时诱发显着的感知变化。我们的探索方法允许为反问题生成多个解决方案，比现有方法快的数量级。我们显示了图像超分辨率和介入问题的结果。

我们介绍了一种使用Nerf式生成模型解决逆问题的新框架。给出了单一的2-D图像和已知相机参数的3-D场景重建问题感兴趣。我们展示了天真地优化潜伏的空间，导致伪影和糟糕的新颖观看渲染。我们将此问题归因于3-D几何形状清晰的音量障碍物，并在新颖视野的渲染中变得可见。我们提出了一种新颖的辐射场正则化方法，以获得更好的3-D表面和改进的新颖观点，给定单一视图观察。我们的方法自然地扩展到一般逆问题，包括若有所述，其中仅部分地观察到单一视图。我们通过实验评估我们的方法，实现视觉改进和性能在广泛的任务中升高了基线。与以前的现有技术相比，我们的方法达到了30-40美元的MSE减免和15-25美元的LPIP损失减少。

edu.hk (a) Image Reconstruction (b) Image Colorization (c) Image Super-Resolution (d) Image Denoising (e) Image Inpainting (f) Semantic Manipulation Figure 1: Multi-code GAN prior facilitates many image processing applications using the reconstruction from fixed PGGAN [23] models.

The primary aim of single-image super-resolution is to construct a high-resolution (HR) image from a corresponding low-resolution (LR) input. In previous approaches, which have generally been supervised, the training objective typically measures a pixel-wise average distance between the super-resolved (SR) and HR images. Optimizing such metrics often leads to blurring, especially in high variance (detailed) regions. We propose an alternative formulation of the super-resolution problem based on creating realistic SR images that downscale correctly. We present a novel super-resolution algorithm addressing this problem, PULSE (Photo Upsampling via Latent Space Exploration), which generates high-resolution, realistic images at resolutions previously unseen in the literature. It accomplishes this in an entirely self-supervised fashion and is not confined to a specific degradation operator used during training, unlike previous methods (which require training on databases of LR-HR image pairs for supervised learning). Instead of starting with the LR image and slowly adding detail, PULSE traverses the high-resolution natural image manifold, searching for images that downscale to the original LR image. This is formalized through the "downscaling loss," which guides exploration through the latent space of a generative model. By leveraging properties of high-dimensional Gaussians, we restrict the search space to guarantee that our outputs are realistic. PULSE thereby generates super-resolved images that both are realistic and downscale correctly. We show extensive experimental results demonstrating the efficacy of our approach in the domain of face super-resolution (also known as face hallucination). We also present a discussion of the limitations and biases of the method as currently implemented with an accompanying model card with relevant metrics. Our method outperforms state-of-the-art methods in perceptual quality at higher resolutions and scale factors than previously pos-sible.

机器学习模型通常培训端到端和监督设置，使用配对（输入，输出）数据。示例包括最近的超分辨率方法，用于在（低分辨率，高分辨率）图像上培训。然而，这些端到端的方法每当输入中存在分布偏移时需要重新训练（例如，夜间图像VS日光）或相关的潜在变量（例如，相机模糊或手动运动）。在这项工作中，我们利用最先进的（SOTA）生成模型（这里是Stylegan2）来构建强大的图像前提，这使得贝叶斯定理应用于许多下游重建任务。我们的方法是通过生成模型（BRGM）的贝叶斯重建，使用单个预先训练的发生器模型来解决不同的图像恢复任务，即超级分辨率和绘画，通过与不同的前向腐败模型相结合。我们将发电机模型的重量保持固定，并通过估计产生重建图像的输入潜在的跳过载体来重建图像来估计图像。我们进一步使用变分推理来近似潜伏向量的后部分布，我们对多种解决方案进行采样。我们在三个大型和多样化的数据集中展示了BRGM：（i）来自Flick的60,000个图像面向高质量的数据集（II）来自MIMIC III的高质量数据集（II）240,000胸X射线，（III）的组合收集5脑MRI数据集，具有7,329个扫描。在所有三个数据集和没有任何DataSet特定的HyperParameter调整，我们的简单方法会在超级分辨率和绘画上对当前的特定任务最先进的方法产生性能竞争力，同时更加稳定，而不需要任何培训。我们的源代码和预先训练的型号可在线获取：https://razvanmarinescu.github.io/brgm/。

通常，层析成像是一个不适合的反问题。通常，从断层扫描测量中获得了拟距对象的单个正则图像估计。但是，可能有多个与相同的测量数据一致的对象。生成此类替代解决方案的能力很重要，因为它可以实现成像系统的新评估。原则上，这可以通过后采样方法来实现。近年来，已经采用了深层神经网络进行后验采样，结果令人鼓舞。但是，此类方法尚未用于大规模断层成像应用。另一方面，经验抽样方法在大规模成像系统上可能是可行的，并且可以对实际应用实现不确定性量化。经验抽样涉及在随机优化框架内求解正规化的逆问题，以获得替代数据一致的解决方案。在这项工作中，提出了一种新的经验抽样方法，该方法计算了与同一获得的测量数据一致的层析成像逆问题的多个解决方案。该方法通过在基于样式的生成对抗网络（stylegan）的潜在空间中反复解决优化问题的运行，并受到通过潜在空间探索（PULSE）方法的照片启发，该方法是为超分辨率任务开发而成的。通过涉及两种程式化的层析成像模式的数值研究来证明和分析所提出的方法。这些研究确定了该方法执行有效的经验抽样和不确定性定量的能力。

Deep convolutional networks have become a popular tool for image generation and restoration. Generally, their excellent performance is imputed to their ability to learn realistic image priors from a large number of example images. In this paper, we show that, on the contrary, the structure of a generator network is sufficient to capture a great deal of low-level image statistics prior to any learning. In order to do so, we show that a randomly-initialized neural network can be used as a handcrafted prior with excellent results in standard inverse problems such as denoising, superresolution, and inpainting. Furthermore, the same prior can be used to invert deep neural representations to diagnose them, and to restore images based on flash-no flash input pairs.

现代数码相机和智能手机主要依赖于图像信号处理（ISP）管道，从而产生逼真的彩色RGB图像。然而，与DSLR相机相比，由于其物理限制，在许多便携式移动设备中通常可以在许多便携式移动设备中获得低质量的图像。低质量的图像具有多种降级，即，由于相机滤色器阵列，由于相机滤色器阵列，由于较小的摄像机传感器而导致的低分辨率，磁割模式，并且其余信息因噪声损坏而导致的镶嵌图案。这种降级限制了从单个低分辨率（LR）图像中恢复高分辨率（HR）图像细节的电流单图像超分辨率（SISR）方法的性能。在这项工作中，我们提出了一种原始的突发超分辨率迭代卷积神经网络（RBSricnn），其作为前向（物理）模型的整体沿着突发拍摄管道。与现有的黑盒数据驱动方法相比，所提出的突发SR方案解决了经典图像正则化，凸优化和深度学习技术的问题。所提出的网络通过中间SR估计的迭代细化产生最终输出。我们展示了我们提出的方法在定量和定性实验中的有效性，这些实验概括为具有可用于培训的ONL合成突发数据的真实LR突发输入。

由于其高质量的重建以及将现有迭代求解器结合起来的易于性，因此最近将扩散模型作为强大的生成反问题解决器研究。但是，大多数工作都专注于在无噪声设置中解决简单的线性逆问题，这显着不足以使实际问题的复杂性不足。在这项工作中，我们将扩散求解器扩展求解器，以通过后采样的拉普拉斯近似有效地处理一般噪声（非）线性反问题。有趣的是，所得的后验采样方案是扩散采样的混合版本，具有歧管约束梯度，而没有严格的测量一致性投影步骤，与先前的研究相比，在嘈杂的设置中产生了更可取的生成路径。我们的方法表明，扩散模型可以结合各种测量噪声统计量，例如高斯和泊松，并且还有效处理嘈杂的非线性反问题，例如傅立叶相检索和不均匀的脱毛。

近年来，深度学习在图像重建方面取得了显着的经验成功。这已经促进了对关键用例中数据驱动方法的正确性和可靠性的精确表征的持续追求，例如在医学成像中。尽管基于深度学习的方法具有出色的性能和功效，但对其稳定性或缺乏稳定性的关注以及严重的实际含义。近年来，已经取得了重大进展，以揭示数据驱动的图像恢复方法的内部运作，从而挑战了其广泛认为的黑盒本质。在本文中，我们将为数据驱动的图像重建指定相关的融合概念，该概念将构成具有数学上严格重建保证的学习方法调查的基础。强调的一个例子是ICNN的作用，提供了将深度学习的力量与经典凸正则化理论相结合的可能性，用于设计被证明是融合的方法。这篇调查文章旨在通过提供对数据驱动的图像重建方法以及从业人员的理解，旨在通过提供可访问的融合概念的描述，并通过将一些现有的经验实践放在可靠的数学上，来推进我们对数据驱动图像重建方法的理解以及从业人员的了解。基础。

由于简单但有效的训练机制和出色的图像产生质量，生成的对抗网络（GAN）引起了极大的关注。具有生成照片现实的高分辨率（例如$ 1024 \ times1024 $）的能力，最近的GAN模型已大大缩小了生成的图像与真实图像之间的差距。因此，许多最近的作品表明，通过利用良好的潜在空间和博学的gan先验来利用预先训练的GAN模型的新兴兴趣。在本文中，我们简要回顾了从三个方面利用预先培训的大规模GAN模型的最新进展，即1）大规模生成对抗网络的培训，2）探索和理解预训练的GAN模型，以及预先培训的GAN模型，以及3）利用这些模型进行后续任务，例如图像恢复和编辑。有关相关方法和存储库的更多信息，请访问https://github.com/csmliu/pretretaining-gans。

最近，由于高性能，深度学习方法已成为生物学图像重建和增强问题的主要研究前沿，以及其超快速推理时间。但是，由于获得监督学习的匹配参考数据的难度，对不需要配对的参考数据的无监督学习方法越来越兴趣。特别是，已成功用于各种生物成像应用的自我监督的学习和生成模型。在本文中，我们概述了在古典逆问题的背景下的连贯性观点，并讨论其对生物成像的应用，包括电子，荧光和去卷积显微镜，光学衍射断层扫描和功能性神经影像。

在过去的几年中，深层神经网络方法的反向成像问题产生了令人印象深刻的结果。在本文中，我们考虑在跨问题方法中使用生成模型。所考虑的正规派对图像进行了惩罚，这些图像远非生成模型的范围，该模型学会了产生类似于训练数据集的图像。我们命名这个家庭\ textit {生成正规派}。生成常规人的成功取决于生成模型的质量，因此我们提出了一组所需的标准来评估生成模型并指导未来的研究。在我们的数值实验中，我们根据我们所需的标准评估了三种常见的生成模型，自动编码器，变异自动编码器和生成对抗网络。我们还测试了三个不同的生成正规疗法仪，关于脱毛，反卷积和断层扫描的逆问题。我们表明，逆问题的限制解决方案完全位于生成模型的范围内可以给出良好的结果，但是允许与发电机范围的小偏差产生更一致的结果。

Learning a good image prior is a long-term goal for image restoration and manipulation. While existing methods like deep image prior (DIP) capture low-level image statistics, there are still gaps toward an image prior that captures rich image semantics including color, spatial coherence, textures, and high-level concepts. This work presents an effective way to exploit the image prior captured by a generative adversarial network (GAN) trained on large-scale natural images. As shown in Fig. 1, the deep generative prior (DGP) provides compelling results to restore missing semantics, e.g., color, patch, resolution, of various degraded images. It also enables diverse image manipulation including random jittering, image morphing, and category transfer. Such highly flexible restoration and manipulation are made possible through relaxing the assumption of existing GAN-inversion methods, which tend to fix the generator. Notably, we allow the generator to be fine-tuned on-the-fly in a progressive manner regularized by feature distance obtained by the discriminator in GAN. We show that these easy-to-implement and practical changes help preserve the reconstruction to remain in the manifold of nature image, and thus lead to more precise and faithful reconstruction for real images. Code is available at https://github.com/XingangPan/deepgenerative-prior.

Generative models learned from training using deep learning methods can be used as priors in inverse under-determined inverse problems, including imaging from sparse set of measurements. In this paper, we present a novel hierarchical deep-generative model MrSARP for SAR imagery that can synthesize SAR images of a target at different resolutions jointly. MrSARP is trained in conjunction with a critic that scores multi resolution images jointly to decide if they are realistic images of a target at different resolutions. We show how this deep generative model can be used to retrieve the high spatial resolution image from low resolution images of the same target. The cost function of the generator is modified to improve its capability to retrieve the input parameters for a given set of resolution images. We evaluate the model's performance using the three standard error metrics used for evaluating super-resolution performance on simulated data and compare it to upsampling and sparsity based image sharpening approaches.

Our goal with this survey is to provide an overview of the state of the art deep learning technologies for face generation and editing. We will cover popular latest architectures and discuss key ideas that make them work, such as inversion, latent representation, loss functions, training procedures, editing methods, and cross domain style transfer. We particularly focus on GAN-based architectures that have culminated in the StyleGAN approaches, which allow generation of high-quality face images and offer rich interfaces for controllable semantics editing and preserving photo quality. We aim to provide an entry point into the field for readers that have basic knowledge about the field of deep learning and are looking for an accessible introduction and overview.

Image super-resolution is a one-to-many problem, but most deep-learning based methods only provide one single solution to this problem. In this work, we tackle the problem of diverse super-resolution by reusing VD-VAE, a state-of-the art variational autoencoder (VAE). We find that the hierarchical latent representation learned by VD-VAE naturally separates the image low-frequency information, encoded in the latent groups at the top of the hierarchy, from the image high-frequency details, determined by the latent groups at the bottom of the latent hierarchy. Starting from this observation, we design a super-resolution model exploiting the specific structure of VD-VAE latent space. Specifically, we train an encoder to encode low-resolution images in the subset of VD-VAE latent space encoding the low-frequency information, and we combine this encoder with VD-VAE generative model to sample diverse super-resolved version of a low-resolution input. We demonstrate the ability of our method to generate diverse solutions to the super-resolution problem on face super-resolution with upsampling factors x4, x8, and x16.

The Bayesian approach to solving inverse problems relies on the choice of a prior. This critical ingredient allows the formulation of expert knowledge or physical constraints in a probabilistic fashion and plays an important role for the success of the inference. Recently, Bayesian inverse problems were solved using generative models as highly informative priors. Generative models are a popular tool in machine learning to generate data whose properties closely resemble those of a given database. Typically, the generated distribution of data is embedded in a low-dimensional manifold. For the inverse problem, a generative model is trained on a database that reflects the properties of the sought solution, such as typical structures of the tissue in the human brain in magnetic resonance (MR) imaging. The inference is carried out in the low-dimensional manifold determined by the generative model which strongly reduces the dimensionality of the inverse problem. However, this proceeding produces a posterior that admits no Lebesgue density in the actual variables and the accuracy reached can strongly depend on the quality of the generative model. For linear Gaussian models we explore an alternative Bayesian inference based on probabilistic generative models which is carried out in the original high-dimensional space. A Laplace approximation is employed to analytically derive the required prior probability density function induced by the generative model. Properties of the resulting inference are investigated. Specifically, we show that derived Bayes estimates are consistent, in contrast to the approach employing the low-dimensional manifold of the generative model. The MNIST data set is used to construct numerical experiments which confirm our theoretical findings.

反事实可以以人类的可解释方式解释神经网络的分类决策。我们提出了一种简单但有效的方法来产生这种反事实。更具体地说，我们执行合适的差异坐标转换，然后在这些坐标中执行梯度上升，以查找反事实，这些反事实是由置信度良好的指定目标类别分类的。我们提出了两种方法来利用生成模型来构建完全或大约差异的合适坐标系。我们使用Riemannian差异几何形状分析了生成过程，并使用各种定性和定量测量方法验证了生成的反事实质量。

已经显示了生成的对抗网络（GaN）的潜在空间在某些子空间内编码丰富的语义。为了识别这些子空间，研究人员通常从合成数据的集合分析统计信息，并且所识别的子空间倾向于在全局控制图像属性（即，操纵属性导致整个图像的变化）。相比之下，这项工作引入了低秩的子空间，使得GaN生成更精确地控制。具体地，给定任意图像和一个感兴趣区域（例如，面部图像的眼睛），我们设法将潜在空间与雅各比矩阵相关联，然后使用低秩分解来发现可转向潜在子空间。我们的方法有三种可区分优势，可以恰当地称为低利纳诺。首先，与现有工作中的分析算法相比，我们的雅各比人的低级别分解能够找到属性歧管的低维表示，使图像编辑更精确和可控。其次，低级别分子化自然地产生空间的属性，使得在其内移动潜在的代码仅影响感兴趣的外部区域。因此，可以通过将属性向量投影到空空间中来简单地实现本地图像编辑，而不依赖于现有方法所做的空间掩模。第三，我们的方法可以从一个图像中鲁布布地与本地区域一起使用，以进行分析，但概括到其他图像，在实践中易于使用。关于各种数据集培训的最先进的GaN模型（包括Stylegan2和Biggan）的大量实验证明了我们的LowRankaN的有效性。