This paper proposes Markovian Generative Adversarial Networks (MGANs), a method for training generative neural networks for efficient texture synthesis. While deep neural network approaches have recently demonstrated remarkable results in terms of synthesis quality, they still come at considerable computational costs (minutes of run-time for low-res images). Our paper addresses this efficiency issue. Instead of a numerical deconvolution in previous work, we precompute a feedforward, strided convolutional network that captures the feature statistics of Markovian patches and is able to directly generate outputs of arbitrary dimensions. Such network can directly decode brown noise to realistic texture, or photos to artistic paintings. With adversarial training, we obtain quality comparable to recent neural texture synthesis methods. As no optimization is required any longer at generation time, our run-time performance (0.25M pixel images at 25Hz) surpasses previous neural texture synthesizers by a significant margin (at least 500 times faster). We apply this idea to texture synthesis, style transfer, and video stylization.

Gatys et al. recently introduced a neural algorithm that renders a content image in the style of another image, achieving so-called style transfer. However, their framework requires a slow iterative optimization process, which limits its practical application. Fast approximations with feed-forward neural networks have been proposed to speed up neural style transfer. Unfortunately, the speed improvement comes at a cost: the network is usually tied to a fixed set of styles and cannot adapt to arbitrary new styles. In this paper, we present a simple yet effective approach that for the first time enables arbitrary style transfer in real-time. At the heart of our method is a novel adaptive instance normalization (AdaIN) layer that aligns the mean and variance of the content features with those of the style features. Our method achieves speed comparable to the fastest existing approach, without the restriction to a pre-defined set of styles. In addition, our approach allows flexible user controls such as content-style trade-off, style interpolation, color & spatial controls, all using a single feed-forward neural network.

我们呈现SeveryGan，一种能够从单个输入示例自动生成砖纹理映射的方法。与大多数现有方法相比，专注于解决合成问题，我们的工作同时解决问题，合成和涤纶性。我们的关键思想是认识到，通过越野落扩展技术训练的生成网络内的潜伏空间产生具有在接缝交叉点的连续性的输出，然后可以通过裁剪中心区域进入彩色图像。由于不是潜在空间的每个值都有有效的来产生高质量的输出，因此我们利用鉴别者作为能够在采样过程中识别无伪纹理的感知误差度量。此外，与之前的深度纹理合成的工作相比，我们的模型设计和优化，以便使用多层纹理表示，使由多个地图组成的纹理，例如Albedo，法线等。我们广泛地测试网络的设计选择架构，丢失功能和采样参数。我们在定性和定量上展示我们的方法优于以前的方法和适用于不同类型的纹理。

Single image super-resolution is the task of inferring a high-resolution image from a single low-resolution input. Traditionally, the performance of algorithms for this task is measured using pixel-wise reconstruction measures such as peak signal-to-noise ratio (PSNR) which have been shown to correlate poorly with the human perception of image quality. As a result, algorithms minimizing these metrics tend to produce over-smoothed images that lack highfrequency textures and do not look natural despite yielding high PSNR values.We propose a novel application of automated texture synthesis in combination with a perceptual loss focusing on creating realistic textures rather than optimizing for a pixelaccurate reproduction of ground truth images during training. By using feed-forward fully convolutional neural networks in an adversarial training setting, we achieve a significant boost in image quality at high magnification ratios. Extensive experiments on a number of datasets show the effectiveness of our approach, yielding state-of-the-art results in both quantitative and qualitative benchmarks.

基于补丁的方法和深度网络已经采用了解决图像染色问题，具有自己的优势和劣势。基于补丁的方法能够通过从未遮盖区域搜索最近的邻居修补程序来恢复具有高质量纹理的缺失区域。但是，这些方法在恢复大缺失区域时会带来问题内容。另一方面，深度网络显示有希望的成果完成大区域。尽管如此，结果往往缺乏类似周围地区的忠诚和尖锐的细节。通过汇集两个范式中，我们提出了一种新的深度染色框架，其中纹理生成是由从未掩蔽区域提取的补丁样本的纹理记忆引导的。该框架具有一种新颖的设计，允许使用深度修复网络训练纹理存储器检索。此外，我们还介绍了贴片分配损失，以鼓励高质量的贴片合成。所提出的方法在三个具有挑战性的图像基准测试中，即地位，Celeba-HQ和巴黎街道视图数据集来说，该方法显示出质量和定量的卓越性能。

现有的神经样式传输方法需要参考样式图像来将样式图像的纹理信息传输到内容图像。然而，在许多实际情况中，用户可能没有参考样式图像，但仍然有兴趣通过想象它们来传输样式。为了处理此类应用程序，我们提出了一个新的框架，它可以实现样式转移`没有'风格图像，但仅使用所需风格的文本描述。使用预先训练的文本图像嵌入模型的剪辑，我们仅通过单个文本条件展示了内容图像样式的调制。具体而言，我们提出了一种针对现实纹理传输的多视图增强的修补程序文本图像匹配丢失。广泛的实验结果证实了具有反映语义查询文本的现实纹理的成功图像风格转移。

任意样式转移生成了艺术图像，该图像仅使用一个训练有素的网络结合了内容图像的结构和艺术风格的结合。此方法中使用的图像表示包含内容结构表示和样式模式表示形式，这通常是预训练的分类网络中高级表示的特征表示。但是，传统的分类网络是为分类而设计的，该分类通常集中在高级功能上并忽略其他功能。结果，风格化的图像在整个图像中均匀地分布了样式元素，并使整体图像结构无法识别。为了解决这个问题，我们通过结合全球和局部损失，引入了一种新型的任意风格转移方法，并通过结构增强。局部结构细节由LapStyle表示，全局结构由图像深度控制。实验结果表明，与其他最新方法相比，我们的方法可以在几个常见数据集中生成具有令人印象深刻的视觉效果的更高质量图像。

最近的方法（例如材料gan）已使用无条件的gan来生成每像素材料图，或作为从输入照片重建材料之前的材料。这些模型可以生成各种随机材料外观，但没有任何将生成材料限制为特定类别或控制生成材料的粗体结构的机制，例如砖墙上的精确砖布局。此外，从单个输入照片中重建的材料通常具有伪像，并且通常不可易换，这限制了它们在实际内容创建管道中的使用。我们提出了Tilegen，这是一种针对SVBRDFS的生成模型，该模型特定于材料类别，始终可易换，并且在提供的输入结构模式上有条件。 Tilegen是Stylegan的变体，其架构经过修改以始终生成可易于的（周期性）材料图。除了标准的“样式”潜在代码外，Tilegen还可以选择拍摄条件图像，从而使用户直接控制材料的主要空间（和可选的颜色）功能。例如，在砖块中，用户可以指定砖布局和砖块，或者在皮革材料中，皱纹和褶皱的位置。我们的反渲染方法可以通过优化找到一种材料，从而感知到单个目标照片。这种重建也可以以用户提供的模式为条件。所得的材料是可拆卸的，可以大于目标图像，并且可以通过改变条件来编辑。

本文的目标是对面部素描合成（FSS）问题进行全面的研究。然而，由于获得了手绘草图数据集的高成本，因此缺乏完整的基准，用于评估过去十年的FSS算法的开发。因此，我们首先向FSS引入高质量的数据集，名为FS2K，其中包括2,104个图像素描对，跨越三种类型的草图样式，图像背景，照明条件，肤色和面部属性。 FS2K与以前的FSS数据集不同于难度，多样性和可扩展性，因此应促进FSS研究的进展。其次，我们通过调查139种古典方法，包括34个手工特征的面部素描合成方法，37个一般的神经式传输方法，43个深映像到图像翻译方法，以及35个图像 - 素描方法。此外，我们详细说明了现有的19个尖端模型的综合实验。第三，我们为FSS提供了一个简单的基准，名为FSGAN。只有两个直截了当的组件，即面部感知屏蔽和风格矢量扩展，FSGAN将超越所提出的FS2K数据集的所有先前最先进模型的性能，通过大边距。最后，我们在过去几年中汲取的经验教训，并指出了几个未解决的挑战。我们的开源代码可在https://github.com/dengpingfan/fsgan中获得。

Despite the breakthroughs in accuracy and speed of single image super-resolution using faster and deeper convolutional neural networks, one central problem remains largely unsolved: how do we recover the finer texture details when we super-resolve at large upscaling factors? The behavior of optimization-based super-resolution methods is principally driven by the choice of the objective function. Recent work has largely focused on minimizing the mean squared reconstruction error. The resulting estimates have high peak signal-to-noise ratios, but they are often lacking high-frequency details and are perceptually unsatisfying in the sense that they fail to match the fidelity expected at the higher resolution. In this paper, we present SRGAN, a generative adversarial network (GAN) for image superresolution (SR). To our knowledge, it is the first framework capable of inferring photo-realistic natural images for 4× upscaling factors. To achieve this, we propose a perceptual loss function which consists of an adversarial loss and a content loss. The adversarial loss pushes our solution to the natural image manifold using a discriminator network that is trained to differentiate between the super-resolved images and original photo-realistic images. In addition, we use a content loss motivated by perceptual similarity instead of similarity in pixel space. Our deep residual network is able to recover photo-realistic textures from heavily downsampled images on public benchmarks. An extensive mean-opinion-score (MOS) test shows hugely significant gains in perceptual quality using SRGAN. The MOS scores obtained with SRGAN are closer to those of the original high-resolution images than to those obtained with any state-of-the-art method.

STYLE TRANSED引起了大量的关注，因为它可以在保留图像结构的同时将给定图像更改为一个壮观的艺术风格。然而，常规方法容易丢失图像细节，并且在风格转移期间倾向于产生令人不快的伪影。在本文中，为了解决这些问题，提出了一种具有目标特征调色板的新颖艺术程式化方法，可以准确地传递关键特征。具体而言，我们的方法包含两个模块，即特征调色板组成（FPC）和注意着色（AC）模块。 FPC模块基于K-means群集捕获代表特征，并生成特征目标调色板。以下AC模块计算内容和样式图像之间的注意力映射，并根据注意力映射和目标调色板传输颜色和模式。这些模块使提出的程式化能够专注于关键功能并生成合理的传输图像。因此，所提出的方法的贡献是提出一种新的深度学习的样式转移方法和当前目标特征调色板和注意着色模块，并通过详尽的消融研究提供对所提出的方法的深入分析和洞察。定性和定量结果表明，我们的程式化图像具有最先进的性能，具有保护核心结构和内容图像的细节。

许多传统的计算机视觉算法通过要求生成图像中的每个补丁都与训练图像中的补丁相似，反之亦然。最近，这种经典方法已被补丁歧视者所取代。对抗方法避免了找到最近的贴片邻居的计算负担，但通常需要很长的训练时间，并且可能无法匹配斑块的分布。在本文中，我们利用了最近开发的切片瓦斯坦距离，并开发了一种算法，该算法明确有效地最大程度地减少了两个图像中斑块分布之间的距离。我们的方法在概念上很简单，不需要培训，并且可以在几行代码中实现。在许多图像生成任务上，我们表明我们的结果通常优于单片图像，不需要训练，并且可以在几秒钟内生成高质量的图像。我们的实施可从https://github.com/ariel415el/gpdm获得

最近的研究表明，通用风格转移的成功取得了巨大的成功，将任意视觉样式转移到内容图像中。但是，现有的方法遭受了审美的非现实主义问题，该问题引入了不和谐的模式和明显的人工制品，从而使结果很容易从真实的绘画中发现。为了解决这一限制，我们提出了一种新颖的美学增强风格转移方法，可以在美学上为任意风格产生更现实和令人愉悦的结果。具体而言，我们的方法引入了一种审美歧视者，以从大量的艺术家创造的绘画中学习通用的人类自愿美学特征。然后，合并了美学特征，以通过新颖的美学感知样式（AESSA）模块来增强样式转移过程。这样的AESSA模块使我们的Aesust能够根据样式图像的全局美学通道分布和内容图像的局部语义空间分布有效而灵活地集成样式模式。此外，我们还开发了一种新的两阶段转移培训策略，并通过两种审美正规化来更有效地训练我们的模型，从而进一步改善风格化的性能。广泛的实验和用户研究表明，我们的方法比艺术的状态综合了美学上更加和谐和现实的结果，从而大大缩小了真正的艺术家创造的绘画的差异。我们的代码可在https://github.com/endywon/aesust上找到。

The recent work of Gatys et al., who characterized the style of an image by the statistics of convolutional neural network filters, ignited a renewed interest in the texture generation and image stylization problems. While their image generation technique uses a slow optimization process, recently several authors have proposed to learn generator neural networks that can produce similar outputs in one quick forward pass. While generator networks are promising, they are still inferior in visual quality and diversity compared to generation-by-optimization. In this work, we advance them in two significant ways. First, we introduce an instance normalization module to replace batch normalization with significant improvements to the quality of image stylization. Second, we improve diversity by introducing a new learning formulation that encourages generators to sample unbiasedly from the Julesz texture ensemble, which is the equivalence class of all images characterized by certain filter responses. Together, these two improvements take feed forward texture synthesis and image stylization much closer to the quality of generation-via-optimization, while retaining the speed advantage.

在本文中，我们介绍了纹理改革器，一个快速和通用的神经基础框架，用于使用用户指定的指导进行交互式纹理传输。挑战在三个方面：1）任务的多样性，2）引导图的简单性，以及3）执行效率。为了解决这些挑战，我们的主要思想是使用由i）全球视图结构对准阶段，ii）局部视图纹理细化阶段和III）的新的前馈多视图和多级合成程序。效果增强阶段用相干结构合成高质量结果，并以粗略的方式进行细纹细节。此外，我们还介绍了一种新颖的无学习视图特定的纹理改革（VSTR）操作，具有新的语义地图指导策略，以实现更准确的语义引导和结构保存的纹理传输。关于各种应用场景的实验结果展示了我们框架的有效性和优越性。并与最先进的交互式纹理转移算法相比，它不仅可以实现更高的质量结果，而且更加显着，也是更快的2-5个数量级。代码可在https://github.com/endywon/texture --reformer中找到。

Our goal with this survey is to provide an overview of the state of the art deep learning technologies for face generation and editing. We will cover popular latest architectures and discuss key ideas that make them work, such as inversion, latent representation, loss functions, training procedures, editing methods, and cross domain style transfer. We particularly focus on GAN-based architectures that have culminated in the StyleGAN approaches, which allow generation of high-quality face images and offer rich interfaces for controllable semantics editing and preserving photo quality. We aim to provide an entry point into the field for readers that have basic knowledge about the field of deep learning and are looking for an accessible introduction and overview.

Figure 1: Example inpainting results of our method on images of natural scene, face and texture. Missing regions are shown in white. In each pair, the left is input image and right is the direct output of our trained generative neural networks without any post-processing.

我们提出了一种将任意样式图像的艺术特征转移到3D场景的方法。在点云或网格上执行3D风格的先前方法对复杂的现实世界场景的几何重建错误敏感。取而代之的是，我们建议对更健壮的辐射场字段表示。我们发现，常用的基于克矩阵的损失倾向于在没有忠实笔触的情况下产生模糊的结果，并引入了最近的基于邻居的损失，该损失非常有效地捕获样式的细节，同时保持多视图一致性。我们还提出了一种新颖的递延后传播方法，以使用在全分辨率渲染图像上定义的样式损失来优化记忆密集型辐射场。我们广泛的评估表明，我们的方法通过产生与样式图像更相似的艺术外观来优于基线。请检查我们的项目页面以获取视频结果和开源实现：https：//www.cs.cornell.edu/projects/arf/。

Photo-realistic style transfer aims at migrating the artistic style from an exemplar style image to a content image, producing a result image without spatial distortions or unrealistic artifacts. Impressive results have been achieved by recent deep models. However, deep neural network based methods are too expensive to run in real-time. Meanwhile, bilateral grid based methods are much faster but still contain artifacts like overexposure. In this work, we propose the \textbf{Adaptive ColorMLP (AdaCM)}, an effective and efficient framework for universal photo-realistic style transfer. First, we find the complex non-linear color mapping between input and target domain can be efficiently modeled by a small multi-layer perceptron (ColorMLP) model. Then, in \textbf{AdaCM}, we adopt a CNN encoder to adaptively predict all parameters for the ColorMLP conditioned on each input content and style image pair. Experimental results demonstrate that AdaCM can generate vivid and high-quality stylization results. Meanwhile, our AdaCM is ultrafast and can process a 4K resolution image in 6ms on one V100 GPU.

We consider image transformation problems, where an input image is transformed into an output image. Recent methods for such problems typically train feed-forward convolutional neural networks using a per-pixel loss between the output and ground-truth images. Parallel work has shown that high-quality images can be generated by defining and optimizing perceptual loss functions based on high-level features extracted from pretrained networks. We combine the benefits of both approaches, and propose the use of perceptual loss functions for training feed-forward networks for image transformation tasks. We show results on image style transfer, where a feed-forward network is trained to solve the optimization problem proposed by Gatys et al in real-time. Compared to the optimization-based method, our network gives similar qualitative results but is three orders of magnitude faster. We also experiment with single-image super-resolution, where replacing a per-pixel loss with a perceptual loss gives visually pleasing results.