Neural style transfer is a deep learning technique that produces an unprecedentedly rich style transfer from a style image to a content image and is particularly impressive when it comes to transferring style from a painting to an image. It was originally achieved by solving an optimization problem to match the global style statistics of the style image while preserving the local geometric features of the content image. The two main drawbacks of this original approach is that it is computationally expensive and that the resolution of the output images is limited by high GPU memory requirements. Many solutions have been proposed to both accelerate neural style transfer and increase its resolution, but they all compromise the quality of the produced images. Indeed, transferring the style of a painting is a complex task involving features at different scales, from the color palette and compositional style to the fine brushstrokes and texture of the canvas. This paper provides a solution to solve the original global optimization for ultra-high resolution images, enabling multiscale style transfer at unprecedented image sizes. This is achieved by spatially localizing the computation of each forward and backward passes through the VGG network. Extensive qualitative and quantitative comparisons show that our method produces a style transfer of unmatched quality for such high resolution painting styles.
translated by 谷歌翻译
我们提出了一个极其简单的超分辨率样式转移框架,称为URST,以灵活地处理任意的高分辨率图像(例如,10000x10000像素)第一次转移。由于在处理超高分辨率图像时,由于巨大的内存成本和小行程大小,大多数现有最先进的方法将降低。 URST完全避免了由超高分辨率图像引起的内存问题(1)将图像划分为小块和(2)与新颖的缩略图实例归一化(TIN)执行修补程序样式传输。具体而言,TIN可以提取缩略图功能的归一化统计信息,并将它们应用于小补丁,确保不同补丁之间的风格一致性。总的来说,与现有技术相比,URST框架有三个优点。 (1)我们将输入图像分为小补丁并采用锡,成功传输图像样式,具有任意的高分辨率。 (2)实验表明,我们的URST超越了现有的SOTA方法对超高分辨率图像,从提高行程大小的提出的中风感知损失的有效性中受益。 (3)我们的URST可以轻松插入大多数现有的样式转移方法,即使在没有培训的情况下也直接提高他们的性能。代码可在https://git.io/urst上获得。
translated by 谷歌翻译
Gatys et al. recently introduced a neural algorithm that renders a content image in the style of another image, achieving so-called style transfer. However, their framework requires a slow iterative optimization process, which limits its practical application. Fast approximations with feed-forward neural networks have been proposed to speed up neural style transfer. Unfortunately, the speed improvement comes at a cost: the network is usually tied to a fixed set of styles and cannot adapt to arbitrary new styles. In this paper, we present a simple yet effective approach that for the first time enables arbitrary style transfer in real-time. At the heart of our method is a novel adaptive instance normalization (AdaIN) layer that aligns the mean and variance of the content features with those of the style features. Our method achieves speed comparable to the fastest existing approach, without the restriction to a pre-defined set of styles. In addition, our approach allows flexible user controls such as content-style trade-off, style interpolation, color & spatial controls, all using a single feed-forward neural network.
translated by 谷歌翻译
任意样式转移生成了艺术图像,该图像仅使用一个训练有素的网络结合了内容图像的结构和艺术风格的结合。此方法中使用的图像表示包含内容结构表示和样式模式表示形式,这通常是预训练的分类网络中高级表示的特征表示。但是,传统的分类网络是为分类而设计的,该分类通常集中在高级功能上并忽略其他功能。结果,风格化的图像在整个图像中均匀地分布了样式元素,并使整体图像结构无法识别。为了解决这个问题,我们通过结合全球和局部损失,引入了一种新型的任意风格转移方法,并通过结构增强。局部结构细节由LapStyle表示,全局结构由图像深度控制。实验结果表明,与其他最新方法相比,我们的方法可以在几个常见数据集中生成具有令人印象深刻的视觉效果的更高质量图像。
translated by 谷歌翻译
We consider image transformation problems, where an input image is transformed into an output image. Recent methods for such problems typically train feed-forward convolutional neural networks using a per-pixel loss between the output and ground-truth images. Parallel work has shown that high-quality images can be generated by defining and optimizing perceptual loss functions based on high-level features extracted from pretrained networks. We combine the benefits of both approaches, and propose the use of perceptual loss functions for training feed-forward networks for image transformation tasks. We show results on image style transfer, where a feed-forward network is trained to solve the optimization problem proposed by Gatys et al in real-time. Compared to the optimization-based method, our network gives similar qualitative results but is three orders of magnitude faster. We also experiment with single-image super-resolution, where replacing a per-pixel loss with a perceptual loss gives visually pleasing results.
translated by 谷歌翻译
现有的神经样式传输方法需要参考样式图像来将样式图像的纹理信息传输到内容图像。然而,在许多实际情况中,用户可能没有参考样式图像,但仍然有兴趣通过想象它们来传输样式。为了处理此类应用程序,我们提出了一个新的框架,它可以实现样式转移`没有'风格图像,但仅使用所需风格的文本描述。使用预先训练的文本图像嵌入模型的剪辑,我们仅通过单个文本条件展示了内容图像样式的调制。具体而言,我们提出了一种针对现实纹理传输的多视图增强的修补程序文本图像匹配丢失。广泛的实验结果证实了具有反映语义查询文本的现实纹理的成功图像风格转移。
translated by 谷歌翻译
我们提出了一种将任意样式图像的艺术特征转移到3D场景的方法。在点云或网格上执行3D风格的先前方法对复杂的现实世界场景的几何重建错误敏感。取而代之的是,我们建议对更健壮的辐射场字段表示。我们发现,常用的基于克矩阵的损失倾向于在没有忠实笔触的情况下产生模糊的结果,并引入了最近的基于邻居的损失,该损失非常有效地捕获样式的细节,同时保持多视图一致性。我们还提出了一种新颖的递延后传播方法,以使用在全分辨率渲染图像上定义的样式损失来优化记忆密集型辐射场。我们广泛的评估表明,我们的方法通过产生与样式图像更相似的艺术外观来优于基线。请检查我们的项目页面以获取视频结果和开源实现:https://www.cs.cornell.edu/projects/arf/。
translated by 谷歌翻译
This paper proposes Markovian Generative Adversarial Networks (MGANs), a method for training generative neural networks for efficient texture synthesis. While deep neural network approaches have recently demonstrated remarkable results in terms of synthesis quality, they still come at considerable computational costs (minutes of run-time for low-res images). Our paper addresses this efficiency issue. Instead of a numerical deconvolution in previous work, we precompute a feedforward, strided convolutional network that captures the feature statistics of Markovian patches and is able to directly generate outputs of arbitrary dimensions. Such network can directly decode brown noise to realistic texture, or photos to artistic paintings. With adversarial training, we obtain quality comparable to recent neural texture synthesis methods. As no optimization is required any longer at generation time, our run-time performance (0.25M pixel images at 25Hz) surpasses previous neural texture synthesizers by a significant margin (at least 500 times faster). We apply this idea to texture synthesis, style transfer, and video stylization.
translated by 谷歌翻译
在本文中,我们介绍了纹理改革器,一个快速和通用的神经基础框架,用于使用用户指定的指导进行交互式纹理传输。挑战在三个方面:1)任务的多样性,2)引导图的简单性,以及3)执行效率。为了解决这些挑战,我们的主要思想是使用由i)全球视图结构对准阶段,ii)局部视图纹理细化阶段和III)的新的前馈多视图和多级合成程序。效果增强阶段用相干结构合成高质量结果,并以粗略的方式进行细纹细节。此外,我们还介绍了一种新颖的无学习视图特定的纹理改革(VSTR)操作,具有新的语义地图指导策略,以实现更准确的语义引导和结构保存的纹理传输。关于各种应用场景的实验结果展示了我们框架的有效性和优越性。并与最先进的交互式纹理转移算法相比,它不仅可以实现更高的质量结果,而且更加显着,也是更快的2-5个数量级。代码可在https://github.com/endywon/texture --reformer中找到。
translated by 谷歌翻译
任意神经风格转移是一个重要的主题,具有研究价值和工业应用前景,该主题旨在使用另一个样式呈现一个图像的结构。最近的研究已致力于任意风格转移(AST)的任务,以提高风格化质量。但是,关于AST图像的质量评估的探索很少,即使它可以指导不同算法的设计。在本文中,我们首先构建了一个新的AST图像质量评估数据库(AST-IQAD),该数据库包括150个内容样式的图像对以及由八种典型AST算法产生的相应的1200个风格化图像。然后,在我们的AST-IQAD数据库上进行了一项主观研究,该研究获得了三种主观评估(即内容保存(CP),样式相似(SR)和整体视觉(OV),该数据库获得了所有风格化图像的主观评分评分。 。为了定量测量AST图像的质量,我们提出了一个新的基于稀疏表示的图像质量评估度量(SRQE),该指标(SRQE)使用稀疏特征相似性来计算质量。 AST-IQAD的实验结果证明了该方法的优越性。数据集和源代码将在https://github.com/hangwei-chen/ast-iqad-srqe上发布
translated by 谷歌翻译
Single image super-resolution is the task of inferring a high-resolution image from a single low-resolution input. Traditionally, the performance of algorithms for this task is measured using pixel-wise reconstruction measures such as peak signal-to-noise ratio (PSNR) which have been shown to correlate poorly with the human perception of image quality. As a result, algorithms minimizing these metrics tend to produce over-smoothed images that lack highfrequency textures and do not look natural despite yielding high PSNR values.We propose a novel application of automated texture synthesis in combination with a perceptual loss focusing on creating realistic textures rather than optimizing for a pixelaccurate reproduction of ground truth images during training. By using feed-forward fully convolutional neural networks in an adversarial training setting, we achieve a significant boost in image quality at high magnification ratios. Extensive experiments on a number of datasets show the effectiveness of our approach, yielding state-of-the-art results in both quantitative and qualitative benchmarks.
translated by 谷歌翻译
最新的馈送前向神经方法的任意图像样式转移主要使用的编码特征映射到其二阶统计数据,即线性转换内容图像的编码特征映射,以具有相同的均值和方差(或协方差)(或协方差)功能图。在这项工作中,我们将二阶统计特征匹配扩展到一般分布匹配,以理解图像的样式由接收场的响应分布表示表示。对于此概括,首先,我们提出了一个新的特征转换层,该层与内容图像的特征映射分布完全匹配到目标样式图像的特征图层。其次,我们分析了与我们的新功能变换层一致的最新样式损失,以训练一个解码器网络,该网络生成了从变换的功能映射传输图像的样式。根据我们的实验结果,证明使用我们的方法获得的风格化图像与所有现有样式测量中的目标样式图像更相似,而不会丢失内容清晰度。
translated by 谷歌翻译
Photorealistic style transfer aims to transfer the artistic style of an image onto an input image or video while keeping photorealism. In this paper, we think it's the summary statistics matching scheme in existing algorithms that leads to unrealistic stylization. To avoid employing the popular Gram loss, we propose a self-supervised style transfer framework, which contains a style removal part and a style restoration part. The style removal network removes the original image styles, and the style restoration network recovers image styles in a supervised manner. Meanwhile, to address the problems in current feature transformation methods, we propose decoupled instance normalization to decompose feature transformation into style whitening and restylization. It works quite well in ColoristaNet and can transfer image styles efficiently while keeping photorealism. To ensure temporal coherency, we also incorporate optical flow methods and ConvLSTM to embed contextual information. Experiments demonstrates that ColoristaNet can achieve better stylization effects when compared with state-of-the-art algorithms.
translated by 谷歌翻译
Photo-realistic style transfer aims at migrating the artistic style from an exemplar style image to a content image, producing a result image without spatial distortions or unrealistic artifacts. Impressive results have been achieved by recent deep models. However, deep neural network based methods are too expensive to run in real-time. Meanwhile, bilateral grid based methods are much faster but still contain artifacts like overexposure. In this work, we propose the \textbf{Adaptive ColorMLP (AdaCM)}, an effective and efficient framework for universal photo-realistic style transfer. First, we find the complex non-linear color mapping between input and target domain can be efficiently modeled by a small multi-layer perceptron (ColorMLP) model. Then, in \textbf{AdaCM}, we adopt a CNN encoder to adaptively predict all parameters for the ColorMLP conditioned on each input content and style image pair. Experimental results demonstrate that AdaCM can generate vivid and high-quality stylization results. Meanwhile, our AdaCM is ultrafast and can process a 4K resolution image in 6ms on one V100 GPU.
translated by 谷歌翻译
Arbitrary style transfer (AST) transfers arbitrary artistic styles onto content images. Despite the recent rapid progress, existing AST methods are either incapable or too slow to run at ultra-resolutions (e.g., 4K) with limited resources, which heavily hinders their further applications. In this paper, we tackle this dilemma by learning a straightforward and lightweight model, dubbed MicroAST. The key insight is to completely abandon the use of cumbersome pre-trained Deep Convolutional Neural Networks (e.g., VGG) at inference. Instead, we design two micro encoders (content and style encoders) and one micro decoder for style transfer. The content encoder aims at extracting the main structure of the content image. The style encoder, coupled with a modulator, encodes the style image into learnable dual-modulation signals that modulate both intermediate features and convolutional filters of the decoder, thus injecting more sophisticated and flexible style signals to guide the stylizations. In addition, to boost the ability of the style encoder to extract more distinct and representative style signals, we also introduce a new style signal contrastive loss in our model. Compared to the state of the art, our MicroAST not only produces visually superior results but also is 5-73 times smaller and 6-18 times faster, for the first time enabling super-fast (about 0.5 seconds) AST at 4K ultra-resolutions. Code is available at https://github.com/EndyWon/MicroAST.
translated by 谷歌翻译
STYLE TRANSED引起了大量的关注,因为它可以在保留图像结构的同时将给定图像更改为一个壮观的艺术风格。然而,常规方法容易丢失图像细节,并且在风格转移期间倾向于产生令人不快的伪影。在本文中,为了解决这些问题,提出了一种具有目标特征调色板的新颖艺术程式化方法,可以准确地传递关键特征。具体而言,我们的方法包含两个模块,即特征调色板组成(FPC)和注意着色(AC)模块。 FPC模块基于K-means群集捕获代表特征,并生成特征目标调色板。以下AC模块计算内容和样式图像之间的注意力映射,并根据注意力映射和目标调色板传输颜色和模式。这些模块使提出的程式化能够专注于关键功能并生成合理的传输图像。因此,所提出的方法的贡献是提出一种新的深度学习的样式转移方法和当前目标特征调色板和注意着色模块,并通过详尽的消融研究提供对所提出的方法的深入分析和洞察。定性和定量结果表明,我们的程式化图像具有最先进的性能,具有保护核心结构和内容图像的细节。
translated by 谷歌翻译
基于图像的艺术渲染可以使用算法图像过滤合成各种表达式。与基于深度学习的方法相反,这些基于启发式的过滤技术可以在高分辨率图像上运行,可以解释,并且可以根据各个设计方面进行参数化。但是,适应或扩展这些技术生产新样式通常是一项乏味且容易出错的任务,需要专家知识。我们提出了一个新的范式来减轻此问题:实现算法图像过滤技术作为可区分的操作,可以学习与某些参考样式一致的参数化。为此,我们提出了明智的,这是一个基于示例的图像处理系统,可以在公共框架内处理多种风格化技术,例如水彩,油或卡通风格。通过训练全局和本地滤波器参数化的参数预测网络,我们可以同时适应参考样式和图像内容,例如增强面部特征。我们的方法可以在样式转移框架中进行优化,也可以在用于图像到图像翻译的生成对流设置中学习。我们证明,共同训练XDOG滤波器和用于后处理的CNN可以与基于GAN的最新方法获得可比的结果。
translated by 谷歌翻译
我们呈现SeveryGan,一种能够从单个输入示例自动生成砖纹理映射的方法。与大多数现有方法相比,专注于解决合成问题,我们的工作同时解决问题,合成和涤纶性。我们的关键思想是认识到,通过越野落扩展技术训练的生成网络内的潜伏空间产生具有在接缝交叉点的连续性的输出,然后可以通过裁剪中心区域进入彩色图像。由于不是潜在空间的每个值都有有效的来产生高质量的输出,因此我们利用鉴别者作为能够在采样过程中识别无伪纹理的感知误差度量。此外,与之前的深度纹理合成的工作相比,我们的模型设计和优化,以便使用多层纹理表示,使由多个地图组成的纹理,例如Albedo,法线等。我们广泛地测试网络的设计选择架构,丢失功能和采样参数。我们在定性和定量上展示我们的方法优于以前的方法和适用于不同类型的纹理。
translated by 谷歌翻译
许多传统的计算机视觉算法通过要求生成图像中的每个补丁都与训练图像中的补丁相似,反之亦然。最近,这种经典方法已被补丁歧视者所取代。对抗方法避免了找到最近的贴片邻居的计算负担,但通常需要很长的训练时间,并且可能无法匹配斑块的分布。在本文中,我们利用了最近开发的切片瓦斯坦距离,并开发了一种算法,该算法明确有效地最大程度地减少了两个图像中斑块分布之间的距离。我们的方法在概念上很简单,不需要培训,并且可以在几行代码中实现。在许多图像生成任务上,我们表明我们的结果通常优于单片图像,不需要训练,并且可以在几秒钟内生成高质量的图像。我们的实施可从https://github.com/ariel415el/gpdm获得
translated by 谷歌翻译
Image harmonization aims to produce visually harmonious composite images by adjusting the foreground appearance to be compatible with the background. When the composite image has photographic foreground and painterly background, the task is called painterly image harmonization. There are only few works on this task, which are either time-consuming or weak in generating well-harmonized results. In this work, we propose a novel painterly harmonization network consisting of a dual-domain generator and a dual-domain discriminator, which harmonizes the composite image in both spatial domain and frequency domain. The dual-domain generator performs harmonization by using AdaIn modules in the spatial domain and our proposed ResFFT modules in the frequency domain. The dual-domain discriminator attempts to distinguish the inharmonious patches based on the spatial feature and frequency feature of each patch, which can enhance the ability of generator in an adversarial manner. Extensive experiments on the benchmark dataset show the effectiveness of our method. Our code and model are available at https://github.com/bcmi/PHDNet-Painterly-Image-Harmonization.
translated by 谷歌翻译