我们提出了一种将任意样式图像的艺术特征转移到3D场景的方法。在点云或网格上执行3D风格的先前方法对复杂的现实世界场景的几何重建错误敏感。取而代之的是，我们建议对更健壮的辐射场字段表示。我们发现，常用的基于克矩阵的损失倾向于在没有忠实笔触的情况下产生模糊的结果，并引入了最近的基于邻居的损失，该损失非常有效地捕获样式的细节，同时保持多视图一致性。我们还提出了一种新颖的递延后传播方法，以使用在全分辨率渲染图像上定义的样式损失来优化记忆密集型辐射场。我们广泛的评估表明，我们的方法通过产生与样式图像更相似的艺术外观来优于基线。请检查我们的项目页面以获取视频结果和开源实现：https：//www.cs.cornell.edu/projects/arf/。

本文提出了一种程式化的新型视图合成方法。将最新的风格化方法应用于新型视图框架上，通常由于缺乏跨视图一致性而引起抖动的伪像。因此，本文研究了3D场景样式，该风格为一致的新型视图综合提供了强烈的诱导偏置。具体而言，我们采用新兴的神经光辉领域（NERF）作为我们选择的3D场景表示，因为它们有能力为各种场景提供高质量的新颖观点。但是，由于从NERF呈现新颖的视图需要大量样品，因此训练风格化的NERF需要大量的GPU内存，这超出了现成的GPU容量。我们引入了一种新的培训方法，通过交替进行NERF和样式优化步骤来解决此问题。这样的方法使我们能够充分利用自己的硬件记忆能力以更高的分辨率生成图像，又采用更具表现力的图像样式传输方法。我们的实验表明，我们的方法生成了针对各种内容的风格化的NERF，包括室内，室外和动态场景，并综合具有跨视图一致性的高质量小说视图。

3D场景感性风格化旨在根据给定的样式图像从任意新颖的视图中生成光真逼真的图像，同时在从不同观点呈现时确保一致性。一些带有神经辐射场的现有风格化方法可以通过将样式图像的特征与多视图图像结合到训练3D场景来有效地预测风格化的场景。但是，这些方法生成了包含令人反感的伪影的新型视图图像。此外，他们无法为3D场景实现普遍的影迷风格化。因此，样式图像必须根据神经辐射场重新训练3D场景表示网络。我们提出了一个新颖的3D场景，逼真的风格转移框架来解决这些问题。它可以通过2D样式图像实现感性3D场景样式转移。我们首先预先训练了2D逼真的样式传输网络，该网络可以符合任何给定内容图像和样式图像之间的影片风格转移。然后，我们使用体素特征来优化3D场景并获得场景的几何表示。最后，我们共同优化了一个超级网络，以实现场景的逼真风格传输的任意样式图像。在转移阶段，我们使用预先训练的2D影视网络来限制3D场景中不同视图和不同样式图像的感性风格。实验结果表明，我们的方法不仅实现了任意样式图像的3D影像风格转移，而且还优于视觉质量和一致性方面的现有方法。项目页面：https：//semchan.github.io/upst_nerf。

通过隐式表示表示视觉信号（例如，基于坐标的深网）在许多视觉任务中都占了上风。这项工作探讨了一个新的有趣的方向：使用可以适用于各种2D和3D场景的广义方法训练风格化的隐式表示。我们对各种隐式函数进行了试点研究，包括基于2D坐标的表示，神经辐射场和签名距离函数。我们的解决方案是一个统一的隐式神经风化框架，称为INS。与Vanilla隐式表示相反，INS将普通隐式函数分解为样式隐式模块和内容隐式模块，以便从样式图像和输入场景中分别编码表示表示。然后，应用合并模块来汇总这些信息并合成样式化的输出。为了使3D场景中的几何形状进行正规化，我们提出了一种新颖的自我鉴定几何形状一致性损失，该损失保留了风格化场景的几何忠诚度。全面的实验是在多个任务设置上进行的，包括对复杂场景的新型综合，隐式表面的风格化以及使用MLP拟合图像。我们进一步证明，学到的表示不仅是连续的，而且在风格上都是连续的，从而导致不同样式之间毫不费力地插值，并以新的混合样式生成图像。请参阅我们的项目页面上的视频以获取更多查看综合结果：https：//zhiwenfan.github.io/ins。

Stylized view generation of scenes captured casually using a camera has received much attention recently. The geometry and appearance of the scene are typically captured as neural point sets or neural radiance fields in the previous work. An image stylization method is used to stylize the captured appearance by training its network jointly or iteratively with the structure capture network. The state-of-the-art SNeRF method trains the NeRF and stylization network in an alternating manner. These methods have high training time and require joint optimization. In this work, we present StyleTRF, a compact, quick-to-optimize strategy for stylized view generation using TensoRF. The appearance part is fine-tuned using sparse stylized priors of a few views rendered using the TensoRF representation for a few iterations. Our method thus effectively decouples style-adaption from view capture and is much faster than the previous methods. We show state-of-the-art results on several scenes used for this purpose.

Volumetric neural rendering methods like NeRF generate high-quality view synthesis results but are optimized per-scene leading to prohibitive reconstruction time. On the other hand, deep multi-view stereo methods can quickly reconstruct scene geometry via direct network inference. Point-NeRF combines the advantages of these two approaches by using neural 3D point clouds, with associated neural features, to model a radiance field. Point-NeRF can be rendered efficiently by aggregating neural point features near scene surfaces, in a ray marching-based rendering pipeline. Moreover, Point-NeRF can be initialized via direct inference of a pre-trained deep network to produce a neural point cloud; this point cloud can be finetuned to surpass the visual quality of NeRF with 30X faster training time. Point-NeRF can be combined with other 3D reconstruction methods and handles the errors and outliers in such methods via a novel pruning and growing mechanism. The experiments on the DTU, the NeRF Synthetics , the ScanNet and the Tanks and Temples datasets demonstrate Point-NeRF can surpass the existing methods and achieve the state-of-the-art results.

As a powerful representation of 3D scenes, the neural radiance field (NeRF) enables high-quality novel view synthesis from multi-view images. Stylizing NeRF, however, remains challenging, especially on simulating a text-guided style with both the appearance and the geometry altered simultaneously. In this paper, we present NeRF-Art, a text-guided NeRF stylization approach that manipulates the style of a pre-trained NeRF model with a simple text prompt. Unlike previous approaches that either lack sufficient geometry deformations and texture details or require meshes to guide the stylization, our method can shift a 3D scene to the target style characterized by desired geometry and appearance variations without any mesh guidance. This is achieved by introducing a novel global-local contrastive learning strategy, combined with the directional constraint to simultaneously control both the trajectory and the strength of the target style. Moreover, we adopt a weight regularization method to effectively suppress cloudy artifacts and geometry noises which arise easily when the density field is transformed during geometry stylization. Through extensive experiments on various styles, we demonstrate that our method is effective and robust regarding both single-view stylization quality and cross-view consistency. The code and more results can be found in our project page: https://cassiepython.github.io/nerfart/.

在移动摄影和AR / VR中，视觉内容创建刺激了飙升的兴趣。作为两个代表性任务的样式转移和单像3D摄影迄今为止独立发展。在本文中，我们在两者之间进行了联系，并解决了3D照片风格化的具有挑战性的任务 - 从单个图像中生成了一个任意映像的程式化的小说视图。我们的关键直观是，风格转移和视图综合必须为此任务共同建模。为此，我们提出了一个深入的模型，可以从场景的点云表示，从场景的点云表示，学习几何风格感知内容特征，从而导致跨视图一致的高质量风格化图像。此外，我们介绍了一种新颖的训练协议，以使学习仅使用2D图像。我们通过广泛的定性和定量研究展示了我们的方法的优越性，以及鉴于从2D图像资产的3D内容创建的需求不断增长，展示我们方法的关键应用。

Recent advances in neural radiance fields have enabled the high-fidelity 3D reconstruction of complex scenes for novel view synthesis. However, it remains underexplored how the appearance of such representations can be efficiently edited while maintaining photorealism. In this work, we present PaletteNeRF, a novel method for photorealistic appearance editing of neural radiance fields (NeRF) based on 3D color decomposition. Our method decomposes the appearance of each 3D point into a linear combination of palette-based bases (i.e., 3D segmentations defined by a group of NeRF-type functions) that are shared across the scene. While our palette-based bases are view-independent, we also predict a view-dependent function to capture the color residual (e.g., specular shading). During training, we jointly optimize the basis functions and the color palettes, and we also introduce novel regularizers to encourage the spatial coherence of the decomposition. Our method allows users to efficiently edit the appearance of the 3D scene by modifying the color palettes. We also extend our framework with compressed semantic features for semantic-aware appearance editing. We demonstrate that our technique is superior to baseline methods both quantitatively and qualitatively for appearance editing of complex real-world scenes.

我们在室内场景的网格重建时应用样式转移。这使得VR应用程序能够体验绘制艺术家风格的3D环境。风格传输通常在2D图像上运行，使网格挑战的程式化。当优化多种姿势时，风格化图案变得延伸并尺寸不一致。另一方面，存在基于模型的3D样式传输方法，其允许从稀疏图像集中的程式化，但是它们需要推动时间的网络。为此，我们优化了场景的重建网格的显式纹理，并从所有可用的输入图像共同地体现它。我们的深度和角度感知优化利用底层网格的表面正常和深度数据，为整个场景创建统一和一致的程式化。我们的实验表明，我们的方法为完整场景创造了锐利，详细的结果，而无需查看依赖文物。通过广泛的消融研究，我们表明所提出的3D意识使得风格转移能够应用于网格的3D域。我们的方法可用于实时与传统的渲染管道实时呈现风格化网格。

Existing 3D scene stylization methods employ an arbitrary style reference to transfer textures and colors as styles without establishing meaningful semantic correspondences. We present Reference-Based Non-Photorealistic Radiance Fields, i.e., Ref-NPR. It is a controllable scene stylization method utilizing radiance fields to stylize a 3D scene, with a single stylized 2D view taken as reference. To achieve decent results, we propose a ray registration process based on the stylized reference view to obtain pseudo-ray supervision in novel views, and exploit the semantic correspondence in content images to fill occluded regions with perceptually similar styles. Combining these operations, Ref-NPR generates non-photorealistic and continuous novel view sequences with a single reference while obtaining reasonable stylization in occluded regions. Experiments show that Ref-NPR significantly outperforms other scene and video stylization methods in terms of both visual quality and semantic correspondence. Code and data will be made publicly available.

b) MVS-NeRF no fine-tuning c) MVS-NeRF 6 min fine-tuning d) NeRF 5.1h optimization a) Source views SSIM:0.766 SSIM: 0.923 SSIM:0.924 * Equal contribution Research done when Anpei Chen was in a remote internship with UCSD.generalizable radiance field reconstruction. Moreover, if dense images are captured, our estimated radiance field representation can be easily fine-tuned; this leads to fast per-scene reconstruction with higher rendering quality and substantially less optimization time than NeRF.

https://video-nerf.github.io Figure 1. Our method takes a single casually captured video as input and learns a space-time neural irradiance field. (Top) Sample frames from the input video. (Middle) Novel view images rendered from textured meshes constructed from depth maps. (Bottom) Our results rendered from the proposed space-time neural irradiance field.

我们人类正在进入虚拟时代，确实想将动物带到虚拟世界中。然而，计算机生成的（CGI）毛茸茸的动物受到乏味的离线渲染的限制，更不用说交互式运动控制了。在本文中，我们提出了Artemis，这是一种新型的神经建模和渲染管道，用于生成具有外观和运动合成的清晰神经宠物。我们的Artemis可以实现互动运动控制，实时动画和毛茸茸的动物的照片真实渲染。我们的Artemis的核心是神经生成的（NGI）动物引擎，该动物发动机采用了有效的基于OCTREE的动物动画和毛皮渲染的代表。然后，该动画等同于基于显式骨骼翘曲的体素级变形。我们进一步使用快速的OCTREE索引和有效的体积渲染方案来生成外观和密度特征地图。最后，我们提出了一个新颖的阴影网络，以在外观和密度特征图中生成外观和不透明度的高保真细节。对于Artemis中的运动控制模块，我们将最新动物运动捕获方法与最近的神经特征控制方案相结合。我们引入了一种有效的优化方案，以重建由多视图RGB和Vicon相机阵列捕获的真实动物的骨骼运动。我们将所有捕获的运动馈送到神经角色控制方案中，以生成具有运动样式的抽象控制信号。我们将Artemis进一步整合到支持VR耳机的现有引擎中，提供了前所未有的沉浸式体验，用户可以与各种具有生动动作和光真实外观的虚拟动物进行紧密互动。我们可以通过https://haiminluo.github.io/publication/artemis/提供我们的Artemis模型和动态毛茸茸的动物数据集。

We present a method that achieves state-of-the-art results for synthesizing novel views of complex scenes by optimizing an underlying continuous volumetric scene function using a sparse set of input views. Our algorithm represents a scene using a fully-connected (nonconvolutional) deep network, whose input is a single continuous 5D coordinate (spatial location (x, y, z) and viewing direction (θ, φ)) and whose output is the volume density and view-dependent emitted radiance at that spatial location. We synthesize views by querying 5D coordinates along camera rays and use classic volume rendering techniques to project the output colors and densities into an image. Because volume rendering is naturally differentiable, the only input required to optimize our representation is a set of images with known camera poses. We describe how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrate results that outperform prior work on neural rendering and view synthesis. View synthesis results are best viewed as videos, so we urge readers to view our supplementary video for convincing comparisons.

综合照片 - 现实图像和视频是计算机图形的核心，并且是几十年的研究焦点。传统上，使用渲染算法（如光栅化或射线跟踪）生成场景的合成图像，其将几何形状和材料属性的表示为输入。统称，这些输入定义了实际场景和呈现的内容，并且被称为场景表示（其中场景由一个或多个对象组成）。示例场景表示是具有附带纹理的三角形网格（例如，由艺术家创建），点云（例如，来自深度传感器），体积网格（例如，来自CT扫描）或隐式曲面函数（例如，截短的符号距离）字段）。使用可分辨率渲染损耗的观察结果的这种场景表示的重建被称为逆图形或反向渲染。神经渲染密切相关，并将思想与经典计算机图形和机器学习中的思想相结合，以创建用于合成来自真实观察图像的图像的算法。神经渲染是朝向合成照片现实图像和视频内容的目标的跨越。近年来，我们通过数百个出版物显示了这一领域的巨大进展，这些出版物显示了将被动组件注入渲染管道的不同方式。这种最先进的神经渲染进步的报告侧重于将经典渲染原则与学习的3D场景表示结合的方法，通常现在被称为神经场景表示。这些方法的一个关键优势在于它们是通过设计的3D-一致，使诸如新颖的视点合成捕获场景的应用。除了处理静态场景的方法外，我们还涵盖了用于建模非刚性变形对象的神经场景表示...

View-dependent effects such as reflections pose a substantial challenge for image-based and neural rendering algorithms. Above all, curved reflectors are particularly hard, as they lead to highly non-linear reflection flows as the camera moves. We introduce a new point-based representation to compute Neural Point Catacaustics allowing novel-view synthesis of scenes with curved reflectors, from a set of casually-captured input photos. At the core of our method is a neural warp field that models catacaustic trajectories of reflections, so complex specular effects can be rendered using efficient point splatting in conjunction with a neural renderer. One of our key contributions is the explicit representation of reflections with a reflection point cloud which is displaced by the neural warp field, and a primary point cloud which is optimized to represent the rest of the scene. After a short manual annotation step, our approach allows interactive high-quality renderings of novel views with accurate reflection flow. Additionally, the explicit representation of reflection flow supports several forms of scene manipulation in captured scenes, such as reflection editing, cloning of specular objects, reflection tracking across views, and comfortable stereo viewing. We provide the source code and other supplemental material on https://repo-sam.inria.fr/ fungraph/neural_catacaustics/

我们提出了HRF-NET，这是一种基于整体辐射场的新型视图合成方法，该方法使用一组稀疏输入来呈现新视图。最近的概括视图合成方法还利用了光辉场，但渲染速度不是实时的。现有的方法可以有效地训练和呈现新颖的观点，但它们无法概括地看不到场景。我们的方法解决了用于概括视图合成的实时渲染问题，并由两个主要阶段组成：整体辐射场预测指标和基于卷积的神经渲染器。该架构不仅基于隐式神经场的一致场景几何形状，而且还可以使用单个GPU有效地呈现新视图。我们首先在DTU数据集的多个3D场景上训练HRF-NET，并且网络只能仅使用光度损耗就看不见的真实和合成数据产生合理的新视图。此外，我们的方法可以利用单个场景的密集参考图像集来产生准确的新颖视图，而无需依赖其他明确表示，并且仍然保持了预训练模型的高速渲染。实验结果表明，HRF-NET优于各种合成和真实数据集的最先进的神经渲染方法。

Figure 1: Our method can synthesize novel views in both space and time from a single monocular video of a dynamic scene. Here we show video results with various configurations of fixing and interpolating view and time (left), as well as a visualization of the recovered scene geometry (right). Please view with Adobe Acrobat or KDE Okular to see animations.

现有的神经样式传输方法需要参考样式图像来将样式图像的纹理信息传输到内容图像。然而，在许多实际情况中，用户可能没有参考样式图像，但仍然有兴趣通过想象它们来传输样式。为了处理此类应用程序，我们提出了一个新的框架，它可以实现样式转移`没有'风格图像，但仅使用所需风格的文本描述。使用预先训练的文本图像嵌入模型的剪辑，我们仅通过单个文本条件展示了内容图像样式的调制。具体而言，我们提出了一种针对现实纹理传输的多视图增强的修补程序文本图像匹配丢失。广泛的实验结果证实了具有反映语义查询文本的现实纹理的成功图像风格转移。