Neural Radiance Fields (NeRF) methods have proved effective as compact, high-quality and versatile representations for 3D scenes, and enable downstream tasks such as editing, retrieval, navigation, etc. Various neural architectures are vying for the core structure of NeRF, including the plain Multi-Layer Perceptron (MLP), sparse tensors, low-rank tensors, hashtables and their compositions. Each of these representations has its particular set of trade-offs. For example, the hashtable-based representations admit faster training and rendering but their lack of clear geometric meaning hampers downstream tasks like spatial-relation-aware editing. In this paper, we propose Progressive Volume Distillation (PVD), a systematic distillation method that allows any-to-any conversions between different architectures, including MLP, sparse or low-rank tensors, hashtables and their compositions. PVD consequently empowers downstream applications to optimally adapt the neural representations for the task at hand in a post hoc fashion. The conversions are fast, as distillation is progressively performed on different levels of volume representations, from shallower to deeper. We also employ special treatment of density to deal with its specific numerical instability problem. Empirical evidence is presented to validate our method on the NeRF-Synthetic, LLFF and TanksAndTemples datasets. For example, with PVD, an MLP-based NeRF model can be distilled from a hashtable-based Instant-NGP model at a 10X~20X faster speed than being trained the original NeRF from scratch, while achieving a superior level of synthesis quality. Code is available at https://github.com/megvii-research/AAAI2023-PVD.

We present TensoRF, a novel approach to model and reconstruct radiance fields. Unlike NeRF that purely uses MLPs, we model the radiance field of a scene as a 4D tensor, which represents a 3D voxel grid with per-voxel multi-channel features. Our central idea is to factorize the 4D scene tensor into multiple compact low-rank tensor components. We demonstrate that applying traditional CP decomposition -- that factorizes tensors into rank-one components with compact vectors -- in our framework leads to improvements over vanilla NeRF. To further boost performance, we introduce a novel vector-matrix (VM) decomposition that relaxes the low-rank constraints for two modes of a tensor and factorizes tensors into compact vector and matrix factors. Beyond superior rendering quality, our models with CP and VM decompositions lead to a significantly lower memory footprint in comparison to previous and concurrent works that directly optimize per-voxel features. Experimentally, we demonstrate that TensoRF with CP decomposition achieves fast reconstruction (<30 min) with better rendering quality and even a smaller model size (<4 MB) compared to NeRF. Moreover, TensoRF with VM decomposition further boosts rendering quality and outperforms previous state-of-the-art methods, while reducing the reconstruction time (<10 min) and retaining a compact model size (<75 MB).

Neural radiance field (NeRF) attracts attention as a promising approach to reconstructing the 3D scene. As NeRF emerges, subsequent studies have been conducted to model dynamic scenes, which include motions or topological changes. However, most of them use an additional deformation network, slowing down the training and rendering speed. Tensorial radiance field (TensoRF) recently shows its potential for fast, high-quality reconstruction of static scenes with compact model size. In this paper, we present D-TensoRF, a tensorial radiance field for dynamic scenes, enabling novel view synthesis at a specific time. We consider the radiance field of a dynamic scene as a 5D tensor. The 5D tensor represents a 4D grid in which each axis corresponds to X, Y, Z, and time and has 1D multi-channel features per element. Similar to TensoRF, we decompose the grid either into rank-one vector components (CP decomposition) or low-rank matrix components (newly proposed MM decomposition). We also use smoothing regularization to reflect the relationship between features at different times (temporal dependency). We conduct extensive evaluations to analyze our models. We show that D-TensoRF with CP decomposition and MM decomposition both have short training times and significantly low memory footprints with quantitatively and qualitatively competitive rendering results in comparison to the state-of-the-art methods in 3D dynamic scene modeling.

Recent efforts in Neural Rendering Fields (NeRF) have shown impressive results on novel view synthesis by utilizing implicit neural representation to represent 3D scenes. Due to the process of volumetric rendering, the inference speed for NeRF is extremely slow, limiting the application scenarios of utilizing NeRF on resource-constrained hardware, such as mobile devices. Many works have been conducted to reduce the latency of running NeRF models. However, most of them still require high-end GPU for acceleration or extra storage memory, which is all unavailable on mobile devices. Another emerging direction utilizes the neural light field (NeLF) for speedup, as only one forward pass is performed on a ray to predict the pixel color. Nevertheless, to reach a similar rendering quality as NeRF, the network in NeLF is designed with intensive computation, which is not mobile-friendly. In this work, we propose an efficient network that runs in real-time on mobile devices for neural rendering. We follow the setting of NeLF to train our network. Unlike existing works, we introduce a novel network architecture that runs efficiently on mobile devices with low latency and small size, i.e., saving $15\times \sim 24\times$ storage compared with MobileNeRF. Our model achieves high-resolution generation while maintaining real-time inference for both synthetic and real-world scenes on mobile devices, e.g., $18.04$ms (iPhone 13) for rendering one $1008\times756$ image of real 3D scenes. Additionally, we achieve similar image quality as NeRF and better quality than MobileNeRF (PSNR $26.15$ vs. $25.91$ on the real-world forward-facing dataset).

关于神经辐射场（NERF）的最新研究爆炸表明，具有神经网络的复杂场面具有令人鼓舞的潜力。 NERF的一个主要缺点是它的推理时间：渲染单像素需要数百次查询NERF网络。为了解决它，现有的努力主要试图减少所需的采样点的数量。但是，迭代采样的问题仍然存在。另一方面，神经光场（NELF）在新型视图合成中对NERF提出了更直接的表示 - 像素的渲染相当于一个单一的正向通行，而无需射线建设。在这项工作中，我们提出了一个深层残留的MLP网络（88层），以有效地学习光场。我们展示了成功学习这种深度NELF网络的关键，就是拥有足够的数据，我们通过数据蒸馏从预训练的NERF模型中转移知识。在合成和现实世界场景上进行的广泛实验表明，我们方法比其他对应算法的优点。在合成场景中，我们实现了26-35倍的拖鞋（每个摄像头射线）和28-31倍的运行时加速，同时提供了比NERF的呈现质量（1.4-2.8 dB的平均PSNR改善），而无需任何定制的并行性要求。

NeRF synthesizes novel views of a scene with unprecedented quality by fitting a neural radiance field to RGB images. However, NeRF requires querying a deep Multi-Layer Perceptron (MLP) millions of times, leading to slow rendering times, even on modern GPUs. In this paper, we demonstrate that real-time rendering is possible by utilizing thousands of tiny MLPs instead of one single large MLP. In our setting, each individual MLP only needs to represent parts of the scene, thus smaller and faster-to-evaluate MLPs can be used. By combining this divide-and-conquer strategy with further optimizations, rendering is accelerated by three orders of magnitude compared to the original NeRF model without incurring high storage costs. Further, using teacher-student distillation for training, we show that this speed-up can be achieved without sacrificing visual quality.

Photo-realistic free-viewpoint rendering of real-world scenes using classical computer graphics techniques is challenging, because it requires the difficult step of capturing detailed appearance and geometry models. Recent studies have demonstrated promising results by learning scene representations that implicitly encode both geometry and appearance without 3D supervision. However, existing approaches in practice often show blurry renderings caused by the limited network capacity or the difficulty in finding accurate intersections of camera rays with the scene geometry. Synthesizing high-resolution imagery from these representations often requires time-consuming optical ray marching. In this work, we introduce Neural Sparse Voxel Fields (NSVF), a new neural scene representation for fast and high-quality free-viewpoint rendering. NSVF defines a set of voxel-bounded implicit fields organized in a sparse voxel octree to model local properties in each cell. We progressively learn the underlying voxel structures with a diffentiable ray-marching operation from only a set of posed RGB images. With the sparse voxel octree structure, rendering novel views can be accelerated by skipping the voxels containing no relevant scene content. Our method is typically over 10 times faster than the state-of-the-art (namely, NeRF (Mildenhall et al., 2020)) at inference time while achieving higher quality results. Furthermore, by utilizing an explicit sparse voxel representation, our method can easily be applied to scene editing and scene composition. We also demonstrate several challenging tasks, including multi-scene learning, free-viewpoint rendering of a moving human, and large-scale scene rendering. Code and data are available at our website: https://github.com/facebookresearch/NSVF.

We introduce a method to render Neural Radiance Fields (NeRFs) in real time using PlenOctrees, an octree-based 3D representation which supports view-dependent effects. Our method can render 800×800 images at more than 150 FPS, which is over 3000 times faster than conventional NeRFs. We do so without sacrificing quality while preserving the ability of NeRFs to perform free-viewpoint rendering of scenes with arbitrary geometry and view-dependent effects. Real-time performance is achieved by pre-tabulating the NeRF into a PlenOctree. In order to preserve viewdependent effects such as specularities, we factorize the appearance via closed-form spherical basis functions. Specifically, we show that it is possible to train NeRFs to predict a spherical harmonic representation of radiance, removing the viewing direction as an input to the neural network. Furthermore, we show that PlenOctrees can be directly optimized to further minimize the reconstruction loss, which leads to equal or better quality compared to competing methods. Moreover, this octree optimization step can be used to reduce the training time, as we no longer need to wait for the NeRF training to converge fully. Our real-time neural rendering approach may potentially enable new applications such as 6-DOF industrial and product visualizations, as well as next generation AR/VR systems. PlenOctrees are amenable to in-browser rendering as well; please visit the project page for the interactive online demo, as well as video and code: https://alexyu. net/plenoctrees.

最近，神经辐射场（NERF）正在彻底改变新型视图合成（NVS）的卓越性能。但是，NERF及其变体通常需要进行冗长的每场训练程序，其中将多层感知器（MLP）拟合到捕获的图像中。为了解决挑战，已经提出了体素网格表示，以显着加快训练的速度。但是，这些现有方法只能处理静态场景。如何开发有效，准确的动态视图合成方法仍然是一个开放的问题。将静态场景的方法扩展到动态场景并不简单，因为场景几何形状和外观随时间变化。在本文中，基于素素网格优化的最新进展，我们提出了一种快速变形的辐射场方法来处理动态场景。我们的方法由两个模块组成。第一个模块采用变形网格来存储3D动态功能，以及使用插值功能将观测空间中的3D点映射到规范空间的变形的轻巧MLP。第二个模块包含密度和颜色网格，以建模场景的几何形状和密度。明确对阻塞进行了建模，以进一步提高渲染质量。实验结果表明，我们的方法仅使用20分钟的训练就可以实现与D-NERF相当的性能，该训练比D-NERF快70倍以上，这清楚地证明了我们提出的方法的效率。

我们介绍了一种超快速的收敛方法来重建从一组图像中捕获具有已知姿势的场景的图像的每场辐射场。该任务通常适用于新颖的视图综合，最近是由神经辐射领域（NERF）彻底改革为其最先进的质量和灵活性。然而，NERF及其变体需要漫长的训练时间来为单个场景的数小时到几天。相比之下，我们的方法实现了NERF相当的质量，并通过单个GPU在不到15分钟内从划痕中迅速收敛。我们采用由密度体素网格组成的表示，用于场景几何形状和具有浅网络的特征体素网格，用于复杂的视图依赖性外观。用明确和离散化卷表示的建模并不是新的，但我们提出了两种简单而非琐碎的技术，有助于快速收敛速度和高质量的输出。首先，我们介绍了体素密度的激活后插值，其能够以较低的网格分辨率产生尖锐的表面。其次，直接体素密度优化容易发生次优几何解决方案，因此我们通过施加多个前沿来强制优化过程。最后，对五个内向的基准评估表明，我们的方法匹配，如果没有超越Nerf的质量，但它只需15分钟即可从头开始训练新场景。

神经辐射场（NERF）在代表3D场景和合成新颖视图中示出了很大的潜力，但是在推理阶段的NERF的计算开销仍然很重。为了减轻负担，我们进入了NERF的粗细分，分层采样过程，并指出粗阶段可以被我们命名神经样本场的轻量级模块代替。所提出的示例场地图光线进入样本分布，可以将其转换为点坐标并进料到radiance字段以进行体积渲染。整体框架被命名为Neusample。我们在现实合成360 $ ^ {\ circ} $和真正的前瞻性，两个流行的3D场景集上进行实验，并表明Neusample在享受更快推理速度时比NERF实现更好的渲染质量。Neusample进一步压缩，以提出的样品场提取方法朝向质量和速度之间的更好的权衡。

学习辐射场对新型视图综合显示出了显着的结果。学习过程通常会花费大量时间，这激发了最新方法，通过没有神经网络或使用更有效的数据结构来通过学习来加快学习过程。但是，这些专门设计的方法不适用于大多数基于辐射的方法的方法。为了解决此问题，我们引入了一项一般策略，以加快几乎所有基于辐射的方法的学习过程。我们的关键想法是通过在多视图卷渲染过程中拍摄较少的射线来减少冗余，这是几乎所有基于辐射的方法的基础。我们发现，在具有巨大色彩变化的像素上的射击不仅显着减轻了训练负担，而且几乎不会影响学到的辐射场的准确性。此外，我们还根据树中每个节点的平均渲染误差将每个视图自适应地细分为Quadtree，这使我们在更复杂的区域中动态射击更多的射线，并具有较大的渲染误差。我们在广泛使用的基准下使用不同的基于辐射的方法评估我们的方法。实验结果表明，我们的方法通过更快的训练获得了与最先进的可比精度。

神经辐射场（NERF）在建模3D场景和合成新型视图图像方面取得了巨大成功。但是，大多数以前的NERF方法需要大量时间来优化一个场景。显式数据结构，例如体素特征，显示出加速训练过程的巨大潜力。但是，体素特征面临两个大挑战，要应用于动态场景，即建模时间信息并捕获不同的点运动尺度。我们通过用时间感知的体素特征（称为Tineuvox）表示场景来提出一个辐射现场框架。引入了一个微小的坐标变形网络，以模拟粗糙运动轨迹，并在辐射网络中进一步增强了时间信息。提出了一种多距离插值方法，并应用于体素特征，以模拟小运动和大型运动。我们的框架大大加快了动态光芒度场的优化，同时保持高渲染质量。经验评估均在合成场景和真实场景上进行。我们的Tineuvox仅需8分钟和8 MB的存储成本即可完成培训，同时表现出比以前的动态NERF方法相似甚至更好的渲染性能。

本文提出了一种用等值的全向图像重建神经辐射场的方法。带有辐射场的隐式神经场景表示可以在有限的空间区域内连续重建场景的3D形状。但是，培训商用PC硬件的完全隐式表示需要大量时间和计算资源（15 $ \ sim $ 20小时每场景20小时）。因此，我们提出了一种显着加速此过程的方法（每个场景20 $ \ sim $ 40分钟）。我们采用特征体素，而不是使用辐射场重建的光线的完全隐式表示，而是在张量中包含密度和颜色特征的特征体素。考虑全向等值输入和相机布局，我们使用球形素化来表示表示而不是立方表示。我们的体素化方法可以平衡内部场景和外部场景的重建质量。此外，我们在颜色特征上采用了与轴对准的位置编码方法，以提高总图像质量。我们的方法可以在随机摄像头姿势上实现满足合成数据集的经验性能。此外，我们使用包含复杂几何形状并实现最先进性能的真实场景测试我们的方法。我们的代码和完整数据集将与纸质出版物同时发布。

3D reconstruction and novel view synthesis of dynamic scenes from collections of single views recently gained increased attention. Existing work shows impressive results for synthetic setups and forward-facing real-world data, but is severely limited in the training speed and angular range for generating novel views. This paper addresses these limitations and proposes a new method for full 360{\deg} novel view synthesis of non-rigidly deforming scenes. At the core of our method are: 1) An efficient deformation module that decouples the processing of spatial and temporal information for acceleration at training and inference time; and 2) A static module representing the canonical scene as a fast hash-encoded neural radiance field. We evaluate the proposed approach on the established synthetic D-NeRF benchmark, that enables efficient reconstruction from a single monocular view per time-frame randomly sampled from a full hemisphere. We refer to this form of inputs as monocularized data. To prove its practicality for real-world scenarios, we recorded twelve challenging sequences with human actors by sampling single frames from a synchronized multi-view rig. In both cases, our method is trained significantly faster than previous methods (minutes instead of days) while achieving higher visual accuracy for generated novel views. Our source code and data is available at our project page https://graphics.tu-bs.de/publications/kappel2022fast.

我们介绍了Plenoxels（plenoptic voxels），是一种光电型观测合成系统。Plenoxels表示作为具有球形谐波的稀疏3D网格的场景。该表示可以通过梯度方法和正则化从校准图像进行优化，而没有任何神经元件。在标准，基准任务中，Plenoxels优化了比神经辐射场更快的两个数量级，无需视觉质量损失。

神经辐射场（NERF）最近在新型视图合成中取得了令人印象深刻的结果。但是，以前的NERF作品主要关注以对象为中心的方案。在这项工作中，我们提出了360ROAM，这是一种新颖的场景级NERF系统，可以实时合成大型室内场景的图像并支持VR漫游。我们的系统首先从多个输入$ 360^\ circ $图像构建全向神经辐射场360NERF。然后，我们逐步估算一个3D概率的占用图，该概率占用图代表了空间密度形式的场景几何形状。跳过空的空间和上采样占据的体素本质上可以使我们通过以几何学意识的方式使用360NERF加速量渲染。此外，我们使用自适应划分和扭曲策略来减少和调整辐射场，以进一步改进。从占用地图中提取的场景的平面图可以为射线采样提供指导，并促进现实的漫游体验。为了显示我们系统的功效，我们在各种场景中收集了$ 360^\ Circ $图像数据集并进行广泛的实验。基线之间的定量和定性比较说明了我们在复杂室内场景的新型视图合成中的主要表现。

潜水员在NERF的关键思想和其变体 - 密度模型和体积渲染的关键思想中建立 - 学习可以从少量图像实际渲染的3D对象模型。与所有先前的NERF方法相比，潜水员使用确定性而不是体积渲染积分的随机估计。潜水员的表示是基于体素的功能领域。为了计算卷渲染积分，将光线分为间隔，每个体素;使用MLP的每个间隔的特征估计体渲染积分的组件，并且组件聚合。结果，潜水员可以呈现其他集成商错过的薄半透明结构。此外，潜水员的表示与其他这样的方法相比相对暴露的语义 - 在体素空间中的运动特征向量导致自然编辑。对当前最先进的方法的广泛定性和定量比较表明，潜水员产生（1）在最先进的质量或高于最先进的质量，（2）的情况下非常小而不会被烘烤，（3）在不被烘烤的情况下渲染非常快，并且（4）可以以自然方式编辑。

神经表面重建旨在基于多视图图像重建准确的3D表面。基于神经量的先前方法主要训练完全隐式的模型，它们需要单个场景的数小时培训。最近的努力探讨了明确的体积表示，该表示通过记住可学习的素网格中的重要信息，从而大大加快了优化过程。但是，这些基于体素的方法通常在重建细粒几何形状方面遇到困难。通过实证研究，我们发现高质量的表面重建取决于两个关键因素：构建相干形状的能力和颜色几何依赖性的精确建模。特别是，后者是准确重建细节的关键。受这些发现的启发，我们开发了Voxurf，这是一种基于体素的方法，用于有效，准确的神经表面重建，该方法由两个阶段组成：1）利用可学习的特征网格来构建颜色场并获得连贯的粗糙形状，并且2）使用双色网络来完善详细的几何形状，可捕获精确的颜色几何依赖性。我们进一步引入了层次几何特征，以启用跨体素的信息共享。我们的实验表明，Voxurf同时达到了高效率和高质量。在DTU基准测试中，与最先进的方法相比，Voxurf获得了更高的重建质量，训练的加速度为20倍。

本文提出了一个逐步连接的光场网络（Prolif），以构成复杂的前向场景的新观点。扩散编码一个4D光场，该场允许在一个训练步骤中渲染大量射线，以实现图像或贴片级损失。直接从图像中学习神经光场很难呈现多视图一致的图像，因为它对基础3D几何形状的不了解。为了解决这个问题，我们提出了一种渐进培训计划和正则化损失，以推断训练过程中的基础几何形状，这两者都会实现多视图一致性，从而极大地提高了渲染质量。实验表明，与香草神经光场相比，我们的方法能够实现明显更好的渲染质量，并且与挑战性的LLFF数据集和闪亮对象数据集的类似NERF的渲染方法相当。此外，我们证明了与LPIP的损失更好的兼容性，以实现与不同的光条件和剪辑损失的稳健性，以控制场景的渲染方式。项目页面：https：//totoro97.github.io/projects/prolif。