Implicit shape representations, such as Level Sets, provide a very elegant formulation for performing computations involving curves and surfaces. However, including implicit representations into canonical Neural Network formulations is far from straightforward. This has consequently restricted existing approaches to shape inference, to significantly less effective representations, perhaps most commonly voxels occupancy maps or sparse point clouds.To overcome this limitation we propose a novel formulation that permits the use of implicit representations of curves and surfaces, of arbitrary topology, as individual layers in Neural Network architectures with end-to-end trainability. Specifically, we propose to represent the output as an oriented level set of a continuous and discretised embedding function. We investigate the benefits of our approach on the task of 3D shape prediction from a single image and demonstrate its ability to produce a more accurate reconstruction compared to voxel-based representations. We further show that our model is flexible and can be applied to a variety of shape inference problems.

Figure 1: DeepSDF represents signed distance functions (SDFs) of shapes via latent code-conditioned feed-forward decoder networks. Above images are raycast renderings of DeepSDF interpolating between two shapes in the learned shape latent space. Best viewed digitally.

With the advent of deep neural networks, learning-based approaches for 3D reconstruction have gained popularity. However, unlike for images, in 3D there is no canonical representation which is both computationally and memory efficient yet allows for representing high-resolution geometry of arbitrary topology. Many of the state-of-the-art learningbased 3D reconstruction approaches can hence only represent very coarse 3D geometry or are limited to a restricted domain. In this paper, we propose Occupancy Networks, a new representation for learning-based 3D reconstruction methods. Occupancy networks implicitly represent the 3D surface as the continuous decision boundary of a deep neural network classifier. In contrast to existing approaches, our representation encodes a description of the 3D output at infinite resolution without excessive memory footprint. We validate that our representation can efficiently encode 3D structure and can be inferred from various kinds of input. Our experiments demonstrate competitive results, both qualitatively and quantitatively, for the challenging tasks of 3D reconstruction from single images, noisy point clouds and coarse discrete voxel grids. We believe that occupancy networks will become a useful tool in a wide variety of learning-based 3D tasks.

Implicit fields have been very effective to represent and learn 3D shapes accurately. Signed distance fields and occupancy fields are the preferred representations, both with well-studied properties, despite their restriction to closed surfaces. Several other variations and training principles have been proposed with the goal to represent all classes of shapes. In this paper, we develop a novel and yet fundamental representation by considering the unit vector field defined on 3D space: at each point in $\mathbb{R}^3$ the vector points to the closest point on the surface. We theoretically demonstrate that this vector field can be easily transformed to surface density by applying the vector field divergence. Unlike other standard representations, it directly encodes an important physical property of the surface, which is the surface normal. We further show the advantages of our vector field representation, specifically in learning general (open, closed, or multi-layered) surfaces as well as piecewise planar surfaces. We compare our method on several datasets including ShapeNet where the proposed new neural implicit field shows superior accuracy in representing any type of shape, outperforming other standard methods. The code will be released at https://github.com/edomel/ImplicitVF

在视觉计算中，3D几何形状以许多不同的形式表示，包括网格，点云，体素电网，水平集和深度图像。每个表示都适用于不同的任务，从而使一个表示形式转换为另一个表示（前向地图）是一个重要且常见的问题。我们提出了全向距离字段（ODF），这是一种新的3D形状表示形式，该表示通过将深度从任何观看方向从任何3D位置存储到对象的表面来编码几何形状。由于射线是ODF的基本单元，因此可以轻松地从通用的3D表示和点云等常见的3D表示。与限制代表封闭表面的水平集方法不同，ODF是未签名的，因此可以对开放表面进行建模（例如服装）。我们证明，尽管在遮挡边界处存在固有的不连续性，但可以通过神经网络（Neururodf）有效地学习ODF。我们还引入了有效的前向映射算法，以转换odf to＆从常见的3D表示。具体而言，我们引入了一种有效的跳跃立方体算法，用于从ODF生成网格。实验表明，神经模型可以通过过度拟合单个对象学会学会捕获高质量的形状，并学会概括对共同的形状类别。

Intelligent mesh generation (IMG) refers to a technique to generate mesh by machine learning, which is a relatively new and promising research field. Within its short life span, IMG has greatly expanded the generalizability and practicality of mesh generation techniques and brought many breakthroughs and potential possibilities for mesh generation. However, there is a lack of surveys focusing on IMG methods covering recent works. In this paper, we are committed to a systematic and comprehensive survey describing the contemporary IMG landscape. Focusing on 110 preliminary IMG methods, we conducted an in-depth analysis and evaluation from multiple perspectives, including the core technique and application scope of the algorithm, agent learning goals, data types, targeting challenges, advantages and limitations. With the aim of literature collection and classification based on content extraction, we propose three different taxonomies from three views of key technique, output mesh unit element, and applicable input data types. Finally, we highlight some promising future research directions and challenges in IMG. To maximize the convenience of readers, a project page of IMG is provided at \url{https://github.com/xzb030/IMG_Survey}.

综合照片 - 现实图像和视频是计算机图形的核心，并且是几十年的研究焦点。传统上，使用渲染算法（如光栅化或射线跟踪）生成场景的合成图像，其将几何形状和材料属性的表示为输入。统称，这些输入定义了实际场景和呈现的内容，并且被称为场景表示（其中场景由一个或多个对象组成）。示例场景表示是具有附带纹理的三角形网格（例如，由艺术家创建），点云（例如，来自深度传感器），体积网格（例如，来自CT扫描）或隐式曲面函数（例如，截短的符号距离）字段）。使用可分辨率渲染损耗的观察结果的这种场景表示的重建被称为逆图形或反向渲染。神经渲染密切相关，并将思想与经典计算机图形和机器学习中的思想相结合，以创建用于合成来自真实观察图像的图像的算法。神经渲染是朝向合成照片现实图像和视频内容的目标的跨越。近年来，我们通过数百个出版物显示了这一领域的巨大进展，这些出版物显示了将被动组件注入渲染管道的不同方式。这种最先进的神经渲染进步的报告侧重于将经典渲染原则与学习的3D场景表示结合的方法，通常现在被称为神经场景表示。这些方法的一个关键优势在于它们是通过设计的3D-一致，使诸如新颖的视点合成捕获场景的应用。除了处理静态场景的方法外，我们还涵盖了用于建模非刚性变形对象的神经场景表示...

Generative models, as an important family of statistical modeling, target learning the observed data distribution via generating new instances. Along with the rise of neural networks, deep generative models, such as variational autoencoders (VAEs) and generative adversarial network (GANs), have made tremendous progress in 2D image synthesis. Recently, researchers switch their attentions from the 2D space to the 3D space considering that 3D data better aligns with our physical world and hence enjoys great potential in practice. However, unlike a 2D image, which owns an efficient representation (i.e., pixel grid) by nature, representing 3D data could face far more challenges. Concretely, we would expect an ideal 3D representation to be capable enough to model shapes and appearances in details, and to be highly efficient so as to model high-resolution data with fast speed and low memory cost. However, existing 3D representations, such as point clouds, meshes, and recent neural fields, usually fail to meet the above requirements simultaneously. In this survey, we make a thorough review of the development of 3D generation, including 3D shape generation and 3D-aware image synthesis, from the perspectives of both algorithms and more importantly representations. We hope that our discussion could help the community track the evolution of this field and further spark some innovative ideas to advance this challenging task.

从单视图重建3D形状是一个长期的研究问题。在本文中，我们展示了深度隐式地面网络，其可以通过预测底层符号距离场来从2D图像产生高质量的细节的3D网格。除了利用全局图像特征之外，禁止2D图像上的每个3D点的投影位置，并从图像特征映射中提取本地特征。结合全球和局部特征显着提高了符合距离场预测的准确性，特别是对于富含细节的区域。据我们所知，伪装是一种不断捕获从单视图图像中存在于3D形状中存在的孔和薄结构等细节的方法。 Disn在从合成和真实图像重建的各种形状类别上实现最先进的单视性重建性能。代码可在https://github.com/xharlie/disn提供补充可以在https://xharlie.github.io/images/neUrips_2019_Supp.pdf中找到补充

近年来，由于其表达力和灵活性，神经隐式表示在3D重建中获得了普及。然而，神经隐式表示的隐式性质导致缓慢的推理时间并且需要仔细初始化。在本文中，我们重新审视经典且无处不在的点云表示，并使用泊松表面重建（PSR）的可分辨率配方引入可分化的点对网格层，其允许给予定向的GPU加速的指示灯的快速解决方案点云。可微分的PSR层允许我们通过隐式指示器字段有效地和分散地桥接与3D网格的显式3D点表示，从而实现诸如倒角距离的表面重建度量的端到端优化。因此，点和网格之间的这种二元性允许我们以面向点云表示形状，这是显式，轻量级和富有表现力的。与神经内隐式表示相比，我们的形状 - 点（SAP）模型更具可解释，轻量级，并通过一个级别加速推理时间。与其他显式表示相比，如点，补丁和网格，SA​​P产生拓扑无关的水密歧管表面。我们展示了SAP对无知点云和基于学习的重建的表面重建任务的有效性。

机器学习的最近进步已经创造了利用一类基于坐标的神经网络来解决视觉计算问题的兴趣，该基于坐标的神经网络在空间和时间跨空间和时间的场景或对象的物理属性。我们称之为神经领域的这些方法已经看到在3D形状和图像的合成中成功应用，人体的动画，3D重建和姿势估计。然而，由于在短时间内的快速进展，许多论文存在，但尚未出现全面的审查和制定问题。在本报告中，我们通过提供上下文，数学接地和对神经领域的文学进行广泛综述来解决这一限制。本报告涉及两种维度的研究。在第一部分中，我们通过识别神经字段方法的公共组件，包括不同的表示，架构，前向映射和泛化方法来专注于神经字段的技术。在第二部分中，我们专注于神经领域的应用在视觉计算中的不同问题，超越（例如，机器人，音频）。我们的评论显示了历史上和当前化身的视觉计算中已覆盖的主题的广度，展示了神经字段方法所带来的提高的质量，灵活性和能力。最后，我们展示了一个伴随着贡献本综述的生活版本，可以由社区不断更新。

Our method completes a partial 3D scan using a 3D Encoder-Predictor network that leverages semantic features from a 3D classification network. The predictions are correlated with a shape database, which we use in a multi-resolution 3D shape synthesis step. We obtain completed high-resolution meshes that are inferred from partial, low-resolution input scans.

Recently, implicit neural representations have gained popularity for learning-based 3D reconstruction. While demonstrating promising results, most implicit approaches are limited to comparably simple geometry of single objects and do not scale to more complicated or large-scale scenes. The key limiting factor of implicit methods is their simple fullyconnected network architecture which does not allow for integrating local information in the observations or incorporating inductive biases such as translational equivariance. In this paper, we propose Convolutional Occupancy Networks, a more flexible implicit representation for detailed reconstruction of objects and 3D scenes. By combining convolutional encoders with implicit occupancy decoders, our model incorporates inductive biases, enabling structured reasoning in 3D space. We investigate the effectiveness of the proposed representation by reconstructing complex geometry from noisy point clouds and low-resolution voxel representations. We empirically find that our method enables the fine-grained implicit 3D reconstruction of single objects, scales to large indoor scenes, and generalizes well from synthetic to real data.

我们介绍DMTET，深度3D条件生成模型，可以使用诸如粗体素的简单用户指南来合成高分辨率3D形状。它通过利用新型混合3D表示来结婚隐式和显式3D表示的优点。与当前隐含的方法相比，培训涉及符号距离值，DMTET直接针对重建的表面进行了优化，这使我们能够用更少的伪像来合成更精细的几何细节。与直接生成诸如网格之类的显式表示的深度3D生成模型不同，我们的模型可以合成具有任意拓扑的形状。 DMTET的核心包括可变形的四面体网格，其编码离散的符号距离函数和可分行的行进Tetrahedra层，其将隐式符号距离表示转换为显式谱图表示。这种组合允许使用在表面网格上明确定义的重建和对抗性损耗来联合优化表面几何形状和拓扑以及生成细分层次结构。我们的方法显着优于来自粗体素输入的条件形状合成的现有工作，培训在复杂的3D动物形状的数据集上。项目页面：https://nv-tlabs.github.io/dmtet/

Training parts from ShapeNet. (b) t-SNE plot of part embeddings. (c) Reconstructing entire scenes with Local Implicit Grids Figure 1:We learn an embedding of parts from objects in ShapeNet [3] using a part autoencoder with an implicit decoder. We show that this representation of parts is generalizable across object categories, and easily scalable to large scenes. By localizing implicit functions in a grid, we are able to reconstruct entire scenes from points via optimization of the latent grid.

我们基于最近普及的隐式神经形状表示，探索了从点云进行基于学习形状重建的新想法。我们将这个问题作为对特征空间中隐式神经签名距离函数的几次学习，我们使用基于梯度的元学习来处理。我们使用卷积编码器在给定输入点云的情况下构建特征空间。隐式解码器学会了预测此特征空间中表示的签名距离值。设置输入点云，即从目标形状函数的零级别设置中的样本，作为支持（即上下文）的少数学习术语的支持（即上下文），我们训练解码器，以便它可以通过使用该上下文的基础形状使其重新调整。几（5）个调整步骤。因此，我们首次同时结合了两种类型的隐式神经网络调节机制，即具有编码和元学习。我们的数值和定性评估表明，在稀疏点云中隐性重建的背景下，我们提出的策略，即在特征空间中的元学习，优于现有的替代方案，即特征空间中的标准监督学习，以及在欧几里得空间中的元学习。 ，同时仍提供快速推理。

where the highest resolution is required, using facial performance capture as a case in point.

隐式神经网络已成功用于点云的表面重建。然而，它们中的许多人面临着可扩展性问题，因为它们将整个对象或场景的异构面功能编码为单个潜在载体。为了克服这种限制，一些方法在粗略普通的3D网格或3D补丁上推断潜伏向量，并将它们插入以应对占用查询。在这样做时，它们可以与对象表面上采样的输入点进行直接连接，并且它们在空间中均匀地附加信息，而不是其最重要的信息，即在表面附近。此外，依赖于固定的补丁大小可能需要离散化调整。要解决这些问题，我们建议使用点云卷积并计算每个输入点的潜伏向量。然后，我们使用推断的权重在最近的邻居上执行基于学习的插值。对象和场景数据集的实验表明，我们的方法在大多数古典指标上显着优于其他方法，产生更精细的细节和更好的重建更薄的卷。代码可在https://github.com/valeoai/poco获得。