We propose an end-to-end deep learning architecture that produces a 3D shape in triangular mesh from a single color image. Limited by the nature of deep neural network, previous methods usually represent a 3D shape in volume or point cloud, and it is non-trivial to convert them to the more ready-to-use mesh model. Unlike the existing methods, our network represents 3D mesh in a graph-based convolutional neural network and produces correct geometry by progressively deforming an ellipsoid, leveraging perceptual features extracted from the input image. We adopt a coarse-to-fine strategy to make the whole deformation procedure stable, and define various of mesh related losses to capture properties of different levels to guarantee visually appealing and physically accurate 3D geometry. Extensive experiments show that our method not only qualitatively produces mesh model with better details, but also achieves higher 3D shape estimation accuracy compared to the state-of-the-art.

单视图3D对象重建是一项基本且具有挑战性的计算机视觉任务，旨在从单视RGB图像中恢复3D形状。大多数现有的基于深度学习的重建方法都是​​在同一类别上培训和评估的，并且在处理训练过程中未见的新颖类别的物体时，它们无法正常工作。本文着眼于这个问题，解决了零照片的单视3D网格重建，以研究对看不见类别的模型概括，并鼓励模型从字面上重建对象。具体而言，我们建议一个端到端的两阶段网络Zeromesh，以打破重建中的类别边界。首先，我们将复杂的图像到网格映射分解为两个较简单的映射，即图像对点映射和点对点映射，而后者主要是几何问题，而不是对象类别的依赖。其次，我们在2D和3D特征空间中设计了局部特征采样策略，以捕获跨对象共享的局部几何形状，以增强模型概括。第三，除了传统的点对点监督外，我们还引入了多视图轮廓损失以监督表面生成过程，该过程提供了其他正则化，并进一步缓解了过度拟合的问题。实验结果表明，我们的方法在不同方案和各种指标下，特别是对于新颖对象而言，在Shapenet和Pix3D上的现有作品显着优于Shapenet和Pix3D的现有作品。

Intelligent mesh generation (IMG) refers to a technique to generate mesh by machine learning, which is a relatively new and promising research field. Within its short life span, IMG has greatly expanded the generalizability and practicality of mesh generation techniques and brought many breakthroughs and potential possibilities for mesh generation. However, there is a lack of surveys focusing on IMG methods covering recent works. In this paper, we are committed to a systematic and comprehensive survey describing the contemporary IMG landscape. Focusing on 110 preliminary IMG methods, we conducted an in-depth analysis and evaluation from multiple perspectives, including the core technique and application scope of the algorithm, agent learning goals, data types, targeting challenges, advantages and limitations. With the aim of literature collection and classification based on content extraction, we propose three different taxonomies from three views of key technique, output mesh unit element, and applicable input data types. Finally, we highlight some promising future research directions and challenges in IMG. To maximize the convenience of readers, a project page of IMG is provided at \url{https://github.com/xzb030/IMG_Survey}.

从单视图重建3D形状是一个长期的研究问题。在本文中，我们展示了深度隐式地面网络，其可以通过预测底层符号距离场来从2D图像产生高质量的细节的3D网格。除了利用全局图像特征之外，禁止2D图像上的每个3D点的投影位置，并从图像特征映射中提取本地特征。结合全球和局部特征显着提高了符合距离场预测的准确性，特别是对于富含细节的区域。据我们所知，伪装是一种不断捕获从单视图图像中存在于3D形状中存在的孔和薄结构等细节的方法。 Disn在从合成和真实图像重建的各种形状类别上实现最先进的单视性重建性能。代码可在https://github.com/xharlie/disn提供补充可以在https://xharlie.github.io/images/neUrips_2019_Supp.pdf中找到补充

Rapid advances in 2D perception have led to systems that accurately detect objects in real-world images. However, these systems make predictions in 2D, ignoring the 3D structure of the world. Concurrently, advances in 3D shape prediction have mostly focused on synthetic benchmarks and isolated objects. We unify advances in these two areas. We propose a system that detects objects in real-world images and produces a triangle mesh giving the full 3D shape of each detected object. Our system, called Mesh R-CNN, augments Mask R-CNN with a mesh prediction branch that outputs meshes with varying topological structure by first predicting coarse voxel representations which are converted to meshes and refined with a graph convolution network operating over the mesh's vertices and edges. We validate our mesh prediction branch on ShapeNet, where we outperform prior work on single-image shape prediction. We then deploy our full Mesh R-CNN system on Pix3D, where we jointly detect objects and predict their 3D shapes. Project page: https://gkioxari.github.io/meshrcnn/.

我们介绍DMTET，深度3D条件生成模型，可以使用诸如粗体素的简单用户指南来合成高分辨率3D形状。它通过利用新型混合3D表示来结婚隐式和显式3D表示的优点。与当前隐含的方法相比，培训涉及符号距离值，DMTET直接针对重建的表面进行了优化，这使我们能够用更少的伪像来合成更精细的几何细节。与直接生成诸如网格之类的显式表示的深度3D生成模型不同，我们的模型可以合成具有任意拓扑的形状。 DMTET的核心包括可变形的四面体网格，其编码离散的符号距离函数和可分行的行进Tetrahedra层，其将隐式符号距离表示转换为显式谱图表示。这种组合允许使用在表面网格上明确定义的重建和对抗性损耗来联合优化表面几何形状和拓扑以及生成细分层次结构。我们的方法显着优于来自粗体素输入的条件形状合成的现有工作，培训在复杂的3D动物形状的数据集上。项目页面：https://nv-tlabs.github.io/dmtet/

With the advent of deep neural networks, learning-based approaches for 3D reconstruction have gained popularity. However, unlike for images, in 3D there is no canonical representation which is both computationally and memory efficient yet allows for representing high-resolution geometry of arbitrary topology. Many of the state-of-the-art learningbased 3D reconstruction approaches can hence only represent very coarse 3D geometry or are limited to a restricted domain. In this paper, we propose Occupancy Networks, a new representation for learning-based 3D reconstruction methods. Occupancy networks implicitly represent the 3D surface as the continuous decision boundary of a deep neural network classifier. In contrast to existing approaches, our representation encodes a description of the 3D output at infinite resolution without excessive memory footprint. We validate that our representation can efficiently encode 3D structure and can be inferred from various kinds of input. Our experiments demonstrate competitive results, both qualitatively and quantitatively, for the challenging tasks of 3D reconstruction from single images, noisy point clouds and coarse discrete voxel grids. We believe that occupancy networks will become a useful tool in a wide variety of learning-based 3D tasks.

近年来，由于其表达力和灵活性，神经隐式表示在3D重建中获得了普及。然而，神经隐式表示的隐式性质导致缓慢的推理时间并且需要仔细初始化。在本文中，我们重新审视经典且无处不在的点云表示，并使用泊松表面重建（PSR）的可分辨率配方引入可分化的点对网格层，其允许给予定向的GPU加速的指示灯的快速解决方案点云。可微分的PSR层允许我们通过隐式指示器字段有效地和分散地桥接与3D网格的显式3D点表示，从而实现诸如倒角距离的表面重建度量的端到端优化。因此，点和网格之间的这种二元性允许我们以面向点云表示形状，这是显式，轻量级和富有表现力的。与神经内隐式表示相比，我们的形状 - 点（SAP）模型更具可解释，轻量级，并通过一个级别加速推理时间。与其他显式表示相比，如点，补丁和网格，SA​​P产生拓扑无关的水密歧管表面。我们展示了SAP对无知点云和基于学习的重建的表面重建任务的有效性。

卷积神经网络（CNNS）在2D计算机视觉中取得了很大的突破。然而，它们的不规则结构使得难以在网格上直接利用CNNS的潜力。细分表面提供分层多分辨率结构，其中闭合的2 - 歧管三角网格中的每个面正恰好邻近三个面。本文推出了这两种观察，介绍了具有环形细分序列连接的3D三角形网格的创新和多功能CNN框架。在2D图像中的网格面和像素之间进行类比允许我们呈现网状卷积操作者以聚合附近面的局部特征。通过利用面部街区，这种卷积可以支持标准的2D卷积网络概念，例如，可变内核大小，步幅和扩张。基于多分辨率层次结构，我们利用汇集层，将四个面均匀地合并成一个和上采样方法，该方法将一个面分为四个。因此，许多流行的2D CNN架构可以容易地适应处理3D网格。可以通过自我参数化来回收具有任意连接的网格，以使循环细分序列连接，使子变量是一般的方法。广泛的评估和各种应用展示了SubDIVNet的有效性和效率。

We propose a differentiable sphere tracing algorithm to bridge the gap between inverse graphics methods and the recently proposed deep learning based implicit signed distance function. Due to the nature of the implicit function, the rendering process requires tremendous function queries, which is particularly problematic when the function is represented as a neural network. We optimize both the forward and backward passes of our rendering layer to make it run efficiently with affordable memory consumption on a commodity graphics card. Our rendering method is fully differentiable such that losses can be directly computed on the rendered 2D observations, and the gradients can be propagated backwards to optimize the 3D geometry. We show that our rendering method can effectively reconstruct accurate 3D shapes from various inputs, such as sparse depth and multi-view images, through inverse optimization. With the geometry based reasoning, our 3D shape prediction methods show excellent generalization capability and robustness against various noises. * Work done while Shaohui Liu was an academic guest at ETH Zurich.

We propose an analysis-by-synthesis method for fast multi-view 3D reconstruction of opaque objects with arbitrary materials and illumination. State-of-the-art methods use both neural surface representations and neural rendering. While flexible, neural surface representations are a significant bottleneck in optimization runtime. Instead, we represent surfaces as triangle meshes and build a differentiable rendering pipeline around triangle rasterization and neural shading. The renderer is used in a gradient descent optimization where both a triangle mesh and a neural shader are jointly optimized to reproduce the multi-view images. We evaluate our method on a public 3D reconstruction dataset and show that it can match the reconstruction accuracy of traditional baselines and neural approaches while surpassing them in optimization runtime. Additionally, we investigate the shader and find that it learns an interpretable representation of appearance, enabling applications such as 3D material editing.

随着几个行业正在朝着建模大规模的3D虚拟世界迈进，因此需要根据3D内容的数量，质量和多样性来扩展的内容创建工具的需求变得显而易见。在我们的工作中，我们旨在训练Parterant 3D生成模型，以合成纹理网格，可以通过3D渲染引擎直接消耗，因此立即在下游应用中使用。 3D生成建模的先前工作要么缺少几何细节，因此在它们可以生成的网格拓扑中受到限制，通常不支持纹理，或者在合成过程中使用神经渲染器，这使得它们在常见的3D软件中使用。在这项工作中，我们介绍了GET3D，这是一种生成模型，该模型直接生成具有复杂拓扑，丰富几何细节和高保真纹理的显式纹理3D网格。我们在可区分的表面建模，可区分渲染以及2D生成对抗网络中桥接了最新成功，以从2D图像集合中训练我们的模型。 GET3D能够生成高质量的3D纹理网格，从汽车，椅子，动物，摩托车和人类角色到建筑物，对以前的方法进行了重大改进。

Recently, implicit neural representations have gained popularity for learning-based 3D reconstruction. While demonstrating promising results, most implicit approaches are limited to comparably simple geometry of single objects and do not scale to more complicated or large-scale scenes. The key limiting factor of implicit methods is their simple fullyconnected network architecture which does not allow for integrating local information in the observations or incorporating inductive biases such as translational equivariance. In this paper, we propose Convolutional Occupancy Networks, a more flexible implicit representation for detailed reconstruction of objects and 3D scenes. By combining convolutional encoders with implicit occupancy decoders, our model incorporates inductive biases, enabling structured reasoning in 3D space. We investigate the effectiveness of the proposed representation by reconstructing complex geometry from noisy point clouds and low-resolution voxel representations. We empirically find that our method enables the fine-grained implicit 3D reconstruction of single objects, scales to large indoor scenes, and generalizes well from synthetic to real data.

Point cloud completion is a generation and estimation issue derived from the partial point clouds, which plays a vital role in the applications in 3D computer vision. The progress of deep learning (DL) has impressively improved the capability and robustness of point cloud completion. However, the quality of completed point clouds is still needed to be further enhanced to meet the practical utilization. Therefore, this work aims to conduct a comprehensive survey on various methods, including point-based, convolution-based, graph-based, and generative model-based approaches, etc. And this survey summarizes the comparisons among these methods to provoke further research insights. Besides, this review sums up the commonly used datasets and illustrates the applications of point cloud completion. Eventually, we also discussed possible research trends in this promptly expanding field.

Figure 1. Given input as either a 2D image or a 3D point cloud (a), we automatically generate a corresponding 3D mesh (b) and its atlas parameterization (c). We can use the recovered mesh and atlas to apply texture to the output shape (d) as well as 3D print the results (e).

基于单个草图图像重建3D形状是由于稀疏，不规则的草图和常规，密集的3D形状之间的较大域间隙而具有挑战性的。现有的作品尝试采用从草图提取的全局功能来直接预测3D坐标，但通常会遭受失去对输入草图不忠心的细节。通过分析3D到2D投影过程，我们注意到表征2D点云分布的密度图（即，投影平面每个位置的点的概率）可以用作代理，以促进该代理重建过程。为此，我们首先通过图像翻译网络将草图翻译成一个更有信息的2D表示，可用于生成密度映射。接下来，通过两个阶段的概率采样过程重建一个3D点云：首先通过对密度映射进行采样，首先恢复2D点（即X和Y坐标）；然后通过在每个2D点确定的射线处采样深度值来预测深度​​（即Z坐标）。进行了广泛的实验，定量和定性结果都表明，我们提出的方法显着优于其他基线方法。

4D隐式表示中的最新进展集中在全球控制形状和运动的情况下，低维潜在向量，这很容易缺少表面细节和累积跟踪误差。尽管许多深层的本地表示显示了3D形状建模的有希望的结果，但它们的4D对应物尚不存在。在本文中，我们通过提出一个新颖的局部4D隐性代表来填补这一空白，以动态穿衣人，名为Lord，具有4D人类建模和局部代表的优点，并实现具有详细的表面变形的高保真重建，例如衣服皱纹。特别是，我们的主要见解是鼓励网络学习本地零件级表示的潜在代码，能够解释本地几何形状和时间变形。为了在测试时间进行推断，我们首先估计内部骨架运动在每个时间步中跟踪本地零件，然后根据不同类型的观察到的数据通过自动编码来优化每个部分的潜在代码。广泛的实验表明，该提出的方法具有强大的代表4D人类的能力，并且在实际应用上胜过最先进的方法，包括从稀疏点，非刚性深度融合（质量和定量）进行的4D重建。

神经隐式功能的最新发展已在高质量的3D形状重建方面表现出巨大的成功。但是，大多数作品将空间分为形状的内部和外部，从而将其代表力量限制为单层和水密形状。这种局限性导致乏味的数据处理（将非紧密的原始数据转换为水密度），以及代表现实世界中一般对象形状的无能。在这项工作中，我们提出了一种新颖的方法来表示一般形状，包括具有多层表面的非水平形状和形状。我们介绍了3D形状（GIF）的一般隐式函数，该功能建模了每两个点之间的关系，而不是点和表面之间的关系。 GIF没有将3D空间分为预定义的内部区域，而是编码是否将两个点分开。 Shapenet上的实验表明，在重建质量，渲染效率和视觉保真度方面，GIF的表现优于先前的最先进方法。项目页面可从https://jianglongye.com/gifs获得。

您将如何通过一些错过来修复物理物体？您可能会想象它的原始形状从先前捕获的图像中，首先恢复其整体（全局）但粗大的形状，然后完善其本地细节。我们有动力模仿物理维修程序以解决点云完成。为此，我们提出了一个跨模式的形状转移双转化网络（称为CSDN），这是一种带有全循环参与图像的粗到精细范式，以完成优质的点云完成。 CSDN主要由“ Shape Fusion”和“ Dual-Refinect”模块组成，以应对跨模式挑战。第一个模块将固有的形状特性从单个图像传输，以指导点云缺失区域的几何形状生成，在其中，我们建议iPadain嵌入图像的全局特征和部分点云的完成。第二个模块通过调整生成点的位置来完善粗糙输出，其中本地改进单元通过图卷积利用了小说和输入点之间的几何关系，而全局约束单元则利用输入图像来微调生成的偏移。与大多数现有方法不同，CSDN不仅探讨了图像中的互补信息，而且还可以在整个粗到精细的完成过程中有效利用跨模式数据。实验结果表明，CSDN对十个跨模式基准的竞争对手表现出色。

大多数现有的动物姿势和形状估计方法用参数模型重建动物网格。这是因为Smal模型的低维姿势和形状参数使得深网络更容易学习高维动物网。然而，Smal模型从具有限制和形状变化的玩具动物的扫描学习，因此可能无法良好地代表高度不同的真实动物。这可能导致估计网格的差，例如2D证据的差。 2d关键点或剪影。为了缓解此问题，我们提出了一种从单个图像重建3D动物网格的粗细方法。粗略估计阶段首先估计Smal模型的姿势，形状和翻译参数。然后将估计的网格用作图表卷积网络（GCN）的起点，以预测细化阶段的每顶顶点变形。基于SMAL和基于顶点的表示的这种组合来自参数和非参数表示。我们将网眼细化GCN（MRGCN）设计为具有分层特征表示的编码器解码器结构，以克服传统GCN的有限接收领域。此外，我们观察到，现有动物网格重建工作所使用的全局图像特征无法捕获用于网格细化的详细形状信息。因此，我们引入了本地特征提取器来检索顶点级别功能，并将其与全局功能一起用作MRGCN的输入。我们在Stanfordextra DataSet上测试我们的方法，实现最先进的结果。此外，我们在动物姿势和BADJA数据集中测试我们方法的泛化能力。我们的代码可在项目网站上获得。