A longstanding question in computer vision concerns the representation of 3D shapes for recognition: should 3D shapes be represented with descriptors operating on their native 3D formats, such as voxel grid or polygon mesh, or can they be effectively represented with view-based descriptors? We address this question in the context of learning to recognize 3D shapes from a collection of their rendered views on 2D images. We first present a standard CNN architecture trained to recognize the shapes' rendered views independently of each other, and show that a 3D shape can be recognized even from a single view at an accuracy far higher than using state-of-the-art 3D shape descriptors. Recognition rates further increase when multiple views of the shapes are provided. In addition, we present a novel CNN architecture that combines information from multiple views of a 3D shape into a single and compact shape descriptor offering even better recognition performance. The same architecture can be applied to accurately recognize human hand-drawn sketches of shapes. We conclude that a collection of 2D views can be highly informative for 3D shape recognition and is amenable to emerging CNN architectures and their derivatives.

3D shape models are becoming widely available and easier to capture, making available 3D information crucial for progress in object classification. Current state-of-theart methods rely on CNNs to address this problem. Recently, we witness two types of CNNs being developed: CNNs based upon volumetric representations versus CNNs based upon multi-view representations. Empirical results from these two types of CNNs exhibit a large gap, indicating that existing volumetric CNN architectures and approaches are unable to fully exploit the power of 3D representations. In this paper, we aim to improve both volumetric CNNs and multi-view CNNs according to extensive analysis of existing approaches. To this end, we introduce two distinct network architectures of volumetric CNNs. In addition, we examine multi-view CNNs, where we introduce multiresolution filtering in 3D. Overall, we are able to outperform current state-of-the-art methods for both volumetric CNNs and multi-view CNNs. We provide extensive experiments designed to evaluate underlying design choices, thus providing a better understanding of the space of methods available for object classification on 3D data.

Multi-view projection techniques have shown themselves to be highly effective in achieving top-performing results in the recognition of 3D shapes. These methods involve learning how to combine information from multiple view-points. However, the camera view-points from which these views are obtained are often fixed for all shapes. To overcome the static nature of current multi-view techniques, we propose learning these view-points. Specifically, we introduce the Multi-View Transformation Network (MVTN), which uses differentiable rendering to determine optimal view-points for 3D shape recognition. As a result, MVTN can be trained end-to-end with any multi-view network for 3D shape classification. We integrate MVTN into a novel adaptive multi-view pipeline that is capable of rendering both 3D meshes and point clouds. Our approach demonstrates state-of-the-art performance in 3D classification and shape retrieval on several benchmarks (ModelNet40, ScanObjectNN, ShapeNet Core55). Further analysis indicates that our approach exhibits improved robustness to occlusion compared to other methods. We also investigate additional aspects of MVTN, such as 2D pretraining and its use for segmentation. To support further research in this area, we have released MVTorch, a PyTorch library for 3D understanding and generation using multi-view projections.

同时对象识别和姿势估计是机器人安全与人类和环境安全相互作用的两个关键功能。尽管对象识别和姿势估计都使用视觉输入，但大多数最先进的问题将它们作为两个独立的问题解决，因为前者需要视图不变的表示，而对象姿势估计需要一个与观点有关的描述。如今，多视图卷积神经网络（MVCNN）方法显示出最新的分类性能。尽管已广泛探索了MVCNN对象识别，但对多视图对象构成估计方法的研究很少，而同时解决这两个问题的研究更少。 MVCNN方法中虚拟摄像机的姿势通常是预先定义的，从而绑定了这种方法的应用。在本文中，我们提出了一种能够同时处理对象识别和姿势估计的方法。特别是，我们开发了一个深度的对象不合时宜的熵估计模型，能够预测给定3D对象的最佳观点。然后将对象的视图馈送到网络中，以同时预测目标对象的姿势和类别标签。实验结果表明，从此类位置获得的观点足以达到良好的精度得分。此外，我们设计了现实生活中的饮料场景，以证明拟议方法在真正的机器人任务中的运作效果如何。代码可在线获得：github.com/subhadityamukherjee/more_mvcnn

We tackle the problem of large scale visual place recognition, where the task is to quickly and accurately recognize the location of a given query photograph. We present the following three principal contributions. First, we develop a convolutional neural network (CNN) architecture that is trainable in an end-to-end manner directly for the place recognition task. The main component of this architecture, NetVLAD, is a new generalized VLAD layer, inspired by the "Vector of Locally Aggregated Descriptors" image representation commonly used in image retrieval. The layer is readily pluggable into any CNN architecture and amenable to training via backpropagation. Second, we develop a training procedure, based on a new weakly supervised ranking loss, to learn parameters of the architecture in an end-to-end manner from images depicting the same places over time downloaded from Google Street View Time Machine. Finally, we show that the proposed architecture significantly outperforms non-learnt image representations and off-the-shelf CNN descriptors on two challenging place recognition benchmarks, and improves over current stateof-the-art compact image representations on standard image retrieval benchmarks.

多视图投影方法在3D理解任务等方面表现出有希望的性能，如3D分类和分割。然而，它仍然不明确如何将这种多视图方法与广泛可用的3D点云组合。以前的方法使用未受忘掉的启发式方法在点级别结合功能。为此，我们介绍了多视图点云（vinoint云）的概念，表示每个3D点作为从多个视图点提取的一组功能。这种新颖的3D Vintor云表示将3D点云表示的紧凑性与多视图表示的自然观。当然，我们可以用卷积和汇集操作配备这一新的表示。我们以理论上建立的功能形式部署了Voint神经网络（vointnet），以学习vinite空间中的表示。我们的小说代表在ScanObjectnn，ModelNet40和ShapEnet​​ Core55上实现了3D分类和检索的最先进的性能。此外，我们在ShapeNet零件上实现了3D语义细分的竞争性能。进一步的分析表明，与其他方法相比，求力提高了旋转和闭塞的鲁棒性。

We propose bilinear models, a recognition architecture that consists of two feature extractors whose outputs are multiplied using outer product at each location of the image and pooled to obtain an image descriptor. This architecture can model local pairwise feature interactions in a translationally invariant manner which is particularly useful for fine-grained categorization. It also generalizes various orderless texture descriptors such as the Fisher vector, VLAD and O2P. We present experiments with bilinear models where the feature extractors are based on convolutional neural networks. The bilinear form simplifies gradient computation and allows end-to-end training of both networks using image labels only. Using networks initialized from the ImageNet dataset followed by domain specific fine-tuning we obtain 84.1% accuracy of the CUB-200-2011 dataset requiring only category labels at training time. We present experiments and visualizations that analyze the effects of fine-tuning and the choice two networks on the speed and accuracy of the models. Results show that the architecture compares favorably to the existing state of the art on a number of fine-grained datasets while being substantially simpler and easier to train. Moreover, our most accurate model is fairly efficient running at 8 frames/sec on a NVIDIA Tesla K40 GPU.

Image descriptors based on activations of Convolutional Neural Networks (CNNs) have become dominant in image retrieval due to their discriminative power, compactness of representation, and search efficiency. Training of CNNs, either from scratch or fine-tuning, requires a large amount of annotated data, where a high quality of annotation is often crucial. In this work, we propose to fine-tune CNNs for image retrieval on a large collection of unordered images in a fully automated manner. Reconstructed 3D models obtained by the state-of-the-art retrieval and structure-from-motion methods guide the selection of the training data. We show that both hard-positive and hard-negative examples, selected by exploiting the geometry and the camera positions available from the 3D models, enhance the performance of particular-object retrieval. CNN descriptor whitening discriminatively learned from the same training data outperforms commonly used PCA whitening. We propose a novel trainable Generalized-Mean (GeM) pooling layer that generalizes max and average pooling and show that it boosts retrieval performance. Applying the proposed method to the VGG network achieves state-of-the-art performance on the standard benchmarks: Oxford Buildings, Paris, and Holidays datasets.

3D shape is a crucial but heavily underutilized cue in today's computer vision systems, mostly due to the lack of a good generic shape representation. With the recent availability of inexpensive 2.5D depth sensors (e.g. Microsoft Kinect), it is becoming increasingly important to have a powerful 3D shape representation in the loop. Apart from category recognition, recovering full 3D shapes from viewbased 2.5D depth maps is also a critical part of visual understanding. To this end, we propose to represent a geometric 3D shape as a probability distribution of binary variables on a 3D voxel grid, using a Convolutional Deep Belief Network. Our model, 3D ShapeNets, learns the distribution of complex 3D shapes across different object categories and arbitrary poses from raw CAD data, and discovers hierarchical compositional part representations automatically. It naturally supports joint object recognition and shape completion from 2.5D depth maps, and it enables active object recognition through view planning. To train our 3D deep learning model, we construct ModelNet -a large-scale 3D CAD model dataset. Extensive experiments show that our 3D deep representation enables significant performance improvement over the-state-of-the-arts in a variety of tasks.

基于草图的3D形状检索（SBSR）是一项重要但艰巨的任务，近年来引起了越来越多的关注。现有方法在限制设置中解决了该问题，而无需适当模拟真实的应用程序方案。为了模仿现实的设置，在此曲目中，我们采用了不同级别的绘图技能的业余爱好者以及各种3D形状的大规模草图，不仅包括CAD型号，而且还可以从真实对象扫描的模型。我们定义了两个SBSR任务，并构建了两个基准，包括46,000多个CAD型号，1,700个现实型号和145,000个草图。四个团队参加了这一轨道，并为这两个任务提交了15次跑步，由7个常用指标评估。我们希望，基准，比较结果和开源评估法会在3D对象检索社区中促进未来的研究。

Figure 1: We provide evidence that state-of-the-art single-view 3D reconstruction methods (AtlasNet (light green, 0.38 IoU) [12], OGN (green, 0.46 IoU) [46], Matryoshka Networks (dark green, 0.47 IoU) [37]) do not actually perform reconstruction but image classification. We explicitly design pure recognition baselines (Clustering (light blue, 0.46 IoU) and Retrieval (dark blue, 0.57 IoU)) and show that they produce similar or better results both qualitatively and quantitatively. For reference, we show the ground truth (white) and a nearest neighbor from the training set (red, 0.76 IoU). The inset shows the input image.

Point cloud is an important type of geometric data structure. Due to its irregular format, most researchers transform such data to regular 3D voxel grids or collections of images. This, however, renders data unnecessarily voluminous and causes issues. In this paper, we design a novel type of neural network that directly consumes point clouds, which well respects the permutation invariance of points in the input. Our network, named PointNet, provides a unified architecture for applications ranging from object classification, part segmentation, to scene semantic parsing. Though simple, PointNet is highly efficient and effective. Empirically, it shows strong performance on par or even better than state of the art. Theoretically, we provide analysis towards understanding of what the network has learnt and why the network is robust with respect to input perturbation and corruption.

作为3D对象的两个基本表示方式，2D多视图图像和3D点云反映了来自视觉外观和几何结构各个方面的形状信息。与基于深度学习的2D多视图图像建模不同，该模型在各种3D形状分析任务中展示了领先的性能，基于3D点云的几何建模仍然遭受学习能力不足。在本文中，我们创新地构建了一个统一的跨模式知识转移框架，该框架将2D图像的歧视性视觉描述器提炼成3D点云的几何描述符。从技术上讲，在经典的教师学习范式下，我们提出了多视觉愿景到几何的蒸馏，由深入的2D图像编码器作为老师和深层的3D点云编码器组成。为了实现异质特征对齐，我们进一步提出了可见性感知的特征投影，通过该投影可以通过该投影将每个点嵌入可以汇总到多视图几何描述符中。对3D形状分类，部分分割和无监督学习的广泛实验验证了我们方法的优势。我们将公开提供代码和数据。

我们介绍了Amazon Berkeley对象（ABO），这是一个新的大型数据集，旨在帮助弥合真实和虚拟3D世界之间的差距。ABO包含产品目录图像，元数据和艺术家创建的3D模型，具有复杂的几何形状和与真实的家用物体相对应的物理基础材料。我们得出了具有挑战性的基准，这些基准利用ABO的独特属性，并测量最先进的对象在三个开放问题上的最新限制，以了解实际3D对象：单视3D 3D重建，材料估计和跨域多视图对象检索。

近年来，已经产生了大量的视觉内容，并从许多领域共享，例如社交媒体平台，医学成像和机器人。这种丰富的内容创建和共享引入了新的挑战，特别是在寻找类似内容内容的图像检索（CBIR）-A的数据库中，即长期建立的研究区域，其中需要改进的效率和准确性来实时检索。人工智能在CBIR中取得了进展，并大大促进了实例搜索过程。在本调查中，我们审查了最近基于深度学习算法和技术开发的实例检索工作，通过深网络架构类型，深度功能，功能嵌入方法以及网络微调策略组织了调查。我们的调查考虑了各种各样的最新方法，在那里，我们识别里程碑工作，揭示各种方法之间的联系，并呈现常用的基准，评估结果，共同挑战，并提出未来的未来方向。

We present a new deep learning architecture (called Kdnetwork) that is designed for 3D model recognition tasks and works with unstructured point clouds. The new architecture performs multiplicative transformations and shares parameters of these transformations according to the subdivisions of the point clouds imposed onto them by kdtrees. Unlike the currently dominant convolutional architectures that usually require rasterization on uniform twodimensional or three-dimensional grids, Kd-networks do not rely on such grids in any way and therefore avoid poor scaling behavior. In a series of experiments with popular shape recognition benchmarks, Kd-networks demonstrate competitive performance in a number of shape recognition tasks such as shape classification, shape retrieval and shape part segmentation.

我们建议在2D域中利用自我监督的技术来实现细粒度的3D形状分割任务。这是受到观察的启发：基于视图的表面表示比基于点云或体素占用率的3D对应物更有效地建模高分辨率表面细节和纹理。具体而言，给定3D形状，我们将其从多个视图中渲染，并在对比度学习框架内建立密集的对应学习任务。结果，与仅在2D或3D中使用自学的替代方案相比，学到的2D表示是视图不变和几何一致的，在对有限的标记形状进行培训时，可以更好地概括概括。对纹理（渲染peple）和未纹理（partnet）3D数据集的实验表明，我们的方法在细粒部分分割中优于最先进的替代方案。当仅一组稀疏的视图可供训练或形状纹理时，对基准的改进就会更大，这表明MVDecor受益于2D处理和3D几何推理。

我们提出了基于零件的复发多视距聚集网络（PREMA），以消除实际视图缺陷的不利影响，例如观点，闭塞或背景腔，以及增强形状表示的辨别能力。灵感来自人类主要通过其判别部分识别对象的事实，我们定义了多视图相干部分（MCP），判别部分在不同视图中再次进行。我们的Prema可以可靠地定位并有效地利用MCP来构建强大的形状表示。全面地，我们在Prema中设计一个新的区域关注单元（RAU），以计算每个视图的置信度图，并通过应用这些映射来查看特征来提取MCP。Prema通过关联不同视图的功能来突出MCP，并汇总为形状表示的零件感知功能。

综合照片 - 现实图像和视频是计算机图形的核心，并且是几十年的研究焦点。传统上，使用渲染算法（如光栅化或射线跟踪）生成场景的合成图像，其将几何形状和材料属性的表示为输入。统称，这些输入定义了实际场景和呈现的内容，并且被称为场景表示（其中场景由一个或多个对象组成）。示例场景表示是具有附带纹理的三角形网格（例如，由艺术家创建），点云（例如，来自深度传感器），体积网格（例如，来自CT扫描）或隐式曲面函数（例如，截短的符号距离）字段）。使用可分辨率渲染损耗的观察结果的这种场景表示的重建被称为逆图形或反向渲染。神经渲染密切相关，并将思想与经典计算机图形和机器学习中的思想相结合，以创建用于合成来自真实观察图像的图像的算法。神经渲染是朝向合成照片现实图像和视频内容的目标的跨越。近年来，我们通过数百个出版物显示了这一领域的巨大进展，这些出版物显示了将被动组件注入渲染管道的不同方式。这种最先进的神经渲染进步的报告侧重于将经典渲染原则与学习的3D场景表示结合的方法，通常现在被称为神经场景表示。这些方法的一个关键优势在于它们是通过设计的3D-一致，使诸如新颖的视点合成捕获场景的应用。除了处理静态场景的方法外，我们还涵盖了用于建模非刚性变形对象的神经场景表示...