同时对象识别和姿势估计是机器人安全与人类和环境安全相互作用的两个关键功能。尽管对象识别和姿势估计都使用视觉输入，但大多数最先进的问题将它们作为两个独立的问题解决，因为前者需要视图不变的表示，而对象姿势估计需要一个与观点有关的描述。如今，多视图卷积神经网络（MVCNN）方法显示出最新的分类性能。尽管已广泛探索了MVCNN对象识别，但对多视图对象构成估计方法的研究很少，而同时解决这两个问题的研究更少。 MVCNN方法中虚拟摄像机的姿势通常是预先定义的，从而绑定了这种方法的应用。在本文中，我们提出了一种能够同时处理对象识别和姿势估计的方法。特别是，我们开发了一个深度的对象不合时宜的熵估计模型，能够预测给定3D对象的最佳观点。然后将对象的视图馈送到网络中，以同时预测目标对象的姿势和类别标签。实验结果表明，从此类位置获得的观点足以达到良好的精度得分。此外，我们设计了现实生活中的饮料场景，以证明拟议方法在真正的机器人任务中的运作效果如何。代码可在线获得：github.com/subhadityamukherjee/more_mvcnn

在以人为本的环境中工作的机器人需要知道场景中存在哪种物体，以及如何掌握和操纵不同情况下的各种对象，以帮助人类在日常任务中。因此，对象识别和抓握是此类机器人的两个关键功能。最先进的解决物体识别并将其抓握为两个单独的问题，同时都使用可视输入。此外，在训练阶段之后，机器人的知识是固定的。在这种情况下，如果机器人面临新的对象类别，则必须从划痕中重新培训以结合新信息而无需灾难性干扰。为了解决这个问题，我们提出了一个深入的学习架构，具有增强的存储器能力来处理开放式对象识别和同时抓握。特别地，我们的方法将物体的多视图作为输入，并共同估计像素 - 方向掌握配置以及作为输出的深度和旋转不变表示。然后通过元主动学习技术使用所获得的表示用于开放式对象识别。我们展示了我们掌握从未见过的对象的方法的能力，并在模拟和现实世界中使用非常少数的例子在现场使用很少的例子快速学习新的对象类别。

如今，机器人在我们的日常生活中起着越来越重要的作用。在以人为本的环境中，机器人经常会遇到成堆的对象，包装的项目或孤立的对象。因此，机器人必须能够在各种情况下掌握和操纵不同的物体，以帮助人类进行日常任务。在本文中，我们提出了一种多视图深度学习方法，以处理以人为中心的域中抓住强大的对象。特别是，我们的方法将任意对象的点云作为输入，然后生成给定对象的拼字图。获得的视图最终用于估计每个对象的像素抓握合成。我们使用小对象抓住数据集训练模型端到端，并在模拟和现实世界数据上对其进行测试，而无需进行任何进一步的微调。为了评估所提出方法的性能，我们在三种情况下进行了广泛的实验集，包括孤立的对象，包装的项目和一堆对象。实验结果表明，我们的方法在所有仿真和现实机器人方案中都表现出色，并且能够在各种场景配置中实现新颖对象的可靠闭环抓握。

服务机器人越来越多地整合到我们的日常生活中，以帮助我们完成各种任务。在这样的环境中，机器人在环境中工作时经常面临新对象，并且需要以开放式的方式学习它们。此外，此类机器人必须能够识别广泛的对象类别。在本文中，我们基于多种表示形式提出了终生的合奏学习方法，以解决少数弹出的对象识别问题。特别是，我们根据深度表示和手工制作的3D形状描述符形成集合方法。为了促进终身学习，每种方法都配备了一个可立即存储和检索对象信息的内存单元。所提出的模型适用于3D对象类别的数量未固定并且可以随着时间而增长的开放式学习方案。我们已经进行了广泛的实验集，以评估离线方法和开放式场景中提出的方法的性能。出于评估目的，除了真实的对象数据集外，我们还生成了一个大型合成家庭对象数据集，该对象由27000个对象组成。实验结果证明了所提出的方法在在线少数3D对象识别任务中的有效性，以及其在最先进的开放式学习方法上的出色表现。此外，我们的结果表明，虽然合奏学习在离线设置中是适度的好处，但它在终生的几次学习情况下是显着有益的。此外，我们在模拟和现实机器人设置中证明了方法的有效性，在该设置中，机器人从有限的示例中迅速学习了新类别。

3D shape models are becoming widely available and easier to capture, making available 3D information crucial for progress in object classification. Current state-of-theart methods rely on CNNs to address this problem. Recently, we witness two types of CNNs being developed: CNNs based upon volumetric representations versus CNNs based upon multi-view representations. Empirical results from these two types of CNNs exhibit a large gap, indicating that existing volumetric CNN architectures and approaches are unable to fully exploit the power of 3D representations. In this paper, we aim to improve both volumetric CNNs and multi-view CNNs according to extensive analysis of existing approaches. To this end, we introduce two distinct network architectures of volumetric CNNs. In addition, we examine multi-view CNNs, where we introduce multiresolution filtering in 3D. Overall, we are able to outperform current state-of-the-art methods for both volumetric CNNs and multi-view CNNs. We provide extensive experiments designed to evaluate underlying design choices, thus providing a better understanding of the space of methods available for object classification on 3D data.

实时机器人掌握，支持随后的精确反对操作任务，是高级高级自治系统的优先目标。然而，尚未找到这样一种可以用时间效率进行充分准确的掌握的算法。本文提出了一种新的方法，其具有2阶段方法，它使用深神经网络结合快速的2D对象识别，以及基于点对特征框架的随后的精确和快速的6D姿态估计来形成实时3D对象识别和抓握解决方案能够多对象类场景。所提出的解决方案有可能在实时应用上稳健地进行，需要效率和准确性。为了验证我们的方法，我们进行了广泛且彻底的实验，涉及我们自己的数据集的费力准备。实验结果表明，该方法在5CM5DEG度量标准中的精度97.37％，平均距离度量分数99.37％。实验结果显示了通过使用该方法的总体62％的相对改善（5cm5deg度量）和52.48％（平均距离度量）。此外，姿势估计执行也显示出运行时间的平均改善47.6％。最后，为了说明系统在实时操作中的整体效率，进行了一个拾取和放置的机器人实验，并显示了90％的准确度的令人信服的成功率。此实验视频可在https://sites.google.com/view/dl-ppf6dpose/上获得。

A longstanding question in computer vision concerns the representation of 3D shapes for recognition: should 3D shapes be represented with descriptors operating on their native 3D formats, such as voxel grid or polygon mesh, or can they be effectively represented with view-based descriptors? We address this question in the context of learning to recognize 3D shapes from a collection of their rendered views on 2D images. We first present a standard CNN architecture trained to recognize the shapes' rendered views independently of each other, and show that a 3D shape can be recognized even from a single view at an accuracy far higher than using state-of-the-art 3D shape descriptors. Recognition rates further increase when multiple views of the shapes are provided. In addition, we present a novel CNN architecture that combines information from multiple views of a 3D shape into a single and compact shape descriptor offering even better recognition performance. The same architecture can be applied to accurately recognize human hand-drawn sketches of shapes. We conclude that a collection of 2D views can be highly informative for 3D shape recognition and is amenable to emerging CNN architectures and their derivatives.

抓握是通过在一组触点上施加力和扭矩来挑选对象的过程。深度学习方法的最新进展允许在机器人对象抓地力方面快速进步。我们在过去十年中系统地调查了出版物，特别感兴趣使用最终效果姿势的所有6度自由度抓住对象。我们的综述发现了四种用于机器人抓钩的常见方法：基于抽样的方法，直接回归，强化学习和示例方法。此外，我们发现了围绕抓握的两种“支持方法”，这些方法使用深入学习来支持抓握过程，形状近似和负担能力。我们已经将本系统评论（85篇论文）中发现的出版物提炼为十个关键要点，我们认为对未来的机器人抓握和操纵研究至关重要。该调查的在线版本可从https://rhys-newbury.github.io/projects/6dof/获得

许多涉及某种形式的3D视觉感知的机器人任务极大地受益于对工作环境的完整知识。但是，机器人通常必须应对非结构化的环境，并且由于工作空间有限，混乱或对象自我划分，它们的车载视觉传感器只能提供不完整的信息。近年来，深度学习架构的形状完成架构已开始将牵引力作为从部分视觉数据中推断出完整的3D对象表示的有效手段。然而，大多数现有的最新方法都以体素电网形式提供了固定的输出分辨率，这与神经网络输出阶段的大小严格相关。尽管这足以完成某些任务，例如导航，抓握和操纵的障碍需要更精细的分辨率，并且简单地扩大神经网络输出在计算上是昂贵的。在本文中，我们通过基于隐式3D表示的对象形状完成方法来解决此限制，该方法为每个重建点提供了置信值。作为第二个贡献，我们提出了一种基于梯度的方法，用于在推理时在任意分辨率下有效地采样这种隐式函数。我们通过将重建的形状与地面真理进行比较，并通过在机器人握把管道中部署形状完成算法来实验验证我们的方法。在这两种情况下，我们将结果与最先进的形状完成方法进行了比较。

我们提出了一种称为DPODV2（密集姿势对象检测器）的三个阶段6 DOF对象检测方法，该方法依赖于致密的对应关系。我们将2D对象检测器与密集的对应关系网络和多视图姿势细化方法相结合，以估计完整的6 DOF姿势。与通常仅限于单眼RGB图像的其他深度学习方法不同，我们提出了一个统一的深度学习网络，允许使用不同的成像方式（RGB或DEPTH）。此外，我们提出了一种基于可区分渲染的新型姿势改进方法。主要概念是在多个视图中比较预测并渲染对应关系，以获得与所有视图中预测的对应关系一致的姿势。我们提出的方法对受控设置中的不同数据方式和培训数据类型进行了严格的评估。主要结论是，RGB在对应性估计中表现出色，而如果有良好的3D-3D对应关系，则深度有助于姿势精度。自然，他们的组合可以实现总体最佳性能。我们进行广泛的评估和消融研究，以分析和验证几个具有挑战性的数据集的结果。 DPODV2在所有这些方面都取得了出色的成果，同时仍然保持快速和可扩展性，独立于使用的数据模式和培训数据的类型

Point cloud learning has lately attracted increasing attention due to its wide applications in many areas, such as computer vision, autonomous driving, and robotics. As a dominating technique in AI, deep learning has been successfully used to solve various 2D vision problems. However, deep learning on point clouds is still in its infancy due to the unique challenges faced by the processing of point clouds with deep neural networks. Recently, deep learning on point clouds has become even thriving, with numerous methods being proposed to address different problems in this area. To stimulate future research, this paper presents a comprehensive review of recent progress in deep learning methods for point clouds. It covers three major tasks, including 3D shape classification, 3D object detection and tracking, and 3D point cloud segmentation. It also presents comparative results on several publicly available datasets, together with insightful observations and inspiring future research directions.

We introduce MegaPose, a method to estimate the 6D pose of novel objects, that is, objects unseen during training. At inference time, the method only assumes knowledge of (i) a region of interest displaying the object in the image and (ii) a CAD model of the observed object. The contributions of this work are threefold. First, we present a 6D pose refiner based on a render&compare strategy which can be applied to novel objects. The shape and coordinate system of the novel object are provided as inputs to the network by rendering multiple synthetic views of the object's CAD model. Second, we introduce a novel approach for coarse pose estimation which leverages a network trained to classify whether the pose error between a synthetic rendering and an observed image of the same object can be corrected by the refiner. Third, we introduce a large-scale synthetic dataset of photorealistic images of thousands of objects with diverse visual and shape properties and show that this diversity is crucial to obtain good generalization performance on novel objects. We train our approach on this large synthetic dataset and apply it without retraining to hundreds of novel objects in real images from several pose estimation benchmarks. Our approach achieves state-of-the-art performance on the ModelNet and YCB-Video datasets. An extensive evaluation on the 7 core datasets of the BOP challenge demonstrates that our approach achieves performance competitive with existing approaches that require access to the target objects during training. Code, dataset and trained models are available on the project page: https://megapose6d.github.io/.

Estimating 6D poses of objects from images is an important problem in various applications such as robot manipulation and virtual reality. While direct regression of images to object poses has limited accuracy, matching rendered images of an object against the input image can produce accurate results. In this work, we propose a novel deep neural network for 6D pose matching named DeepIM. Given an initial pose estimation, our network is able to iteratively refine the pose by matching the rendered image against the observed image. The network is trained to predict a relative pose transformation using a disentangled representation of 3D location and 3D orientation and an iterative training process. Experiments on two commonly used benchmarks for 6D pose estimation demonstrate that DeepIM achieves large improvements over stateof-the-art methods. We furthermore show that DeepIM is able to match previously unseen objects.

We present a learnt system for multi-view stereopsis. In contrast to recent learning based methods for 3D reconstruction, we leverage the underlying 3D geometry of the problem through feature projection and unprojection along viewing rays. By formulating these operations in a differentiable manner, we are able to learn the system end-to-end for the task of metric 3D reconstruction. End-to-end learning allows us to jointly reason about shape priors while conforming to geometric constraints, enabling reconstruction from much fewer images (even a single image) than required by classical approaches as well as completion of unseen surfaces. We thoroughly evaluate our approach on the ShapeNet dataset and demonstrate the benefits over classical approaches and recent learning based methods.

当代掌握检测方法采用深度学习，实现传感器和物体模型不确定性的鲁棒性。这两个主导的方法设计了掌握质量评分或基于锚的掌握识别网络。本文通过将其视为图像空间中的关键点检测来掌握掌握检测的不同方法。深网络检测每个掌握候选者作为一对关键点，可转换为掌握代表= {x，y，w，{\ theta}} t，而不是转角点的三态或四重奏。通过将关键点分组成对来降低检测难度提高性能。为了促进捕获关键点之间的依赖关系，将非本地模块结合到网络设计中。基于离散和连续定向预测的最终过滤策略消除了错误的对应关系，并进一步提高了掌握检测性能。此处提出的方法GKNET在康奈尔和伸缩的提花数据集上的精度和速度之间实现了良好的平衡（在41.67和23.26 fps的96.9％和98.39％）之间。操纵器上的后续实验使用4种类型的抓取实验来评估GKNet，反映不同滋扰的速度：静态抓握，动态抓握，在各种相机角度抓住，夹住。 GKNet优于静态和动态掌握实验中的参考基线，同时表现出变化的相机观点和中度杂波的稳健性。结果证实了掌握关键点是深度掌握网络的有效输出表示的假设，为预期的滋扰因素提供鲁棒性。

6D object pose estimation problem has been extensively studied in the field of Computer Vision and Robotics. It has wide range of applications such as robot manipulation, augmented reality, and 3D scene understanding. With the advent of Deep Learning, many breakthroughs have been made; however, approaches continue to struggle when they encounter unseen instances, new categories, or real-world challenges such as cluttered backgrounds and occlusions. In this study, we will explore the available methods based on input modality, problem formulation, and whether it is a category-level or instance-level approach. As a part of our discussion, we will focus on how 6D object pose estimation can be used for understanding 3D scenes.

We introduce an approach for recovering the 6D pose of multiple known objects in a scene captured by a set of input images with unknown camera viewpoints. First, we present a single-view single-object 6D pose estimation method, which we use to generate 6D object pose hypotheses. Second, we develop a robust method for matching individual 6D object pose hypotheses across different input images in order to jointly estimate camera viewpoints and 6D poses of all objects in a single consistent scene. Our approach explicitly handles object symmetries, does not require depth measurements, is robust to missing or incorrect object hypotheses, and automatically recovers the number of objects in the scene. Third, we develop a method for global scene refinement given multiple object hypotheses and their correspondences across views. This is achieved by solving an object-level bundle adjustment problem that refines the poses of cameras and objects to minimize the reprojection error in all views. We demonstrate that the proposed method, dubbed Cosy-Pose, outperforms current state-of-the-art results for single-view and multi-view 6D object pose estimation by a large margin on two challenging benchmarks: the YCB-Video and T-LESS datasets. Code and pre-trained models are available on the project webpage. 5

Understanding the 3D world without supervision is currently a major challenge in computer vision as the annotations required to supervise deep networks for tasks in this domain are expensive to obtain on a large scale. In this paper, we address the problem of unsupervised viewpoint estimation. We formulate this as a self-supervised learning task, where image reconstruction provides the supervision needed to predict the camera viewpoint. Specifically, we make use of pairs of images of the same object at training time, from unknown viewpoints, to self-supervise training by combining the viewpoint information from one image with the appearance information from the other. We demonstrate that using a perspective spatial transformer allows efficient viewpoint learning, outperforming existing unsupervised approaches on synthetic data, and obtains competitive results on the challenging PASCAL3D+ dataset.

形状通知如何将对象掌握，无论是如何以及如何。因此，本文介绍了一种基于分割的架构，用于将用深度摄像机进行分解为多个基本形状的对象，以及用于机器人抓握的后处理管道。分段采用深度网络，称为PS-CNN，在具有6个类的原始形状和使用模拟引擎生成的合成数据上培训。每个原始形状都设计有参数化掌握家族，允许管道识别每个形状区域的多个掌握候选者。掌握是排序的排名，选择用于执行的第一个可行的。对于无任务掌握单个对象，该方法达到94.2％的成功率将其放置在顶部执行掌握方法中，与自上而下和SE（3）基础相比。涉及变量观点和杂波的其他测试展示了设置的鲁棒性。对于面向任务的掌握，PS-CNN实现了93.0％的成功率。总体而言，结果支持该假设，即在抓地管道内明确地编码形状原语应该提高掌握性能，包括无任务和任务相关的掌握预测。

Grasp learning has become an exciting and important topic in robotics. Just a few years ago, the problem of grasping novel objects from unstructured piles of clutter was considered a serious research challenge. Now, it is a capability that is quickly becoming incorporated into industrial supply chain automation. How did that happen? What is the current state of the art in robotic grasp learning, what are the different methodological approaches, and what machine learning models are used? This review attempts to give an overview of the current state of the art of grasp learning research.