我们介绍了一种简单而有效的算法，它使用卷积神经网络直接从视频中估计对象。我们的方法利用了视频序列的时间信息，并计算了支持机器人和AR域的计算上高效且鲁棒。我们所提出的网络采用预先训练的2D对象检测器作为输入，并通过经常性神经网络聚合视觉特征以在每个帧处进行预测。YCB-Video数据集的实验评估表明，我们的方法与最先进的算法相提并论。此外，通过30 FPS的速度，它也比现有技术更有效，因此适用于需要实时对象姿态估计的各种应用。

We introduce a Transformer based 6D Object Pose Estimation framework VideoPose, comprising an end-to-end attention based modelling architecture, that attends to previous frames in order to estimate accurate 6D Object Poses in videos. Our approach leverages the temporal information from a video sequence for pose refinement, along with being computationally efficient and robust. Compared to existing methods, our architecture is able to capture and reason from long-range dependencies efficiently, thus iteratively refining over video sequences. Experimental evaluation on the YCB-Video dataset shows that our approach is on par with the state-of-the-art Transformer methods, and performs significantly better relative to CNN based approaches. Further, with a speed of 33 fps, it is also more efficient and therefore applicable to a variety of applications that require real-time object pose estimation. Training code and pretrained models are available at https://github.com/ApoorvaBeedu/VideoPose

Estimating the 6D pose of known objects is important for robots to interact with the real world. The problem is challenging due to the variety of objects as well as the complexity of a scene caused by clutter and occlusions between objects. In this work, we introduce PoseCNN, a new Convolutional Neural Network for 6D object pose estimation. PoseCNN estimates the 3D translation of an object by localizing its center in the image and predicting its distance from the camera. The 3D rotation of the object is estimated by regressing to a quaternion representation. We also introduce a novel loss function that enables PoseCNN to handle symmetric objects. In addition, we contribute a large scale video dataset for 6D object pose estimation named the YCB-Video dataset. Our dataset provides accurate 6D poses of 21 objects from the YCB dataset observed in 92 videos with 133,827 frames. We conduct extensive experiments on our YCB-Video dataset and the OccludedLINEMOD dataset to show that PoseCNN is highly robust to occlusions, can handle symmetric objects, and provide accurate pose estimation using only color images as input. When using depth data to further refine the poses, our approach achieves state-of-the-art results on the challenging OccludedLINEMOD dataset. Our code and dataset are available at https://rse-lab.cs.washington.edu/projects/posecnn/.

Estimating 6D poses of objects from images is an important problem in various applications such as robot manipulation and virtual reality. While direct regression of images to object poses has limited accuracy, matching rendered images of an object against the input image can produce accurate results. In this work, we propose a novel deep neural network for 6D pose matching named DeepIM. Given an initial pose estimation, our network is able to iteratively refine the pose by matching the rendered image against the observed image. The network is trained to predict a relative pose transformation using a disentangled representation of 3D location and 3D orientation and an iterative training process. Experiments on two commonly used benchmarks for 6D pose estimation demonstrate that DeepIM achieves large improvements over stateof-the-art methods. We furthermore show that DeepIM is able to match previously unseen objects.

A key technical challenge in performing 6D object pose estimation from RGB-D image is to fully leverage the two complementary data sources. Prior works either extract information from the RGB image and depth separately or use costly post-processing steps, limiting their performances in highly cluttered scenes and real-time applications. In this work, we present DenseFusion, a generic framework for estimating 6D pose of a set of known objects from RGB-D images. DenseFusion is a heterogeneous architecture that processes the two data sources individually and uses a novel dense fusion network to extract pixel-wise dense feature embedding, from which the pose is estimated. Furthermore, we integrate an end-to-end iterative pose refinement procedure that further improves the pose estimation while achieving near real-time inference. Our experiments show that our method outperforms state-of-the-art approaches in two datasets, YCB-Video and LineMOD. We also deploy our proposed method to a real robot to grasp and manipulate objects based on the estimated pose. Our code and video are available at https://sites.google.com/view/densefusion/.

估计对象的6D姿势是必不可少的计算机视觉任务。但是，大多数常规方法从单个角度依赖相机数据，因此遭受遮挡。我们通过称为MV6D的新型多视图6D姿势估计方法克服了这个问题，该方法从多个角度根据RGB-D图像准确地预测了混乱场景中所有对象的6D姿势。我们将方法以PVN3D网络为基础，该网络使用单个RGB-D图像来预测目标对象的关键点。我们通过从多个视图中使用组合点云来扩展此方法，并将每个视图中的图像与密集层层融合。与当前的多视图检测网络（例如Cosypose）相反，我们的MV6D可以以端到端的方式学习多个观点的融合，并且不需要多个预测阶段或随后对预测的微调。此外，我们介绍了三个新颖的影像学数据集，这些数据集具有沉重的遮挡的混乱场景。所有这些都从多个角度包含RGB-D图像，例如语义分割和6D姿势估计。即使在摄像头不正确的情况下，MV6D也明显优于多视图6D姿势估计中最新的姿势估计。此外，我们表明我们的方法对动态相机设置具有强大的态度，并且其准确性随着越来越多的观点而逐渐增加。

本文提出了一种类别级别的6D对象姿势和形状估计方法IDAPS，其允许在类别中跟踪6D姿势并估计其3D形状。我们使用深度图像作为输入开发类别级别自动编码器网络，其中来自自动编码器编码的特征嵌入在类别中对象的姿势。自动编码器可用于粒子过滤器框架，以估计和跟踪类别中的对象的姿势。通过利用基于符号距离函数的隐式形状表示，我们构建延迟网络以估计给定对象的估计姿势的3D形状的潜在表示。然后，估计的姿势和形状可用于以迭代方式互相更新。我们的类别级别6D对象姿势和形状估计流水线仅需要2D检测和分段进行初始化。我们在公开的数据集中评估我们的方法，并展示其有效性。特别是，我们的方法在形状估计上实现了相对高的准确性。

对象姿态估计有多个重要应用，例如机器人抓握和增强现实。我们提出了一种估计了提高当前提案的准确性的6D对象的6D姿势，仍然可以实时使用。我们的方法使用RGB-D数据作为段对象的输入并估计它们的姿势。它使用具有多个头部的神经网络，一个头估计对象分类并生成掩码，第二估计转换向量的值，最后一个头估计表示对象旋转的四元轴的值。这些头部利用特征提取和特征融合期间使用的金字塔架构。我们的方法可以实时使用，其低推理时间为0.12秒并具有高精度。通过这种快速推理和良好准确性的组合，可以在机器人挑选和放置任务和/或增强现实应用中使用我们的方法。

This paper addresses the challenge of 6DoF pose estimation from a single RGB image under severe occlusion or truncation. Many recent works have shown that a two-stage approach, which first detects keypoints and then solves a Perspective-n-Point (PnP) problem for pose estimation, achieves remarkable performance. However, most of these methods only localize a set of sparse keypoints by regressing their image coordinates or heatmaps, which are sensitive to occlusion and truncation. Instead, we introduce a Pixel-wise Voting Network (PVNet) to regress pixel-wise unit vectors pointing to the keypoints and use these vectors to vote for keypoint locations using RANSAC. This creates a flexible representation for localizing occluded or truncated keypoints. Another important feature of this representation is that it provides uncertainties of keypoint locations that can be further leveraged by the PnP solver. Experiments show that the proposed approach outperforms the state of the art on the LINEMOD, Occlusion LINEMOD and YCB-Video datasets by a large margin, while being efficient for real-time pose estimation. We further create a Truncation LINEMOD dataset to validate the robustness of our approach against truncation. The code will be avaliable at https://zju-3dv.github.io/pvnet/.

估计没有先验知识的新对象的相对姿势是一个困难的问题，而它是机器人技术和增强现实中非常需要的能力。我们提出了一种方法，可以在训练图像和对象的3D几何形状都没有可用时跟踪对象中对象的6D运动。因此，与以前的作品相反，我们的方法可以立即考虑开放世界中的未知对象，而无需任何先前的信息或特定的培训阶段。我们考虑两个架构，一个基于两个帧，另一个依赖于变压器编码器，它们可以利用任意数量的过去帧。我们仅使用具有域随机化的合成渲染训练架构。我们在具有挑战性的数据集上的结果与以前需要更多信息的作品（训练目标对象，3D模型和/或深度数据的培训图像）相当。我们的源代码可从https://github.com/nv-nguyen/pizza获得

We propose a single-shot approach for simultaneously detecting an object in an RGB image and predicting its 6D pose without requiring multiple stages or having to examine multiple hypotheses. Unlike a recently proposed single-shot technique for this task [11] that only predicts an approximate 6D pose that must then be refined, ours is accurate enough not to require additional post-processing. As a result, it is much faster -50 fps on a Titan X (Pascal) GPU -and more suitable for real-time processing. The key component of our method is a new CNN architecture inspired by [28,29] that directly predicts the 2D image locations of the projected vertices of the object's 3D bounding box. The object's 6D pose is then estimated using a PnP algorithm.For single object and multiple object pose estimation on the LINEMOD and OCCLUSION datasets, our approach substantially outperforms other recent 26] when they are all used without postprocessing. During post-processing, a pose refinement step can be used to boost the accuracy of these two methods, but at 10 fps or less, they are much slower than our method.

我们提出了一种称为DPODV2（密集姿势对象检测器）的三个阶段6 DOF对象检测方法，该方法依赖于致密的对应关系。我们将2D对象检测器与密集的对应关系网络和多视图姿势细化方法相结合，以估计完整的6 DOF姿势。与通常仅限于单眼RGB图像的其他深度学习方法不同，我们提出了一个统一的深度学习网络，允许使用不同的成像方式（RGB或DEPTH）。此外，我们提出了一种基于可区分渲染的新型姿势改进方法。主要概念是在多个视图中比较预测并渲染对应关系，以获得与所有视图中预测的对应关系一致的姿势。我们提出的方法对受控设置中的不同数据方式和培训数据类型进行了严格的评估。主要结论是，RGB在对应性估计中表现出色，而如果有良好的3D-3D对应关系，则深度有助于姿势精度。自然，他们的组合可以实现总体最佳性能。我们进行广泛的评估和消融研究，以分析和验证几个具有挑战性的数据集的结果。 DPODV2在所有这些方面都取得了出色的成果，同时仍然保持快速和可扩展性，独立于使用的数据模式和培训数据的类型

从RGB-D图像中对刚性对象的6D姿势估计对于机器人技术中的对象抓握和操纵至关重要。尽管RGB通道和深度（d）通道通常是互补的，分别提供了外观和几何信息，但如何完全从两个跨模式数据中完全受益仍然是非平凡的。从简单而新的观察结果来看，当对象旋转时，其语义标签是姿势不变的，而其关键点偏移方向是姿势的变体。为此，我们提出了So（3）pose，这是一个新的表示学习网络，可以探索SO（3）equivariant和So（3） - 从深度通道中进行姿势估计的特征。 SO（3） - 激素特征有助于学习更独特的表示，以分割来自RGB通道外观相似的对象。 SO（3） - 等级特征与RGB功能通信，以推导（缺失的）几何形状，以检测从深度通道的反射表面的对象的关键点。与大多数现有的姿势估计方法不同，我们的SO（3） - 不仅可以实现RGB和深度渠道之间的信息通信，而且自然会吸收SO（3） - 等级的几何学知识，从深度图像中，导致更好的外观和更好的外观和更好几何表示学习。综合实验表明，我们的方法在三个基准测试中实现了最先进的性能。

We introduce MegaPose, a method to estimate the 6D pose of novel objects, that is, objects unseen during training. At inference time, the method only assumes knowledge of (i) a region of interest displaying the object in the image and (ii) a CAD model of the observed object. The contributions of this work are threefold. First, we present a 6D pose refiner based on a render&compare strategy which can be applied to novel objects. The shape and coordinate system of the novel object are provided as inputs to the network by rendering multiple synthetic views of the object's CAD model. Second, we introduce a novel approach for coarse pose estimation which leverages a network trained to classify whether the pose error between a synthetic rendering and an observed image of the same object can be corrected by the refiner. Third, we introduce a large-scale synthetic dataset of photorealistic images of thousands of objects with diverse visual and shape properties and show that this diversity is crucial to obtain good generalization performance on novel objects. We train our approach on this large synthetic dataset and apply it without retraining to hundreds of novel objects in real images from several pose estimation benchmarks. Our approach achieves state-of-the-art performance on the ModelNet and YCB-Video datasets. An extensive evaluation on the 7 core datasets of the BOP challenge demonstrates that our approach achieves performance competitive with existing approaches that require access to the target objects during training. Code, dataset and trained models are available on the project page: https://megapose6d.github.io/.

We introduce a novel method for 3D object detection and pose estimation from color images only. We first use segmentation to detect the objects of interest in 2D even in presence of partial occlusions and cluttered background. By contrast with recent patch-based methods, we rely on a "holistic" approach: We apply to the detected objects a Convolutional Neural Network (CNN) trained to predict their 3D poses in the form of 2D projections of the corners of their 3D bounding boxes. This, however, is not sufficient for handling objects from the recent T-LESS dataset: These objects exhibit an axis of rotational symmetry, and the similarity of two images of such an object under two different poses makes training the CNN challenging. We solve this problem by restricting the range of poses used for training, and by introducing a classifier to identify the range of a pose at run-time before estimating it. We also use an optional additional step that refines the predicted poses. We improve the state-of-the-art on the LINEMOD dataset from 73.7% [2] to 89.3% of correctly registered RGB frames. We are also the first to report results on the Occlusion dataset [1] using color images only. We obtain 54% of frames passing the Pose 6D criterion on average on several sequences of the T-LESS dataset, compared to the 67% of the state-of-the-art [10] on the same sequences which uses both color and depth. The full approach is also scalable, as a single network can be trained for multiple objects simultaneously.

很少有6D姿势估计方法使用骨干网络从RGB和深度图像中提取功能，而Uni6D是这样做的先驱。我们发现UNI6D中性能限制的主要原因是实例外部和实例 - 内噪声。 uni6d不可避免地会由于其固有的直接管道设计而从接收场中的背景像素引入实例外部噪声，并忽略了输入深度数据中的实例 - 内侧噪声。在这项工作中，我们提出了一种两步的denoising方法，以处理UNI6D中上述噪声。在第一步中，实例分割网络用于裁剪和掩盖实例，以消除非实施区域的噪声。在第二步中，提出了一个轻巧的深度剥夺模块，以校准深度特征，然后再将其输入姿势回归网络。广泛的实验表明，我们称为uni6dv2的方法能够有效，稳健地消除噪声，在不牺牲过多的推理效率的情况下超过UNI6D。它还减少了对需要昂贵标签的注释真实数据的需求。

We introduce an approach for recovering the 6D pose of multiple known objects in a scene captured by a set of input images with unknown camera viewpoints. First, we present a single-view single-object 6D pose estimation method, which we use to generate 6D object pose hypotheses. Second, we develop a robust method for matching individual 6D object pose hypotheses across different input images in order to jointly estimate camera viewpoints and 6D poses of all objects in a single consistent scene. Our approach explicitly handles object symmetries, does not require depth measurements, is robust to missing or incorrect object hypotheses, and automatically recovers the number of objects in the scene. Third, we develop a method for global scene refinement given multiple object hypotheses and their correspondences across views. This is achieved by solving an object-level bundle adjustment problem that refines the poses of cameras and objects to minimize the reprojection error in all views. We demonstrate that the proposed method, dubbed Cosy-Pose, outperforms current state-of-the-art results for single-view and multi-view 6D object pose estimation by a large margin on two challenging benchmarks: the YCB-Video and T-LESS datasets. Code and pre-trained models are available on the project webpage. 5

从RGB图像中对刚性对象进行精确的6D构成估计是机器人技术和增强现实中的一项至关重要的任务。为了解决这个问题，我们提出了DeepRM，这是一种新型的经过精炼的新型经过的网络体系结构。 DeepRM利用初始粗姿势估计来渲染目标对象的合成图像。然后将渲染图像与观察到的图像匹配，以预测更新先前姿势估计值的刚性变换。重复此过程以逐步完善每次迭代的估计值。 LSTM单元用于通过每个完善步骤来传播信息，从而显着提高整体性能。与许多基于2阶段的透视点解决方案相反，DEEPRM是端到端训练的，并使用可扩展的主链，可以通过单个参数调整以提高准确性和效率。在训练过程中，添加了多尺度的光流头，以预测观察到的和合成图像之间的光流。光流预测稳定了训练过程，并强制学习与姿势估计任务相关的功能。我们的结果表明，DEEPRM在两个广泛接受的具有挑战性的数据集上实现了最先进的性能。

我们提出了一个基于按键的对象级别的SLAM框架，该框架可以为对称和不对称对象提供全球一致的6DOF姿势估计。据我们所知，我们的系统是最早利用来自SLAM的相机姿势信息的系统之一，以提供先验知识，以跟踪对称对象的关键点 - 确保新测量与当前的3D场景一致。此外，我们的语义关键点网络经过训练，可以预测捕获预测的真实错误的关键点的高斯协方差，因此不仅可以作为系统优化问题中残留物的权重，而且还可以作为检测手段有害的统计异常值，而无需选择手动阈值。实验表明，我们的方法以6DOF对象姿势估算和实时速度为最先进的状态提供了竞争性能。我们的代码，预培训模型和关键点标签可用https://github.com/rpng/suo_slam。

在这项工作中，我们解决了共同跟踪手对象姿势并从野外深度点云序列重建形状的具有挑战性，HandTrackNet，以估计框架间的手动运动。我们的HandTrackNet提出了一个新型的手姿势构成典型化模块，以简化跟踪任务，从而产生准确且稳健的手工关节跟踪。然后，我们的管道通过将预测的手关节转换为基于模板的参数手模型mano来重建全手。对于对象跟踪，我们设计了一个简单而有效的模块，该模块从第一帧估算对象SDF并执行基于优化的跟踪。最后，采用联合优化步骤执行联合手和物体推理，从而减轻了闭塞引起的歧义并进一步完善了手姿势。在训练过程中，整个管道仅看到纯粹的合成数据，这些数据与足够的变化并通过深度模拟合成，以易于概括。整个管道与概括差距有关，因此可以直接传输到真实的野外数据。我们在两个真实的手对象交互数据集上评估我们的方法，例如HO3D和DEXYCB，没有任何填充。我们的实验表明，所提出的方法显着优于先前基于深度的手和对象姿势估计和跟踪方法，以9 fps的帧速率运行。