3D感知最近的进展在了解3DACHAPES甚至场景的几何结构方面表现出令人印象深刻的进展。灵感来自这些进步的几何理解，我们旨在利用几何约束下学到的表示基于图像的感知。我们介绍一种基于多视图RGB-D数据学习View-Invariant的方法，用于网络预训练的网络预训练的几何感知表示，然后可以将其有效地传送到下游2D任务。我们建议在多视图IM-ysge约束和图像 - 几何约束下采用对比学习，以便在学习的2D表示中进行编码。这不仅仅是在几乎非仅对图像的语义分割，实例分段和对象检测的基于图像的基于图像的基于图像的TASK上学习而改进，而且，但是，在低数据方案中提供了显着的改进。我们对全数据的语义细分显示6.0％的显着提高，以及剪刀上的基线20％数据上的11.9％。

我们提出了一种新的方法来将4D动态对象前瞻灌输到学习的3D表示，通过无监督的预训练。我们观察到对象通过环境的动态移动提供了关于其对象的重要提示，因此提出了利用这种动态理解的学习学习的3D表示，然后可以有效地传送到下游3D语义场景中的改进性能。我们提出了一种新的数据增强方案，利用静态3D环境中移动的合成3D形状，并在3D-4D约束下采用对比学习，该约束将4D Imormces编码到学习的3D表示中。实验表明，我们无监督的代表学习导致下游3D语义分割，对象检测和实例分割任务的改进，而且，显着提高了数据稀缺方案的性能。

Arguably one of the top success stories of deep learning is transfer learning. The finding that pre-training a network on a rich source set (e.g., ImageNet) can help boost performance once fine-tuned on a usually much smaller target set, has been instrumental to many applications in language and vision. Yet, very little is known about its usefulness in 3D point cloud understanding. We see this as an opportunity considering the effort required for annotating data in 3D. In this work, we aim at facilitating research on 3D representation learning. Different from previous works, we focus on high-level scene understanding tasks. To this end, we select a suite of diverse datasets and tasks to measure the effect of unsupervised pre-training on a large source set of 3D scenes. Our findings are extremely encouraging: using a unified triplet of architecture, source dataset, and contrastive loss for pre-training, we achieve improvement over recent best results in segmentation and detection across 6 different benchmarks for indoor and outdoor, real and synthetic datasets -demonstrating that the learned representation can generalize across domains. Furthermore, the improvement was similar to supervised pre-training, suggesting that future efforts should favor scaling data collection over more detailed annotation. We hope these findings will encourage more research on unsupervised pretext task design for 3D deep learning. Our code is publicly available at https://github.com/facebookresearch/PointContrast

由于缺乏大规模标记的3D数据集，大多数3D神经网络都是从划痕训练。在本文中，我们通过利用来自丰富的2D数据集学习的2D网络来介绍一种新的3D预预测方法。我们提出了通过将像素级和点级别特征映射到同一嵌入空间中的对比度的像素到点知识转移来有效地利用2D信息。由于2D和3D网络之间的异构性质，我们介绍了后投影功能以对准2D和3D之间的功能以使转移成为可能。此外，我们设计了一个上采样功能投影层，以增加高级2D特征图的空间分辨率，这使得能够学习细粒度的3D表示。利用普雷累染的2D网络，所提出的预介绍过程不需要额外的2D或3D标记数据，进一步缓解了昂贵的3D数据注释成本。据我们所知，我们是第一个利用现有的2D培训的权重，以预先rain 3D深度神经网络。我们的密集实验表明，使用2D知识预订的3D模型可以通过各种真实世界3D下游任务进行3D网络的性能。

我们建议在2D域中利用自我监督的技术来实现细粒度的3D形状分割任务。这是受到观察的启发：基于视图的表面表示比基于点云或体素占用率的3D对应物更有效地建模高分辨率表面细节和纹理。具体而言，给定3D形状，我们将其从多个视图中渲染，并在对比度学习框架内建立密集的对应学习任务。结果，与仅在2D或3D中使用自学的替代方案相比，学到的2D表示是视图不变和几何一致的，在对有限的标记形状进行培训时，可以更好地概括概括。对纹理（渲染peple）和未纹理（partnet）3D数据集的实验表明，我们的方法在细粒部分分割中优于最先进的替代方案。当仅一组稀疏的视图可供训练或形状纹理时，对基准的改进就会更大，这表明MVDecor受益于2D处理和3D几何推理。

深度神经网络的3D语义分割的最新进展已取得了显着的成功，并且可用数据集的性能快速提高。但是，当前的3D语义分割基准仅包含少数类别 - 例如，扫描仪和semantickitti少于30个类别，这些类别不足以反映真实环境的多样性（例如，语义图像涵盖数百到数千个类别的类别）。因此，我们建议研究3D语义分割的较大词汇，并在扫描仪数据上具有新的扩展基准测试，其中有200个类别类别，比以前研究的数量级要多。大量的类别类别也引起了巨大的自然级别不平衡，这两者对于现有的3D语义分割方法都具有挑战性。为了在这种情况下了解更多强大的3D功能，我们提出了一种以语言为导向的预训练方法来鼓励学习的3D功能，该方法可能有限的培训示例以靠近其预训练的文本嵌入。广泛的实验表明，我们的方法始终优于我们所提出的基准测试（ +9％相对MIOU）的3D语义分割的最先进的3D预训练，包括仅使用5％的 +25％相对MIOU的有限数据方案注释。

近年来，3D视觉的自我监督预训练引起了研究的兴趣。为了学习信息的表示，许多以前的作品都利用了3D功能的不向导，\ eg，同一场景的视图之间的透视感，深度和RGB图像之间的模态侵权次数，点云和voxels之间的格式不变。尽管他们取得了令人鼓舞的结果，但以前的研究缺乏对这些不稳定的系统性比较。为了解决这个问题，我们的工作首次引入了一个统一的框架，根据该框架可以研究各种预培训方法。我们进行了广泛的实验，并仔细研究了3D预训练中不同不变的贡献。另外，我们提出了一种简单但有效的方法，该方法可以共同预先培训3D编码器和使用对比度学习的深度图编码器。通过我们的方法进行预训练的模型在下游任务方面具有显着的性能提高。例如，预先训练的投票表现优于Sun RGB-D和扫描对象检测基准的先前方法，并具有明显的利润。

To date, most existing self-supervised learning methods are designed and optimized for image classification. These pre-trained models can be sub-optimal for dense prediction tasks due to the discrepancy between image-level prediction and pixel-level prediction. To fill this gap, we aim to design an effective, dense self-supervised learning method that directly works at the level of pixels (or local features) by taking into account the correspondence between local features. We present dense contrastive learning (DenseCL), which implements self-supervised learning by optimizing a pairwise contrastive (dis)similarity loss at the pixel level between two views of input images.Compared to the baseline method MoCo-v2, our method introduces negligible computation overhead (only <1% slower), but demonstrates consistently superior performance when transferring to downstream dense prediction tasks including object detection, semantic segmentation and instance segmentation; and outperforms the state-of-the-art methods by a large margin. Specifically, over the strong MoCo-v2 baseline, our method achieves significant improvements of 2.0% AP on PASCAL VOC object detection, 1.1% AP on COCO object detection, 0.9% AP on COCO instance segmentation, 3.0% mIoU on PASCAL VOC semantic segmentation and 1.8% mIoU on Cityscapes semantic segmentation.

最近的进展表明，使用对比图像文本对的大规模预训练可以是从自然语言监督的高质量视觉表演学习的有前途的替代方案。从更广泛的监督来源受益，这种新的范例展示了对下游分类任务和数据集的令人印象深刻的可转移性。然而，从图像文本对中学习的知识转移到更复杂的密集预测任务的问题几乎没有访问过。在这项工作中，我们通过隐式和明确地利用来自剪辑的预先训练的知识来提出了一种新的密集预测框架。具体地，我们将剪辑中的原始图像文本匹配问题转换为像素文本匹配问题，并使用像素文本分数图来指导致密预测模型的学习。通过进一步使用图像中的上下文信息来提示语言模型，我们能够促进我们的模型来更好地利用预先接受训练的知识。我们的方法是模型 - 不可行的，它可以应用于任意密集的预测系统和各种预先训练的视觉底座，包括夹模型和想象成预先训练的模型。广泛的实验证明了我们对语义分割，对象检测和实例分段任务的方法的卓越性能。代码可在https://github.com/raoyongming/denseclip获得

预训练已成为许多计算机视觉任务中的标准范式。但是，大多数方法通常都设计在RGB图像域上。由于二维图像平面和三维空间之间的差异，这种预先训练的模型无法感知空间信息，并用作3D相关任务的子最优解。为了弥合这种差距，我们的目标是学习可以描述三维空间的空间感知视觉表示，并且对这些任务更适合和有效。为了利用点云，在与图像相比提供空间信息时更有优越，我们提出了一个简单而有效的2D图像和3D点云无监督的预训练策略，称为Simipu。具体而言，我们开发了一种多模态对比学习框架，包括模态空间感知模块，用于从点云和模态特征交互模块中学习空间感知表示，以从点传输感知空间信息的能力云编码器分别到图像编码器。匹配算法和投影矩阵建立了用于对比损耗的正对。整个框架培训以无人监督的端到端时尚。据我们所知，这是第一项探索户外多模态数据集的对比学习训练策略的研究，其中包含配对的相机图像和LIDAR点云。 HTTPS：//github.com/zhever/simipu提供代码和模型。

Contrastive learning methods for unsupervised visual representation learning have reached remarkable levels of transfer performance. We argue that the power of contrastive learning has yet to be fully unleashed, as current methods are trained only on instance-level pretext tasks, leading to representations that may be sub-optimal for downstream tasks requiring dense pixel predictions. In this paper, we introduce pixel-level pretext tasks for learning dense feature representations. The first task directly applies contrastive learning at the pixel level. We additionally propose a pixel-to-propagation consistency task that produces better results, even surpassing the state-of-the-art approaches by a large margin. Specifically, it achieves 60.2 AP, 41.4 / 40.5 mAP and 77.2 mIoU when transferred to Pascal VOC object detection (C4), COCO object detection (FPN / C4) and Cityscapes semantic segmentation using a ResNet-50 backbone network, which are 2.6 AP, 0.8 / 1.0 mAP and 1.0 mIoU better than the previous best methods built on instance-level contrastive learning. Moreover, the pixel-level pretext tasks are found to be effective for pretraining not only regular backbone networks but also head networks used for dense downstream tasks, and are complementary to instance-level contrastive methods. These results demonstrate the strong potential of defining pretext tasks at the pixel level, and suggest a new path forward in unsupervised visual representation learning. Code is available at https://github.com/zdaxie/PixPro.

跨图像建立视觉对应是一项具有挑战性且必不可少的任务。最近，已经提出了大量的自我监督方法，以更好地学习视觉对应的表示。但是，我们发现这些方法通常无法利用语义信息，并且在低级功能的匹配方面过度融合。相反，人类的视觉能够将不同的物体区分为跟踪的借口。受此范式的启发，我们建议学习语义意识的细粒对应关系。首先，我们证明语义对应是通过一组丰富的图像级别自我监督方法隐式获得的。我们进一步设计了一个像素级的自我监督学习目标，该目标专门针对细粒的对应关系。对于下游任务，我们将这两种互补的对应表示形式融合在一起，表明它们是协同增强性能的。我们的方法超过了先前的最先进的自我监督方法，使用卷积网络在各种视觉通信任务上，包括视频对象分割，人姿势跟踪和人类部分跟踪。

我们提出了一种适用于半全球任务的自学学习（SSL）方法，例如对象检测和语义分割。我们通过在训练过程中最大程度地减少像素级局部对比度（LC）损失，代表了同一图像转换版本的相应图像位置之间的局部一致性。可以将LC-LOSS添加到以最小开销的现有自我监督学习方法中。我们使用可可，Pascal VOC和CityScapes数据集评估了两个下游任务的SSL方法 - 对象检测和语义细分。我们的方法的表现优于现有的最新SSL方法可可对象检测的方法1.9％，Pascal VOC检测1.4％，而CityScapes Sementation则为0.6％。

We propose a novel approach to self-supervised learning of point cloud representations by differentiable neural rendering. Motivated by the fact that informative point cloud features should be able to encode rich geometry and appearance cues and render realistic images, we train a point-cloud encoder within a devised point-based neural renderer by comparing the rendered images with real images on massive RGB-D data. The learned point-cloud encoder can be easily integrated into various downstream tasks, including not only high-level tasks like 3D detection and segmentation, but low-level tasks like 3D reconstruction and image synthesis. Extensive experiments on various tasks demonstrate the superiority of our approach compared to existing pre-training methods.

We present ObjectMatch, a semantic and object-centric camera pose estimation for RGB-D SLAM pipelines. Modern camera pose estimators rely on direct correspondences of overlapping regions between frames; however, they cannot align camera frames with little or no overlap. In this work, we propose to leverage indirect correspondences obtained via semantic object identification. For instance, when an object is seen from the front in one frame and from the back in another frame, we can provide additional pose constraints through canonical object correspondences. We first propose a neural network to predict such correspondences on a per-pixel level, which we then combine in our energy formulation with state-of-the-art keypoint matching solved with a joint Gauss-Newton optimization. In a pairwise setting, our method improves registration recall of state-of-the-art feature matching from 77% to 87% overall and from 21% to 52% in pairs with 10% or less inter-frame overlap. In registering RGB-D sequences, our method outperforms cutting-edge SLAM baselines in challenging, low frame-rate scenarios, achieving more than 35% reduction in trajectory error in multiple scenes.

We propose a new self-supervised method for pre-training the backbone of deep perception models operating on point clouds. The core idea is to train the model on a pretext task which is the reconstruction of the surface on which the 3D points are sampled, and to use the underlying latent vectors as input to the perception head. The intuition is that if the network is able to reconstruct the scene surface, given only sparse input points, then it probably also captures some fragments of semantic information, that can be used to boost an actual perception task. This principle has a very simple formulation, which makes it both easy to implement and widely applicable to a large range of 3D sensors and deep networks performing semantic segmentation or object detection. In fact, it supports a single-stream pipeline, as opposed to most contrastive learning approaches, allowing training on limited resources. We conducted extensive experiments on various autonomous driving datasets, involving very different kinds of lidars, for both semantic segmentation and object detection. The results show the effectiveness of our method to learn useful representations without any annotation, compared to existing approaches. Code is available at \href{https://github.com/valeoai/ALSO}{github.com/valeoai/ALSO}

大规模数据集的预培训模型，如想象成，是计算机视觉中的标准实践。此范例对于具有小型培训套的任务特别有效，其中高容量模型往往会过度装备。在这项工作中，我们考虑一个自我监督的预训练场景，只能利用目标任务数据。我们考虑数据集，如斯坦福汽车，草图或可可，这是比想象成小的数量的顺序。我们的研究表明，在本文中介绍的Beit或诸如Beit或Variant的去噪对预训练数据的类型和大小比通过比较图像嵌入来训练的流行自我监督方法更加强大。我们获得了竞争性能与ImageNet预训练相比，来自不同域的各种分类数据集。在Coco上，当专注于使用Coco Images进行预训练时，检测和实例分割性能超过了可比设置中的监督Imagenet预训练。

Masked Modeling (MM) has demonstrated widespread success in various vision challenges, by reconstructing masked visual patches. Yet, applying MM for large-scale 3D scenes remains an open problem due to the data sparsity and scene complexity. The conventional random masking paradigm used in 2D images often causes a high risk of ambiguity when recovering the masked region of 3D scenes. To this end, we propose a novel informative-preserved reconstruction, which explores local statistics to discover and preserve the representative structured points, effectively enhancing the pretext masking task for 3D scene understanding. Integrated with a progressive reconstruction manner, our method can concentrate on modeling regional geometry and enjoy less ambiguity for masked reconstruction. Besides, such scenes with progressive masking ratios can also serve to self-distill their intrinsic spatial consistency, requiring to learn the consistent representations from unmasked areas. By elegantly combining informative-preserved reconstruction on masked areas and consistency self-distillation from unmasked areas, a unified framework called MM-3DScene is yielded. We conduct comprehensive experiments on a host of downstream tasks. The consistent improvement (e.g., +6.1 mAP@0.5 on object detection and +2.2% mIoU on semantic segmentation) demonstrates the superiority of our approach.

蒙面图像建模（MIM）在各种视觉任务上取得了令人鼓舞的结果。但是，学到的表示形式的有限可区分性表现出来，使一个更强大的视力学习者还有很多值得一试。为了实现这一目标，我们提出了对比度蒙面的自动编码器（CMAE），这是一种新的自我监督的预训练方法，用于学习更全面和有能力的视觉表示。通过详细统一的对比度学习（CL）和掩盖图像模型（MIM），CMAE利用了它们各自的优势，并以强大的实例可辨别性和局部的可感知来学习表示形式。具体而言，CMAE由两个分支组成，其中在线分支是不对称的编码器编码器，而目标分支是动量更新的编码器。在培训期间，在线编码器从蒙面图像的潜在表示中重建了原始图像，以学习整体特征。馈送完整图像的目标编码器通过其在线学习通过对比度学习增强了功能可区分性。为了使CL与MIM兼容，CMAE引入了两个新组件，即用于生成合理的正视图和特征解码器的像素移位，以补充对比度对的特征。多亏了这些新颖的设计，CMAE可以有效地提高了MIM对应物的表示质量和转移性能。 CMAE在图像分类，语义分割和对象检测的高度竞争基准上实现了最先进的性能。值得注意的是，CMAE-BASE在Imagenet上获得了$ 85.3 \％$ $ TOP-1的准确性和$ 52.5 \％$ MIOU的ADE20K，分别超过了$ 0.7 \％\％$ $和$ 1.8 \％$ $。代码将公开可用。

本文介绍了密集的暹罗网络（Denseiam），这是一个简单的无监督学习框架，用于密集的预测任务。它通过以两种类型的一致性（即像素一致性和区域一致性）之间最大化一个图像的两个视图之间的相似性来学习视觉表示。具体地，根据重叠区域中的确切位置对应关系，Denseiam首先最大化像素级的空间一致性。它还提取一批与重叠区域中某些子区域相对应的区域嵌入，以形成区域一致性。与以前需要负像素对，动量编码器或启发式面膜的方法相反，Denseiam受益于简单的暹罗网络，并优化了不同粒度的一致性。它还证明了简单的位置对应关系和相互作用的区域嵌入足以学习相似性。我们将Denseiam应用于ImageNet，并在各种下游任务上获得竞争性改进。我们还表明，只有在一些特定于任务的损失中，简单的框架才能直接执行密集的预测任务。在现有的无监督语义细分基准中，它以2.1 miou的速度超过了最新的细分方法，培训成本为28％。代码和型号在https://github.com/zwwwayne/densesiam上发布。