近年来，3D视觉的自我监督预训练引起了研究的兴趣。为了学习信息的表示，许多以前的作品都利用了3D功能的不向导，\ eg，同一场景的视图之间的透视感，深度和RGB图像之间的模态侵权次数，点云和voxels之间的格式不变。尽管他们取得了令人鼓舞的结果，但以前的研究缺乏对这些不稳定的系统性比较。为了解决这个问题，我们的工作首次引入了一个统一的框架，根据该框架可以研究各种预培训方法。我们进行了广泛的实验，并仔细研究了3D预训练中不同不变的贡献。另外，我们提出了一种简单但有效的方法，该方法可以共同预先培训3D编码器和使用对比度学习的深度图编码器。通过我们的方法进行预训练的模型在下游任务方面具有显着的性能提高。例如，预先训练的投票表现优于Sun RGB-D和扫描对象检测基准的先前方法，并具有明显的利润。

由于缺乏大规模标记的3D数据集，大多数3D神经网络都是从划痕训练。在本文中，我们通过利用来自丰富的2D数据集学习的2D网络来介绍一种新的3D预预测方法。我们提出了通过将像素级和点级别特征映射到同一嵌入空间中的对比度的像素到点知识转移来有效地利用2D信息。由于2D和3D网络之间的异构性质，我们介绍了后投影功能以对准2D和3D之间的功能以使转移成为可能。此外，我们设计了一个上采样功能投影层，以增加高级2D特征图的空间分辨率，这使得能够学习细粒度的3D表示。利用普雷累染的2D网络，所提出的预介绍过程不需要额外的2D或3D标记数据，进一步缓解了昂贵的3D数据注释成本。据我们所知，我们是第一个利用现有的2D培训的权重，以预先rain 3D深度神经网络。我们的密集实验表明，使用2D知识预订的3D模型可以通过各种真实世界3D下游任务进行3D网络的性能。

我们建议在2D域中利用自我监督的技术来实现细粒度的3D形状分割任务。这是受到观察的启发：基于视图的表面表示比基于点云或体素占用率的3D对应物更有效地建模高分辨率表面细节和纹理。具体而言，给定3D形状，我们将其从多个视图中渲染，并在对比度学习框架内建立密集的对应学习任务。结果，与仅在2D或3D中使用自学的替代方案相比，学到的2D表示是视图不变和几何一致的，在对有限的标记形状进行培训时，可以更好地概括概括。对纹理（渲染peple）和未纹理（partnet）3D数据集的实验表明，我们的方法在细粒部分分割中优于最先进的替代方案。当仅一组稀疏的视图可供训练或形状纹理时，对基准的改进就会更大，这表明MVDecor受益于2D处理和3D几何推理。

3D感知最近的进展在了解3DACHAPES甚至场景的几何结构方面表现出令人印象深刻的进展。灵感来自这些进步的几何理解，我们旨在利用几何约束下学到的表示基于图像的感知。我们介绍一种基于多视图RGB-D数据学习View-Invariant的方法，用于网络预训练的网络预训练的几何感知表示，然后可以将其有效地传送到下游2D任务。我们建议在多视图IM-ysge约束和图像 - 几何约束下采用对比学习，以便在学习的2D表示中进行编码。这不仅仅是在几乎非仅对图像的语义分割，实例分段和对象检测的基于图像的基于图像的基于图像的TASK上学习而改进，而且，但是，在低数据方案中提供了显着的改进。我们对全数据的语义细分显示6.0％的显着提高，以及剪刀上的基线20％数据上的11.9％。

Arguably one of the top success stories of deep learning is transfer learning. The finding that pre-training a network on a rich source set (e.g., ImageNet) can help boost performance once fine-tuned on a usually much smaller target set, has been instrumental to many applications in language and vision. Yet, very little is known about its usefulness in 3D point cloud understanding. We see this as an opportunity considering the effort required for annotating data in 3D. In this work, we aim at facilitating research on 3D representation learning. Different from previous works, we focus on high-level scene understanding tasks. To this end, we select a suite of diverse datasets and tasks to measure the effect of unsupervised pre-training on a large source set of 3D scenes. Our findings are extremely encouraging: using a unified triplet of architecture, source dataset, and contrastive loss for pre-training, we achieve improvement over recent best results in segmentation and detection across 6 different benchmarks for indoor and outdoor, real and synthetic datasets -demonstrating that the learned representation can generalize across domains. Furthermore, the improvement was similar to supervised pre-training, suggesting that future efforts should favor scaling data collection over more detailed annotation. We hope these findings will encourage more research on unsupervised pretext task design for 3D deep learning. Our code is publicly available at https://github.com/facebookresearch/PointContrast

预训练已成为许多计算机视觉任务中的标准范式。但是，大多数方法通常都设计在RGB图像域上。由于二维图像平面和三维空间之间的差异，这种预先训练的模型无法感知空间信息，并用作3D相关任务的子最优解。为了弥合这种差距，我们的目标是学习可以描述三维空间的空间感知视觉表示，并且对这些任务更适合和有效。为了利用点云，在与图像相比提供空间信息时更有优越，我们提出了一个简单而有效的2D图像和3D点云无监督的预训练策略，称为Simipu。具体而言，我们开发了一种多模态对比学习框架，包括模态空间感知模块，用于从点云和模态特征交互模块中学习空间感知表示，以从点传输感知空间信息的能力云编码器分别到图像编码器。匹配算法和投影矩阵建立了用于对比损耗的正对。整个框架培训以无人监督的端到端时尚。据我们所知，这是第一项探索户外多模态数据集的对比学习训练策略的研究，其中包含配对的相机图像和LIDAR点云。 HTTPS：//github.com/zhever/simipu提供代码和模型。

现有的无监督点云预训练的方法被限制在场景级或点/体素级实例歧视上。场景级别的方法往往会失去对识别道路对象至关重要的本地细节，而点/体素级方法固有地遭受了有限的接收领域，而这种接收领域无力感知大型对象或上下文环境。考虑到区域级表示更适合3D对象检测，我们设计了一个新的无监督点云预训练框架，称为proposalcontrast，该框架通过对比的区域建议来学习强大的3D表示。具体而言，通过从每个点云中采样一组详尽的区域建议，每个提案中的几何点关系都是建模用于创建表达性建议表示形式的。为了更好地适应3D检测属性，提案contrast可以通过群体间和统一分离来优化，即提高跨语义类别和对象实例的提议表示的歧视性。在各种3D检测器（即PV-RCNN，Centerpoint，Pointpillars和Pointrcnn）和数据集（即Kitti，Waymo和一次）上验证了提案cont抗对流的概括性和可传递性。

We propose a novel approach to self-supervised learning of point cloud representations by differentiable neural rendering. Motivated by the fact that informative point cloud features should be able to encode rich geometry and appearance cues and render realistic images, we train a point-cloud encoder within a devised point-based neural renderer by comparing the rendered images with real images on massive RGB-D data. The learned point-cloud encoder can be easily integrated into various downstream tasks, including not only high-level tasks like 3D detection and segmentation, but low-level tasks like 3D reconstruction and image synthesis. Extensive experiments on various tasks demonstrate the superiority of our approach compared to existing pre-training methods.

我们提出了一种新的方法来将4D动态对象前瞻灌输到学习的3D表示，通过无监督的预训练。我们观察到对象通过环境的动态移动提供了关于其对象的重要提示，因此提出了利用这种动态理解的学习学习的3D表示，然后可以有效地传送到下游3D语义场景中的改进性能。我们提出了一种新的数据增强方案，利用静态3D环境中移动的合成3D形状，并在3D-4D约束下采用对比学习，该约束将4D Imormces编码到学习的3D表示中。实验表明，我们无监督的代表学习导致下游3D语义分割，对象检测和实例分割任务的改进，而且，显着提高了数据稀缺方案的性能。

许多3D表示（例如，点云）是下面连续3D表面的离散样本。该过程不可避免地介绍了底层的3D形状上的采样变化。在学习3D表示中，应忽略应忽略变化，而应捕获基础3D形状的可转换知识。这成为现有代表学习范式的大挑战。本文在点云上自动编码。标准自动编码范例强制编码器捕获这种采样变体，因为解码器必须重建具有采样变化的原始点云。我们介绍了隐式AutoEncoder（IAE），这是一种简单而有效的方法，通过用隐式解码器替换点云解码器来解决这一挑战。隐式解码器输出与相同模型的不同点云采样之间共享的连续表示。在隐式表示下重建可以优先考虑编码器丢弃采样变体，引入更多空间以学习有用的功能。在一个简单的线性AutoEncoder下，理论上理论地证明这一索赔。此外，隐式解码器提供丰富的空间来为不同的任务设计合适的隐式表示。我们展示了IAE对3D对象和3D场景的各种自我监督学习任务的有用性。实验结果表明，IAE在每项任务中始终如一地优于最先进的。

大规模点云的注释仍然耗时，并且对于许多真实世界任务不可用。点云预训练是用于获得快速适配的可扩展模型的一个潜在解决方案。因此，在本文中，我们调查了一种新的自我监督学习方法，称为混合和解除戒（MD），用于点云预培训。顾名思义，我们探索如何将原始点云与混合点云分开，并利用这一具有挑战的任务作为模型培训的借口优化目标。考虑到原始数据集中的有限培训数据，这远低于普遍的想象，混合过程可以有效地产生更高质量的样本。我们构建一个基线网络以验证我们的直觉，只包含两个模块，编码器和解码器。给定混合点云，首先预先训练编码器以提取语义嵌入。然后，利用实例 - 自适应解码器根据嵌入来解除点云。尽管简单，编码器本质上是能够在训练后捕获点云关键点，并且可以快速适应下游任务，包括预先训练和微调范例的分类和分割。在两个数据集上的广泛实验表明编码器+我们的（MD）显着超越了从头划痕培训的编码器和快速收敛的编码器。在消融研究中，我们进一步研究了每个部件的效果，并讨论了拟议的自我监督学习策略的优势。我们希望这种自我监督的学习尝试点云可以铺平了减少对大规模标记数据的深度学习模型依赖的方式，并在将来节省了大量的注释成本。

Reducing the quantity of annotations required for supervised training is vital when labels are scarce and costly. This reduction is especially important for semantic segmentation tasks involving 3D datasets that are often significantly smaller and more challenging to annotate than their image-based counterparts. Self-supervised pre-training on large unlabelled datasets is one way to reduce the amount of manual annotations needed. Previous work has focused on pre-training with point cloud data exclusively; this approach often requires two or more registered views. In the present work, we combine image and point cloud modalities, by first learning self-supervised image features and then using these features to train a 3D model. By incorporating image data, which is often included in many 3D datasets, our pre-training method only requires a single scan of a scene. We demonstrate that our pre-training approach, despite using single scans, achieves comparable performance to other multi-scan, point cloud-only methods.

Masked Modeling (MM) has demonstrated widespread success in various vision challenges, by reconstructing masked visual patches. Yet, applying MM for large-scale 3D scenes remains an open problem due to the data sparsity and scene complexity. The conventional random masking paradigm used in 2D images often causes a high risk of ambiguity when recovering the masked region of 3D scenes. To this end, we propose a novel informative-preserved reconstruction, which explores local statistics to discover and preserve the representative structured points, effectively enhancing the pretext masking task for 3D scene understanding. Integrated with a progressive reconstruction manner, our method can concentrate on modeling regional geometry and enjoy less ambiguity for masked reconstruction. Besides, such scenes with progressive masking ratios can also serve to self-distill their intrinsic spatial consistency, requiring to learn the consistent representations from unmasked areas. By elegantly combining informative-preserved reconstruction on masked areas and consistency self-distillation from unmasked areas, a unified framework called MM-3DScene is yielded. We conduct comprehensive experiments on a host of downstream tasks. The consistent improvement (e.g., +6.1 mAP@0.5 on object detection and +2.2% mIoU on semantic segmentation) demonstrates the superiority of our approach.

The past few years have witnessed the prevalence of self-supervised representation learning within the language and 2D vision communities. However, such advancements have not been fully migrated to the community of 3D point cloud learning. Different from previous pre-training pipelines for 3D point clouds that generally fall into the scope of either generative modeling or contrastive learning, in this paper, we investigate a translative pre-training paradigm, namely PointVST, driven by a novel self-supervised pretext task of cross-modal translation from an input 3D object point cloud to its diverse forms of 2D rendered images (e.g., silhouette, depth, contour). Specifically, we begin with deducing view-conditioned point-wise embeddings via the insertion of the viewpoint indicator, and then adaptively aggregate a view-specific global codeword, which is further fed into the subsequent 2D convolutional translation heads for image generation. We conduct extensive experiments on common task scenarios of 3D shape analysis, where our PointVST shows consistent and prominent performance superiority over current state-of-the-art methods under diverse evaluation protocols. Our code will be made publicly available.

The understanding capabilities of current state-of-the-art 3D models are limited by datasets with a small number of annotated data and a pre-defined set of categories. In its 2D counterpart, recent advances have shown that similar problems can be significantly alleviated by employing knowledge from other modalities, such as language. Inspired by this, leveraging multimodal information for 3D modality could be promising to improve 3D understanding under the restricted data regime, but this line of research is not well studied. Therefore, we introduce ULIP to learn a unified representation of image, text, and 3D point cloud by pre-training with object triplets from the three modalities. To overcome the shortage of training triplets, ULIP leverages a pre-trained vision-language model that has already learned a common visual and textual space by training with massive image-text pairs. Then, ULIP learns a 3D representation space aligned with the common image-text space, using a small number of automatically synthesized triplets. ULIP is agnostic to 3D backbone networks and can easily be integrated into any 3D architecture. Experiments show that ULIP effectively improves the performance of multiple recent 3D backbones by simply pre-training them on ShapeNet55 using our framework, achieving state-of-the-art performance in both standard 3D classification and zero-shot 3D classification on ModelNet40 and ScanObjectNN. ULIP also improves the performance of PointMLP by around 3% in 3D classification on ScanObjectNN, and outperforms PointCLIP by 28.8% on top-1 accuracy for zero-shot 3D classification on ModelNet40. Our code and pre-trained models will be released.

随着相机和激光雷达传感器捕获用于自主驾驶的互补信息，已经做出了巨大的努力，通过多模式数据融合来开发语义分割算法。但是，基于融合的方法需要配对的数据，即具有严格的点对像素映射的激光点云和相机图像，因为培训和推理的输入都严重阻碍了在实际情况下的应用。因此，在这项工作中，我们建议通过充分利用具有丰富外观的2D图像来提高对点云上的代表性学习的2D先验辅助语义分割（2DPass），以增强对点云的表示。实际上，通过利用辅助模态融合和多尺度融合到单个知识蒸馏（MSFSKD），2DAPS从多模式数据中获取更丰富的语义和结构信息，然后在线蒸馏到纯3D网络。结果，配备了2DAPS，我们的基线仅使用点云输入显示出显着的改进。具体而言，它在两个大规模的基准（即Semantickitti和Nuscenes）上实现了最先进的方法，其中包括TOP-1的semantickitti的单扫描和多次扫描竞赛。

室内场景云的无监督对比学习取得了巨大的成功。但是，室外场景中无监督的学习点云仍然充满挑战，因为以前的方法需要重建整个场景并捕获对比度目标的部分视图。这在带有移动物体，障碍物和传感器的室外场景中是不可行的。在本文中，我们提出了CO^3，即合作对比度学习和上下文形状的预测，以无监督的方式学习3D表示室外景点云。与现有方法相比，Co^3具有几种优点。 （1）它利用了从车辆侧和基础架构侧来的激光点云来构建差异，但同时维护对比度学习的通用语义信息，这比以前的方法构建的视图更合适。 （2）在对比度目标的同时，提出了形状上下文预测作为预训练目标，并为无监督的3D点云表示学习带来了更多与任务相关的信息，这在将学习的表示形式转移到下游检测任务时是有益的。 （3）与以前的方法相比，CO^3学到的表示形式可以通过不同类型的LIDAR传感器收集到不同的室外场景数据集。 （4）CO^3将一次和Kitti数据集的当前最新方法提高到2.58地图。代码和模型将发布。我们认为Co^3将有助于了解室外场景中的LiDar Point云。

Current outdoor LiDAR-based 3D object detection methods mainly adopt the training-from-scratch paradigm. Unfortunately, this paradigm heavily relies on large-scale labeled data, whose collection can be expensive and time-consuming. Self-supervised pre-training is an effective and desirable way to alleviate this dependence on extensive annotated data. Recently, masked modeling has become a successful self-supervised learning approach for point clouds. However, current works mainly focus on synthetic or indoor datasets. When applied to large-scale and sparse outdoor point clouds, they fail to yield satisfactory results. In this work, we present BEV-MAE, a simple masked autoencoder pre-training framework for 3D object detection on outdoor point clouds. Specifically, we first propose a bird's eye view (BEV) guided masking strategy to guide the 3D encoder learning feature representation in a BEV perspective and avoid complex decoder design during pre-training. Besides, we introduce a learnable point token to maintain a consistent receptive field size of the 3D encoder with fine-tuning for masked point cloud inputs. Finally, based on the property of outdoor point clouds, i.e., the point clouds of distant objects are more sparse, we propose point density prediction to enable the 3D encoder to learn location information, which is essential for object detection. Experimental results show that BEV-MAE achieves new state-of-the-art self-supervised results on both Waymo and nuScenes with diverse 3D object detectors. Furthermore, with only 20% data and 7% training cost during pre-training, BEV-MAE achieves comparable performance with the state-of-the-art method ProposalContrast. The source code and pre-trained models will be made publicly available.

在深度学习研究中，自学学习（SSL）引起了极大的关注，引起了计算机视觉和遥感社区的兴趣。尽管计算机视觉取得了很大的成功，但SSL在地球观测领域的大部分潜力仍然锁定。在本文中，我们对在遥感的背景下为计算机视觉的SSL概念和最新发展提供了介绍，并回顾了SSL中的概念和最新发展。此外，我们在流行的遥感数据集上提供了现代SSL算法的初步基准，从而验证了SSL在遥感中的潜力，并提供了有关数据增强的扩展研究。最后，我们确定了SSL未来研究的有希望的方向的地球观察（SSL4EO），以铺平了两个领域的富有成效的相互作用。

跨图像建立视觉对应是一项具有挑战性且必不可少的任务。最近，已经提出了大量的自我监督方法，以更好地学习视觉对应的表示。但是，我们发现这些方法通常无法利用语义信息，并且在低级功能的匹配方面过度融合。相反，人类的视觉能够将不同的物体区分为跟踪的借口。受此范式的启发，我们建议学习语义意识的细粒对应关系。首先，我们证明语义对应是通过一组丰富的图像级别自我监督方法隐式获得的。我们进一步设计了一个像素级的自我监督学习目标，该目标专门针对细粒的对应关系。对于下游任务，我们将这两种互补的对应表示形式融合在一起，表明它们是协同增强性能的。我们的方法超过了先前的最先进的自我监督方法，使用卷积网络在各种视觉通信任务上，包括视频对象分割，人姿势跟踪和人类部分跟踪。