Although DETR-based 3D detectors can simplify the detection pipeline and achieve direct sparse predictions, their performance still lags behind dense detectors with post-processing for 3D object detection from point clouds. DETRs usually adopt a larger number of queries than GTs (e.g., 300 queries v.s. 40 objects in Waymo) in a scene, which inevitably incur many false positives during inference. In this paper, we propose a simple yet effective sparse 3D detector, named Query Contrast Voxel-DETR (ConQueR), to eliminate the challenging false positives, and achieve more accurate and sparser predictions. We observe that most false positives are highly overlapping in local regions, caused by the lack of explicit supervision to discriminate locally similar queries. We thus propose a Query Contrast mechanism to explicitly enhance queries towards their best-matched GTs over all unmatched query predictions. This is achieved by the construction of positive and negative GT-query pairs for each GT, and a contrastive loss to enhance positive GT-query pairs against negative ones based on feature similarities. ConQueR closes the gap of sparse and dense 3D detectors, and reduces up to ~60% false positives. Our single-frame ConQueR achieves new state-of-the-art (sota) 71.6 mAPH/L2 on the challenging Waymo Open Dataset validation set, outperforming previous sota methods (e.g., PV-RCNN++) by over 2.0 mAPH/L2.

Large-scale cross-modal pre-training paradigms have recently shown ubiquitous success on a wide range of downstream tasks, e.g., zero-shot classification, retrieval and image captioning. However, their successes highly rely on the scale and quality of web-crawled data that naturally contain incomplete and noisy information (e.g., wrong or irrelevant content). Existing works either design manual rules to clean data or generate pseudo-targets as auxiliary signals for reducing noise impact, which do not explicitly tackle both the incorrect and incomplete challenges simultaneously. In this paper, to automatically mitigate the impact of noise by solely mining over existing data, we propose a principled Noise-robust Language-Image Pre-training framework (NLIP) to stabilize pre-training via two schemes: noise-harmonization and noise-completion. First, in noise-harmonization scheme, NLIP estimates the noise probability of each pair according to the memorization effect of cross-modal transformers, then adopts noise-adaptive regularization to harmonize the cross-modal alignments with varying degrees. Second, in noise-completion scheme, to enrich the missing object information of text, NLIP injects a concept-conditioned cross-modal decoder to obtain semantic-consistent synthetic captions to complete noisy ones, which uses the retrieved visual concepts (i.e., objects' names) for the corresponding image to guide captioning generation. By collaboratively optimizing noise-harmonization and noise-completion schemes, our NLIP can alleviate the common noise effects during image-text pre-training in a more efficient way. Extensive experiments show the significant performance improvements of our NLIP using only 26M data over existing pre-trained models (e.g., CLIP, FILIP and BLIP) on 12 zero-shot classification datasets, MSCOCO image captioning and zero-shot image-text retrieval tasks.

We study the problem of learning online packing skills for irregular 3D shapes, which is arguably the most challenging setting of bin packing problems. The goal is to consecutively move a sequence of 3D objects with arbitrary shapes into a designated container with only partial observations of the object sequence. Meanwhile, we take physical realizability into account, involving physics dynamics and constraints of a placement. The packing policy should understand the 3D geometry of the object to be packed and make effective decisions to accommodate it in the container in a physically realizable way. We propose a Reinforcement Learning (RL) pipeline to learn the policy. The complex irregular geometry and imperfect object placement together lead to huge solution space. Direct training in such space is prohibitively data intensive. We instead propose a theoretically-provable method for candidate action generation to reduce the action space of RL and the learning burden. A parameterized policy is then learned to select the best placement from the candidates. Equipped with an efficient method of asynchronous RL acceleration and a data preparation process of simulation-ready training sequences, a mature packing policy can be trained in a physics-based environment within 48 hours. Through extensive evaluation on a variety of real-life shape datasets and comparisons with state-of-the-art baselines, we demonstrate that our method outperforms the best-performing baseline on all datasets by at least 12.8% in terms of packing utility.

Text-guided 3D object generation aims to generate 3D objects described by user-defined captions, which paves a flexible way to visualize what we imagined. Although some works have been devoted to solving this challenging task, these works either utilize some explicit 3D representations (e.g., mesh), which lack texture and require post-processing for rendering photo-realistic views; or require individual time-consuming optimization for every single case. Here, we make the first attempt to achieve generic text-guided cross-category 3D object generation via a new 3D-TOGO model, which integrates a text-to-views generation module and a views-to-3D generation module. The text-to-views generation module is designed to generate different views of the target 3D object given an input caption. prior-guidance, caption-guidance and view contrastive learning are proposed for achieving better view-consistency and caption similarity. Meanwhile, a pixelNeRF model is adopted for the views-to-3D generation module to obtain the implicit 3D neural representation from the previously-generated views. Our 3D-TOGO model generates 3D objects in the form of the neural radiance field with good texture and requires no time-cost optimization for every single caption. Besides, 3D-TOGO can control the category, color and shape of generated 3D objects with the input caption. Extensive experiments on the largest 3D object dataset (i.e., ABO) are conducted to verify that 3D-TOGO can better generate high-quality 3D objects according to the input captions across 98 different categories, in terms of PSNR, SSIM, LPIPS and CLIP-score, compared with text-NeRF and Dreamfields.

鉴于其广泛的应用，已经对人面部交换的任务进行了许多尝试。尽管现有的方法主要依赖于乏味的网络和损失设计，但它们仍然在源和目标面之间的信息平衡中挣扎，并倾向于产生可见的人工制品。在这项工作中，我们引入了一个名为StylesWap的简洁有效的框架。我们的核心想法是利用基于样式的生成器来增强高保真性和稳健的面部交换，因此可以采用发电机的优势来优化身份相似性。我们仅通过最小的修改来确定，StyleGAN2体系结构可以成功地处理来自源和目标的所需信息。此外，受到TORGB层的启发，进一步设计了交换驱动的面具分支以改善信息的融合。此外，可以采用stylegan倒置的优势。特别是，提出了交换引导的ID反转策略来优化身份相似性。广泛的实验验证了我们的框架会产生高质量的面部交换结果，从而超过了最先进的方法，既有定性和定量。

开放世界对象检测是一个更具笼统和挑战性的目标，旨在识别和本地化由任意类别名称描述的对象。最近的工作GLIP通过将检测数据集的所有类别名称连接到句子中，从而将此问题作为接地问题，从而导致类别名称之间的效率低下的相互作用。本文介绍了Distclip，这是一种通过诉诸于设计概念词典的知识富集，是一种平行的视觉概念训练预训练方法，用于开放世界检测。为了提高学习效率，我们提出了一种新型的并行概念公式，该公式分别提取概念，以更好地利用异质数据集（即检测，接地和图像文本对）进行培训。我们进一步设计了来自各种在线资源和检测数据集的概念字典〜（带有描述），以提供每个概念的先验知识。通过用描述丰富这些概念，我们明确地建立了各种概念之间的关系，以促进开放域学习。所提出的概念词典进一步用于提供足够的负面概念，用于构建单词区域对齐损失\，并完成图像对文本对数据标题中缺少描述的对象的标签。所提出的框架显示出强烈的零射击性能性能，例如，在LVIS数据集上，我们的DETCLIP-T优于9.9％的地图GLIPT-T优于GLIP-T，并且与完全避免的型号相比，稀有类别的稀有类别提高了13.5％。作为我们的。

为了同时朝着对多个下游任务的整体理解，需要提取具有更好可传递性的功能。尽管许多最新的自我监管的预训练方法在普遍的预处理前范式下在各种视觉任务上取得了令人印象深刻的表现，但它们对多任务学习方案的概括能力尚待探索。在本文中，我们在三个下游任务上进行了广泛研究各种类型的自我监督方法的转移性能，例如Moco和Simc​​lr，包括语义细分，可驱动的区域细分和交通对象检测，在大规模驾驶数据集中BDD100K。我们出人意料地发现，他们的表现是最佳的甚至落后于单任务基线的滞后，这可能是由于训练目标和建筑设计的区别在于预处理范式。为了克服这一难题，并避免重新设计资源密集的预培训阶段，我们提出了一种简单而有效的预处理 - 适应性 - 赛范围，用于一般的多任务培训，可以有效地适应现行预审预周态的模型没有增加培训开销。在自适应阶段，我们利用可学习的多尺度适配器来动态调整由多任务目标监督的预验证的模型权重，同时使经过预告片的知识未经触及。此外，我们将视觉语言预训练模型剪辑视为对预处理 - 适应 - 最终范式的强烈补充，并提出了一个名为LV-Adapter的新型适配器，该适配器通过任务特定的提示将语言先验纳入了多任务的模型中和视觉和文本特征之间的对齐。

对比性语言图像预训练（剪辑）通过随时可用的自然语言监督学习丰富的表示。它可以改善下游视觉任务的一般性能，包括但不限于零射击，长尾巴，细分，检索，标题和视频。但是，据我们所知，尚未研究剪辑的视觉解释性。为了提供其预测的视觉解释，我们提出了图像文本相似性图（ITSM）。基于它，我们出人意料地发现，剪辑比前景更喜欢背景区域，并且对人类理解提出了错误的可视化。在实验上，我们发现魔鬼在汇总部分，其中不适当的合并方法导致一种称为语义转移的现象。为了纠正和提高可视化结果，我们提出了蒙版的最大池，并使用自我监督图像编码器的注意力图。同时，解释性任务和识别任务需要不同的表示。为了解决这个问题，我们提出了双重预测，以满足这一要求。我们将上述方法整合为可解释的对比度图像预训练（ICLIP）。实验表明ICLIP极大地提高了可解释性。例如，在VOC 2012数据集中，非平凡的改进分别为$ 32.85 \％$和$ 49.10 \％$。

从单眼图像中学习的自我监督深度学习通常依赖于暂时相邻图像帧之间的2D像素光度关系。但是，他们既没有完全利用3D点的几何对应关系，也没有有效地应对闭塞或照明不一致引起的光度扭曲中的歧义。为了解决这些问题，这项工作提出了密度量构建网络（DEVNET），这是一种新型的自我监管的单眼深度学习框架，可以考虑3D空间信息，并利用相邻的相机flustums中的更强的几何约束。我们的DEVNET不是直接从单个图像中回归像素值，而是将摄像头划分为多个平行的平面，并预测每个平面上的点闭塞概率密度。最终的深度图是通过沿相应射线集成密度来生成的。在训练过程中，引入了新颖的正则化策略和损失功能，以减轻光度歧义和过度拟合。如果没有明显放大的模型参数的大小或运行时间，DEVNET在Kitti-2015室外数据集和NYU-V2室内数据集上均优于几个代表性基准。特别是，在深度估计的任务中，在Kitti-2015和NYU-V2上，DEVNET均减少了4％的根平方。代码可在https://github.com/gitkaichenzhou/devnet上找到。

流行的图神经网络模型在图表学习方面取得了重大进展。但是，在本文中，我们发现了一个不断被忽视的现象：用完整图测试的预训练的图表学习模型的表现不佳，该模型用良好的图表测试。该观察结果表明，图中存在混杂因素，这可能会干扰模型学习语义信息，而当前的图表表示方法并未消除其影响。为了解决这个问题，我们建议强大的因果图表示学习（RCGRL）学习可靠的图形表示，以防止混杂效应。 RCGRL引入了一种主动方法，可以在无条件的力矩限制下生成仪器变量，该方法使图表学习模型能够消除混杂因素，从而捕获与下游预测有因果关系的歧视性信息。我们提供定理和证明，以保证拟议方法的理论有效性。从经验上讲，我们对合成数据集和多个基准数据集进行了广泛的实验。结果表明，与最先进的方法相比，RCGRL实现了更好的预测性能和泛化能力。