除了图像分类外，对比性语言图像预训练（剪辑）还为广泛的视觉任务（包括对象级别和3D空间理解）取得了非凡的成功。但是，将从剪辑中学到的语义知识转移到更复杂的量化目标任务，例如使用几何信息的深度估计。在本文中，我们建议将剪辑应用于零拍的单眼估计，称为Depthclip。我们发现，输入图像的斑块可以响应一定的语义距离令牌，然后将其投影到量化的深度箱中进行粗估算。在没有任何培训的情况下，我们的深度算法超过了现有的无监督方法，甚至可以接近早期全面监督的网络。据我们最大的知识，我们是第一个从语义语言知识进行零拍调整的人，以量化下游任务并执行零拍的单眼估计。我们希望我们的工作能够阐明未来的研究。该代码可在https://github.com/adonis-galaxy/depthclip上找到。

随着大型预训练的Vison语言模型（如剪辑）的出现，可以通过及时调整来调整可转让表示形式。及时调整试图从存储在预训练的视觉模型的图像和文本编码器中的常识中探索有益信息，以探索下游任务。最近提出的名为“上下文优化”（COP）的方法将一组可学习的向量从语言侧引入文本提示符，而单独调整文本提示符则不会影响图像编码器的计算视觉特征，从而导致了次级优势。在本文中，我们通过学习文本提示并同时为文本和图像编码器提供双重模式提示调整范式。此外，为了使视觉提示更多地集中在目标视觉概念上，我们提出了类感知的视觉及时调整（CAVPT），该调整是通过在模板提示和视觉类别令牌嵌入的语言描述之间进行交叉注意来动态生成的。我们的方法提供了一种新的范式来调整大型预训练的视觉模型，并在8个数据集上进行了广泛的实验结果，证明了该方法的有效性。我们的代码在补充材料中可用。

对比性语言图像预训练（剪辑）已被证明可以学习具有出色传递性的视觉表示，从而实现了零击分类的有希望的准确性。为了进一步提高其下游性能，现有作品在剪辑上提出了其他可学习的模块，并通过几次训练集对其进行微调。但是，由此产生的额外培训成本和数据要求严重阻碍了模型部署和知识转移的效率。在本文中，我们引入了一种自由午餐的增强方法CALIP，以通过无参数注意模块来提高Clip的零拍摄性能。具体而言，我们指导视觉和文本表示相互交互，并通过注意探索跨模式的信息特征。由于预训练大大降低了两种方式之间的嵌入距离，因此我们在注意力中丢弃所有可学习的参数，并在双向更新多模式特征，从而使整个过程无参数且无培训。通过这种方式，图像与文本感知信号混合在一起，文本表示形式被视觉引导以获得更好的自适应零射击对齐。我们在14个数据集的各种基准上评估CALIP，用于2D图像和3D Point Cloud几乎没有分类，显示出一致的零弹性性能改进了夹子。基于此，我们进一步在Calip的注意模块中插入了少量线性层，并在少量射击设置下验证我们的鲁棒性，与现有方法相比，这也可以实现领先的性能。这些广泛的实验证明了我们的方法在有效增强夹子方面的优势。

最近，通过对比视觉 - 语言预训练（CLIP）的零射击和少量学习已经在2D视觉识别上显示了鼓舞人心的性能，从而了解在开放词汇设置中将图像与其相应的文本匹配。然而，它仍然在探索中，是否通过2D中的大规模图像文本对预先训练的剪辑可以推广到3D识别。在本文中，我们通过提出引人点来识别这种设置是可行的，这在剪辑编码点云和3D类别文本之间进行对准。具体地，我们通过将点云投射到多视图深度映射而不呈现，并聚合视图零拍摄预测以实现从2D到3D的知识转移。首先，我们设计了一个视图间适配器，以更好地提取全局特征，并自适应地融合从3D到2D预培训的剪辑中学到的几次拍摄知识。只需在几次拍摄设置中微调轻量级适配器，可以在很大程度上提高要素的性能。此外，我们遵守CONTCLIP和古典3D监督网络之间的互补财产。通过简单的合奏，PointClip提高了基线的性能，甚至超越了最先进的模型。因此，PointClip是在低资源成本和数据制度下通过剪辑的有效3D点云理解的有希望的替代方案。我们在广泛采用的ModelNet10，ModelNet40和挑战ScanObjectnn上进行彻底的实验，以证明Pointclip的有效性。代码在https://github.com/zrrskywalker/pointclip发布。

We address the problem of estimating a high quality dense depth map from a single RGB input image. We start out with a baseline encoder-decoder convolutional neural network architecture and pose the question of how the global processing of information can help improve overall depth estimation. To this end, we propose a transformerbased architecture block that divides the depth range into bins whose center value is estimated adaptively per image. The final depth values are estimated as linear combinations of the bin centers. We call our new building block AdaBins. Our results show a decisive improvement over the state-ofthe-art on several popular depth datasets across all metrics. We also validate the effectiveness of the proposed block with an ablation study and provide the code and corresponding pre-trained weights of the new state-of-the-art model 1 .

最近，Vision-Language预训练的零拍图像分类已经表现出令人难以置信的成就，即该模型可以对任意类别进行分类而不看到该类别的其他注释图像。然而，目前尚不清楚如何在更广泛的视觉问题上进行零射识别，例如对象检测和语义分割。在本文中，我们通过在现成的预训练的视觉模型，即剪辑上建立零拍语义分割来定位零拍语义分割。很难因为语义分割和剪辑模型在不同的视觉粒度上执行，该语义分段处理在像素上时，而剪辑在图像上执行。为了解决处理粒度的差异，我们拒绝使用普遍的一级FCN基于FCN的框架，并倡导一个两级语义分割框架，其中第一阶段提取一个完全提取的掩模提案和第二阶段利用基于图像的剪辑模型在第一阶段生成的蒙版图像作物上执行零拍分类。我们的实验结果表明，这种简单的框架通过大型利润率超越了先前的最先进：+29.5 Hiou On Pascal VOC 2012 DataSet，+8.9 Hiou On Coco Stuff DataSet。凭借其简单性和强大的表现，我们希望本框架成为促进未来研究的基准。

作为许多自主驾驶和机器人活动的基本组成部分，如自我运动估计，障碍避免和场景理解，单眼深度估计（MDE）引起了计算机视觉和机器人社区的极大关注。在过去的几十年中，已经开发了大量方法。然而，据我们所知，对MDE没有全面调查。本文旨在通过审查1970年至2021年之间发布的197个相关条款来弥补这一差距。特别是，我们为涵盖各种方法的MDE提供了全面的调查，介绍了流行的绩效评估指标并汇总公开的数据集。我们还总结了一些代表方法的可用开源实现，并比较了他们的表演。此外，我们在一些重要的机器人任务中审查了MDE的应用。最后，我们通过展示一些有希望的未来研究方向来结束本文。预计本调查有助于读者浏览该研究领域。

Prompt Tuning, conditioning on task-specific learned prompt vectors, has emerged as a data-efficient and parameter-efficient method for adapting large pretrained vision-language models to multiple downstream tasks. However, existing approaches usually consider learning prompt vectors for each task independently from scratch, thereby failing to exploit the rich shareable knowledge across different vision-language tasks. In this paper, we propose multitask vision-language prompt tuning (MVLPT), which incorporates cross-task knowledge into prompt tuning for vision-language models. Specifically, (i) we demonstrate the effectiveness of learning a single transferable prompt from multiple source tasks to initialize the prompt for each target task; (ii) we show many target tasks can benefit each other from sharing prompt vectors and thus can be jointly learned via multitask prompt tuning. We benchmark the proposed MVLPT using three representative prompt tuning methods, namely text prompt tuning, visual prompt tuning, and the unified vision-language prompt tuning. Results in 20 vision tasks demonstrate that the proposed approach outperforms all single-task baseline prompt tuning methods, setting the new state-of-the-art on the few-shot ELEVATER benchmarks and cross-task generalization benchmarks. To understand where the cross-task knowledge is most effective, we also conduct a large-scale study on task transferability with 20 vision tasks in 400 combinations for each prompt tuning method. It shows that the most performant MVLPT for each prompt tuning method prefers different task combinations and many tasks can benefit each other, depending on their visual similarity and label similarity. Code is available at https://github.com/sIncerass/MVLPT.

现有的时间动作检测（TAD）方法依赖于大型培训数据，包括细分级注释，仅限于在推理期间单独识别先前看到的课程。为每类兴趣收集和注释一个大型培训集是昂贵的，因此无法计算。零射TAD（ZS-TAD）通过启用预训练的模型来识别任何看不见的动作类别来解决这一障碍。同时，ZS-TAD的调查大大降低，ZS-Tad也更具挑战性。受零摄像图像分类的成功的启发，我们旨在解决更复杂的TAD任务。一种直观的方法是将现成的建议探测器与剪辑样式分类集成。但是，由于顺序定位（例如，提案生成）和分类设计，它很容易进行定位误差传播。为了克服这个问题，在本文中，我们通过视觉提示（陈旧）提出了一种新型的零射击时间动作检测模型。这种新颖的设计通过破坏介于两者之间的错误传播途径来有效地消除了定位和分类之间的依赖性。我们进一步介绍了分类和定位之间的相互作用机制，以改善优化。对标准ZS-TAD视频基准测试的广泛实验表明，我们的陈旧的表现明显优于最先进的替代方案。此外，我们的模型还与最近的强大竞争对手相比，在受到监督的TAD上还能产生卓越的成果。 Stale的Pytorch实现可从https://github.com/sauradip/stale获得。

While monocular depth estimation (MDE) is an important problem in computer vision, it is difficult due to the ambiguity that results from the compression of a 3D scene into only 2 dimensions. It is common practice in the field to treat it as simple image-to-image translation, without consideration for the semantics of the scene and the objects within it. In contrast, humans and animals have been shown to use higher-level information to solve MDE: prior knowledge of the nature of the objects in the scene, their positions and likely configurations relative to one another, and their apparent sizes have all been shown to help resolve this ambiguity. In this paper, we present a novel method to enhance MDE performance by encouraging use of known-useful information about the semantics of objects and inter-object relationships within a scene. Our novel ObjCAViT module sources world-knowledge from language models and learns inter-object relationships in the context of the MDE problem using transformer attention, incorporating apparent size information. Our method produces highly accurate depth maps, and we obtain competitive results on the NYUv2 and KITTI datasets. Our ablation experiments show that the use of language and cross-attention within the ObjCAViT module increases performance. Code is released at https://github.com/DylanAuty/ObjCAViT.

在低标签制度中，解决图像的多标签识别（MLR）是许多现实世界应用的一项艰巨任务。最近的工作学会了文本和视觉空间之间的一致性，以补偿图像标签不足，但由于可用的MLR注释量有限，因此失去了准确性。在这项工作中，我们利用数百万辅助图像文本对预测的文本和视觉特征的牢固对齐，并提出双背景优化（dualCoop）作为部分标签MLR和零发射MLR的统一框架。 DualCoop用类名来编码正面和负面的上下文，作为语言输入的一部分（即提示）。由于DualCoop仅在验证的视觉语言框架上引入了非常轻松的开销，因此它可以迅速适应具有有限的注释甚至看不见的类别的多标签识别任务。对两个挑战性低标签设置的标准多标签识别基准测试的实验证明了我们方法比最新方法的优势。

建立新型观点综合的最近进展后，我们提出了改善单眼深度估计的应用。特别是，我们提出了一种在三个主要步骤中分开的新颖训练方法。首先，单眼深度网络的预测结果被扭转到额外的视点。其次，我们应用一个额外的图像综合网络，其纠正并提高了翘曲的RGB图像的质量。通过最小化像素-WISE RGB重建误差，该网络的输出需要尽可能类似地查看地面真实性视图。第三，我们将相同的单眼深度估计重新应用于合成的第二视图点，并确保深度预测与相关的地面真理深度一致。实验结果证明，我们的方法在Kitti和Nyu-Deaft-V2数据集上实现了最先进的或可比性，具有轻量级和简单的香草U-Net架构。

Current computer vision models, unlike the human visual system, cannot yet achieve general-purpose visual understanding. Existing efforts to create a general vision model are limited in the scope of assessed tasks and offer no overarching framework to perform them holistically. We present a new comprehensive benchmark, General-purpose Visual Understanding Evaluation (G-VUE), covering the full spectrum of visual cognitive abilities with four functional domains $\unicode{x2014}$ Perceive, Ground, Reason, and Act. The four domains are embodied in 11 carefully curated tasks, from 3D reconstruction to visual reasoning and manipulation. Along with the benchmark, we provide a general encoder-decoder framework to allow for the evaluation of arbitrary visual representation on all 11 tasks. We evaluate various pre-trained visual representations with our framework and observe that (1) Transformer-based visual backbone generally outperforms CNN-based backbone on G-VUE, (2) visual representations from vision-language pre-training are superior to those with vision-only pre-training across visual tasks. With G-VUE, we provide a holistic evaluation standard to motivate research toward building general-purpose visual systems via obtaining more general-purpose visual representations.

The understanding capabilities of current state-of-the-art 3D models are limited by datasets with a small number of annotated data and a pre-defined set of categories. In its 2D counterpart, recent advances have shown that similar problems can be significantly alleviated by employing knowledge from other modalities, such as language. Inspired by this, leveraging multimodal information for 3D modality could be promising to improve 3D understanding under the restricted data regime, but this line of research is not well studied. Therefore, we introduce ULIP to learn a unified representation of image, text, and 3D point cloud by pre-training with object triplets from the three modalities. To overcome the shortage of training triplets, ULIP leverages a pre-trained vision-language model that has already learned a common visual and textual space by training with massive image-text pairs. Then, ULIP learns a 3D representation space aligned with the common image-text space, using a small number of automatically synthesized triplets. ULIP is agnostic to 3D backbone networks and can easily be integrated into any 3D architecture. Experiments show that ULIP effectively improves the performance of multiple recent 3D backbones by simply pre-training them on ShapeNet55 using our framework, achieving state-of-the-art performance in both standard 3D classification and zero-shot 3D classification on ModelNet40 and ScanObjectNN. ULIP also improves the performance of PointMLP by around 3% in 3D classification on ScanObjectNN, and outperforms PointCLIP by 28.8% on top-1 accuracy for zero-shot 3D classification on ModelNet40. Our code and pre-trained models will be released.

本文提出了一个简单而有效的框架蒙版，该框架将新提出的掩盖自distillation纳入对比的语言图像预处理中。掩盖自distillation的核心思想是将表示从完整的图像提取到蒙版图像预测的表示形式。这种合并享有两个重要的好处。首先，掩盖的自我验证目标是本地贴片表示学习，这与视觉对比度的互补，专注于与文本相关的表示。二，掩盖的自我验证也与视觉语言对比符合训练目标的视野对比是一致的。视觉编码器用于功能对齐，因此能够学习本地语义从该语言中获得间接监督。我们提供了专门设计的实验，并进行了全面的分析，以验证这两个好处。从经验上讲，我们表明，当MaskClip应用于各种具有挑战性的下游任务时，可以在线性探测，填充和零拍摄中取得卓越的结果，并在语言编码器的指导下取得了卓越的结果。

基于注意力的模型（例如变压器）在密集的预测任务（例如语义分割）上表现出出色的性能，因为它们可以捕获图像中的长期依赖性。但是，到目前为止，很少探索变压器对单眼深度预测的好处。本文基于室内NYUV2数据集和室外KITTI数据集的深度估计任务的各种基于变压器的模型。我们提出了一种新型的基于注意力的架构，即单眼深度估计的深度构建器，该估计使用多头自我注意力来生成多尺度特征图，这些图由我们提出的解码器网络有效地组合。我们还提出了一个跨键模块，该模块将深度范围划分为每个图像可自适应估计的中心值的垃圾箱。估计的最终深度是每个像素的垃圾箱中心的线性组合。 TransBins模块在编码阶段使用变压器模块利用全局接收场。 NYUV2和KITTI深度估计基准的实验结果表明，我们提出的方法将最新方法提高了3.3％，在根平方误差（RMSE）方面分别将最新方法提高了3.3％。

We introduce Patch Aligned Contrastive Learning (PACL), a modified compatibility function for CLIP's contrastive loss, intending to train an alignment between the patch tokens of the vision encoder and the CLS token of the text encoder. With such an alignment, a model can identify regions of an image corresponding to a given text input, and therefore transfer seamlessly to the task of open vocabulary semantic segmentation without requiring any segmentation annotations during training. Using pre-trained CLIP encoders with PACL, we are able to set the state-of-the-art on the task of open vocabulary zero-shot segmentation on 4 different segmentation benchmarks: Pascal VOC, Pascal Context, COCO Stuff and ADE20K. Furthermore, we show that PACL is also applicable to image-level predictions and when used with a CLIP backbone, provides a general improvement in zero-shot classification accuracy compared to CLIP, across a suite of 12 image classification datasets.

Prompt tuning is a new few-shot transfer learning technique that only tunes the learnable prompt for pre-trained vision and language models such as CLIP. However, existing prompt tuning methods tend to learn spurious or entangled representations, which leads to poor generalization to unseen concepts. Towards non-spurious and efficient prompt learning from limited examples, this paper presents a novel \underline{\textbf{C}}ounterfactual \underline{\textbf{P}}rompt \underline{\textbf{L}}earning (CPL) method for vision and language models, which simultaneously employs counterfactual generation and contrastive learning in a joint optimization framework. Particularly, CPL constructs counterfactual by identifying minimal non-spurious feature change between semantically-similar positive and negative samples that causes concept change, and learns more generalizable prompt representation from both factual and counterfactual examples via contrastive learning. Extensive experiments demonstrate that CPL can obtain superior few-shot performance on different vision and language tasks than previous prompt tuning methods on CLIP. On image classification, we achieve 3.55\% average relative improvement on unseen classes across seven datasets; on image-text retrieval and visual question answering, we gain up to 4.09\% and 25.08\% relative improvements across three few-shot scenarios on unseen test sets respectively.

对于单眼深度估计，获取真实数据的地面真相并不容易，因此通常使用监督的合成数据采用域适应方法。但是，由于缺乏实际数据的监督，这仍然可能会导致较大的域间隙。在本文中，我们通过从真实数据中生成可靠的伪基础真理来开发一个域适应框架，以提供直接的监督。具体而言，我们提出了两种用于伪标记的机制：1）通过测量图像具有相同内容但不同样式的深度预测的一致性，通过测量深度预测的一致性； 2）通过点云完成网络的3D感知伪标记，该网络学会完成3D空间中的深度值，从而在场景中提供更多的结构信息，以完善并生成更可靠的伪标签。在实验中，我们表明我们的伪标记方法改善了各种环境中的深度估计，包括在训练过程中使用立体声对。此外，该提出的方法对现实世界数据集中的几种最新无监督域的适应方法表现出色。

Recently, CLIP has been applied to pixel-level zero-shot learning tasks via a two-stage scheme. The general idea is to first generate class-agnostic region proposals and then feed the cropped proposal regions to CLIP to utilize its image-level zero-shot classification capability. While effective, such a scheme requires two image encoders, one for proposal generation and one for CLIP, leading to a complicated pipeline and high computational cost. In this work, we pursue a simpler-and-efficient one-stage solution that directly extends CLIP's zero-shot prediction capability from image to pixel level. Our investigation starts with a straightforward extension as our baseline that generates semantic masks by comparing the similarity between text and patch embeddings extracted from CLIP. However, such a paradigm could heavily overfit the seen classes and fail to generalize to unseen classes. To handle this issue, we propose three simple-but-effective designs and figure out that they can significantly retain the inherent zero-shot capacity of CLIP and improve pixel-level generalization ability. Incorporating those modifications leads to an efficient zero-shot semantic segmentation system called ZegCLIP. Through extensive experiments on three public benchmarks, ZegCLIP demonstrates superior performance, outperforming the state-of-the-art methods by a large margin under both "inductive" and "transductive" zero-shot settings. In addition, compared with the two-stage method, our one-stage ZegCLIP achieves a speedup of about 5 times faster during inference. We release the code at https://github.com/ZiqinZhou66/ZegCLIP.git.