尽管达到了最新的零击性能，但现有的视觉语言模型仍然缺乏针对域特异性问题的几乎没有传输能力。经典的微调通常无法阻止高度表达模型利用虚假相关性。尽管模型不足的元学习（MAML）作为几次转移学习的天然替代方案，但由于隐式二阶优化而引起的昂贵计算限制了其在大规模视觉语言模型（例如剪辑）上的使用。尽管许多文献都致力于探索替代优化策略，但我们确定了有效的几次转移学习，任务抽样的另一个基本方面，以前仅将其视为MAML中数据预处理的一部分。为了显示任务采样的影响，我们提出了一种简单的算法，模型不合时宜的多任务微调（MAMF），该算法仅在均匀地采样多个任务上区分了经典的微调。尽管它很简单，但我们表明，MAMF在五个几乎没有视觉语言分类任务上始终优于经典的微调。我们进一步表明，MAML中BI级优化的有效性对在几乎没有射击视觉分类的上下文中对任务的零弹性性能高度敏感。本文的目的是提供有关几乎没有成功学习工作的新见解，并鼓励更多的研究来研究更好的任务抽样策略。

从自然语言监督中学习视觉表示，最近在许多开创性的作品中表现出了巨大的希望。通常，这些具有语言的视觉模型表现出对各种数据集和任务的强大可传递性。但是，由于缺乏易于使用的评估工具包和公共基准，评估这些模型的可转让性仍然很具有挑战性。为了解决这个问题，我们构建了高级版（评估语言的视觉任务级传输），这是用于评估（预训练）语言增强视觉模型的第一个基准和工具包。升华由三个组成部分组成。 （i）数据集。作为下游评估套件，它由20个图像分类数据集和35个对象检测数据集组成，每个数据集都用外部知识来增强。 （ii）工具包。开发了自动高参数调谐工具包，以促进下游任务的模型评估。 （iii）指标。多种评估指标用于测量样品效率（零射击和少量）和参数效率（线性探测和完整模型微调）。我们在https://computer-vision-in-the-wild.github.io/elevater/上公开发布leverater

Prompt tuning is a new few-shot transfer learning technique that only tunes the learnable prompt for pre-trained vision and language models such as CLIP. However, existing prompt tuning methods tend to learn spurious or entangled representations, which leads to poor generalization to unseen concepts. Towards non-spurious and efficient prompt learning from limited examples, this paper presents a novel \underline{\textbf{C}}ounterfactual \underline{\textbf{P}}rompt \underline{\textbf{L}}earning (CPL) method for vision and language models, which simultaneously employs counterfactual generation and contrastive learning in a joint optimization framework. Particularly, CPL constructs counterfactual by identifying minimal non-spurious feature change between semantically-similar positive and negative samples that causes concept change, and learns more generalizable prompt representation from both factual and counterfactual examples via contrastive learning. Extensive experiments demonstrate that CPL can obtain superior few-shot performance on different vision and language tasks than previous prompt tuning methods on CLIP. On image classification, we achieve 3.55\% average relative improvement on unseen classes across seven datasets; on image-text retrieval and visual question answering, we gain up to 4.09\% and 25.08\% relative improvements across three few-shot scenarios on unseen test sets respectively.

视觉模型最近在许多计算机视觉任务上显示出巨大的潜力。同时，与线性探针相比，先前的工作表明，与线性探针相比，这是较少的图像识别的迅速调整，可以在很少的图像识别上获得卓越的性能。在实际应用程序中，相关的几个射击任务是相关的，尤其是在专业领域。但是，以前的工作忽略了此类信息。受到以下事实的启发，即通过多任务学习通常可以提高性能，我们提出了一种新颖的方法softcpt（迅速调整的软上下文共享），以微调多个目标几个目标任务的预训练的视觉模型， 同时。具体来说，我们设计了一个任务共享的元网络，以使用预定义的任务名称以及可学习的元提示为输入为每个任务生成提示向量。因此，所有任务的迅速向量将以软的方式共享。该共享的元网络的参数以及元提示向量都在所有目标任务的联合培训集中调整。在三个多任务少量数据集上进行的广泛实验表明，SoftCpt的表现优于代表性的单任务提示方法Coop [78]，这意味着多任务学习在视觉及时及时调整中的有效性。源代码和数据将公开可用。

预训练的视觉模型（例如，剪辑）在许多下游任务中显示出有希望的零弹性概括，并具有正确设计的文本提示。最近的作品不依赖手工设计的提示，而是使用下游任务的培训数据来学习提示。虽然有效，但针对领域数据的培训却降低了模型的概括能力，使其无法看到新领域。在这项工作中，我们提出了测试时间提示调整（TPT），该方法可以通过单个测试样本即时学习自适应提示。对于图像分类，TPT通过使用置信度选择最小化熵来优化提示，以便模型在每个测试样本的不同增强视图上都具有一致的预测。在评估对自然分布变化的概括时，TPT平均将零击的TOP-1精度提高了3.6％，超过了先前需要其他特定于任务的训练数据的迅速调整方法。在评估看不见类别的跨数据集泛化时，TPT与使用其他培训数据的最先进方法相当。项目页面：https：//azshue.github.io/tpt。

Finetuning image-text models such as CLIP achieves state-of-the-art accuracies on a variety of benchmarks. However, recent works like WiseFT (Wortsman et al., 2021) and LP-FT (Kumar et al., 2022) have shown that even subtle differences in the finetuning process can lead to surprisingly large differences in the final performance, both for in-distribution (ID) and out-of-distribution (OOD) data. In this work, we show that a natural and simple approach of mimicking contrastive pretraining consistently outperforms alternative finetuning approaches. Specifically, we cast downstream class labels as text prompts and continue optimizing the contrastive loss between image embeddings and class-descriptive prompt embeddings (contrastive finetuning). Our method consistently outperforms baselines across 7 distribution shifts, 6 transfer learning, and 3 few-shot learning benchmarks. On WILDS-iWILDCam, our proposed approach FLYP outperforms the top of the leaderboard by $2.3\%$ ID and $2.7\%$ OOD, giving the highest reported accuracy. Averaged across 7 OOD datasets (2 WILDS and 5 ImageNet associated shifts), FLYP gives gains of $4.2\%$ OOD over standard finetuning and outperforms the current state of the art (LP-FT) by more than $1\%$ both ID and OOD. Similarly, on 3 few-shot learning benchmarks, our approach gives gains up to $4.6\%$ over standard finetuning and $4.4\%$ over the state of the art. In total, these benchmarks establish contrastive finetuning as a simple, intuitive, and state-of-the-art approach for supervised finetuning of image-text models like CLIP. Code is available at https://github.com/locuslab/FLYP.

从任务不足的预训练的深层模型中转移知识以进行下游任务是计算机视觉研究中的一个重要主题。随着计算能力的增长，我们现在拥有大规模的模型体系结构和数据量的开源视觉语言预培训模型。在这项研究中，我们专注于转移视力分类任务的知识。传统方法随机初始化线性分类器头进行视觉分类，但是它们将文本编码器的用法留为未发现的下游视觉识别任务。在本文中，我们修改了线性分类器的角色，并用对象类别的嵌入式语言表示替换分类器。这些语言表示是从视觉语言预训练模型的文本编码器初始化的，以进一步利用其良好的语言模型参数。实证研究表明，我们的方法提高了视频分类的性能和训练速度，模型的变化微不足道。特别是，我们的范式在动力学400上实现了87.3％的最新准确性。

Prompt learning is one of the most effective and trending ways to adapt powerful vision-language foundation models like CLIP to downstream datasets by tuning learnable prompt vectors with very few samples. However, although prompt learning achieves excellent performance over in-domain data, it still faces the major challenge of generalizing to unseen classes and domains. Some existing prompt learning methods tackle this issue by adaptively generating different prompts for different tokens or domains but neglecting the ability of learned prompts to generalize to unseen domains. In this paper, we propose a novel prompt learning paradigm that directly generates domain invariant prompt generalizable to unseen domains, called MetaPrompt. Specifically, a dual-modality prompt tuning network is proposed to generate prompts for inputs from both image and text modalities. More importantly, we propose a meta-learning-based prompt tuning algorithm that explicitly constrains the prompt tuned on a specific domain or class also to achieve good performance on another domain or class. Extensive experiments on 11 datasets for base-to-new generalization and four datasets for domain generalization demonstrate that our method consistently and significantly outperforms existing methods.

Pre-trained representations are becoming crucial for many NLP and perception tasks. While representation learning in NLP has transitioned to training on raw text without human annotations, visual and vision-language representations still rely heavily on curated training datasets that are expensive or require expert knowledge. For vision applications, representations are mostly learned using datasets with explicit class labels such as Ima-geNet or OpenImages. For vision-language, popular datasets like Conceptual Captions, MSCOCO, or CLIP all involve a non-trivial data collection (and cleaning) process. This costly curation process limits the size of datasets and hence hinders the scaling of trained models. In this paper, we leverage a noisy dataset of over one billion image alt-text pairs, obtained without expensive filtering or post-processing steps in the Conceptual Captions dataset. A simple dual-encoder architecture learns to align visual and language representations of the image and text pairs using a contrastive loss. We show that the scale of our corpus can make up for its noise and leads to state-of-the-art representations even with such a simple learning scheme. Our visual representation achieves strong performance when transferred to classification tasks such as ImageNet and VTAB. The aligned visual and language representations enables zero-shot image classification and also set new state-of-the-art results on Flickr30K and MSCOCO image-text retrieval benchmarks, even when compared with more sophisticated crossattention models. The representations also enable cross-modality search with complex text and text + image queries.

随着大型预训练的Vison语言模型（如剪辑）的出现，可以通过及时调整来调整可转让表示形式。及时调整试图从存储在预训练的视觉模型的图像和文本编码器中的常识中探索有益信息，以探索下游任务。最近提出的名为“上下文优化”（COP）的方法将一组可学习的向量从语言侧引入文本提示符，而单独调整文本提示符则不会影响图像编码器的计算视觉特征，从而导致了次级优势。在本文中，我们通过学习文本提示并同时为文本和图像编码器提供双重模式提示调整范式。此外，为了使视觉提示更多地集中在目标视觉概念上，我们提出了类感知的视觉及时调整（CAVPT），该调整是通过在模板提示和视觉类别令牌嵌入的语言描述之间进行交叉注意来动态生成的。我们的方法提供了一种新的范式来调整大型预训练的视觉模型，并在8个数据集上进行了广泛的实验结果，证明了该方法的有效性。我们的代码在补充材料中可用。

Contrastive Language-Image Pre-training (CLIP) has emerged as a simple yet effective way to train large-scale vision-language models. CLIP demonstrates impressive zero-shot classification and retrieval on diverse downstream tasks. However, to leverage its full potential, fine-tuning still appears to be necessary. Fine-tuning the entire CLIP model can be resource-intensive and unstable. Moreover, recent methods that aim to circumvent this need for fine-tuning still require access to images from the target distribution. In this paper, we pursue a different approach and explore the regime of training-free "name-only transfer" in which the only knowledge we possess about the downstream task comprises the names of downstream target categories. We propose a novel method, SuS-X, consisting of two key building blocks -- SuS and TIP-X, that requires neither intensive fine-tuning nor costly labelled data. SuS-X achieves state-of-the-art zero-shot classification results on 19 benchmark datasets. We further show the utility of TIP-X in the training-free few-shot setting, where we again achieve state-of-the-art results over strong training-free baselines. Code is available at https://github.com/vishaal27/SuS-X.

Vision-language foundation models pretrained on large-scale data provide a powerful tool for many visual understanding tasks. Notably, many vision-language models build two encoders (visual and textual) that can map two modalities into the same embedding space. As a result, the learned representations achieve good zero-shot performance on tasks like image classification. However, when there are only a few examples per category, the potential of large vision-language models is often underperformed, mainly due to the gap between a large number of parameters and a relatively small amount of training data. This paper shows that we can significantly improve the performance of few-shot classification by using the category names to initialize the classification head. More interestingly, we can borrow the non-perfect category names, or even names from a foreign language, to improve the few-shot classification performance compared with random initialization. With the proposed category name initialization method, our model obtains the state-of-the-art performance on a number of few-shot image classification benchmarks (e.g., 87.37\% on ImageNet and 96.08\% on Stanford Cars, both using five-shot learning). We also investigate and analyze when the benefit of category names diminishes and how to use distillation to improve the performance of smaller models, providing guidance for future research.

Prompt Tuning, conditioning on task-specific learned prompt vectors, has emerged as a data-efficient and parameter-efficient method for adapting large pretrained vision-language models to multiple downstream tasks. However, existing approaches usually consider learning prompt vectors for each task independently from scratch, thereby failing to exploit the rich shareable knowledge across different vision-language tasks. In this paper, we propose multitask vision-language prompt tuning (MVLPT), which incorporates cross-task knowledge into prompt tuning for vision-language models. Specifically, (i) we demonstrate the effectiveness of learning a single transferable prompt from multiple source tasks to initialize the prompt for each target task; (ii) we show many target tasks can benefit each other from sharing prompt vectors and thus can be jointly learned via multitask prompt tuning. We benchmark the proposed MVLPT using three representative prompt tuning methods, namely text prompt tuning, visual prompt tuning, and the unified vision-language prompt tuning. Results in 20 vision tasks demonstrate that the proposed approach outperforms all single-task baseline prompt tuning methods, setting the new state-of-the-art on the few-shot ELEVATER benchmarks and cross-task generalization benchmarks. To understand where the cross-task knowledge is most effective, we also conduct a large-scale study on task transferability with 20 vision tasks in 400 combinations for each prompt tuning method. It shows that the most performant MVLPT for each prompt tuning method prefers different task combinations and many tasks can benefit each other, depending on their visual similarity and label similarity. Code is available at https://github.com/sIncerass/MVLPT.

如今，基于变压器的模型逐渐成为人工智能先驱的默认选择。即使在几个镜头的情况下，这些模型也会显示出优势。在本文中，我们重新审视了经典方法，并提出了一种新的几次替代方法。具体而言，我们研究了几个镜头的单级问题，该问题实际上以已知样本为参考来检测未知实例是否属于同一类。可以从序列匹配的角度研究此问题。结果表明，使用元学习，经典序列匹配方法，即比较聚集，显着优于变压器。经典方法所需的培训成本要少得多。此外，我们在简单的微调和元学习下进行两种序列匹配方法之间进行了经验比较。元学习导致变压器模型的特征具有高相关尺寸。原因与变压器模型的层和头数密切相关。实验代码和数据可从https://github.com/hmt2014/fewone获得

我们引入了构图软提示（CSP），这是一种参数有效的学习技术，可改善大规模预处理视觉模型（VLMS）的零摄像组成性。 VLM可以在其灵活的文本编码器中代表任意类作为自然语言提示，但在组成零击基准任务上的表现不佳。为了改善VLM，我们提出了一种新颖的软提示形式。我们将构成的属性和对象视为将类定义为词汇的可学习令牌，并在多个及时的构图上调整它们。在推断期间，我们在新组合中重新组装了学习的属性对象词汇。我们表明，CSP在基准数据集上的原始VLM的表现平均为AUC上的10.9个百分点。 CSP还胜过Coop，这是一种调谐前缀上下文的软提示方法，在AUC上平均要点5.8个百分点。我们执行其他实验，以表明CSP对仅属性分类，高阶属性 - 属性对象组成以及预验证属性和微调对象的组合进行了改进。

Contrastive Language-Image Pre-trained (CLIP) models have zero-shot ability of classifying an image belonging to "[CLASS]" by using similarity between the image and the prompt sentence "a [CONTEXT] of [CLASS]". Based on exhaustive text cues in "[CONTEXT]", CLIP model is aware of different contexts, e.g. background, style, viewpoint, and exhibits unprecedented robustness against a wide range of distribution shifts. However, recent works find further fine-tuning of CLIP models improves accuracy but sacrifices the robustness on downstream tasks. We conduct an empirical investigation to show fine-tuning will corrupt the context-aware ability of pre-trained CLIP features. To solve this problem, we propose Context-Aware Robust Fine-tuning (CAR-FT). CAR-FT regularizes the model during fine-tuning to capture the context information. Specifically, we use zero-shot prompt weights to get the context distribution contained in the image. By minimizing the Kullback-Leibler Divergence (KLD) between context distributions induced by original/fine-tuned CLIP models, CAR-FT makes the context-aware ability of CLIP inherited into downstream tasks, and achieves both higher In-Distribution (ID) and Out-Of-Distribution (OOD) accuracy. The experimental results show CAR-FT achieves superior robustness on five OOD test datasets of ImageNet, and meanwhile brings accuracy gains on nine downstream tasks. Additionally, CAR-FT surpasses previous Domain Generalization (DG) methods and gets 78.5% averaged accuracy on DomainBed benchmark, building the new state-of-the-art.

视觉世界自然地展现了一个长尾的开放类分布，这对现代视觉系统带来了巨大挑战。现有方法可以执行类重新平衡策略或直接改进网络模块以解决问题。然而，他们仍然用有限一套预定义标签训练模型，限制了他们的监督信息并限制了他们对新颖实例的可转移性。新途径上的大型对比视觉普瑞宁普雷宁闪光灯的最新进展，可视识别。利用开放词汇监督，预先染色的对比视觉语言模型学习强大的多模式表示，这是对处理数据缺陷和看不见的概念。通过计算视觉和文本输入之间的语义相似性，可视识别被转换为vision语言匹配问题。灵感来自于此，我们提出了民谣，利用了对比尾识别的对比视觉模型。我们首先通过对特定的长尾目标数据集进行对比学习继续预先预留视觉语言骨干。之后，我们冻结了骨干，进一步采用了额外的适配器层，以增强通过重新采样策略构建的平衡训练样本上的尾级课程的表示。已经在三个流行的长尾识别基准测试中进行了广泛的实验。因此，我们简单有效的方法设定了新的最先进的表演，优于具有大边距的竞争基础。代码在https://github.com/gaopengcuhk/ballad发布。

元学习可以从先前的学习体验中提取归纳偏见，并协助培训新任务。通常通过优化特定于任务求解器的评估损失来实现META模型来实现。大多数现有算法样本非重叠$ \ mathit {support} $ sets和$ \ mathit {查询} $ sets以分别为培训和评估求解器（$ \ mathcal {s} $ / $ \ mathcal {q} $协议）。不同于$ \ mathcal {s} $ / $ \ mathcal {q} $协议，我们还可以通过将其与目标型号$ \ mathcal {t} $进行比较来评估任务特定的求解器，这是它的最佳模型任务或在此任务中足够好的模型（$ \ mathcal {s} $ / $ \ mathcal {t} $协议）。虽然研究短缺，但$ \ mathcal {s} $ / $ \ mathcal {t} $协议具有独特的优势，如提供更具信息性的监督，但它是计算昂贵的。本文研究了这种特殊的评估方法，迈出了将它付诸实践。我们发现，通过掌握目标模型的任务比例小，可以提高经典的元学习算法，而不会消耗许多资源。我们在典型的Meta-Learning，$ \ Mathit {i.} $，几秒钟学习中验证$ \ mathcal {s} $ / $ \ mathcal {t} $协议的有效性。详细地，在通过微调预先调整那些硬任务的预先训练网络之后构建目标模型之后，我们通过知识蒸馏匹配任务特定的求解器和目标模型。

State-of-the-art computer vision systems are trained to predict a fixed set of predetermined object categories. This restricted form of supervision limits their generality and usability since additional labeled data is needed to specify any other visual concept. Learning directly from raw text about images is a promising alternative which leverages a much broader source of supervision. We demonstrate that the simple pre-training task of predicting which caption goes with which image is an efficient and scalable way to learn SOTA image representations from scratch on a dataset of 400 million (image, text) pairs collected from the internet. After pre-training, natural language is used to reference learned visual concepts (or describe new ones) enabling zero-shot transfer of the model to downstream tasks. We study the performance of this approach by benchmarking on over 30 different existing computer vision datasets, spanning tasks such as OCR, action recognition in videos, geo-localization, and many types of fine-grained object classification. The model transfers non-trivially to most tasks and is often competitive with a fully supervised baseline without the need for any dataset specific training. For instance, we match the accuracy of the original ResNet-50 on ImageNet zero-shot without needing to use any of the 1.28 million training examples it was trained on. We release our code and pre-trained model weights at https://github.com/OpenAI/CLIP.

最先进的愿景和愿景和语言模型依靠大规模的Visio-linguisting预借鉴，以获得各种下游任务的良好性能。通常，这种模型通常是跨模态（对比）或多模态（具有早期融合）但不是两者;它们通常只针对特定的方式或任务。有希望的方向将是使用单一整体普遍模型，作为“基础”，目标是一次性的所有方式 - 真正的视觉和语言基础模型应该擅长视力任务，语言任务和交叉和多数模态视觉和语言任务。我们将Flava介绍在这样的模型中，并在跨越这些目标模式的广泛的35个任务上展示令人印象深刻的性能。