Recent visuolinguistic pre-trained models show promising progress on various end tasks such as image retrieval and video captioning. Yet, they fail miserably on the recently proposed Winoground dataset, which challenges models to match paired images and English captions, with items constructed to overlap lexically but differ in meaning (e.g., "there is a mug in some grass" vs. "there is some grass in a mug"). By annotating the dataset using new fine-grained tags, we show that solving the Winoground task requires not just compositional language understanding, but a host of other abilities like commonsense reasoning or locating small, out-of-focus objects in low-resolution images. In this paper, we identify the dataset's main challenges through a suite of experiments on related tasks (probing task, image retrieval task), data augmentation, and manual inspection of the dataset. Our analysis suggests that a main challenge in visuolinguistic models may lie in fusing visual and textual representations, rather than in compositional language understanding. We release our annotation and code at https://github.com/ajd12342/why-winoground-hard .

State-of-the-art computer vision systems are trained to predict a fixed set of predetermined object categories. This restricted form of supervision limits their generality and usability since additional labeled data is needed to specify any other visual concept. Learning directly from raw text about images is a promising alternative which leverages a much broader source of supervision. We demonstrate that the simple pre-training task of predicting which caption goes with which image is an efficient and scalable way to learn SOTA image representations from scratch on a dataset of 400 million (image, text) pairs collected from the internet. After pre-training, natural language is used to reference learned visual concepts (or describe new ones) enabling zero-shot transfer of the model to downstream tasks. We study the performance of this approach by benchmarking on over 30 different existing computer vision datasets, spanning tasks such as OCR, action recognition in videos, geo-localization, and many types of fine-grained object classification. The model transfers non-trivially to most tasks and is often competitive with a fully supervised baseline without the need for any dataset specific training. For instance, we match the accuracy of the original ResNet-50 on ImageNet zero-shot without needing to use any of the 1.28 million training examples it was trained on. We release our code and pre-trained model weights at https://github.com/OpenAI/CLIP.

Large-scale pre-training methods of learning cross-modal representations on image-text pairs are becoming popular for vision-language tasks. While existing methods simply concatenate image region features and text features as input to the model to be pre-trained and use selfattention to learn image-text semantic alignments in a brute force manner, in this paper, we propose a new learning method Oscar 1 , which uses object tags detected in images as anchor points to significantly ease the learning of alignments. Our method is motivated by the observation that the salient objects in an image can be accurately detected, and are often mentioned in the paired text. We pre-train an Oscar model on the public corpus of 6.5 million text-image pairs, and fine-tune it on downstream tasks, creating new state-of-the-arts on six well-established vision-language understanding and generation tasks. 2

Large-scale vision-language models such as CLIP have shown impressive performance on zero-shot image classification and image-to-text retrieval. However, such zero-shot performance of CLIP-based models does not realize in tasks that require a finer-grained correspondence between vision and language, such as Visual Question Answering (VQA). We investigate why this is the case, and report an interesting phenomenon of CLIP, which we call the Concept Association Bias (CAB), as a potential cause of the difficulty of applying CLIP to VQA and similar tasks. CAB is especially apparent when two concepts are present in the given image while a text prompt only contains a single concept. In such a case, we find that CLIP tends to treat input as a bag of concepts and attempts to fill in the other missing concept crossmodally, leading to an unexpected zero-shot prediction. For example, when asked for the color of a lemon in an image, CLIP predicts ``purple'' if the image contains a lemon and an eggplant. We demonstrate the Concept Association Bias of CLIP by showing that CLIP's zero-shot classification performance greatly suffers when there is a strong concept association between an object (e.g. lemon) and an attribute (e.g. its color). On the other hand, when the association between object and attribute is weak, we do not see this phenomenon. Furthermore, we show that CAB is significantly mitigated when we enable CLIP to learn deeper structure across image and text embeddings by adding an additional Transformer on top of CLIP and fine-tuning it on VQA. We find that across such fine-tuned variants of CLIP, the strength of CAB in a model predicts how well it performs on VQA.

随着图像文本对的大量数据以及视觉和语言（V＆L）任务的多样性，学者在该研究领域引入了大量的深度学习模型。此外，近年来，转移学习还显示出在计算机愿景中的巨大成功，例如图像分类，对象检测等以及在自然语言处理中以进行问答，机器翻译等的自然语言处理。继承转移学习的精神， V＆L的研究工作已经在大规模数据集上设计了多种预训练技术，以增强下游任务的性能。本文的目的是提供当代V＆L预审前模型的全面修订。特别是，我们对预处理的方法进行了分类和描述，以及最先进的视觉和语言预训练模型的摘要。此外，还提供了培训数据集和下游任务的列表，以进一步提高V＆L预处理的观点。最后，我们决定采取进一步的一步，讨论众多未来研究的方向。

我们提出Valse（视觉和语言结构化评估），这是一种新的基准，专为测试通用净化的视觉和语言（V＆L）模型而设计，用于对特定语言现象的视野 - 语言接地能力。Valse提供涵盖各种语言构建体的六种测试套件。解决这些需要模型在视觉模型中地对语言现象，允许比迄今为止更细粒度的评估。我们使用支持有效箔的构造的方法构建Valse，并通过评估五种广泛使用的V＆L模型的报告结果。我们的实验表明，目前的模型有很大的困难解决了大多数现象。因此，我们预计Valse就可以作为一种重要的基准，从语言角度来衡量预训过的V＆L模型的未来进展，补充规范任务为中心的V＆L评价。

Image captioning models tend to describe images in an object-centric way, emphasising visible objects. But image descriptions can also abstract away from objects and describe the type of scene depicted. In this paper, we explore the potential of a state-of-the-art Vision and Language model, VinVL, to caption images at the scene level using (1) a novel dataset which pairs images with both object-centric and scene descriptions. Through (2) an in-depth analysis of the effect of the fine-tuning, we show (3) that a small amount of curated data suffices to generate scene descriptions without losing the capability to identify object-level concepts in the scene; the model acquires a more holistic view of the image compared to when object-centric descriptions are generated. We discuss the parallels between these results and insights from computational and cognitive science research on scene perception.

Pre-trained representations are becoming crucial for many NLP and perception tasks. While representation learning in NLP has transitioned to training on raw text without human annotations, visual and vision-language representations still rely heavily on curated training datasets that are expensive or require expert knowledge. For vision applications, representations are mostly learned using datasets with explicit class labels such as Ima-geNet or OpenImages. For vision-language, popular datasets like Conceptual Captions, MSCOCO, or CLIP all involve a non-trivial data collection (and cleaning) process. This costly curation process limits the size of datasets and hence hinders the scaling of trained models. In this paper, we leverage a noisy dataset of over one billion image alt-text pairs, obtained without expensive filtering or post-processing steps in the Conceptual Captions dataset. A simple dual-encoder architecture learns to align visual and language representations of the image and text pairs using a contrastive loss. We show that the scale of our corpus can make up for its noise and leads to state-of-the-art representations even with such a simple learning scheme. Our visual representation achieves strong performance when transferred to classification tasks such as ImageNet and VTAB. The aligned visual and language representations enables zero-shot image classification and also set new state-of-the-art results on Flickr30K and MSCOCO image-text retrieval benchmarks, even when compared with more sophisticated crossattention models. The representations also enable cross-modality search with complex text and text + image queries.

The availability of large-scale image captioning and visual question answering datasets has contributed significantly to recent successes in vision-and-language pretraining. However, these datasets are often collected with overrestrictive requirements inherited from their original target tasks (e.g., image caption generation), which limit the resulting dataset scale and diversity. We take a step further in pushing the limits of vision-and-language pretraining data by relaxing the data collection pipeline used in Conceptual Captions 3M (CC3M) [70] and introduce the Conceptual 12M (CC12M), a dataset with 12 million image-text pairs specifically meant to be used for visionand-language pre-training. We perform an analysis of this dataset and benchmark its effectiveness against CC3M on multiple downstream tasks with an emphasis on long-tail visual recognition. Our results clearly illustrate the benefit of scaling up pre-training data for vision-and-language tasks, as indicated by the new state-of-the-art results on both the nocaps and Conceptual Captions benchmarks. 1

随着网络和在线百科全书的可访问性的增加，要管理的数据量正在不断增加。例如，在Wikipedia中，有数百万页用多种语言编写。这些页面包含通常缺乏文本上下文的图像，在概念上保持浮动，因此很难找到和管理。在这项工作中，我们介绍了我们设计的系统，用于参加Kaggle上的Wikipedia图像捕捉匹配挑战，其目的是使用与图像（URL和视觉数据）相关的数据来在大量可用图像中找到正确的标题。能够执行此任务的系统将改善大型在线百科全书上多媒体内容的可访问性和完整性。具体而言，我们提出了一个由最近的变压器模型提供支持的两个模型的级联，能够有效地推断出查询图像数据和字幕之间的相关得分。我们通过广泛的实验来验证，提出的两模型方法是处理大量图像和标题的有效方法，同时保持了推理时的整体计算复杂性。我们的方法取得了显着的结果，在Kaggle Challenge的私人排行榜上获得了0.53的归一化折扣累积增益（NDCG）值。

已经开发了许多Visio语言（V + L）表示学习方法，但现有数据集不会评估它们在统一空间中代表视觉和语言概念的程度。灵感来自于奇妙的转移和精神语言学文献，我们提出了一个新的V + L型号的评价设置：零射频跨模型转移。现有的V + L基准也经常在整个数据集上报告全局精度分数，渲染难以确定模型失败并成功的具体推理任务。要解决此问题并启用对跨模型传输的评估，我们存在TRAVLR，包括四个V + L推理任务的合成数据集。每个示例对场景进行了双倍，使得在训练/测试期间可以丢弃无论是没有相关信息的丢失。 Travlr的培训和测试分布也沿任务相关维度约束，从而可以评估分配外概括。我们评估了四个最先进的V + L型号，发现它们在从同一模态的测试集上表现良好，但所有型号都无法转移交叉模态，并且成功有限，容纳一个模态的添加或删除。在与事先工作的对齐中，我们还发现这些模型需要大量数据来学习简单的空间关系。我们将Travlr释放为研究界的开放挑战。

注意机制已被用作跨视觉和语言（VL）任务的重要组成部分，以弥合视觉和文本特征之间的语义差距。尽管注意力已被广泛用于VL任务，但尚未研究其在弥合视觉和文本线索之间语义差距方面的不同注意对准计算的能力。在这项研究中，我们通过研究注意力评分计算方法，并检查其实际代表视觉区域的作用以及文本令牌对全球评估的重要性，对了解注意力对齐的作用进行全面分析。我们还分析了注意力分数计算机制的条件更多（或更少）可解释，并且可能会影响三个不同VL任务的模型性能，包括视觉问题答案，文本到图像生成，文本和图像匹配（句子和图像检索）。我们的分析是同类产品中的第一个，并提供了在VL任务的训练阶段应用的每个注意力对齐得分计算的重要性，这通常在基于注意力的交叉模态模型和/或预审前的模型中被忽略。

We introduce a new dataset for joint reasoning about natural language and images, with a focus on semantic diversity, compositionality, and visual reasoning challenges. The data contains 107,292 examples of English sentences paired with web photographs. The task is to determine whether a natural language caption is true about a pair of photographs. We crowdsource the data using sets of visually rich images and a compare-and-contrast task to elicit linguistically diverse language. Qualitative analysis shows the data requires compositional joint reasoning, including about quantities, comparisons, and relations. Evaluation using state-of-the-art visual reasoning methods shows the data presents a strong challenge. * Contributed equally. † Work done as an undergraduate at Cornell University. 1 In parts of this paper, we use the term compositional differently than it is commonly used in linguistics to refer to reasoning that requires composition. This type of reasoning often manifests itself in highly compositional language.2 Appendix G contains license information for all photographs used in this paper. 3 The top example is True, while the bottom is False.

图像文本匹配（ITM）是评估视觉和语言（VL）模型的常见任务。但是，现有的ITM基准有一个重大限制。他们有许多缺失的信件，源自数据构建过程本身。例如，标题仅与一个图像匹配，尽管标题可以与其他类似图像匹配，反之亦然。为了纠正大规模的虚假负面因素，我们通过提供与机器和人类注释者的缺失关联来构建扩展的可可验证（ECCV）标题数据集。我们在注释过程中采用五个具有不同属性的最先进的ITM模型。与原始的MS-Coco相比，我们的数据集提供了X3.6的X3.6积极图像到支撑关联和X8.5字幕到图像关联。我们还建议使用基于等级的公制映射@r，而不是流行的召回@k（r@k）。我们在现有和拟议的基准测试中重新评估了现有的25个VL模型。我们的发现是现有的基准测试，例如可可1K r@k，可可5k r@k，cxc r@1彼此高度相关，而当我们转移到eccv map@r时，排名会改变。最后，我们深入研究机器注释者选择引入的偏差的效果。源代码和数据集可从https://github.com/naver-ai/eccv-caption获得

连接视觉和语言在生成智能中起着重要作用。因此，已经致力于图像标题的大型研究工作，即用句法和语义有意义的句子描述图像。从2015年开始，该任务通常通过由Visual Encoder组成的管道和文本生成的语言模型来解决任务。在这些年来，两种组件通过对象区域，属性，介绍多模态连接，完全关注方法和伯特早期融合策略的利用而显着发展。但是，无论令人印象深刻的结果，图像标题的研究还没有达到结论性答案。这项工作旨在提供图像标题方法的全面概述，从视觉编码和文本生成到培训策略，数据集和评估度量。在这方面，我们量化地比较了许多相关的最先进的方法来确定架构和培训策略中最有影响力的技术创新。此外，讨论了问题的许多变体及其开放挑战。这项工作的最终目标是作为理解现有文献的工具，并突出显示计算机视觉和自然语言处理的研究领域的未来方向可以找到最佳的协同作用。

误导现在是由于其核心民主和社会价值观和订单的潜在高风险导致的主要问题。外观的错误信息是对病毒假故事进行的对手使用的最简单有效的方法之一。在这种威胁中，通过歪曲其上下文和/或元素来重新设计真实的图像以支持其他叙述。互联网被用作使用不同来源和模态的信息来验证信息。我们的目标是一种可防止的方法，通过使用Web证据来检查图像标题配对来自动实现这一耗时和推理的密集流程。要从两种方式集成证据和提示，我们介绍了“多模态周期 - 一致性检查”的概念;从图像/标题开始，我们收集文本/视觉证据，将分别与其他配对的字幕/图像进行比较。此外，我们提出了一种新颖的架构，一致性检查网络（CCN），其模拟了相同和不同的方式的分层人工理学：标题与文本证据，图像与视觉证据和图像与标题。我们的工作为开放式，基于内容，多模态事实检查提供的第一步和基准，并且显着优于未杠杆效率的基准。

This paper focuses on analyzing and improving the commonsense ability of recent popular vision-language (VL) models. Despite the great success, we observe that existing VL-models still lack commonsense knowledge/reasoning ability (e.g., "Lemons are sour"), which is a vital component towards artificial general intelligence. Through our analysis, we find one important reason is that existing large-scale VL datasets do not contain much commonsense knowledge, which motivates us to improve the commonsense of VL-models from the data perspective. Rather than collecting a new VL training dataset, we propose a more scalable strategy, i.e., "Data Augmentation with kNowledge graph linearization for CommonsensE capability" (DANCE). It can be viewed as one type of data augmentation technique, which can inject commonsense knowledge into existing VL datasets on the fly during training. More specifically, we leverage the commonsense knowledge graph (e.g., ConceptNet) and create variants of text description in VL datasets via bidirectional sub-graph sequentialization. For better commonsense evaluation, we further propose the first retrieval-based commonsense diagnostic benchmark. By conducting extensive experiments on some representative VL-models, we demonstrate that our DANCE technique is able to significantly improve the commonsense ability while maintaining the performance on vanilla retrieval tasks. The code and data are available at https://github.com/pleaseconnectwifi/DANCE

使用自然语言作为培训视觉识别模型的监督持有巨大的承诺。最近的作品表明，如果在大型训练数据集中的图像和标题之间的对齐形式使用此类监督，则结果对齐模型在零拍摄分类中表现出色，如下游任务2。在本文中，我们专注于挑逗语言监督的哪些部分对于训练零拍摄图像分类模型至关重要。通过广泛和仔细的实验​​，我们表明：1）可以将简单的单词（弓）标题用作数据集中大多数图像标题的替代品。令人惊讶的是，我们观察到这种方法在与单词平衡结合时提高了零拍分类性能。 2）使用船首净化模型，我们可以通过在没有标题的图像上生成伪弓标题来获得更多培训数据。使用真实和伪弓形标题培训的模型达到了更强的零射性能。在ImageNet-1K零拍评估中，我们只使用3M图像标题对的最佳模型，使用15M图像标题对培训的剪辑模型（31.5％VS 31.3％）进行剪辑。

Many high-level skills that are required for computer vision tasks, such as parsing questions, comparing and contrasting semantics, and writing descriptions, are also required in other domains such as natural language processing. In this paper, we ask whether this makes it possible to learn those skills from text data and then use them to complete vision tasks without ever training on visual training data. Key to our approach is exploiting the joint embedding space of contrastively trained vision and language encoders. In practice, there can be systematic differences between embedding spaces for different modalities in contrastive models, and we analyze how these differences affect our approach and study a variety of strategies to mitigate this concern. We produce models using only text training data on three tasks: image captioning, visual entailment and visual question answering, and evaluate them on standard benchmarks using images. We find that this kind of transfer is possible and results in only a small drop in performance relative to models trained on images. We also showcase a variety of stylistic image captioning models that were trained using no image data and no human-curated language data, but instead text data from books, the web, or language models.

Vision and language models (VL) are known to exploit unrobust indicators in individual modalities (e.g., introduced by distributional biases), instead of focusing on relevant information in each modality. A small drop in accuracy obtained on a VL task with a unimodal model suggests that so-called unimodal collapse occurred. But how to quantify the amount of unimodal collapse reliably, at dataset and instance-level, to diagnose and combat unimodal collapse in a targeted way? We present MM-SHAP, a performance-agnostic multimodality score that quantifies the proportion by which a model uses individual modalities in multimodal tasks. MM-SHAP is based on Shapley values and will be applied in two ways: (1) to compare models for their degree of multimodality, and (2) to measure the contribution of individual modalities for a given task and dataset. Experiments with 6 VL models -- LXMERT, CLIP and four ALBEF variants -- on four VL tasks highlight that unimodal collapse can occur to different degrees and in different directions, contradicting the wide-spread assumption that unimodal collapse is one-sided. We recommend MM-SHAP for analysing multimodal tasks, to diagnose and guide progress towards multimodal integration. Code available at: https://github.com/Heidelberg-NLP/MM-SHAP