Large-scale models combining text and images have made incredible progress in recent years. However, they can still fail at tasks requiring compositional knowledge, such as correctly picking out a red cube from a picture of multiple shapes. We examine the ability of CLIP (Radford et al., 2021), to caption images requiring compositional knowledge. We implement five compositional language models to probe the kinds of structure that CLIP may be using, and develop a novel training algorithm, Compositional Skipgram for Images (CoSI), to train these models. We look at performance in attribute-based tasks, requiring the identification of a particular combination of attribute and object (such as "red cube"), and in relational settings, where the spatial relation between two shapes (such as "cube behind sphere") must be identified. We find that in some conditions, CLIP is able to learn attribute-object labellings, and to generalize to unseen attribute-object combinations. However, we also see evidence that CLIP is not able to bind features together reliably. Moreover, CLIP is not able to reliably learn relations between objects, whereas some compositional models are able to learn these perfectly. Of the five models we developed, none were able to generalize to unseen relations.

Large-scale vision-language models such as CLIP have shown impressive performance on zero-shot image classification and image-to-text retrieval. However, such zero-shot performance of CLIP-based models does not realize in tasks that require a finer-grained correspondence between vision and language, such as Visual Question Answering (VQA). We investigate why this is the case, and report an interesting phenomenon of CLIP, which we call the Concept Association Bias (CAB), as a potential cause of the difficulty of applying CLIP to VQA and similar tasks. CAB is especially apparent when two concepts are present in the given image while a text prompt only contains a single concept. In such a case, we find that CLIP tends to treat input as a bag of concepts and attempts to fill in the other missing concept crossmodally, leading to an unexpected zero-shot prediction. For example, when asked for the color of a lemon in an image, CLIP predicts ``purple'' if the image contains a lemon and an eggplant. We demonstrate the Concept Association Bias of CLIP by showing that CLIP's zero-shot classification performance greatly suffers when there is a strong concept association between an object (e.g. lemon) and an attribute (e.g. its color). On the other hand, when the association between object and attribute is weak, we do not see this phenomenon. Furthermore, we show that CAB is significantly mitigated when we enable CLIP to learn deeper structure across image and text embeddings by adding an additional Transformer on top of CLIP and fine-tuning it on VQA. We find that across such fine-tuned variants of CLIP, the strength of CAB in a model predicts how well it performs on VQA.

我们周围的视觉世界可以被描述为结构化的对象和相关关系。只有在底层对象的描述及其相关关系的描述中，可以将房间的图像召唤。虽然在设计可能将各个物体组成的深度神经网络上进行了重大工作，但在构图对象之间的各个关系方面取得了更少的工作。主要困难是，虽然对象的放置是相互独立的，但它们的关系彼此纠缠并依赖。为了规避这个问题，现有的作品主要通过利用文本或图形的形式来通过利用整体编码器来构成关系。在这项工作中，我们建议将每个关系作为非正规化密度（基于能量的模型）表示，使我们能够以分解方式构成单独的关系。我们表明这种分解分解允许模型生成和编辑具有多组关系的场景更忠实地。我们进一步表明，分解使我们的模型能够有效地理解底层关系场景结构。项目页面：https://comushvisual relations.github.io/。

Recent visuolinguistic pre-trained models show promising progress on various end tasks such as image retrieval and video captioning. Yet, they fail miserably on the recently proposed Winoground dataset, which challenges models to match paired images and English captions, with items constructed to overlap lexically but differ in meaning (e.g., "there is a mug in some grass" vs. "there is some grass in a mug"). By annotating the dataset using new fine-grained tags, we show that solving the Winoground task requires not just compositional language understanding, but a host of other abilities like commonsense reasoning or locating small, out-of-focus objects in low-resolution images. In this paper, we identify the dataset's main challenges through a suite of experiments on related tasks (probing task, image retrieval task), data augmentation, and manual inspection of the dataset. Our analysis suggests that a main challenge in visuolinguistic models may lie in fusing visual and textual representations, rather than in compositional language understanding. We release our annotation and code at https://github.com/ajd12342/why-winoground-hard .

When building artificial intelligence systems that can reason and answer questions about visual data, we need diagnostic tests to analyze our progress and discover shortcomings. Existing benchmarks for visual question answering can help, but have strong biases that models can exploit to correctly answer questions without reasoning. They also conflate multiple sources of error, making it hard to pinpoint model weaknesses. We present a diagnostic dataset that tests a range of visual reasoning abilities. It contains minimal biases and has detailed annotations describing the kind of reasoning each question requires. We use this dataset to analyze a variety of modern visual reasoning systems, providing novel insights into their abilities and limitations.

已经开发了许多Visio语言（V + L）表示学习方法，但现有数据集不会评估它们在统一空间中代表视觉和语言概念的程度。灵感来自于奇妙的转移和精神语言学文献，我们提出了一个新的V + L型号的评价设置：零射频跨模型转移。现有的V + L基准也经常在整个数据集上报告全局精度分数，渲染难以确定模型失败并成功的具体推理任务。要解决此问题并启用对跨模型传输的评估，我们存在TRAVLR，包括四个V + L推理任务的合成数据集。每个示例对场景进行了双倍，使得在训练/测试期间可以丢弃无论是没有相关信息的丢失。 Travlr的培训和测试分布也沿任务相关维度约束，从而可以评估分配外概括。我们评估了四个最先进的V + L型号，发现它们在从同一模态的测试集上表现良好，但所有型号都无法转移交叉模态，并且成功有限，容纳一个模态的添加或删除。在与事先工作的对齐中，我们还发现这些模型需要大量数据来学习简单的空间关系。我们将Travlr释放为研究界的开放挑战。

我们提出Valse（视觉和语言结构化评估），这是一种新的基准，专为测试通用净化的视觉和语言（V＆L）模型而设计，用于对特定语言现象的视野 - 语言接地能力。Valse提供涵盖各种语言构建体的六种测试套件。解决这些需要模型在视觉模型中地对语言现象，允许比迄今为止更细粒度的评估。我们使用支持有效箔的构造的方法构建Valse，并通过评估五种广泛使用的V＆L模型的报告结果。我们的实验表明，目前的模型有很大的困难解决了大多数现象。因此，我们预计Valse就可以作为一种重要的基准，从语言角度来衡量预训过的V＆L模型的未来进展，补充规范任务为中心的V＆L评价。

我们介绍了CLEVR-MATH，这是一个多模式数学单词问题数据集，该数据集由涉及加法/减法的简单数学单词问题组成，部分地表示文本描述，部分地是由图像说明了场景。文本描述了图像中描述的场景上执行的动作。由于提出的问题可能与图像中的场景有关，而是针对采用动作之前或之后的场景状态，因此求解器设想或想象由于这些动作而导致的状态发生了变化。解决这些单词问题需要语言，视觉和数学推理的结合。我们将最新的神经和神经符号模型应用于CLEVR-MATH的视觉问题，并经验评估其表现。我们的结果表明，两种方法如何推广到操作链。我们讨论了两者在解决多模式单词问题解决的任务时的局限性。

Despite progress in perceptual tasks such as image classification, computers still perform poorly on cognitive tasks such as image description and question answering. Cognition is core to tasks that involve not just recognizing, but reasoning about our visual world. However, models used to tackle the rich content in images for cognitive tasks are still being trained using the same datasets designed for perceptual tasks. To achieve success at cognitive tasks, models need to understand the interactions and relationships between objects in

Visual question answering is fundamentally compositional in nature-a question like where is the dog? shares substructure with questions like what color is the dog? and where is the cat? This paper seeks to simultaneously exploit the representational capacity of deep networks and the compositional linguistic structure of questions. We describe a procedure for constructing and learning neural module networks, which compose collections of jointly-trained neural "modules" into deep networks for question answering. Our approach decomposes questions into their linguistic substructures, and uses these structures to dynamically instantiate modular networks (with reusable components for recognizing dogs, classifying colors, etc.). The resulting compound networks are jointly trained. We evaluate our approach on two challenging datasets for visual question answering, achieving state-of-the-art results on both the VQA natural image dataset and a new dataset of complex questions about abstract shapes.

We introduce a new dataset for joint reasoning about natural language and images, with a focus on semantic diversity, compositionality, and visual reasoning challenges. The data contains 107,292 examples of English sentences paired with web photographs. The task is to determine whether a natural language caption is true about a pair of photographs. We crowdsource the data using sets of visually rich images and a compare-and-contrast task to elicit linguistically diverse language. Qualitative analysis shows the data requires compositional joint reasoning, including about quantities, comparisons, and relations. Evaluation using state-of-the-art visual reasoning methods shows the data presents a strong challenge. * Contributed equally. † Work done as an undergraduate at Cornell University. 1 In parts of this paper, we use the term compositional differently than it is commonly used in linguistics to refer to reasoning that requires composition. This type of reasoning often manifests itself in highly compositional language.2 Appendix G contains license information for all photographs used in this paper. 3 The top example is True, while the bottom is False.

我们引入了构图软提示（CSP），这是一种参数有效的学习技术，可改善大规模预处理视觉模型（VLMS）的零摄像组成性。 VLM可以在其灵活的文本编码器中代表任意类作为自然语言提示，但在组成零击基准任务上的表现不佳。为了改善VLM，我们提出了一种新颖的软提示形式。我们将构成的属性和对象视为将类定义为词汇的可学习令牌，并在多个及时的构图上调整它们。在推断期间，我们在新组合中重新组装了学习的属性对象词汇。我们表明，CSP在基准数据集上的原始VLM的表现平均为AUC上的10.9个百分点。 CSP还胜过Coop，这是一种调谐前缀上下文的软提示方法，在AUC上平均要点5.8个百分点。我们执行其他实验，以表明CSP对仅属性分类，高阶属性 - 属性对象组成以及预验证属性和微调对象的组合进行了改进。

A fundamental characteristic common to both human vision and natural language is their compositional nature. Yet, despite the performance gains contributed by large vision and language pretraining, we find that - across 6 architectures trained with 4 algorithms on massive datasets - they exhibit little compositionality. To arrive at this conclusion, we introduce a new compositionality evaluation benchmark CREPE which measures two important aspects of compositionality identified by cognitive science literature: systematicity and productivity. To measure systematicity, CREPE consists of three test datasets. The three test sets are designed to test models trained on three of the popular training datasets: CC-12M, YFCC-15M, and LAION-400M. They contain 385K, 385K, and 373K image-text pairs and 237K, 210K, and 178K hard negative captions. To test productivity, CREPE contains 17K image-text pairs with nine different complexities plus 246K hard negative captions with atomic, swapping, and negation foils. The datasets are generated by repurposing the Visual Genome scene graphs and region descriptions and applying handcrafted templates and GPT-3. For systematicity, we find that model performance decreases consistently when novel compositions dominate the retrieval set, with Recall@1 dropping by up to 8%. For productivity, models' retrieval success decays as complexity increases, frequently nearing random chance at high complexity. These results hold regardless of model and training dataset size.

Relational reasoning is a central component of generally intelligent behavior, but has proven difficult for neural networks to learn. In this paper we describe how to use Relation Networks (RNs) as a simple plug-and-play module to solve problems that fundamentally hinge on relational reasoning. We tested RN-augmented networks on three tasks: visual question answering using a challenging dataset called CLEVR, on which we achieve state-of-the-art, super-human performance; text-based question answering using the bAbI suite of tasks; and complex reasoning about dynamic physical systems. Then, using a curated dataset called Sort-of-CLEVR we show that powerful convolutional networks do not have a general capacity to solve relational questions, but can gain this capacity when augmented with RNs. Our work shows how a deep learning architecture equipped with an RN module can implicitly discover and learn to reason about entities and their relations.

通用视觉（GPV）系统是旨在解决各种视觉任务的模型，而无需进行架构更改。如今，GPV主要从大型完全监督的数据集中学习技能和概念。通过获取数据以迅速学习每个技能的每个概念，将GPV扩展到数万个概念都变得令人望而却步。这项工作提出了一种有效且廉价的替代方法：从监督数据集中学习技能，从Web图像搜索中学习概念，并利用GPV的关键特征：跨技能传递视觉知识的能力。我们使用跨越10K+视觉概念的1M+图像的数据集来演示3个基准上的两个现有GPV（GPV-1和VL-T5）的Webly Supumented概念扩展：5个基于可可的数据集（80个主要概念），这是一个新的策划系列，这是一个新的策划系列。基于OpenImages和VisualGenome存储库（〜500个概念）以及Web衍生的数据集（10K+概念）的5个数据集。我们还提出了一种新的体系结构GPV-2，该架构支持各种任务 - 从分类和本地化等视觉任务到Qu Viewer+语言任务，例如QA和字幕，再到更多的利基市场，例如人类对象互动检测。 GPV-2从Web数据中受益匪浅，并且在这些基准测试中胜过GPV-1和VL-T5。我们的数据，代码和Web演示可在https://prior.allenai.org/projects/gpv2上获得。

Top-down visual attention mechanisms have been used extensively in image captioning and visual question answering (VQA) to enable deeper image understanding through fine-grained analysis and even multiple steps of reasoning. In this work, we propose a combined bottom-up and topdown attention mechanism that enables attention to be calculated at the level of objects and other salient image regions. This is the natural basis for attention to be considered. Within our approach, the bottom-up mechanism (based on Faster R-CNN) proposes image regions, each with an associated feature vector, while the top-down mechanism determines feature weightings. Applying this approach to image captioning, our results on the MSCOCO test server establish a new state-of-the-art for the task, achieving CIDEr / SPICE / BLEU-4 scores of 117.9, 21.5 and 36.9, respectively. Demonstrating the broad applicability of the method, applying the same approach to VQA we obtain first place in the 2017 VQA Challenge.

Artificial Intelligence (AI) and its applications have sparked extraordinary interest in recent years. This achievement can be ascribed in part to advances in AI subfields including Machine Learning (ML), Computer Vision (CV), and Natural Language Processing (NLP). Deep learning, a sub-field of machine learning that employs artificial neural network concepts, has enabled the most rapid growth in these domains. The integration of vision and language has sparked a lot of attention as a result of this. The tasks have been created in such a way that they properly exemplify the concepts of deep learning. In this review paper, we provide a thorough and an extensive review of the state of the arts approaches, key models design principles and discuss existing datasets, methods, their problem formulation and evaluation measures for VQA and Visual reasoning tasks to understand vision and language representation learning. We also present some potential future paths in this field of research, with the hope that our study may generate new ideas and novel approaches to handle existing difficulties and develop new applications.

十年自2010年以来，人工智能成功一直处于计算机科学和技术的最前沿，传染媒介空间模型已经巩固了人工智能最前沿的位置。与此同时，量子计算机已经变得更加强大，主要进步的公告经常在新闻中。这些区域的基础的数学技术比有时意识到更多的共同之处。传染媒介空间在20世纪30年代的量子力学的公理心脏上采取了位置，这一采用是从矢量空间的线性几何形状推导逻辑和概率的关键动机。粒子之间的量子相互作用是使用张量产品进行建模的，其也用于表达人工神经网络中的物体和操作。本文介绍了这些常见的数学区域中的一些，包括如何在人工智能（AI）中使用的示例，特别是在自动推理和自然语言处理（NLP）中。讨论的技术包括矢量空间，标量产品，子空间和含义，正交投影和否定，双向矩阵，密度矩阵，正算子和张量产品。应用领域包括信息检索，分类和含义，建模字传感和歧义，知识库的推断和语义构成。其中一些方法可能会在量子硬件上实现。该实施中的许多实际步骤都处于早期阶段，其中一些已经实现了。解释一些常见的数学工具可以帮助AI和量子计算中的研究人员进一步利用这些重叠，识别和沿途探索新方向。

Large-scale diffusion models have achieved state-of-the-art results on text-to-image synthesis (T2I) tasks. Despite their ability to generate high-quality yet creative images, we observe that attribution-binding and compositional capabilities are still considered major challenging issues, especially when involving multiple objects. In this work, we improve the compositional skills of T2I models, specifically more accurate attribute binding and better image compositions. To do this, we incorporate linguistic structures with the diffusion guidance process based on the controllable properties of manipulating cross-attention layers in diffusion-based T2I models. We observe that keys and values in cross-attention layers have strong semantic meanings associated with object layouts and content. Therefore, we can better preserve the compositional semantics in the generated image by manipulating the cross-attention representations based on linguistic insights. Built upon Stable Diffusion, a SOTA T2I model, our structured cross-attention design is efficient that requires no additional training samples. We achieve better compositional skills in qualitative and quantitative results, leading to a 5-8% advantage in head-to-head user comparison studies. Lastly, we conduct an in-depth analysis to reveal potential causes of incorrect image compositions and justify the properties of cross-attention layers in the generation process.

本文介绍了牧羊犬：一个简单但有效的模型，用于持续学习语言如何在视野中接地。考虑到预先训练的多模式嵌入模型，其中语言和图像在同一语义空间中投影（在这种情况下通过Openai剪辑），牧灵学习一个转换功能，在需要时调整语言嵌入式以适应新的语言使用。与传统的少量学习不同，该模型不仅仅是学习新的类和标签，还可以概括与类似的语言使用。我们验证了模型对两种不同的持续学习任务的性能，并表明它可以仅从几个例子中有效地学习和概括，几乎没有干扰模型的原始零射击性能。