视觉模型可以评估图像中的视觉上下文并生成描述性文本。尽管生成的文本可能是准确且句法正确的，但通常过于笼统。为了解决这个问题，最近的工作使用光学特征识别来补充视觉信息，并从图像中提取的文本进行补充。在这项工作中，我们认为，视觉模型可以受益于可以从图像中提取但不使用当前模型使用的其他信息。我们修改了以前的多模式框架，以接受来自任意数量的辅助分类器的相关信息。特别是，我们将重点放在人的名字作为附加令牌上，并创建一个新颖的图像捕获数据集，以促进用人名称的字幕。标题（PAC）中的数据集，政客和运动员包括背景下知名人士的字幕图像。通过使用此数据集对预处理的模型进行微调，我们演示了一个模型，该模型可以自然地将面部识别令牌纳入生成的文本中，通过培训有限的数据。对于PAC数据集，我们提供有关集合和基线基准分数的讨论。

The availability of large-scale image captioning and visual question answering datasets has contributed significantly to recent successes in vision-and-language pretraining. However, these datasets are often collected with overrestrictive requirements inherited from their original target tasks (e.g., image caption generation), which limit the resulting dataset scale and diversity. We take a step further in pushing the limits of vision-and-language pretraining data by relaxing the data collection pipeline used in Conceptual Captions 3M (CC3M) [70] and introduce the Conceptual 12M (CC12M), a dataset with 12 million image-text pairs specifically meant to be used for visionand-language pre-training. We perform an analysis of this dataset and benchmark its effectiveness against CC3M on multiple downstream tasks with an emphasis on long-tail visual recognition. Our results clearly illustrate the benefit of scaling up pre-training data for vision-and-language tasks, as indicated by the new state-of-the-art results on both the nocaps and Conceptual Captions benchmarks. 1

我们提出了一种用于场景文本视觉问题的新型多模式架构（STVQA），命名为布局感知变压器（LatR）。 STVQA的任务需要模型以推理不同的方式。因此，我们首先调查每种方式的影响，并揭示语言模块的重要性，尤其是在丰富布局信息时。考虑到这一点，我们提出了一种客观预培训计划，只需要文本和空间线索。我们表明，尽管域间隙差距，但仍然对扫描文件进行了对扫描文件的培训方案具有某些优点。扫描的文档易于采购，文本密集并具有各种布局，帮助模型通过捆绑语言和布局信息来学习各种空间线索（例如，下面等等）。与现有方法相比，我们的方法执行无词汇解码，如图所示，概括到超出培训词汇。我们进一步证明Latr改善了对OCR错误的鲁棒性，在STVQA失败的常见原因。另外，通过利用视觉变压器，我们消除了对外部物体检测器的需求。 Latr在多个数据集上赢得最先进的STVQA方法。特别是+ 7.6％的TextVQA，ST-VQA上的10.8％，+ 4.0％在OCR-VQA（所有绝对精度数字）。

Top-down visual attention mechanisms have been used extensively in image captioning and visual question answering (VQA) to enable deeper image understanding through fine-grained analysis and even multiple steps of reasoning. In this work, we propose a combined bottom-up and topdown attention mechanism that enables attention to be calculated at the level of objects and other salient image regions. This is the natural basis for attention to be considered. Within our approach, the bottom-up mechanism (based on Faster R-CNN) proposes image regions, each with an associated feature vector, while the top-down mechanism determines feature weightings. Applying this approach to image captioning, our results on the MSCOCO test server establish a new state-of-the-art for the task, achieving CIDEr / SPICE / BLEU-4 scores of 117.9, 21.5 and 36.9, respectively. Demonstrating the broad applicability of the method, applying the same approach to VQA we obtain first place in the 2017 VQA Challenge.

连接视觉和语言在生成智能中起着重要作用。因此，已经致力于图像标题的大型研究工作，即用句法和语义有意义的句子描述图像。从2015年开始，该任务通常通过由Visual Encoder组成的管道和文本生成的语言模型来解决任务。在这些年来，两种组件通过对象区域，属性，介绍多模态连接，完全关注方法和伯特早期融合策略的利用而显着发展。但是，无论令人印象深刻的结果，图像标题的研究还没有达到结论性答案。这项工作旨在提供图像标题方法的全面概述，从视觉编码和文本生成到培训策略，数据集和评估度量。在这方面，我们量化地比较了许多相关的最先进的方法来确定架构和培训策略中最有影响力的技术创新。此外，讨论了问题的许多变体及其开放挑战。这项工作的最终目标是作为理解现有文献的工具，并突出显示计算机视觉和自然语言处理的研究领域的未来方向可以找到最佳的协同作用。

图像标题是自动生成句子的任务，以最好的方式生成描述输入图像。最近用于自动生成图像标题的最成功的技术最近使用了细心的深度学习模型。设计了深入学习模型的设计方式有变化。在本调查中，我们为图像标题的细心深度学习模型提供了相关的文献述评。而不是对深度图像标题模型的所有先前工作进行全面审查，我们解释了用于深度学习模型中的图像标题任务的各种类型的注意机制。用于图像标题的最成功的深度学习模型遵循编码器解码器架构，尽管这些模型采用注意机制的方式存在差异。通过分析图像标题的不同细节深层模型的性能结果，我们的目标是在图像标题中找到深度模型中最成功的注意机制。柔软的关注，自下而上的关注和多主题是一种广泛应用于图像标题的最先进的深度学习模型的关注机构的类型。在当前时，最佳结果是从多针关注的变体实现的，以自下而上的关注。

We present ViLBERT (short for Vision-and-Language BERT), a model for learning task-agnostic joint representations of image content and natural language. We extend the popular BERT architecture to a multi-modal two-stream model, processing both visual and textual inputs in separate streams that interact through co-attentional transformer layers. We pretrain our model through two proxy tasks on the large, automatically collected Conceptual Captions dataset and then transfer it to multiple established vision-and-language tasks -visual question answering, visual commonsense reasoning, referring expressions, and caption-based image retrieval -by making only minor additions to the base architecture. We observe significant improvements across tasks compared to existing task-specific modelsachieving state-of-the-art on all four tasks. Our work represents a shift away from learning groundings between vision and language only as part of task training and towards treating visual grounding as a pretrainable and transferable capability.Preprint. Under review.

视觉问题应答（VQA）任务利用视觉图像和语言分析来回回答图像的文本问题。它是一个流行的研究课题，在过去十年中越来越多的现实应用。本文介绍了我们最近对AliceMind-MMU的研究（阿里巴巴的编码器 - 解码器来自Damo Academy - 多媒体理解的机器智能实验室），其比人类在VQA上获得相似甚至略微更好的结果。这是通过系统地改善VQA流水线来实现的，包括：（1）具有全面的视觉和文本特征表示的预培训; （2）与学习参加的有效跨模型互动; （3）一个新颖的知识挖掘框架，具有专门的专业专家模块，适用于复杂的VQA任务。处理不同类型的视觉问题，需要具有相应的专业知识在提高我们的VQA架构的表现方面发挥着重要作用，这取决于人力水平。进行了广泛的实验和分析，以证明新的研究工作的有效性。

文本VQA的开放式问题回答任务通常需要读取和推理图像中很少见或完全看不见的场景文本内容。我们通过提出广义使用外部知识来增强我们对场景文本的理解来解决问题的零射击性质。我们设计一个框架，使用标准的多模式变压器来提取，验证和理性，以了解视觉语言理解任务。通过经验证据和定性结果，我们证明了外部知识如何突出实例的线索，从而有助于应对培训数据偏见，提高答案实体类型的正确性并检测名为“实体”的多字。在类似上游OCR系统和培训数据的限制下，我们生成的结果与三个公开数据集的最新结果相当。

This paper presents a unified Vision-Language Pre-training (VLP) model. The model is unified in that (1) it can be finetuned for either vision-language generation (e.g., image captioning) or understanding (e.g., visual question answering) tasks, and (2) it uses a shared multi-layer transformer network for both encoding and decoding, which differs from many existing methods where the encoder and decoder are implemented using separate models. The unified VLP model is pre-trained on a large amount of image-text pairs using the unsupervised learning objectives of two tasks: bidirectional and sequence-to-sequence (seq2seq) masked vision-language prediction. The two tasks differ solely in what context the prediction conditions on. This is controlled by utilizing specific self-attention masks for the shared transformer network. To the best of our knowledge, VLP is the first reported model that achieves state-of-the-art results on both vision-language generation and understanding tasks, as disparate as image captioning and visual question answering, across three challenging benchmark datasets: COCO Captions, Flickr30k Captions, and VQA 2.0. The code and the pre-trained models are available at https://github.com/LuoweiZhou/VLP.

Image captioning models tend to describe images in an object-centric way, emphasising visible objects. But image descriptions can also abstract away from objects and describe the type of scene depicted. In this paper, we explore the potential of a state-of-the-art Vision and Language model, VinVL, to caption images at the scene level using (1) a novel dataset which pairs images with both object-centric and scene descriptions. Through (2) an in-depth analysis of the effect of the fine-tuning, we show (3) that a small amount of curated data suffices to generate scene descriptions without losing the capability to identify object-level concepts in the scene; the model acquires a more holistic view of the image compared to when object-centric descriptions are generated. We discuss the parallels between these results and insights from computational and cognitive science research on scene perception.

在这项工作中，我们提出了一种开放式摄制对象检测方法，该方法基于图像映射对，学会了检测新颖对象类别以及给定的一组已知类别。这是一种两阶段的训练方法，首先使用位置引导的图像捕获匹配技术以弱监督的方式学习新颖和已知类别的类标签，第二个使用已知的类注释专用于对象检测任务的模型。我们表明，一个简单的语言模型比检测新对象的大型上下文化语言模型更适合。此外，我们引入了一种一致性调查技术，以更好地利用图像捕获对信息。我们的方法比较与现有的开放式检测方法相比，同时具有数据效率。源代码可从https://github.com/lmb-freiburg/locov获得。

人类利用先验知识来描述图像，并能够使其解释适应特定的上下文信息，即使在上下文信息和图像不匹配时，也可以在发明合理的解释的范围内。在这项工作中，我们提出了通过整合上下文知识来字幕Wikipedia图像的新颖任务。具体而言，我们制作的模型共同推理了Wikipedia文章，Wikimedia图像及其相关描述以产生上下文化的标题。特别是，可以使用类似的Wikimedia图像来说明不同的文章，并且所产生的标题需要适应特定的上下文，因此使我们能够探索模型的限制以调整标题为不同的上下文信息。该领域中的一个特殊挑战性的任务是处理量不多的单词和命名实体。为了解决这个问题，我们提出了一个预训练目标，掩盖了命名实体建模（MNEM），并表明与基线模型相比，此借口任务可以改善。此外，我们验证了Wikipedia中使用MNEM目标预先训练的模型可以很好地推广到新闻字幕数据集。此外，我们根据字幕任务的难度定义了两种不同的测试拆分。我们提供有关每种方式的作用和重要性的见解，并突出我们模型的局限性。接受时，代码，模型和数据拆分可公开可用。

自动在自然语言中自动生成图像的描述称为图像字幕。这是一个积极的研究主题，位于人工智能，计算机视觉和自然语言处理中两个主要领域的交集。图像字幕是图像理解中的重要挑战之一，因为它不仅需要识别图像中的显着对象，还需要其属性及其相互作用的方式。然后，系统必须生成句法和语义上正确的标题，该标题描述了自然语言的图像内容。鉴于深度学习模型的重大进展及其有效编码大量图像并生成正确句子的能力，最近已经提出了几种基于神经的字幕方法，每种方法都试图达到更好的准确性和标题质量。本文介绍了一个基于编码器的图像字幕系统，其中编码器使用以RESNET-101作为骨干为骨干来提取图像中每个区域的空间和全局特征。此阶段之后是一个精致的模型，该模型使用注意力进行注意的机制来提取目标图像对象的视觉特征，然后确定其相互作用。解码器由一个基于注意力的复发模块和一个反思性注意模块组成，该模块会协作地将注意力应用于视觉和文本特征，以增强解码器对长期顺序依赖性建模的能力。在两个基准数据集（MSCOCO和FLICKR30K）上进行的广泛实验显示了提出的方法和生成的字幕的高质量。

在本文中，我们设计和训练生成的图像到文本变压器Git，以统一视觉语言任务，例如图像/视频字幕和问题答案。尽管生成模型在预训练和微调之间提供了一致的网络体系结构，但现有工作通常包含复杂的结构（Uni/多模式编码器/解码器），并取决于外部模块，例如对象检测器/标记器和光学角色识别（OCR） ）。在git中，我们将体系结构简化为一个图像编码器，而在单语言建模任务下将架构简化为一个文本解码器。我们还扩展了预训练数据和模型大小，以提高模型性能。没有铃铛和哨子，我们的git在12个具有挑战性的基准下建立了新的艺术状态。例如，我们的模型在文本贴图上首次超过了人类的表现（138.2 vs. 125.5在苹果酒中）。此外，我们提出了一种新的基于一代的图像分类和场景文本识别的方案，在标准基准上实现了不错的表现。

Many high-level skills that are required for computer vision tasks, such as parsing questions, comparing and contrasting semantics, and writing descriptions, are also required in other domains such as natural language processing. In this paper, we ask whether this makes it possible to learn those skills from text data and then use them to complete vision tasks without ever training on visual training data. Key to our approach is exploiting the joint embedding space of contrastively trained vision and language encoders. In practice, there can be systematic differences between embedding spaces for different modalities in contrastive models, and we analyze how these differences affect our approach and study a variety of strategies to mitigate this concern. We produce models using only text training data on three tasks: image captioning, visual entailment and visual question answering, and evaluate them on standard benchmarks using images. We find that this kind of transfer is possible and results in only a small drop in performance relative to models trained on images. We also showcase a variety of stylistic image captioning models that were trained using no image data and no human-curated language data, but instead text data from books, the web, or language models.

Artificial Intelligence (AI) and its applications have sparked extraordinary interest in recent years. This achievement can be ascribed in part to advances in AI subfields including Machine Learning (ML), Computer Vision (CV), and Natural Language Processing (NLP). Deep learning, a sub-field of machine learning that employs artificial neural network concepts, has enabled the most rapid growth in these domains. The integration of vision and language has sparked a lot of attention as a result of this. The tasks have been created in such a way that they properly exemplify the concepts of deep learning. In this review paper, we provide a thorough and an extensive review of the state of the arts approaches, key models design principles and discuss existing datasets, methods, their problem formulation and evaluation measures for VQA and Visual reasoning tasks to understand vision and language representation learning. We also present some potential future paths in this field of research, with the hope that our study may generate new ideas and novel approaches to handle existing difficulties and develop new applications.

Large-scale pre-training methods of learning cross-modal representations on image-text pairs are becoming popular for vision-language tasks. While existing methods simply concatenate image region features and text features as input to the model to be pre-trained and use selfattention to learn image-text semantic alignments in a brute force manner, in this paper, we propose a new learning method Oscar 1 , which uses object tags detected in images as anchor points to significantly ease the learning of alignments. Our method is motivated by the observation that the salient objects in an image can be accurately detected, and are often mentioned in the paired text. We pre-train an Oscar model on the public corpus of 6.5 million text-image pairs, and fine-tune it on downstream tasks, creating new state-of-the-arts on six well-established vision-language understanding and generation tasks. 2

图像标题是视觉语言理解的基本任务，其中模型将文本信息标题预测到给定输入图像。在本文中，我们提出了一种解决此任务的简单方法。我们使用剪辑编码作为标题的前缀，通过采用简单的映射网络，然后微调语言模型以生成图像标题。最近提出的剪辑模型包含丰富的语义特征，这些功能培训了文本背景，使其最适合视觉语言感知。我们的关键思想与预先接受训练的语言模型（GPT2）一起，我们获得了广泛了解视觉和文本数据。因此，我们的方法只需要相当快速的培训来产生称职的标题模型。如果没有额外的注释或预训练，它有效地为大规模和多样化的数据集生成有意义的标题。令人惊讶的是，即使仅在训练映射网络时，我们的方法也很好地运行良好，而剪辑和语言模型仍然冻结，则允许较轻的培训参数较轻的架构。通过定量评估，我们展示了我们的模型在充满挑战的概念标题和Nocaps数据集上实现了最先进的方法的可比结果，而它更简单，更快，更轻。我们的代码在https://github.com/rmokady/clip_prefix_caption中提供。

We present Answer-Me, a task-aware multi-task framework which unifies a variety of question answering tasks, such as, visual question answering, visual entailment, visual reasoning. In contrast to previous works using contrastive or generative captioning training, we propose a novel and simple recipe to pre-train a vision-language joint model, which is multi-task as well. The pre-training uses only noisy image captioning data, and is formulated to use the entire architecture end-to-end with both a strong language encoder and decoder. Our results show state-of-the-art performance, zero-shot generalization, robustness to forgetting, and competitive single-task results across a variety of question answering tasks. Our multi-task mixture training learns from tasks of various question intents and thus generalizes better, including on zero-shot vision-language tasks. We conduct experiments in the challenging multi-task and open-vocabulary settings and across a variety of datasets and tasks, such as VQA2.0, SNLI-VE, NLVR2, GQA. We observe that the proposed approach is able to generalize to unseen tasks and that more diverse mixtures lead to higher accuracy in both known and novel tasks.