超出经验的概括在开发实用AI系统方面具有重要作用。已经表明，目前的视觉问题回答（VQA）模型过度依赖于火车集中的语言 - 前沿（问题类型和最常见答案之间的虚假相关性），并在分配外构成性能差（ ood）测试集。这一行为限制了它们的概括性，并限制了他们在现实世界中使用的。本文表明，问题编码器中使用的序列模型架构在VQA模型的普遍性中具有重要作用。为了证明这一点，我们对各种现有的基于RNN和基于变压器的问号进行了详细的分析，以及我们提出了一种基于新的曲线图注意网络（GAT）的问题编码器。我们的研究发现，即使不使用任何额外的相对复杂的偏差方法，问题编码器中的序列模型也更好地选择了VQA模型的普遍性。

Artificial Intelligence (AI) and its applications have sparked extraordinary interest in recent years. This achievement can be ascribed in part to advances in AI subfields including Machine Learning (ML), Computer Vision (CV), and Natural Language Processing (NLP). Deep learning, a sub-field of machine learning that employs artificial neural network concepts, has enabled the most rapid growth in these domains. The integration of vision and language has sparked a lot of attention as a result of this. The tasks have been created in such a way that they properly exemplify the concepts of deep learning. In this review paper, we provide a thorough and an extensive review of the state of the arts approaches, key models design principles and discuss existing datasets, methods, their problem formulation and evaluation measures for VQA and Visual reasoning tasks to understand vision and language representation learning. We also present some potential future paths in this field of research, with the hope that our study may generate new ideas and novel approaches to handle existing difficulties and develop new applications.

Vision-and-language reasoning requires an understanding of visual concepts, language semantics, and, most importantly, the alignment and relationships between these two modalities. We thus propose the LXMERT (Learning Cross-Modality Encoder Representations from Transformers) framework to learn these vision-and-language connections. In LXMERT, we build a large-scale Transformer model that consists of three encoders: an object relationship encoder, a language encoder, and a cross-modality encoder. Next, to endow our model with the capability of connecting vision and language semantics, we pre-train the model with large amounts of image-and-sentence pairs, via five diverse representative pre-training tasks: masked language modeling, masked object prediction (feature regression and label classification), cross-modality matching, and image question answering. These tasks help in learning both intra-modality and cross-modality relationships. After fine-tuning from our pretrained parameters, our model achieves the state-of-the-art results on two visual question answering datasets (i.e., VQA and GQA). We also show the generalizability of our pretrained cross-modality model by adapting it to a challenging visual-reasoning task, NLVR 2 , and improve the previous best result by 22% absolute (54% to 76%). Lastly, we demonstrate detailed ablation studies to prove that both our novel model components and pretraining strategies significantly contribute to our strong results; and also present several attention visualizations for the different encoders. 1

我们启动了对MLP架构进行了视觉和语言（VL）融合的第一个实证研究。通过对5 VL任务和5个强大的VQA基准测试的广泛实验，我们发现：（i）没有预先训练，使用MLP进行多模式融合，与变压器相比具有明显的性能差距; （ii）但是，VL预培训可以帮助关闭性能差距; （iii）代替重大的多主头注意力，将微小的单臂注意MLPS增加足以实现对变压器的可比性。此外，我们还发现，当在更难的鲁棒VQA基准测试时，MLP和变压器之间的性能差距不会扩大，建议使用MLP融合可以大致呈现与使用变压器相似的程度。这些结果提示MLP可以有效地学会对准从较低级别的编码器中提取的视觉和文本功能，而不依赖于自我关注。基于此，我们提出了一个更大胆的问题：我们可以为VL建模提供全部MLP架构，其中VL融合和视觉编码器都用MLPS替换吗？我们的结果表明，与最先进的全功能VL模型相比，全部MLP VL模型是当它们都获得预先培训的时型vl模型。然而，预先培训ALL-MLP可能令人惊讶地实现比没有预先训练的完整变压器模型更好的平均分数。这表明VL建模的MLP样架构的大规模预培训的潜力，并激发了未来的研究方向，简化了较少的归纳设计偏差的良好的VL建模。我们的代码可公开提供：https://github.com/easonnie/mlp-vil

Vision-Language预培训是一个新兴和快速发展的研究主题，将多模态知识从丰富的资源预训练任务转移到有限资源下游任务。与主要学习单个通用编码器的现有作品不同，我们提出了一种可训练的通用编码器 - 解码器网络（UNI-EDEN），以促进视觉语言感知（例如，视觉问题应答）和生成（例如，图像标题）。 UNI-EDEN是一种基于双流变换器的结构，由三个模块组成：对象和句子编码器，其单独了解每个模态的表示，以及通过模态交互能够实现多模态推理和句子的句子解码器。考虑到每个图像的语言表示可以跨越该层次结构的不同粒度，包括从简单到全面，个人标签，短语和自然句子，我们通过多粒愿景语言代理任务预先列车UNI-EDEN：屏蔽对象分类（MOC），蒙版区域短语生成（MRPG），图像句匹配（ISM）和屏蔽句生成（MSG）。以这种方式，UNI-EDEN赋予了多模态表示提取和语言建模的功率。广泛的实验证明了通过微调到四个视觉语言感知和发电下游任务来展示Uni-Eden的概括性。

用于视觉语言表示学习的变压器已经引起了很多兴趣，并在视觉问题答案（VQA）和接地方面表现出了巨大的表现。但是，大多数显示出良好性能的系统在培训过程中仍然依赖于预训练的对象探测器，这将其适用性限制在可用于这些检测器的对象类中。为了减轻这种限制，以下论文着重于在变形金刚中的视觉问题答案的背景下进行弱监督的基础问题。该方法通过将每个视觉令牌分组在视觉编码器中，并使用语言自我发项层作为文本引导选择模块来利用胶囊，以在将它们转发到下一层之前掩盖它们。我们评估了针对挑战的GQA以及VQA帽数据集的VQA接地的方法。我们的实验表明：在从标准变压器体系结构中删除蒙版对象的信息的同时，胶囊的集成显着提高了此类系统的接地能力，并提供了与其他新的最先进的结果。在现场接近。

We study joint learning of Convolutional Neural Network (CNN) and Transformer for vision-language pre-training (VLPT) which aims to learn cross-modal alignments from millions of image-text pairs. State-of-the-art approaches extract salient image regions and align regions with words step-by-step. As region-based visual features usually represent parts of an image, it is challenging for existing visionlanguage models to fully understand the semantics from paired natural languages. In this paper, we propose SOHO to "See Out of tHe bOx" that takes a whole image as input, and learns vision-language representation in an endto-end manner. SOHO does not require bounding box annotations which enables inference 10 times faster than regionbased approaches. In particular, SOHO learns to extract comprehensive yet compact image features through a visual dictionary (VD) that facilitates cross-modal understanding. VD is designed to represent consistent visual abstractions of similar semantics. It is updated on-the-fly and utilized in our proposed pre-training task Masked Visual Modeling (MVM). We conduct experiments on four well-established vision-language tasks by following standard VLPT settings. In particular, SOHO achieves absolute gains of 2.0% R@1 score on MSCOCO text retrieval 5k test split, 1.5% accuracy on NLVR 2 test-P split, 6.7% accuracy on SNLI-VE test split, respectively.

Astounding results from Transformer models on natural language tasks have intrigued the vision community to study their application to computer vision problems. Among their salient benefits, Transformers enable modeling long dependencies between input sequence elements and support parallel processing of sequence as compared to recurrent networks e.g., Long short-term memory (LSTM). Different from convolutional networks, Transformers require minimal inductive biases for their design and are naturally suited as set-functions. Furthermore, the straightforward design of Transformers allows processing multiple modalities (e.g., images, videos, text and speech) using similar processing blocks and demonstrates excellent scalability to very large capacity networks and huge datasets. These strengths have led to exciting progress on a number of vision tasks using Transformer networks. This survey aims to provide a comprehensive overview of the Transformer models in the computer vision discipline. We start with an introduction to fundamental concepts behind the success of Transformers i.e., self-attention, large-scale pre-training, and bidirectional feature encoding. We then cover extensive applications of transformers in vision including popular recognition tasks (e.g., image classification, object detection, action recognition, and segmentation), generative modeling, multi-modal tasks (e.g., visual-question answering, visual reasoning, and visual grounding), video processing (e.g., activity recognition, video forecasting), low-level vision (e.g., image super-resolution, image enhancement, and colorization) and 3D analysis (e.g., point cloud classification and segmentation). We compare the respective advantages and limitations of popular techniques both in terms of architectural design and their experimental value. Finally, we provide an analysis on open research directions and possible future works. We hope this effort will ignite further interest in the community to solve current challenges towards the application of transformer models in computer vision.

最近，许多研究表明，通过使用多模式的训练预训练目标扩展BERT体系结构，在各种视觉语言多模式任务（例如图像字幕和视觉问题）上进行了令人印象深刻的表现。在这项工作中，我们探讨了医学领域中的一系列多模式表示任务，专门使用放射学图像和非结构化报告。我们提出了医学视觉语言学习者（MEDVILL），该语言学习者采用基于BERT的建筑与一种新型的多模式注意掩盖方案相结合，以最大程度地提高概括性能，以实现视力语言理解任务（诊断分类，医疗图像报告，医学视觉，医疗视觉效果问答）和视觉生成任务（放射学报告生成）。通过统计和严格评估四个下游任务的拟议模型，该模型具有三个X光摄影图像报告数据集（Mimic-CXR，Open-I和VQA-RAD），我们从经验上凭经验证明了MEDVILL的卓越下游任务，包括各种基准，包括任务 - 特定体系结构。源代码可公开可用：https：//github.com/supersupermoon/medvill

视觉问题应答（VQA）任务利用视觉图像和语言分析来回回答图像的文本问题。它是一个流行的研究课题，在过去十年中越来越多的现实应用。本文介绍了我们最近对AliceMind-MMU的研究（阿里巴巴的编码器 - 解码器来自Damo Academy - 多媒体理解的机器智能实验室），其比人类在VQA上获得相似甚至略微更好的结果。这是通过系统地改善VQA流水线来实现的，包括：（1）具有全面的视觉和文本特征表示的预培训; （2）与学习参加的有效跨模型互动; （3）一个新颖的知识挖掘框架，具有专门的专业专家模块，适用于复杂的VQA任务。处理不同类型的视觉问题，需要具有相应的专业知识在提高我们的VQA架构的表现方面发挥着重要作用，这取决于人力水平。进行了广泛的实验和分析，以证明新的研究工作的有效性。

A number of recent works have proposed attention models for Visual Question Answering (VQA) that generate spatial maps highlighting image regions relevant to answering the question. In this paper, we argue that in addition to modeling "where to look" or visual attention, it is equally important to model "what words to listen to" or question attention. We present a novel co-attention model for VQA that jointly reasons about image and question attention. In addition, our model reasons about the question (and consequently the image via the co-attention mechanism) in a hierarchical fashion via a novel 1-dimensional convolution neural networks (CNN). Our model improves the state-of-the-art on the VQA dataset from 60.3% to 60.5%, and from 61.6% to 63.3% on the COCO-QA dataset. By using ResNet, the performance is further improved to 62.1% for VQA and 65.4% for COCO-QA. 1 .

随着图像文本对的大量数据以及视觉和语言（V＆L）任务的多样性，学者在该研究领域引入了大量的深度学习模型。此外，近年来，转移学习还显示出在计算机愿景中的巨大成功，例如图像分类，对象检测等以及在自然语言处理中以进行问答，机器翻译等的自然语言处理。继承转移学习的精神， V＆L的研究工作已经在大规模数据集上设计了多种预训练技术，以增强下游任务的性能。本文的目的是提供当代V＆L预审前模型的全面修订。特别是，我们对预处理的方法进行了分类和描述，以及最先进的视觉和语言预训练模型的摘要。此外，还提供了培训数据集和下游任务的列表，以进一步提高V＆L预处理的观点。最后，我们决定采取进一步的一步，讨论众多未来研究的方向。

This paper presents a unified Vision-Language Pre-training (VLP) model. The model is unified in that (1) it can be finetuned for either vision-language generation (e.g., image captioning) or understanding (e.g., visual question answering) tasks, and (2) it uses a shared multi-layer transformer network for both encoding and decoding, which differs from many existing methods where the encoder and decoder are implemented using separate models. The unified VLP model is pre-trained on a large amount of image-text pairs using the unsupervised learning objectives of two tasks: bidirectional and sequence-to-sequence (seq2seq) masked vision-language prediction. The two tasks differ solely in what context the prediction conditions on. This is controlled by utilizing specific self-attention masks for the shared transformer network. To the best of our knowledge, VLP is the first reported model that achieves state-of-the-art results on both vision-language generation and understanding tasks, as disparate as image captioning and visual question answering, across three challenging benchmark datasets: COCO Captions, Flickr30k Captions, and VQA 2.0. The code and the pre-trained models are available at https://github.com/LuoweiZhou/VLP.

根据图像回答语义复杂的问题是在视觉问题应答（VQA）任务中的具有挑战性。虽然图像可以通过深度学习来良好代表，但是始终简单地嵌入问题，并且不能很好地表明它的含义。此外，视觉和文本特征具有不同模式的间隙，很难对齐和利用跨模块信息。在本文中，我们专注于这两个问题，并提出了一种匹配关注（GMA）网络的图表。首先，它不仅为图像构建图形，而且在句法和嵌入信息方面构建了该问题的图表。接下来，我们通过双级图形编码器探讨了模特内的关系，然后呈现双边跨模型图匹配注意力以推断图像与问题之间的关系。然后将更新的跨模式特征发送到答案预测模块中以进行最终答案预测。实验表明，我们的网络在GQA数据集和VQA 2.0数据集上达到了最先进的性能。消融研究验证了GMA网络中每个模块的有效性。

变压器架构已经带来了计算语言领域的根本变化，这已经由经常性神经网络主导多年。它的成功还意味着具有语言和愿景的跨模型任务的大幅度变化，许多研究人员已经解决了这个问题。在本文中，我们审查了该领域中的一些最关键的里程碑，以及变压器架构如何纳入Visuol语言跨模型任务的整体趋势。此外，我们讨论了当前的局限性，并推测了我们发现迫在眉睫的一些前景。

3D场景理解是一个相对新兴的研究领域。在本文中，我们介绍了3D现实世界场景（VQA-3D）中的视觉问题应答任务，旨在给出3D场景的所有可能的问题。为了解决这个问题，提出了第一个VQA-3D数据集，即CLEVR3D，其中包含在1,129个现实世界场景中的60k个问题。具体而言，我们开发一个问题发动机利用3D场景图结构来生成不同的推理问题，涵盖物体属性的问题（即，大小，颜色和材料）及其空间关系。建立在此数据集之上，我们进一步设计了第一个VQA-3D基线模型TransVQA3D。 TransVQA3D型号采用精心设计的变压器架构，实现优越的VQA-3D性能，与纯语言基线和先前的3D推理方法直接应用于3D场景。实验结果验证了VQA-3D作为辅助任务可以提高3D场景理解的性能，包括节点明智分类和全图识别的场景图分析。

随着变压器的发展，近年来预先训练的模型已经以突破性的步伐发展。他们在自然语言处理（NLP）和计算机视觉（CV）中主导了主流技术。如何将预训练适应视觉和语言（V-L）学习和改善下游任务绩效成为多模式学习的重点。在本文中，我们回顾了视力语言预训练模型（VL-PTMS）的最新进展。作为核心内容，我们首先简要介绍了几种方法，将原始图像和文本编码为单模式嵌入在预训练之前。然后，我们在建模文本和图像表示之间的相互作用时深入研究VL-PTM的主流体系结构。我们进一步提出了广泛使用的预训练任务，然后我们介绍了一些常见的下游任务。我们终于结束了本文，并提出了一些有前途的研究方向。我们的调查旨在为研究人员提供合成和指向相关研究的指针。

Vision-and语言（VL）预培训已被证明对各种VL下游任务非常有效。虽然最近的工作表明，基于完全变换器的VL模型可以比以前的基于区域特征的方法更有效，但它们在下游任务上的性能通常显着降低。在本文中，我们呈现仪表〜（\ textbf {m} ultimodal \ textbf {e} nd-to-text \ textbf {t} ransform \ textbf {er}），我们通过它系统地调查如何设计和预先列车基于完全变换器的VL模型以端到端的方式。具体而言，我们将模型设计沿多个尺寸分析：视觉编码器（例如，剪辑 - vit，Swin变压器），文本编码器（例如，Roberta，Deberta），多模式融合（例如，合并注意力与共同关注），架构设计（例如，仅编码器与编码器 - 解码器）和预训练目标（例如，屏蔽图像建模）。我们对广泛的VL任务进行全面实验，并提供有关如何在保持快速推理速度的同时培训表演VL变压器的见解。值得注意的是，仪表〜使用仅使用4M图像进行预培训的VQAV2 TEST-STD设置的精度为77.64 \％，超过最先进的区域特征的VINVL模型+1.04 \％，以及优于以前最好的完全变换器的ALBEF模型+1.6 \％。

Large-scale pre-training methods of learning cross-modal representations on image-text pairs are becoming popular for vision-language tasks. While existing methods simply concatenate image region features and text features as input to the model to be pre-trained and use selfattention to learn image-text semantic alignments in a brute force manner, in this paper, we propose a new learning method Oscar 1 , which uses object tags detected in images as anchor points to significantly ease the learning of alignments. Our method is motivated by the observation that the salient objects in an image can be accurately detected, and are often mentioned in the paired text. We pre-train an Oscar model on the public corpus of 6.5 million text-image pairs, and fine-tune it on downstream tasks, creating new state-of-the-arts on six well-established vision-language understanding and generation tasks. 2

We present ViLBERT (short for Vision-and-Language BERT), a model for learning task-agnostic joint representations of image content and natural language. We extend the popular BERT architecture to a multi-modal two-stream model, processing both visual and textual inputs in separate streams that interact through co-attentional transformer layers. We pretrain our model through two proxy tasks on the large, automatically collected Conceptual Captions dataset and then transfer it to multiple established vision-and-language tasks -visual question answering, visual commonsense reasoning, referring expressions, and caption-based image retrieval -by making only minor additions to the base architecture. We observe significant improvements across tasks compared to existing task-specific modelsachieving state-of-the-art on all four tasks. Our work represents a shift away from learning groundings between vision and language only as part of task training and towards treating visual grounding as a pretrainable and transferable capability.Preprint. Under review.