尽管语言任务自然而然地以单个，统一的建模框架（即生成代币序列）表示，但在计算机视觉中并非如此。结果，对于不同的视力任务，不同的架构和损失功能的扩散。在这项工作中，我们表明，如果根据共享像素到序列界面进行配制，也可以统一一组“核心”计算机视觉任务。我们专注于四个任务，即对象检测，实例分割，关键点检测和图像字幕，所有这些任务都具有各种类型的输出，例如边界框或密集的掩码。尽管如此，通过将每个任务的输出作为具有统一界面的离散令牌的顺序，我们表明可以在所有这些任务上训练具有单个模型体系结构和损失功能的神经网络，而没有特定于任务的自定义。为了解决特定的任务，我们使用一个简短的提示作为任务说明，序列输出适应提示，以便它可以产生特定于任务的输出。我们表明，与成熟的特定任务模型相比，这种模型可以实现竞争性能。

我们提出了Unified-io，该模型执行了跨越经典计算机视觉任务的各种AI任务，包括姿势估计，对象检测，深度估计和图像生成，视觉和语言任务，例如区域字幕和引用表达理解，并引用表达理解，进行自然语言处理任务，例如回答和释义。由于与每个任务有关的异质输入和输出，包括RGB图像，每个像素映射，二进制掩码，边界框和语言，开发一个统一模型引起了独特的挑战。我们通过将每个受支持的输入和输出均匀地均匀地统一到一系列离散的词汇令牌来实现这一统一。在所有任务中，这种共同的表示使我们能够在视觉和语言字段中的80多个不同数据集上培训单个基于变压器的体系结构。 Unified-io是第一个能够在砂砾基准上执行所有7个任务的模型，并在NYUV2-DEPTH，Imagenet，VQA2.0，OK-VQA，SWIG，SWIG，VIZWIZ，BOOLQ，BOOLQ和SCITAIL，带有NYUV2-DEPTH，Imagenet，VQA2.0，诸如NYUV2-DEPTH，ImageNet，vqa2.0等16个不同的基准中产生强大的结果。没有任务或基准特定的微调。 unified-io的演示可在https://unified-io.allenai.org上获得。

In-context learning, as a new paradigm in NLP, allows the model to rapidly adapt to various tasks with only a handful of prompts and examples. But in computer vision, the difficulties for in-context learning lie in that tasks vary significantly in the output representations, thus it is unclear how to define the general-purpose task prompts that the vision model can understand and transfer to out-of-domain tasks. In this work, we present Painter, a generalist model which addresses these obstacles with an "image"-centric solution, that is, to redefine the output of core vision tasks as images, and specify task prompts as also images. With this idea, our training process is extremely simple, which performs standard masked image modeling on the stitch of input and output image pairs. This makes the model capable of performing tasks conditioned on visible image patches. Thus, during inference, we can adopt a pair of input and output images from the same task as the input condition, to indicate which task to perform. Without bells and whistles, our generalist Painter can achieve competitive performance compared to well-established task-specific models, on seven representative vision tasks ranging from high-level visual understanding to low-level image processing. Painter significantly outperforms recent generalist models on several challenging tasks. Surprisingly, our model shows capabilities of completing out-of-domain tasks, which do not exist in the training data, such as open-category keypoint detection and object segmentation, validating the powerful task transferability of in-context learning.

我们提出了Findit，这是一个简单而多功能的框架，统一了各种视觉接地和本地化任务，包括引用表达理解，基于文本的本地化和对象检测。我们体系结构的关键是一个有效的多尺度融合模块，该模块统一了整个任务中不同的本地化要求。此外，我们发现标准对象检测器在统一这些任务的无需特定任务设计，损失或预计算检测方面非常有效。我们的端到端可训练框架灵活，准确地响应了零，一个或多个对象的广泛的参考表达，本地化或检测查询。在这些任务上进行了共同培训，发现在引用表达和基于文本的本地化方面，胜过最高的艺术状态，并在对象检测中表现出竞争性的性能。最后，与强大的单任务基准相比，Findit可以更好地推广到分布数据和新型类别。所有这些都是通过一个单一的，统一和有效的模型来完成的。代码将发布。

我们提出了GLIPV2，这是一个接地的VL理解模型，该模型既服务于本地化任务（例如，对象检测，实例分割）和视觉语言（VL）理解任务（例如VQA，图像字幕）。 GLIPV2优雅地将本地化预训练和视觉语言预训练（VLP）具有三个预训练任务：短语接地作为对检测任务的VL重新重新制定，区域词对比度学习作为新型的区域词对比度对比度对比学习任务，以及蒙面的语言建模。这种统一不仅简化了先前的多阶段VLP程序，而且还可以在本地化和理解任务之间实现相互利益。实验结果表明，在各种本地化和理解任务上，单个GLIPV2模型（所有模型权重）在SOTA性能附近实现。该模型还显示了（1）在开放式摄制对象检测任务上进行的强零射击和很少的自适应性能，以及（2）VL理解任务上的卓越接地能力。代码将在https://github.com/microsoft/glip上发布。

在本文中，我们提出了Unicorn，一种vision-language（vl）模型，使文本生成和边界框预测到单个架构中。具体而言，我们将每个框量化为四个离散框令牌，并将其序列化为序列，可以与文本令牌集成。我们将所有VL问题作为一代任务，其中目标序列由集成文本和框令牌组成。然后，我们训练变压器编码器解码器以以自动回归方式预测目标。通过如此统一的框架和输入输出格式，Unicorn在7 VL基准测试中实现了对现有技术的可比性的性能，涵盖了视觉接地，接地字幕，视觉问题应答和图像标题任务。当用多任务FINETUNING培训时，UNICORN可以通过单一的参数方法接近不同的VL任务，从而跨越下游任务边界。我们展示了具有单一模型不仅可以节省参数，而且还可以在某些任务上提高模型性能。最后，Unicorn显示了概括到诸如ImageNet对象本地化的新任务的能力。

将简单的体系结构与大规模预训练相结合已导致图像分类的大量改进。对于对象检测，预训练和缩放方法的确定性不佳，尤其是在长尾和开放式摄影的环境中，训练数据相对较少。在本文中，我们提出了一个强大的配方，用于将图像文本模型转移到开放式对象检测中。我们使用具有最小修改，对比度文本预训练和端到端检测微调的标准视觉变压器体系结构。我们对该设置的缩放属性的分析表明，增加图像级预训练和模型大小在下游检测任务上产生一致的改进。我们提供适应性策略和正规化，以实现零击文本条件和单次图像条件对象检测的非常强劲的性能。代码和型号可在GitHub上找到。

Astounding results from Transformer models on natural language tasks have intrigued the vision community to study their application to computer vision problems. Among their salient benefits, Transformers enable modeling long dependencies between input sequence elements and support parallel processing of sequence as compared to recurrent networks e.g., Long short-term memory (LSTM). Different from convolutional networks, Transformers require minimal inductive biases for their design and are naturally suited as set-functions. Furthermore, the straightforward design of Transformers allows processing multiple modalities (e.g., images, videos, text and speech) using similar processing blocks and demonstrates excellent scalability to very large capacity networks and huge datasets. These strengths have led to exciting progress on a number of vision tasks using Transformer networks. This survey aims to provide a comprehensive overview of the Transformer models in the computer vision discipline. We start with an introduction to fundamental concepts behind the success of Transformers i.e., self-attention, large-scale pre-training, and bidirectional feature encoding. We then cover extensive applications of transformers in vision including popular recognition tasks (e.g., image classification, object detection, action recognition, and segmentation), generative modeling, multi-modal tasks (e.g., visual-question answering, visual reasoning, and visual grounding), video processing (e.g., activity recognition, video forecasting), low-level vision (e.g., image super-resolution, image enhancement, and colorization) and 3D analysis (e.g., point cloud classification and segmentation). We compare the respective advantages and limitations of popular techniques both in terms of architectural design and their experimental value. Finally, we provide an analysis on open research directions and possible future works. We hope this effort will ignite further interest in the community to solve current challenges towards the application of transformer models in computer vision.

Open-vocabulary object detection, which is concerned with the problem of detecting novel objects guided by natural language, has gained increasing attention from the community. Ideally, we would like to extend an open-vocabulary detector such that it can produce bounding box predictions based on user inputs in form of either natural language or exemplar image. This offers great flexibility and user experience for human-computer interaction. To this end, we propose a novel open-vocabulary detector based on DETR -- hence the name OV-DETR -- which, once trained, can detect any object given its class name or an exemplar image. The biggest challenge of turning DETR into an open-vocabulary detector is that it is impossible to calculate the classification cost matrix of novel classes without access to their labeled images. To overcome this challenge, we formulate the learning objective as a binary matching one between input queries (class name or exemplar image) and the corresponding objects, which learns useful correspondence to generalize to unseen queries during testing. For training, we choose to condition the Transformer decoder on the input embeddings obtained from a pre-trained vision-language model like CLIP, in order to enable matching for both text and image queries. With extensive experiments on LVIS and COCO datasets, we demonstrate that our OV-DETR -- the first end-to-end Transformer-based open-vocabulary detector -- achieves non-trivial improvements over current state of the arts.

视觉任务的输出格式和相关内容差异很大，因此很难以相同的结构处理它们。一个主要障碍在于对象级别的视觉任务中的高维输出。在本文中，我们提出了一个以对象为中心的视觉框架OBJ2Seq。 OBJ2Seq将对象作为基本单元，并将大多数对象级的视觉任务视为对象的序列生成问题。因此，这些视觉任务可以分为两个步骤。首先识别给定类别的对象，然后为每个对象生成一个序列。输出序列的定义对于不同的任务有所不同，并且通过将这些序列与地面真相目标匹配来监督模型。 OBJ2SEQ能够灵活地确定输入类别以满足自定义要求，并可以轻松扩展到不同的视觉任务。在对MS Coco进行实验时，OBJ2SEQ在对象检测时可获得45.7％的AP，多标签分类的89.0％AP和人类姿势估计的65.0％AP。这些结果证明了其通常应用于不同视觉任务的潜力。代码已在以下网址提供：https：//github.com/casia-iva-lab/obj2seq。

We present X-Decoder, a generalized decoding model that can predict pixel-level segmentation and language tokens seamlessly. X-Decodert takes as input two types of queries: (i) generic non-semantic queries and (ii) semantic queries induced from text inputs, to decode different pixel-level and token-level outputs in the same semantic space. With such a novel design, X-Decoder is the first work that provides a unified way to support all types of image segmentation and a variety of vision-language (VL) tasks. Further, our design enables seamless interactions across tasks at different granularities and brings mutual benefits by learning a common and rich pixel-level visual-semantic understanding space, without any pseudo-labeling. After pretraining on a mixed set of a limited amount of segmentation data and millions of image-text pairs, X-Decoder exhibits strong transferability to a wide range of downstream tasks in both zero-shot and finetuning settings. Notably, it achieves (1) state-of-the-art results on open-vocabulary segmentation and referring segmentation on eight datasets; (2) better or competitive finetuned performance to other generalist and specialist models on segmentation and VL tasks; and (3) flexibility for efficient finetuning and novel task composition (e.g., referring captioning and image editing). Code, demo, video, and visualization are available at https://x-decoder-vl.github.io.

This paper presents a Generative RegIon-to-Text transformer, GRiT, for object understanding. The spirit of GRiT is to formulate object understanding as <region, text> pairs, where region locates objects and text describes objects. For example, the text in object detection denotes class names while that in dense captioning refers to descriptive sentences. Specifically, GRiT consists of a visual encoder to extract image features, a foreground object extractor to localize objects, and a text decoder to generate open-set object descriptions. With the same model architecture, GRiT can understand objects via not only simple nouns, but also rich descriptive sentences including object attributes or actions. Experimentally, we apply GRiT to object detection and dense captioning tasks. GRiT achieves 60.4 AP on COCO 2017 test-dev for object detection and 15.5 mAP on Visual Genome for dense captioning. Code is available at https://github.com/JialianW/GRiT

Performing 3D dense captioning and visual grounding requires a common and shared understanding of the underlying multimodal relationships. However, despite some previous attempts on connecting these two related tasks with highly task-specific neural modules, it remains understudied how to explicitly depict their shared nature to learn them simultaneously. In this work, we propose UniT3D, a simple yet effective fully unified transformer-based architecture for jointly solving 3D visual grounding and dense captioning. UniT3D enables learning a strong multimodal representation across the two tasks through a supervised joint pre-training scheme with bidirectional and seq-to-seq objectives. With a generic architecture design, UniT3D allows expanding the pre-training scope to more various training sources such as the synthesized data from 2D prior knowledge to benefit 3D vision-language tasks. Extensive experiments and analysis demonstrate that UniT3D obtains significant gains for 3D dense captioning and visual grounding.

通用视觉（GPV）系统是旨在解决各种视觉任务的模型，而无需进行架构更改。如今，GPV主要从大型完全监督的数据集中学习技能和概念。通过获取数据以迅速学习每个技能的每个概念，将GPV扩展到数万个概念都变得令人望而却步。这项工作提出了一种有效且廉价的替代方法：从监督数据集中学习技能，从Web图像搜索中学习概念，并利用GPV的关键特征：跨技能传递视觉知识的能力。我们使用跨越10K+视觉概念的1M+图像的数据集来演示3个基准上的两个现有GPV（GPV-1和VL-T5）的Webly Supumented概念扩展：5个基于可可的数据集（80个主要概念），这是一个新的策划系列，这是一个新的策划系列。基于OpenImages和VisualGenome存储库（〜500个概念）以及Web衍生的数据集（10K+概念）的5个数据集。我们还提出了一种新的体系结构GPV-2，该架构支持各种任务 - 从分类和本地化等视觉任务到Qu Viewer+语言任务，例如QA和字幕，再到更多的利基市场，例如人类对象互动检测。 GPV-2从Web数据中受益匪浅，并且在这些基准测试中胜过GPV-1和VL-T5。我们的数据，代码和Web演示可在https://prior.allenai.org/projects/gpv2上获得。

什么构成一个物体？这是计算机愿景中的长期问题。为了实现这一目标，已经开发了许多基于学习的基于学习的方法来得分对象。但是，它们通常不会划过新域和未经看不见的对象。在本文中，我们倡导现有方法缺乏由人类可理解的语义管理的自上而下的监督信号。为了弥合这一差距，我们探索了已经用对齐的图像文本对培训的多模态视觉变压器（MVIT）。我们对各个域和新型对象的广泛实验显示了MVITS的最先进的性能，以使图像中的通用对象本地化。基于这些发现，我们使用多尺度特征处理和可变形的自我关注来开发一种高效且灵活的MVIT架构，可以自适应地生成给定特定语言查询的提议。我们展示了MVIT提案在各种应用中的重要性，包括开放世界对象检测，突出和伪装对象检测，监督和自我监督的检测任务。此外，MVITS提供了具有可理解文本查询的增强的交互性。代码：https://git.io/j1hpy。

In this paper we present Mask DINO, a unified object detection and segmentation framework. Mask DINO extends DINO (DETR with Improved Denoising Anchor Boxes) by adding a mask prediction branch which supports all image segmentation tasks (instance, panoptic, and semantic). It makes use of the query embeddings from DINO to dot-product a high-resolution pixel embedding map to predict a set of binary masks. Some key components in DINO are extended for segmentation through a shared architecture and training process. Mask DINO is simple, efficient, and scalable, and it can benefit from joint large-scale detection and segmentation datasets. Our experiments show that Mask DINO significantly outperforms all existing specialized segmentation methods, both on a ResNet-50 backbone and a pre-trained model with SwinL backbone. Notably, Mask DINO establishes the best results to date on instance segmentation (54.5 AP on COCO), panoptic segmentation (59.4 PQ on COCO), and semantic segmentation (60.8 mIoU on ADE20K) among models under one billion parameters. Code is available at \url{https://github.com/IDEACVR/MaskDINO}.

Existing object detection methods are bounded in a fixed-set vocabulary by costly labeled data. When dealing with novel categories, the model has to be retrained with more bounding box annotations. Natural language supervision is an attractive alternative for its annotation-free attributes and broader object concepts. However, learning open-vocabulary object detection from language is challenging since image-text pairs do not contain fine-grained object-language alignments. Previous solutions rely on either expensive grounding annotations or distilling classification-oriented vision models. In this paper, we propose a novel open-vocabulary object detection framework directly learning from image-text pair data. We formulate object-language alignment as a set matching problem between a set of image region features and a set of word embeddings. It enables us to train an open-vocabulary object detector on image-text pairs in a much simple and effective way. Extensive experiments on two benchmark datasets, COCO and LVIS, demonstrate our superior performance over the competing approaches on novel categories, e.g. achieving 32.0% mAP on COCO and 21.7% mask mAP on LVIS. Code is available at: https://github.com/clin1223/VLDet.

长期以来，将物体检测推向开放量和几乎没有射击转移一直是计算机视觉研究的挑战。这项工作探讨了一种持续的学习方法，该方法使探测器能够通过多数据远见语言的预训练扩展其零/少量功能。我们使用自然语言作为知识表示，我们探讨了从不同培训数据集积累“视觉词汇”的方法，并将任务统一为语言条件的检测框架。具体而言，我们提出了一种新颖的语言感知探测器OMDET和一种新颖的培训机制。拟议的多模式检测网络可以解决多数据库联合培训中的技术挑战，并且可以推广到任意数量的培训数据集，而无需手动标签分类合并的要求。与单独训练相比，Coco，Pascal VOC和更宽的面部/行人的实验结果通过在关节训练中或更高的分数来证实了疗效。此外，我们对超过400万个独特的对象词汇进行了预先培训，并在ODINW的35个下游任务上评估了所得模型。结果表明，OMDET能够在ODINW上实现最新的微调性能。分析表明，通过扩展提出的预训练方法，OMDET继续改善其零/少量调整性能，这表明了进一步扩展的有希望的方法。

图像字幕的当前最新方法采用基于区域的特征，因为它们提供了对象级信息，对于描述图像的内容至关重要；它们通常由对象检测器（例如更快的R-CNN）提取。但是，他们有几个问题，例如缺乏上下文信息，不准确检测的风险以及高计算成本。可以通过使用基于网格的功能来解决前两个。但是，如何提取和融合这两种功能是未知的。本文提出了一种仅使用变压器的神经结构，称为砂砾（基于网格和区域的图像字幕变压器），该构建物有效地利用了两个视觉特征来生成更好的字幕。粒度用基于DITR的方法代替了以前方法中使用的基于CNN的检测器，从而使其更快地计算。此外，它的整体设计仅由变压器组成，可以对模型进行端到端的训练。这种创新的设计和双重视觉功能的集成带来了重大的性能提高。几个图像字幕基准的实验结果表明，砂砾的推论准确性和速度优于先前的方法。

Generalist models, which are capable of performing diverse multi-modal tasks in a task-agnostic way within a single model, have been explored recently. Being, hopefully, an alternative to approaching general-purpose AI, existing generalist models are still at an early stage, where modality and task coverage is limited. To empower multi-modal task-scaling and speed up this line of research, we release a generalist model learning system, OFASys, built on top of a declarative task interface named multi-modal instruction. At the core of OFASys is the idea of decoupling multi-modal task representations from the underlying model implementations. In OFASys, a task involving multiple modalities can be defined declaratively even with just a single line of code. The system automatically generates task plans from such instructions for training and inference. It also facilitates multi-task training for diverse multi-modal workloads. As a starting point, we provide presets of 7 different modalities and 23 highly-diverse example tasks in OFASys, with which we also develop a first-in-kind, single model, OFA+, that can handle text, image, speech, video, and motion data. The single OFA+ model achieves 95% performance in average with only 16% parameters of 15 task-finetuned models, showcasing the performance reliability of multi-modal task-scaling provided by OFASys. Available at https://github.com/OFA-Sys/OFASys