对象检测是用于测试预先训练的网络参数的中央下游任务是否达到益处，例如提高准确度或训练速度。当新架构（如视觉变压器（VIT）模型到达时，物体检测方法的复杂性可以使该基准是非微不足道的。这些困难（例如，架构不相容，慢训练，高记忆消耗，未知的培训公式等）已经阻止了最近通过标准VIT模型进行了基准测试转移学习的研究。在本文中，我们提出了克服这些挑战的培训技术，使得使用标准的VT模型作为面膜R-CNN的骨干。这些工具促进了我们研究的主要目标：我们比较五种Vit初始化，包括最近的最先进的自我监督的学习方法，监督初始化和强大的随机初始化基线。我们的研究结果表明，最近基于掩蔽的无监督学习方法可能是在COCO的令人信服的转移学习改进，将箱子AP增加到4％（绝对）的监督和先前自我监督的预训练方法。此外，基于掩蔽的初始化比例更好，随着模型尺寸的增加而增长的提高。

我们探索普通的非层次视觉变压器（VIT）作为用于对象检测的骨干网络。该设计使原始的VIT体系结构可以进行微调以进行对象检测，而无需重新设计层次结构的主链以进行预训练。随着微调的最低适应性，我们的纯净背骨检测器可以取得竞争成果。令人惊讶的是，我们观察到：（i）足以从单尺度特征映射（没有常见的FPN设计）构建一个简单的特征金字塔，并且（ii）足以使用窗户注意力（无需转移），很少有帮助跨窗口传播块。凭借普通的VIT骨架作为掩盖自动编码器（MAE），我们的探测器（名为VITDET）可以与先前基于层次结构骨架的先前领先方法竞争，仅使用ImagEnet-1k Pre Pre pre to Coco Dataset上的61.3 ap_box竞争-训练。我们希望我们的研究能够引起人们对普通背骨检测器的研究。 VITDET的代码可在detectron2中获得。

在本文中，我们将多尺度视觉变压器（MVIT）作为图像和视频分类的统一架构，以及对象检测。我们提出了一种改进的MVIT版本，它包含分解的相对位置嵌入和残余汇集连接。我们以五种尺寸实例化此架构，并评估Imagenet分类，COCO检测和动力学视频识别，在此优先效果。我们进一步比较了MVITS的汇集注意力来窗口注意力机制，其中它在准确性/计算中优于后者。如果没有钟声，MVIT在3个域中具有最先进的性能：ImageNet分类的准确性为88.8％，Coco对象检测的56.1盒AP和动力学-400视频分类的86.1％。代码和模型将公开可用。

本文显示屏蔽的自动化器（MAE）是可扩展的自我监督学习者，用于计算机愿景。我们的MAE方法很简单：我们掩盖输入图像的随机补丁并重建缺失像素。它基于两个核心设计。首先，我们开发一个不对称的编码器解码器架构，其中编码器仅在掩码的可见子集（没有掩码令牌）上，以及重量解码器，该重量解码器从潜像和掩码令牌重建原始图像。其次，我们发现掩蔽了高比例的输入图像，例如，75％，产生非凡和有意义的自我监督任务。耦合这两种设计使我们能够有效且有效地培训大型模型：我们加速培训（3倍或更多）并提高准确性。我们可扩展的方法允许学习概括的高容量模型：例如，Vanilla Vit-Maxim模型在使用Imagenet-1K数据的方法中实现最佳准确性（87.8％）。下游任务中的转移性能优于监督预培训并显示有前途的缩放行为。

The combination of transformers and masked image modeling (MIM) pre-training framework has shown great potential in various vision tasks. However, the pre-training computational budget is too heavy and withholds the MIM from becoming a practical training paradigm. This paper presents FastMIM, a simple and generic framework for expediting masked image modeling with the following two steps: (i) pre-training vision backbones with low-resolution input images; and (ii) reconstructing Histograms of Oriented Gradients (HOG) feature instead of original RGB values of the input images. In addition, we propose FastMIM-P to progressively enlarge the input resolution during pre-training stage to further enhance the transfer results of models with high capacity. We point out that: (i) a wide range of input resolutions in pre-training phase can lead to similar performances in fine-tuning phase and downstream tasks such as detection and segmentation; (ii) the shallow layers of encoder are more important during pre-training and discarding last several layers can speed up the training stage with no harm to fine-tuning performance; (iii) the decoder should match the size of selected network; and (iv) HOG is more stable than RGB values when resolution transfers;. Equipped with FastMIM, all kinds of vision backbones can be pre-trained in an efficient way. For example, we can achieve 83.8%/84.1% top-1 accuracy on ImageNet-1K with ViT-B/Swin-B as backbones. Compared to previous relevant approaches, we can achieve comparable or better top-1 accuracy while accelerate the training procedure by $\sim$5$\times$. Code can be found in https://github.com/ggjy/FastMIM.pytorch.

Vision transformer (ViT) models exhibit substandard optimizability. In particular, they are sensitive to the choice of optimizer (AdamW vs. SGD), optimizer hyperparameters, and training schedule length. In comparison, modern convolutional neural networks are easier to optimize. Why is this the case? In this work, we conjecture that the issue lies with the patchify stem of ViT models, which is implemented by a stride-p p×p convolution (p = 16 by default) applied to the input image. This large-kernel plus large-stride convolution runs counter to typical design choices of convolutional layers in neural networks. To test whether this atypical design choice causes an issue, we analyze the optimization behavior of ViT models with their original patchify stem versus a simple counterpart where we replace the ViT stem by a small number of stacked stride-two 3×3 convolutions. While the vast majority of computation in the two ViT designs is identical, we find that this small change in early visual processing results in markedly different training behavior in terms of the sensitivity to optimization settings as well as the final model accuracy. Using a convolutional stem in ViT dramatically increases optimization stability and also improves peak performance (by ∼1-2% top-1 accuracy on ImageNet-1k), while maintaining flops and runtime. The improvement can be observed across the wide spectrum of model complexities (from 1G to 36G flops) and dataset scales (from ImageNet-1k to ImageNet-21k). These findings lead us to recommend using a standard, lightweight convolutional stem for ViT models in this regime as a more robust architectural choice compared to the original ViT model design.

We report competitive results on object detection and instance segmentation on the COCO dataset using standard models trained from random initialization. The results are no worse than their ImageNet pre-training counterparts even when using the hyper-parameters of the baseline system (Mask R-CNN) that were optimized for fine-tuning pretrained models, with the sole exception of increasing the number of training iterations so the randomly initialized models may converge. Training from random initialization is surprisingly robust; our results hold even when: (i) using only 10% of the training data, (ii) for deeper and wider models, and (iii) for multiple tasks and metrics. Experiments show that ImageNet pre-training speeds up convergence early in training, but does not necessarily provide regularization or improve final target task accuracy. To push the envelope we demonstrate 50.9 AP on COCO object detection without using any external data-a result on par with the top COCO 2017 competition results that used ImageNet pre-training. These observations challenge the conventional wisdom of ImageNet pre-training for dependent tasks and we expect these discoveries will encourage people to rethink the current de facto paradigm of 'pretraining and fine-tuning' in computer vision.

我们展示了如何通过基于关注的全球地图扩充任何卷积网络，以实现非本地推理。我们通过基于关注的聚合层替换为单个变压器块的最终平均池，重量贴片如何参与分类决策。我们使用2个参数（宽度和深度）使用简单的补丁卷积网络，使用简单的补丁的卷积网络插入学习的聚合层。与金字塔设计相比，该架构系列在所有层上维护输入补丁分辨率。它在准确性和复杂性之间产生了令人惊讶的竞争权衡，特别是在记忆消耗方面，如我们在各种计算机视觉任务所示：对象分类，图像分割和检测的实验所示。

视觉识别的“咆哮20S”开始引入视觉变压器（VITS），这将被取代的Cummnets作为最先进的图像分类模型。另一方面，vanilla vit，当应用于一般计算机视觉任务等对象检测和语义分割时面临困难。它是重新引入多个ConvNet Priors的等级变压器（例如，Swin变压器），使变压器实际上可作为通用视觉骨干网，并在各种视觉任务上展示了显着性能。然而，这种混合方法的有效性仍然在很大程度上归功于变压器的内在优越性，而不是卷积的固有感应偏差。在这项工作中，我们重新审视设计空间并测试纯粹的Convnet可以实现的限制。我们逐渐“现代化”标准Reset朝着视觉变压器的设计设计，并发现几个有助于沿途绩效差异的关键组件。此探索的结果是一个纯粹的ConvNet型号被称为ConvNext。完全由标准的Convnet模块构建，ConvNexts在准确性和可扩展性方面与变压器竞争，实现了87.8％的ImageNet Top-1精度和表现优于COCO检测和ADE20K分割的Swin变压器，同时保持了标准Convnet的简单性和效率。

This paper presents a new vision Transformer, called Swin Transformer, that capably serves as a general-purpose backbone for computer vision. Challenges in adapting Transformer from language to vision arise from differences between the two domains, such as large variations in the scale of visual entities and the high resolution of pixels in images compared to words in text. To address these differences, we propose a hierarchical Transformer whose representation is computed with Shifted windows. The shifted windowing scheme brings greater efficiency by limiting self-attention computation to non-overlapping local windows while also allowing for cross-window connection. This hierarchical architecture has the flexibility to model at various scales and has linear computational complexity with respect to image size. These qualities of Swin Transformer make it compatible with a broad range of vision tasks, including image classification (87.3 top-1 accuracy on ImageNet-1K) and dense prediction tasks such as object detection (58.7 box AP and 51.1 mask AP on COCO testdev) and semantic segmentation (53.5 mIoU on ADE20K val). Its performance surpasses the previous state-of-theart by a large margin of +2.7 box AP and +2.6 mask AP on COCO, and +3.2 mIoU on ADE20K, demonstrating the potential of Transformer-based models as vision backbones. The hierarchical design and the shifted window approach also prove beneficial for all-MLP architectures. The code and models are publicly available at https://github. com/microsoft/Swin-Transformer.

While the Transformer architecture has become the de-facto standard for natural language processing tasks, its applications to computer vision remain limited. In vision, attention is either applied in conjunction with convolutional networks, or used to replace certain components of convolutional networks while keeping their overall structure in place. We show that this reliance on CNNs is not necessary and a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks. When pre-trained on large amounts of data and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.), Vision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train. 1

We present BoTNet, a conceptually simple yet powerful backbone architecture that incorporates self-attention for multiple computer vision tasks including image classification, object detection and instance segmentation. By just replacing the spatial convolutions with global self-attention in the final three bottleneck blocks of a ResNet and no other changes, our approach improves upon the baselines significantly on instance segmentation and object detection while also reducing the parameters, with minimal overhead in latency. Through the design of BoTNet, we also point out how ResNet bottleneck blocks with self-attention can be viewed as Transformer blocks. Without any bells and whistles, BoTNet achieves 44.4% Mask AP and 49.7% Box AP on the COCO Instance Segmentation benchmark using the Mask R-CNN framework; surpassing the previous best published single model and single scale results of ResNeSt [67] evaluated on the COCO validation set. Finally, we present a simple adaptation of the BoTNet design for image classification, resulting in models that achieve a strong performance of 84.7% top-1 accuracy on the ImageNet benchmark while being up to 1.64x faster in "compute" 1 time than the popular EfficientNet models on TPU-v3 hardware. We hope our simple and effective approach will serve as a strong baseline for future research in self-attention models for vision.

大规模数据集的预培训模型，如想象成，是计算机视觉中的标准实践。此范例对于具有小型培训套的任务特别有效，其中高容量模型往往会过度装备。在这项工作中，我们考虑一个自我监督的预训练场景，只能利用目标任务数据。我们考虑数据集，如斯坦福汽车，草图或可可，这是比想象成小的数量的顺序。我们的研究表明，在本文中介绍的Beit或诸如Beit或Variant的去噪对预训练数据的类型和大小比通过比较图像嵌入来训练的流行自我监督方法更加强大。我们获得了竞争性能与ImageNet预训练相比，来自不同域的各种分类数据集。在Coco上，当专注于使用Coco Images进行预训练时，检测和实例分割性能超过了可比设置中的监督Imagenet预训练。

我们呈现蒙版特征预测（MaskFeat），用于自我监督的视频模型的预训练。我们的方法首先随机地掩盖输入序列的一部分，然后预测蒙面区域的特征。我们研究五种不同类型的功能，找到面向导向渐变（HOG）的直方图，手工制作的特征描述符，在性能和效率方面尤其良好。我们观察到猪中的局部对比标准化对于良好的结果至关重要，这与使用HOG进行视觉识别的早期工作符合。我们的方法可以学习丰富的视觉知识和基于大规模的变压器的模型。在不使用额外的模型重量或监督的情况下，在未标记视频上预先培训的MaskFeat在动力学-400上使用MVIT-L达到86.7％的前所未有的结果，在动力学-600，88.3％上，88.3％，在动力学-700,88.8地图上SSV2上的75.0％。 MaskFeat进一步推广到图像输入，其可以被解释为具有单个帧的视频，并在想象中获得竞争结果。

Driven by improved architectures and better representation learning frameworks, the field of visual recognition has enjoyed rapid modernization and performance boost in the early 2020s. For example, modern ConvNets, represented by ConvNeXt, have demonstrated strong performance in various scenarios. While these models were originally designed for supervised learning with ImageNet labels, they can also potentially benefit from self-supervised learning techniques such as masked autoencoders (MAE). However, we found that simply combining these two approaches leads to subpar performance. In this paper, we propose a fully convolutional masked autoencoder framework and a new Global Response Normalization (GRN) layer that can be added to the ConvNeXt architecture to enhance inter-channel feature competition. This co-design of self-supervised learning techniques and architectural improvement results in a new model family called ConvNeXt V2, which significantly improves the performance of pure ConvNets on various recognition benchmarks, including ImageNet classification, COCO detection, and ADE20K segmentation. We also provide pre-trained ConvNeXt V2 models of various sizes, ranging from an efficient 3.7M-parameter Atto model with 76.7% top-1 accuracy on ImageNet, to a 650M Huge model that achieves a state-of-the-art 88.9% accuracy using only public training data.

视觉变换器（VTS）作为卷积网络（CNNS）的架构范式替代品。与CNN不同，VT可以捕获图像元素之间的全局关系，并且它们可能具有更大的表示容量。然而，缺乏典型的卷积电感偏差使这些模型比普通的CNN更饥饿。实际上，嵌入在CNN架构设计中的某些本地属性，在VTS中应该从样品中学习。在本文中，我们明确地分析了不同的VTS，比较了他们在小型训练制度中的鲁棒性，并且我们表明，尽管在想象中训练时具有可比的准确性，但它们在较小数据集上的性能可能很大程度上不同。此外，我们提出了一种自我监督的任务，可以从图像中提取其他信息，只有可忽略不计的计算开销。这项任务鼓励VTS学习图像内的空间关系，并使VT培训在训练数据稀缺时更加强劲。我们的任务与标准（监督）培训共同使用，它不依赖于特定的架构选择，因此它可以轻松插入现有的VTS。使用与不同的VTS和数据集进行广泛的评估，我们表明我们的方法可以改善（有时显着地）VTS的最终精度。我们的代码可用于：https：//github.com/yhlleo/vts-droc。

视觉变压器（VIT）已被证明可以在广泛的视觉应用中获得高度竞争性的性能，例如图像分类，对象检测和语义图像分割。与卷积神经网络相比，通常发现视觉变压器的较弱的电感偏差会在较小的培训数据集上培训时，会增加对模型正则化或数据增强的依赖（简称为“ AUGREG”）。我们进行了一项系统的实证研究，以便更好地了解培训数据，AUGREG，模型大小和计算预算之间的相互作用。作为这项研究的一个结果，我们发现增加的计算和AUGREG的组合可以产生与在数量级上训练的模型相同的训练数据的模型：我们在公共Imagenet-21K数据集中培训各种尺寸的VIT模型在较大的JFT-300M数据集上匹配或超越其对手的培训。

本文提出了RESTV2，这是一种更简单，更快，更强的多尺度视觉变压器，用于视觉识别。 RESTV2简化了RESTV1中的EMSA结构（即消除了多头相互作用零件），并采用了upplame操作来重建由下采样操作引起的丢失的中等和高频信息。此外，我们探索了不同的技术，以更好地将RESTV2骨架应用于下游任务。我们发现，尽管将EMSAV2和窗户注意力结合起来可以大大减少理论矩阵乘数拖台，但它可能会大大降低计算密度，从而导致较低的实际速度。我们全面验证RESTV2在Imagenet分类，可可检测和ADE20K语义分割方面。实验结果表明，所提出的RESTV2可以大幅度优于最近最新的骨干，这表明RESTV2作为固体骨架的潜力。代码和模型将在\ url {https://github.com/wofmanaf/rest}公开可用

我们介绍克斯内变压器，一种高效且有效的变压器的骨干，用于通用视觉任务。变压器设计的具有挑战性的问题是，全球自我关注来计算成本昂贵，而局部自我关注经常限制每个令牌的相互作用。为了解决这个问题，我们开发了以平行的横向和垂直条纹在水平和垂直条纹中计算自我关注的交叉形窗口自我关注机制，通过将输入特征分成相等的条纹而获得的每个条纹宽度。我们提供了条纹宽度效果的数学分析，并改变变压器网络的不同层的条纹宽度，这在限制计算成本时实现了强大的建模能力。我们还介绍了本地增强的位置编码（LEPE），比现有的编码方案更好地处理本地位置信息。 LEPE自然支持任意输入分辨率，因此对下游任务特别有效和友好。 CSWIN变压器并入其具有这些设计和分层结构，展示了普通愿景任务的竞争性能。具体来说，它在ImageNet-1K上实现了85.4 \％Top-1精度，而无需任何额外的培训数据或标签，53.9盒AP和46.4掩模AP，ADE20K语义分割任务上的52.2 Miou，超过以前的状态 - 在类似的拖鞋设置下，艺术品+1.2，+2.0，+1.4和+2.0分别为+1.2，+2.0，+1.4和+2.0。通过在较大的数据集Imagenet-21k上进行前预先预订，我们在Ave20K上实现了87.5％的成像-1K和高分性能，55.7 miou。代码和模型可在https://github.com/microsoft/cswin-transformer中找到。

Modern object detectors have taken the advantages of backbone networks pre-trained on large scale datasets. Except for the backbone networks, however, other components such as the detector head and the feature pyramid network (FPN) remain trained from scratch, which hinders fully tapping the potential of representation models. In this study, we propose to integrally migrate pre-trained transformer encoder-decoders (imTED) to a detector, constructing a feature extraction path which is ``fully pre-trained" so that detectors' generalization capacity is maximized. The essential differences between imTED with the baseline detector are twofold: (1) migrating the pre-trained transformer decoder to the detector head while removing the randomly initialized FPN from the feature extraction path; and (2) defining a multi-scale feature modulator (MFM) to enhance scale adaptability. Such designs not only reduce randomly initialized parameters significantly but also unify detector training with representation learning intendedly. Experiments on the MS COCO object detection dataset show that imTED consistently outperforms its counterparts by $\sim$2.4 AP. Without bells and whistles, imTED improves the state-of-the-art of few-shot object detection by up to 7.6 AP. Code is available at https://github.com/LiewFeng/imTED.