Transformers have been recently adapted for large scale image classification, achieving high scores shaking up the long supremacy of convolutional neural networks. However the optimization of image transformers has been little studied so far. In this work, we build and optimize deeper transformer networks for image classification. In particular, we investigate the interplay of architecture and optimization of such dedicated transformers. We make two transformers architecture changes that significantly improve the accuracy of deep transformers. This leads us to produce models whose performance does not saturate early with more depth, for instance we obtain 86.5% top-1 accuracy on Imagenet when training with no external data, we thus attain the current SOTA with less FLOPs and parameters. Moreover, our best model establishes the new state of the art on Imagenet with Reassessed labels and Imagenet-V2 / match frequency, in the setting with no additional training data. We share our code and models 1 .

Recently, neural networks purely based on attention were shown to address image understanding tasks such as image classification. These highperforming vision transformers are pre-trained with hundreds of millions of images using a large infrastructure, thereby limiting their adoption.In this work, we produce competitive convolution-free transformers by training on Imagenet only. We train them on a single computer in less than 3 days. Our reference vision transformer (86M parameters) achieves top-1 accuracy of 83.1% (single-crop) on ImageNet with no external data.More importantly, we introduce a teacher-student strategy specific to transformers. It relies on a distillation token ensuring that the student learns from the teacher through attention. We show the interest of this token-based distillation, especially when using a convnet as a teacher. This leads us to report results competitive with convnets for both Imagenet (where we obtain up to 85.2% accuracy) and when transferring to other tasks. We share our code and models.

Recently, neural networks purely based on attention were shown to address image understanding tasks such as image classification. These highperforming vision transformers are pre-trained with hundreds of millions of images using a large infrastructure, thereby limiting their adoption.In this work, we produce competitive convolutionfree transformers trained on ImageNet only using a single computer in less than 3 days. Our reference vision transformer (86M parameters) achieves top-1 accuracy of 83.1% (single-crop) on ImageNet with no external data.We also introduce a teacher-student strategy specific to transformers. It relies on a distillation token ensuring that the student learns from the teacher through attention, typically from a convnet teacher. The learned transformers are competitive (85.2% top-1 acc.) with the state of the art on ImageNet, and similarly when transferred to other tasks. We will share our code and models.

我们展示了如何通过基于关注的全球地图扩充任何卷积网络，以实现非本地推理。我们通过基于关注的聚合层替换为单个变压器块的最终平均池，重量贴片如何参与分类决策。我们使用2个参数（宽度和深度）使用简单的补丁卷积网络，使用简单的补丁的卷积网络插入学习的聚合层。与金字塔设计相比，该架构系列在所有层上维护输入补丁分辨率。它在准确性和复杂性之间产生了令人惊讶的竞争权衡，特别是在记忆消耗方面，如我们在各种计算机视觉任务所示：对象分类，图像分割和检测的实验所示。

We introduce submodel co-training, a regularization method related to co-training, self-distillation and stochastic depth. Given a neural network to be trained, for each sample we implicitly instantiate two altered networks, ``submodels'', with stochastic depth: we activate only a subset of the layers. Each network serves as a soft teacher to the other, by providing a loss that complements the regular loss provided by the one-hot label. Our approach, dubbed cosub, uses a single set of weights, and does not involve a pre-trained external model or temporal averaging. Experimentally, we show that submodel co-training is effective to train backbones for recognition tasks such as image classification and semantic segmentation. Our approach is compatible with multiple architectures, including RegNet, ViT, PiT, XCiT, Swin and ConvNext. Our training strategy improves their results in comparable settings. For instance, a ViT-B pretrained with cosub on ImageNet-21k obtains 87.4% top-1 acc. @448 on ImageNet-val.

We design a family of image classification architectures that optimize the trade-off between accuracy and efficiency in a high-speed regime. Our work exploits recent findings in attention-based architectures, which are competitive on highly parallel processing hardware. We revisit principles from the extensive literature on convolutional neural networks to apply them to transformers, in particular activation maps with decreasing resolutions. We also introduce the attention bias, a new way to integrate positional information in vision transformers.As a result, we propose LeVIT: a hybrid neural network for fast inference image classification. We consider different measures of efficiency on different hardware platforms, so as to best reflect a wide range of application scenarios. Our extensive experiments empirically validate our technical choices and show they are suitable to most architectures. Overall, LeViT significantly outperforms existing convnets and vision transformers with respect to the speed/accuracy tradeoff. For example, at 80% ImageNet top-1 accuracy, LeViT is 5 times faster than EfficientNet on CPU. We release the code at https: //github.com/facebookresearch/LeViT.

我们在视觉变压器上呈现整洁但有效的递归操作，可以提高参数利用而不涉及额外参数。这是通过在变压器网络的深度分享权重来实现的。所提出的方法可以只使用NA \“IVE递归操作来获得大量增益（〜2％），不需要对设计网络原理的特殊或复杂的知识，并引入训练程序的最小计算开销。减少额外的计算通过递归操作，同时保持卓越的准确性，我们通过递归层的多个切片组自行引入近似方法，这可以通过最小的性能损失将成本消耗降低10〜30％。我们称我们的模型切片递归变压器（SRET） ，这与高效视觉变压器的广泛的其他设计兼容。我们最好的模型在含有较少参数的同时，在最先进的方法中对Imagenet建立了重大改进。建议的切片递归操作使我们能够建立一个变压器超过100甚至1000层，仍然仍然小尺寸（13〜15米），以避免困难当模型尺寸太大时，IES在优化中。灵活的可扩展性显示出缩放和构建极深和大维视觉变压器的巨大潜力。我们的代码和模型可在https://github.com/szq0214/sret中找到。

Convolutional architectures have proven extremely successful for vision tasks. Their hard inductive biases enable sample-efficient learning, but come at the cost of a potentially lower performance ceiling. Vision Transformers (ViTs) rely on more flexible self-attention layers, and have recently outperformed CNNs for image classification. However, they require costly pre-training on large external datasets or distillation from pretrained convolutional networks. In this paper, we ask the following question: is it possible to combine the strengths of these two architectures while avoiding their respective limitations? To this end, we introduce gated positional self-attention (GPSA), a form of positional self-attention which can be equipped with a "soft" convolutional inductive bias. We initialize the GPSA layers to mimic the locality of convolutional layers, then give each attention head the freedom to escape locality by adjusting a gating parameter regulating the attention paid to position versus content information. The resulting convolutionallike ViT architecture, ConViT, outperforms the DeiT (Touvron et al., 2020) on ImageNet, while offering a much improved sample efficiency. We further investigate the role of locality in learning by first quantifying how it is encouraged in vanilla self-attention layers, then analyzing how it is escaped in GPSA layers. We conclude by presenting various ablations to better understand the success of the ConViT. Our code and models are released publicly at https://github.com/ facebookresearch/convit.

The recently developed vision transformer (ViT) has achieved promising results on image classification compared to convolutional neural networks. Inspired by this, in this paper, we study how to learn multi-scale feature representations in transformer models for image classification. To this end, we propose a dual-branch transformer to combine image patches (i.e., tokens in a transformer) of different sizes to produce stronger image features. Our approach processes small-patch and large-patch tokens with two separate branches of different computational complexity and these tokens are then fused purely by attention multiple times to complement each other. Furthermore, to reduce computation, we develop a simple yet effective token fusion module based on cross attention, which uses a single token for each branch as a query to exchange information with other branches. Our proposed cross-attention only requires linear time for both computational and memory complexity instead of quadratic time otherwise. Extensive experiments demonstrate that our approach performs better than or on par with several concurrent works on vision transformer, in addition to efficient CNN models. For example, on the ImageNet1K dataset, with some architectural changes, our approach outperforms the recent DeiT by a large margin of 2% with a small to moderate increase in FLOPs and model parameters. Our source codes and models are available at https://github.com/IBM/CrossViT.

While the Transformer architecture has become the de-facto standard for natural language processing tasks, its applications to computer vision remain limited. In vision, attention is either applied in conjunction with convolutional networks, or used to replace certain components of convolutional networks while keeping their overall structure in place. We show that this reliance on CNNs is not necessary and a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks. When pre-trained on large amounts of data and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.), Vision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train. 1

随着变压器作为语言处理的标准及其在计算机视觉方面的进步，参数大小和培训数据的数量相应地增长。许多人开始相信，因此，变形金刚不适合少量数据。这种趋势引起了人们的关注，例如：某些科学领域中数据的可用性有限，并且排除了该领域研究资源有限的人。在本文中，我们旨在通过引入紧凑型变压器来提出一种小规模学习的方法。我们首次表明，具有正确的尺寸，卷积令牌化，变压器可以避免在小数据集上过度拟合和优于最先进的CNN。我们的模型在模型大小方面具有灵活性，并且在获得竞争成果的同时，参数可能仅为0.28亿。当在CIFAR-10上训练Cifar-10，只有370万参数训练时，我们的最佳模型可以达到98％的准确性，这是与以前的基于变形金刚的模型相比，数据效率的显着提高，比其他变压器小于10倍，并且是15％的大小。在实现类似性能的同时，重新NET50。 CCT还表现优于许多基于CNN的现代方法，甚至超过一些基于NAS的方法。此外，我们在Flowers-102上获得了新的SOTA，具有99.76％的TOP-1准确性，并改善了Imagenet上现有基线（82.71％精度，具有29％的VIT参数）以及NLP任务。我们针对变压器的简单而紧凑的设计使它们更可行，可以为那些计算资源和/或处理小型数据集的人学习，同时扩展了在数据高效变压器中的现有研究工作。我们的代码和预培训模型可在https://github.com/shi-labs/compact-transformers上公开获得。

We present Multiscale Vision Transformers (MViT) for video and image recognition, by connecting the seminal idea of multiscale feature hierarchies with transformer models. Multiscale Transformers have several channel-resolution scale stages. Starting from the input resolution and a small channel dimension, the stages hierarchically expand the channel capacity while reducing the spatial resolution. This creates a multiscale pyramid of features with early layers operating at high spatial resolution to model simple low-level visual information, and deeper layers at spatially coarse, but complex, high-dimensional features. We evaluate this fundamental architectural prior for modeling the dense nature of visual signals for a variety of video recognition tasks where it outperforms concurrent vision transformers that rely on large scale external pre-training and are 5-10× more costly in computation and parameters. We further remove the temporal dimension and apply our model for image classification where it outperforms prior work on vision transformers. Code is available at: https: //github.com/facebookresearch/SlowFast.

尽管图像变形金刚与计算机视觉任务中的卷积神经网络显示出竞争性结果，但缺乏诸如区域的电感偏见仍然在模型效率方面构成问题，尤其是对于嵌入式应用程序而言。在这项工作中，我们通过引入注意力面具以将空间位置纳入自我发挥作用来解决这个问题。局部依赖性有效地捕获了掩盖的注意力头，以及由未掩盖的注意力头部捕获的全球依赖性。随着蒙版注意力图像变压器 - MAIT，与CAIT相比，TOP -1的准确性提高了1.7％，与SWIN相比，吞吐量更少，吞吐量提高了1.5倍。使用注意力面罩编码局部性是模型的不可知论，因此它适用于整体，分层或其他新型变压器体系结构。

Vision transformer (ViT) models exhibit substandard optimizability. In particular, they are sensitive to the choice of optimizer (AdamW vs. SGD), optimizer hyperparameters, and training schedule length. In comparison, modern convolutional neural networks are easier to optimize. Why is this the case? In this work, we conjecture that the issue lies with the patchify stem of ViT models, which is implemented by a stride-p p×p convolution (p = 16 by default) applied to the input image. This large-kernel plus large-stride convolution runs counter to typical design choices of convolutional layers in neural networks. To test whether this atypical design choice causes an issue, we analyze the optimization behavior of ViT models with their original patchify stem versus a simple counterpart where we replace the ViT stem by a small number of stacked stride-two 3×3 convolutions. While the vast majority of computation in the two ViT designs is identical, we find that this small change in early visual processing results in markedly different training behavior in terms of the sensitivity to optimization settings as well as the final model accuracy. Using a convolutional stem in ViT dramatically increases optimization stability and also improves peak performance (by ∼1-2% top-1 accuracy on ImageNet-1k), while maintaining flops and runtime. The improvement can be observed across the wide spectrum of model complexities (from 1G to 36G flops) and dataset scales (from ImageNet-1k to ImageNet-21k). These findings lead us to recommend using a standard, lightweight convolutional stem for ViT models in this regime as a more robust architectural choice compared to the original ViT model design.

在本文中，我们询问视觉变形金刚（VIT）是否可以作为改善机器学习模型对抗逃避攻击的对抗性鲁棒性的基础结构。尽管较早的作品集中在改善卷积神经网络上，但我们表明VIT也非常适合对抗训练以实现竞争性能。我们使用自定义的对抗训练配方实现了这一目标，该配方是在Imagenet数据集的一部分上使用严格的消融研究发现的。与卷积相比，VIT的规范培训配方建议强大的数据增强，部分是为了补偿注意力模块的视力归纳偏置。我们表明，该食谱在用于对抗训练时可实现次优性能。相比之下，我们发现省略所有重型数据增强，并添加一些额外的零件（$ \ varepsilon $ -Warmup和更大的重量衰减），从而大大提高了健壮的Vits的性能。我们表明，我们的配方在完整的Imagenet-1k上概括了不同类别的VIT体系结构和大规模模型。此外，调查了模型鲁棒性的原因，我们表明，在使用我们的食谱时，在训练过程中产生强烈的攻击更加容易，这会在测试时提高鲁棒性。最后，我们通过提出一种量化对抗性扰动的语义性质并强调其与模型的鲁棒性的相关性来进一步研究对抗训练的结果。总体而言，我们建议社区应避免将VIT的规范培训食谱转换为在对抗培训的背景下进行强大的培训和重新思考常见的培训选择。

We present in this paper a new architecture, named Convolutional vision Transformer (CvT), that improves Vision Transformer (ViT) in performance and efficiency by introducing convolutions into ViT to yield the best of both designs. This is accomplished through two primary modifications: a hierarchy of Transformers containing a new convolutional token embedding, and a convolutional Transformer block leveraging a convolutional projection. These changes introduce desirable properties of convolutional neural networks (CNNs) to the ViT architecture (i.e. shift, scale, and distortion invariance) while maintaining the merits of Transformers (i.e. dynamic attention, global context, and better generalization). We validate CvT by conducting extensive experiments, showing that this approach achieves state-of-the-art performance over other Vision Transformers and ResNets on ImageNet-1k, with fewer parameters and lower FLOPs. In addition, performance gains are maintained when pretrained on larger datasets (e.g. ImageNet-22k) and fine-tuned to downstream tasks. Pretrained on ImageNet-22k, our CvT-W24 obtains a top-1 accuracy of 87.7% on the ImageNet-1k val set. Finally, our results show that the positional encoding, a crucial component in existing Vision Transformers, can be safely removed in our model, simplifying the design for higher resolution vision tasks. Code will be released at https: //github.com/leoxiaobin/CvT.

最近，视觉变压器变得非常流行。但是，将它们部署在许多应用程序中的计算昂贵部分是由于注意力块中的软磁层。我们引入了一个简单但有效的，无软的注意力块Sima，它使用简单的$ \ ell_1 $ -norm而不是使用SoftMax层，将查询和密钥矩阵归一化。然后，SIMA中的注意力块是三个矩阵的简单乘法，因此SIMA可以在测试时间动态更改计算的顺序，以在令牌数量或通道数量上实现线性计算。我们从经验上表明，SIMA应用于变形金刚，DEIT，XCIT和CVT的三种SOTA变体，与SOTA模型相比，SIMA可在不需要SoftMax层的情况下达到PAR准确性。有趣的是，将SIMA从多头更改为单头只会对精度产生很小的影响，这进一步简化了注意力障碍。该代码可在此处找到：$ \ href {https://github.com/ucdvision/sima} {\ text {this https url}} $

虽然最先进的视觉变压器模型实现了图像分类的有希望的结果，但它们是非常昂贵的并且需要许多GFLOPS。尽管可以通过减少网络中的令牌数量来降低视觉变压器的GFLOPS，但是没有对所有输入图像的最佳设置。因此，在这项工作中，我们引入了可分辨率的无参数自适应令牌采样（ATS）模块，可以插入任何现有的视觉变压器架构。通过评分和自适应采样重要令牌，在视觉变压器上实现视觉变压器。结果，令牌的数量不再静态，但是每个输入图像都变化。通过将ATS集成为当前变压器块内的附加层，我们可以将它们转换为具有自适应令牌的更高效的视觉变压器。由于ATS是一种无参数模块，因此它可以作为即插即用模块添加到从货架上的预制视觉变压器中，从而在没有任何额外训练的情况下减少他们的GFLOP。但是，由于其可分辨动的设计，人们还可以培训配有ATS的视觉变压器。通过将其添加到多个最先进的视觉变压器，我们在想象成数据集上进行评估。我们的评估表明，通过将计算成本（GFLOPS）降低37％，在保留准确性时，该模块通过降低了37％，提高了最先进的模块。

视觉变压器已成为计算机视觉任务最重要的模型之一。虽然它们以前的工作胜过，但它们需要大量计算资源，这是二次为$ n $的规模。这是传统自我关注（SA）算法的主要缺点。在这里，我们提出了单位力操作的视觉变压器（UFO-VIT），一种具有线性复杂性的新型SA机制。这项工作的主要方法是消除原始SA的非线性。我们将SA机构的矩阵乘法构成而没有复杂的线性近似。通过从原始SA仅修改几行代码，所提出的型号在大多数容量制度上以图像分类和密集的预测任务更优于基于变压器的模型。

随着计算机愿景中变压器架构的普及，研究焦点已转向开发计算有效的设计。基于窗口的本地关注是最近作品采用的主要技术之一。这些方法以非常小的贴片尺寸和小的嵌入尺寸开始，然后执行冲击卷积（贴片合并），以减少特征图尺寸并增加嵌入尺寸，因此，形成像设计的金字塔卷积神经网络（CNN）。在这项工作中，我们通过呈现一种新的各向同性架构，调查变压器中的本地和全球信息建模，以便采用当地窗口和特殊令牌，称为超级令牌，以自我关注。具体地，将单个超级令牌分配给每个图像窗口，该窗口捕获该窗口的丰富本地细节。然后使用这些令牌用于跨窗口通信和全局代表学习。因此，大多数学习都独立于较高层次的图像补丁$（n）$，并且仅基于超级令牌$（n / m ^ 2）$何处，从中学习额外的嵌入量窗口大小。在ImageNet-1K上的标准图像分类中，所提出的基于超代币的变压器（STT-S25）实现了83.5 \％的精度，其等同于带有大约一半参数（49M）的Swin变压器（Swin-B）和推断的两倍时间吞吐量。建议的超级令牌变压器为可视识别任务提供轻量级和有前途的骨干。