视觉变压器（VIV）被涌现为图像识别的最先进的架构。虽然最近的研究表明，VITS比卷积对应物更强大，但我们的实验发现，VITS过度依赖于局部特征（例如，滋扰和质地），并且不能充分使用全局背景（例如，形状和结构）。因此，VIT不能概括到分销，现实世界数据。为了解决这一缺陷，我们通过添加由矢量量化编码器产生的离散令牌来向Vit的输入层提出简单有效的架构修改。与标准的连续像素令牌不同，离散令牌在小扰动下不变，并且单独包含较少的信息，这促进了VITS学习不变的全局信息。实验结果表明，在七种想象中的鲁棒性基准中增加了四个架构变体上的离散表示，在七个想象中心坚固的基准中加强了高达12％的鲁棒性，同时保持了在想象成上的性能。

While the Transformer architecture has become the de-facto standard for natural language processing tasks, its applications to computer vision remain limited. In vision, attention is either applied in conjunction with convolutional networks, or used to replace certain components of convolutional networks while keeping their overall structure in place. We show that this reliance on CNNs is not necessary and a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks. When pre-trained on large amounts of data and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.), Vision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train. 1

本文探讨了贝尔视觉变压器预训练的更好的码本。最近的工作成功地转移了从NLP到视野领域的BERT预训练。它直接采用一个简单的离散VAE作为视觉销售器，但尚未考虑由此产生的视觉令牌的语义水平。相比之下，NLP字段中的离散令牌是自然的高度语义。这种差异激励我们学习一个感知码本。我们惊奇地找到了一个简单而有效的想法：在DVAE训练期间强制执行感知相似性。我们证明，所提出的感知码本生成的视觉令牌确实表现出更好的语义含义，随后有助于预训练在各种下游任务中实现卓越的转移性能。例如，我们在Imagenet-1K上实现了84.5前1个精度，vit-B骨干，优于竞争方法Beit +1.3，具有相同的训练纪元。它还可以通过+1.3框AP和+1.0掩模AP，在ADE20K上的语义细分，在ADE20K上提高对象检测和分割任务的性能，+1.0 miou，代码和型号将在\ url {https：// github.com/microsoft/peco}。

在本文中，我们询问视觉变形金刚（VIT）是否可以作为改善机器学习模型对抗逃避攻击的对抗性鲁棒性的基础结构。尽管较早的作品集中在改善卷积神经网络上，但我们表明VIT也非常适合对抗训练以实现竞争性能。我们使用自定义的对抗训练配方实现了这一目标，该配方是在Imagenet数据集的一部分上使用严格的消融研究发现的。与卷积相比，VIT的规范培训配方建议强大的数据增强，部分是为了补偿注意力模块的视力归纳偏置。我们表明，该食谱在用于对抗训练时可实现次优性能。相比之下，我们发现省略所有重型数据增强，并添加一些额外的零件（$ \ varepsilon $ -Warmup和更大的重量衰减），从而大大提高了健壮的Vits的性能。我们表明，我们的配方在完整的Imagenet-1k上概括了不同类别的VIT体系结构和大规模模型。此外，调查了模型鲁棒性的原因，我们表明，在使用我们的食谱时，在训练过程中产生强烈的攻击更加容易，这会在测试时提高鲁棒性。最后，我们通过提出一种量化对抗性扰动的语义性质并强调其与模型的鲁棒性的相关性来进一步研究对抗训练的结果。总体而言，我们建议社区应避免将VIT的规范培训食谱转换为在对抗培训的背景下进行强大的培训和重新思考常见的培训选择。

Image segmentation is often ambiguous at the level of individual image patches and requires contextual information to reach label consensus. In this paper we introduce Segmenter, a transformer model for semantic segmentation. In contrast to convolution-based methods, our approach allows to model global context already at the first layer and throughout the network. We build on the recent Vision Transformer (ViT) and extend it to semantic segmentation. To do so, we rely on the output embeddings corresponding to image patches and obtain class labels from these embeddings with a point-wise linear decoder or a mask transformer decoder. We leverage models pre-trained for image classification and show that we can fine-tune them on moderate sized datasets available for semantic segmentation. The linear decoder allows to obtain excellent results already, but the performance can be further improved by a mask transformer generating class masks. We conduct an extensive ablation study to show the impact of the different parameters, in particular the performance is better for large models and small patch sizes. Segmenter attains excellent results for semantic segmentation. It outperforms the state of the art on both ADE20K and Pascal Context datasets and is competitive on Cityscapes.

积极的数据增强是视觉变压器（VIT）的强大泛化能力的关键组成部分。一种这样的数据增强技术是对抗性培训;然而，许多先前的作品表明，这通常会导致清洁的准确性差。在这项工作中，我们展示了金字塔对抗训练，这是一种简单有效的技术来提高韦维尔的整体性能。我们将其与“匹配”辍学和随机深度正则化配对，这采用了干净和对抗样品的相同辍学和随机深度配置。类似于Advprop的CNNS的改进（不直接适用于VIT），我们的金字塔对抗性训练会破坏分销准确性和vit和相关架构的分配鲁棒性之间的权衡。当Imagenet-1K数据训练时，它导致ImageNet清洁准确性的182美元的vit-B模型的精确度，同时由7美元的稳健性指标同时提高性能，从$ 1.76 \％$至11.45 \％$。我们为Imagenet-C（41.4 MCE），Imagenet-R（$ 53.92 \％$），以及Imagenet-Sketch（41.04美元\％$）的新的最先进，只使用vit-b / 16骨干和我们的金字塔对抗训练。我们的代码将在接受时公开提供。

视觉变压器（VIT）在各种机器视觉问题上表现出令人印象深刻的性能。这些模型基于多头自我关注机制，可以灵活地参加一系列图像修补程序以编码上下文提示。一个重要问题是在给定贴片上参加图像范围内的上下文的这种灵活性是如何促进在自然图像中处理滋扰，例如，严重的闭塞，域移位，空间置换，对抗和天然扰动。我们通过广泛的一组实验来系统地研究了这个问题，包括三个vit家族和具有高性能卷积神经网络（CNN）的比较。我们展示和分析了vit的以下迷恋性质：（a）变压器对严重闭塞，扰动和域移位高度稳健，例如，即使在随机堵塞80％的图像之后，也可以在想象中保持高达60％的前1个精度。内容。 （b）与局部纹理的偏置有抗闭锁的强大性能，与CNN相比，VITS对纹理的偏置显着偏差。当受到适当训练以编码基于形状的特征时，VITS展示与人类视觉系统相当的形状识别能力，以前在文献中无与伦比。 （c）使用VIT来编码形状表示导致准确的语义分割而没有像素级监控的有趣后果。 （d）可以组合从单VIT模型的现成功能，以创建一个功能集合，导致传统和几枪学习范例的一系列分类数据集中的高精度率。我们显示VIT的有效特征是由于自我关注机制可以实现灵活和动态的接受领域。

随着变压器作为语言处理的标准及其在计算机视觉方面的进步，参数大小和培训数据的数量相应地增长。许多人开始相信，因此，变形金刚不适合少量数据。这种趋势引起了人们的关注，例如：某些科学领域中数据的可用性有限，并且排除了该领域研究资源有限的人。在本文中，我们旨在通过引入紧凑型变压器来提出一种小规模学习的方法。我们首次表明，具有正确的尺寸，卷积令牌化，变压器可以避免在小数据集上过度拟合和优于最先进的CNN。我们的模型在模型大小方面具有灵活性，并且在获得竞争成果的同时，参数可能仅为0.28亿。当在CIFAR-10上训练Cifar-10，只有370万参数训练时，我们的最佳模型可以达到98％的准确性，这是与以前的基于变形金刚的模型相比，数据效率的显着提高，比其他变压器小于10倍，并且是15％的大小。在实现类似性能的同时，重新NET50。 CCT还表现优于许多基于CNN的现代方法，甚至超过一些基于NAS的方法。此外，我们在Flowers-102上获得了新的SOTA，具有99.76％的TOP-1准确性，并改善了Imagenet上现有基线（82.71％精度，具有29％的VIT参数）以及NLP任务。我们针对变压器的简单而紧凑的设计使它们更可行，可以为那些计算资源和/或处理小型数据集的人学习，同时扩展了在数据高效变压器中的现有研究工作。我们的代码和预培训模型可在https://github.com/shi-labs/compact-transformers上公开获得。

视觉变压器（VIV）及其变体（例如，Swin，PVT）在各种计算机视觉任务中取得了巨大的成功，这是由于他们学习远程语境信息的能力。层标准化（LN）是这些模型中的必要成分。然而，我们发现普通LN在不同位置处的令牌幅度，因为它标准化每个令牌内的嵌入物。变压器难以捕获诱导偏压，例如用LN的图像中的位置上下文。我们通过提出新的标准化器，称为动态令牌归一化（DTN）来解决这个问题，其中归一化在每个令牌（令牌）和跨不同的标记（令牌互补）中执行归一化。 DTN有几个优点。首先，它基于统一的制定，因此可以代表各种现有的归一化方法。其次，DTN学习在令牌内部和令牌间的互联网上标准化令牌，使变换器能够捕获全局上下文信息和本地位置上下文。 {第三，通过简单地更换LN层，DTN可以容易地插入各种视觉变压器，例如VIT，SWIN，PVT，Levit，T2T-VIT，BIGBIRD和REPLERER。广泛的实验表明，配备DTN的变压器始终如一地优于基线模型，具有最小的额外参数和计算开销。例如，DTN优于0.5 \％$ 0.5 \％$ - $ 1.2 \％$ 1.2 \％$ top-1在Imagenet上的准确性，超过1.2 $ - $ 1.4 $ box ap在Coco基准测试的对象检测中，达到2.3 \％$ - $ 3.9 \％$ mce在ImageNet-C上的鲁棒性实验，在远程竞技场上长浪列表中的0.5 \％$ 0.8 \％$ 0.8 \％。}代码将在\ url {https://github.com/wqshao126/dtn}公开。

The recently developed vision transformer (ViT) has achieved promising results on image classification compared to convolutional neural networks. Inspired by this, in this paper, we study how to learn multi-scale feature representations in transformer models for image classification. To this end, we propose a dual-branch transformer to combine image patches (i.e., tokens in a transformer) of different sizes to produce stronger image features. Our approach processes small-patch and large-patch tokens with two separate branches of different computational complexity and these tokens are then fused purely by attention multiple times to complement each other. Furthermore, to reduce computation, we develop a simple yet effective token fusion module based on cross attention, which uses a single token for each branch as a query to exchange information with other branches. Our proposed cross-attention only requires linear time for both computational and memory complexity instead of quadratic time otherwise. Extensive experiments demonstrate that our approach performs better than or on par with several concurrent works on vision transformer, in addition to efficient CNN models. For example, on the ImageNet1K dataset, with some architectural changes, our approach outperforms the recent DeiT by a large margin of 2% with a small to moderate increase in FLOPs and model parameters. Our source codes and models are available at https://github.com/IBM/CrossViT.

由于具有强大的代表性，变形金刚在包括自然语言处理（NLP），计算机视觉和语音识别在内的广泛应用中越来越受欢迎。但是，利用这种代表性的能力有效地需要大量的数据，强大的正则化或两者兼而有之以减轻过度拟合。最近，基于掩盖的自动编码器的自我监督预处理策略已解锁了变压器的功能，这些策略依赖于直接或从未掩盖的内容对比的掩蔽输入进行重建。这种预训练的策略已在NLP中的BERT模型，Speak2VEC模型中使用，最近在Vision中的MAE模型中，该模型迫使该模型使用自动编码相关的目标来了解输入不同部分中的内容之间的关系。在本文中，我们提出了一种小说但令人惊讶的简单替代内容，以预测内容的位置，而无需为其提供位置信息。这样做需要变压器仅凭内容就可以理解输入不同部分之间的位置关系。这相当于有效的实现，其中借口任务是每个输入令牌所有可能位置之间的分类问题。我们在视觉和语音基准上进行了实验，我们的方法对强有力的监督训练基准进行了改进，并且与现代的无监督/自我监督预审方法相媲美。我们的方法还可以使经过训练的变压器在没有位置嵌入的情况下胜过训练有完整位置信息的训练的变压器。

变形金刚在自然语言处理方面取得了巨大的成功。由于变压器中自我发挥机制的强大能力，研究人员为各种计算机视觉任务（例如图像识别，对象检测，图像分割，姿势估计和3D重建）开发了视觉变压器。本文介绍了有关视觉变形金刚的不同建筑设计和培训技巧（包括自我监督的学习）文献的全面概述。我们的目标是为开放研究机会提供系统的审查。

探讨了语言建模流行的变形金刚，用于近期解决视觉任务，例如，用于图像分类的视觉变压器（VIT）。 VIT模型将每个图像分成具有固定长度的令牌序列，然后应用多个变压器层以模拟它们的全局关系以进行分类。然而，当从像想象中的中型数据集上从头开始训练时，VIT对CNNS达到较差的性能。我们发现它是因为：1）输入图像的简单标记未能模拟相邻像素之间的重要局部结构，例如边缘和线路，导致训练采样效率低。 2）冗余注意骨干骨干设计对固定计算预算和有限的训练样本有限的具有限制性。为了克服这些限制，我们提出了一种新的令牌到令牌视觉变压器（T2T-VIT），它包含1）层 - 明智的代币（T2T）转换，通过递归聚合相邻来逐步地结构于令牌到令牌。代币进入一个令牌（令牌到令牌），这样可以建模由周围令牌所代表的本地结构，并且可以减少令牌长度; 2）一种高效的骨干，具有深度狭窄的结构，用于在实证研究后CNN建筑设计的激励变压器结构。值得注意的是，T2T-VIT将Vanilla Vit的参数计数和Mac减少了一半，同时从想象中从头开始训练时，改善了超过3.0 \％。它还优于Endnets并通过直接培训Imagenet训练来实现与MobileNets相当的性能。例如，T2T-VTO与Reset50（21.5M参数）的可比大小（21.5M参数）可以在图像分辨率384 $ \ Times 384上实现83.3 \％TOP1精度。 （代码：https://github.com/yitu-opensource/t2t-vit）

There still remains an extreme performance gap between Vision Transformers (ViTs) and Convolutional Neural Networks (CNNs) when training from scratch on small datasets, which is concluded to the lack of inductive bias. In this paper, we further consider this problem and point out two weaknesses of ViTs in inductive biases, that is, the spatial relevance and diverse channel representation. First, on spatial aspect, objects are locally compact and relevant, thus fine-grained feature needs to be extracted from a token and its neighbors. While the lack of data hinders ViTs to attend the spatial relevance. Second, on channel aspect, representation exhibits diversity on different channels. But the scarce data can not enable ViTs to learn strong enough representation for accurate recognition. To this end, we propose Dynamic Hybrid Vision Transformer (DHVT) as the solution to enhance the two inductive biases. On spatial aspect, we adopt a hybrid structure, in which convolution is integrated into patch embedding and multi-layer perceptron module, forcing the model to capture the token features as well as their neighboring features. On channel aspect, we introduce a dynamic feature aggregation module in MLP and a brand new "head token" design in multi-head self-attention module to help re-calibrate channel representation and make different channel group representation interacts with each other. The fusion of weak channel representation forms a strong enough representation for classification. With this design, we successfully eliminate the performance gap between CNNs and ViTs, and our DHVT achieves a series of state-of-the-art performance with a lightweight model, 85.68% on CIFAR-100 with 22.8M parameters, 82.3% on ImageNet-1K with 24.0M parameters. Code is available at https://github.com/ArieSeirack/DHVT.

We present in this paper a new architecture, named Convolutional vision Transformer (CvT), that improves Vision Transformer (ViT) in performance and efficiency by introducing convolutions into ViT to yield the best of both designs. This is accomplished through two primary modifications: a hierarchy of Transformers containing a new convolutional token embedding, and a convolutional Transformer block leveraging a convolutional projection. These changes introduce desirable properties of convolutional neural networks (CNNs) to the ViT architecture (i.e. shift, scale, and distortion invariance) while maintaining the merits of Transformers (i.e. dynamic attention, global context, and better generalization). We validate CvT by conducting extensive experiments, showing that this approach achieves state-of-the-art performance over other Vision Transformers and ResNets on ImageNet-1k, with fewer parameters and lower FLOPs. In addition, performance gains are maintained when pretrained on larger datasets (e.g. ImageNet-22k) and fine-tuned to downstream tasks. Pretrained on ImageNet-22k, our CvT-W24 obtains a top-1 accuracy of 87.7% on the ImageNet-1k val set. Finally, our results show that the positional encoding, a crucial component in existing Vision Transformers, can be safely removed in our model, simplifying the design for higher resolution vision tasks. Code will be released at https: //github.com/leoxiaobin/CvT.

在过去的几年中，基于自我注意力的变压器模型一直在主导许多计算机视觉任务。它们的出色模型质量在很大程度上取决于标记过多的图像数据集。为了减少对大型标记数据集的依赖，基于重建的掩盖自动编码器正在获得流行，这些自动编码器从未标记的图像中学习了高质量的可转移表示形式。出于同样的目的，最近弱监督的图像预处理方法探索了图像随附的文本字幕的语言监督。在这项工作中，我们提出了对语言辅助代表的预读图像，称为米兰。我们的预处理目标不是预测原始像素或低级别的特征，而是用使用字幕监督获得的大量语义信号来重建图像特征。此外，为了适应我们的重建目标，我们提出了更有效的促使解码器体系结构和语义意识到的掩码采样机制，从而进一步推进了预告片模型的传输性能。实验结果表明，米兰的精度比以前的工作更高。当掩盖的自动编码器在ImagEnet-1K数据集上进行了预估计并以224x224的输入分辨率进行了填充时，米兰在VITB/16上的前1位准确性达到了85.4％，使以前的先前最先前的艺术品达到1％。在下游的语义分割任务中，米兰在ADE20K数据集上使用VIT-B/16骨架达到52.7 MIOU，表现优于先前的蒙版预读结果4分。

We present a convolution-free approach to video classification built exclusively on self-attention over space and time. Our method, named "TimeSformer," adapts the standard Transformer architecture to video by enabling spatiotemporal feature learning directly from a sequence of framelevel patches. Our experimental study compares different self-attention schemes and suggests that "divided attention," where temporal attention and spatial attention are separately applied within each block, leads to the best video classification accuracy among the design choices considered. Despite the radically new design, TimeSformer achieves state-of-the-art results on several action recognition benchmarks, including the best reported accuracy on Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks, our model is faster to train, it can achieve dramatically higher test efficiency (at a small drop in accuracy), and it can also be applied to much longer video clips (over one minute long). Code and models are available at: https://github.com/ facebookresearch/TimeSformer.

We introduce dense vision transformers, an architecture that leverages vision transformers in place of convolutional networks as a backbone for dense prediction tasks. We assemble tokens from various stages of the vision transformer into image-like representations at various resolutions and progressively combine them into full-resolution predictions using a convolutional decoder. The transformer backbone processes representations at a constant and relatively high resolution and has a global receptive field at every stage. These properties allow the dense vision transformer to provide finer-grained and more globally coherent predictions when compared to fully-convolutional networks. Our experiments show that this architecture yields substantial improvements on dense prediction tasks, especially when a large amount of training data is available. For monocular depth estimation, we observe an improvement of up to 28% in relative performance when compared to a state-of-theart fully-convolutional network. When applied to semantic segmentation, dense vision transformers set a new state of the art on ADE20K with 49.02% mIoU. We further show that the architecture can be fine-tuned on smaller datasets such as NYUv2, KITTI, and Pascal Context where it also sets the new state of the art. Our models are available at https://github.com/intel-isl/DPT.

基于卷积神经网络（CNN）框架对图像支出进行了很好的研究，最近引起了计算机视觉的更多关注。但是，CNN依靠固有的电感偏见来实现有效的样品学习，这可能会降低性能上限。在本文中，以最小的变压器体系结构中的柔性自我发挥机制的启发，我们将广义图像支出问题重新构架为贴片的序列到序列自动估计问题，从而使基于查询的图像映射出现。具体而言，我们提出了一个新型混合视觉转换器基于编码器框架，名为\ textbf {query} \ textbf {o} utpainting \ textbf {trextbf {tr} ansformer（\ textbf {queryotr}）围绕给定的图像。 Patch Mode的全球建模能力使我们可以从注意机制的查询角度推断图像。新颖的查询扩展模块（QEM）旨在根据编码器的输出从预测查询中整合信息，因此即使使用相对较小的数据集，也可以加速纯变压器的收敛性。为了进一步提高每个贴片之间的连接性，提议的贴片平滑模块（PSM）重新分配并平均重叠区域，从而提供无缝的预测图像。我们在实验上表明，QueryOtr可以针对最新的图像支出方法平稳和现实地产生吸引力的结果。

Vision transformer (ViT) models exhibit substandard optimizability. In particular, they are sensitive to the choice of optimizer (AdamW vs. SGD), optimizer hyperparameters, and training schedule length. In comparison, modern convolutional neural networks are easier to optimize. Why is this the case? In this work, we conjecture that the issue lies with the patchify stem of ViT models, which is implemented by a stride-p p×p convolution (p = 16 by default) applied to the input image. This large-kernel plus large-stride convolution runs counter to typical design choices of convolutional layers in neural networks. To test whether this atypical design choice causes an issue, we analyze the optimization behavior of ViT models with their original patchify stem versus a simple counterpart where we replace the ViT stem by a small number of stacked stride-two 3×3 convolutions. While the vast majority of computation in the two ViT designs is identical, we find that this small change in early visual processing results in markedly different training behavior in terms of the sensitivity to optimization settings as well as the final model accuracy. Using a convolutional stem in ViT dramatically increases optimization stability and also improves peak performance (by ∼1-2% top-1 accuracy on ImageNet-1k), while maintaining flops and runtime. The improvement can be observed across the wide spectrum of model complexities (from 1G to 36G flops) and dataset scales (from ImageNet-1k to ImageNet-21k). These findings lead us to recommend using a standard, lightweight convolutional stem for ViT models in this regime as a more robust architectural choice compared to the original ViT model design.