Vision Transformers (ViTs) have gained significant popularity in recent years and have proliferated into many applications. However, it is not well explored how varied their behavior is under different learning paradigms. We compare ViTs trained through different methods of supervision, and show that they learn a diverse range of behaviors in terms of their attention, representations, and downstream performance. We also discover ViT behaviors that are consistent across supervision, including the emergence of Offset Local Attention Heads. These are self-attention heads that attend to a token adjacent to the current token with a fixed directional offset, a phenomenon that to the best of our knowledge has not been highlighted in any prior work. Our analysis shows that ViTs are highly flexible and learn to process local and global information in different orders depending on their training method. We find that contrastive self-supervised methods learn features that are competitive with explicitly supervised features, and they can even be superior for part-level tasks. We also find that the representations of reconstruction-based models show non-trivial similarity to contrastive self-supervised models. Finally, we show how the "best" layer for a given task varies by both supervision method and task, further demonstrating the differing order of information processing in ViTs.
translated by 谷歌翻译
Convolutional neural networks (CNNs) have so far been the de-facto model for visual data. Recent work has shown that (Vision) Transformer models (ViT) can achieve comparable or even superior performance on image classification tasks. This raises a central question: how are Vision Transformers solving these tasks? Are they acting like convolutional networks, or learning entirely different visual representations? Analyzing the internal representation structure of ViTs and CNNs on image classification benchmarks, we find striking differences between the two architectures, such as ViT having more uniform representations across all layers. We explore how these differences arise, finding crucial roles played by self-attention, which enables early aggregation of global information, and ViT residual connections, which strongly propagate features from lower to higher layers. We study the ramifications for spatial localization, demonstrating ViTs successfully preserve input spatial information, with noticeable effects from different classification methods. Finally, we study the effect of (pretraining) dataset scale on intermediate features and transfer learning, and conclude with a discussion on connections to new architectures such as the MLP-Mixer. This breakthrough highlights a fundamental question: how are Vision Transformers solving these image based tasks? Do they act like convolutions, learning the same inductive biases from scratch? Or are they developing novel task representations? What is the role of scale in learning these representations? And are there ramifications for downstream tasks? In this paper, we study these questions, uncovering key representational differences between ViTs and CNNs, the ways in which these difference arise, and effects on classification and transfer learning. Specifically, our contributions are:35th Conference on Neural Information Processing Systems (NeurIPS 2021).
translated by 谷歌翻译
语言变形金刚的成功主要归因于屏蔽语言建模(MLM)的借口任务,其中文本首先被致以语义有意义的作品。在这项工作中,我们研究了蒙面图像建模(MIM),并指出使用语义有意义的视觉销售器的优缺点。我们提出了一个自我监督的框架IBOT,可以使用在线标记器执行蒙版预测。具体而言,我们在蒙面的补丁令牌上进行自我蒸馏,并将教师网络作为在线标记器,以及在课堂上的自蒸馏来获取视觉语义。在线销售器与MIM目标和分配的多级培训管道共同学习,销售器需要预先预先培训。通过在Imagenet-1K上达到81.6%的线性探测精度和86.3%的微调精度来展示IBOT的突出。除了最先进的图像分类结果之外,我们强调了新兴的局部语义模式,这有助于模型对共同损坏获得强大的鲁棒性,并在密集的下游任务中实现领先的结果,例如,对象检测,实例分割和语义细分。
translated by 谷歌翻译
视觉变压器(VIT)在各种机器视觉问题上表现出令人印象深刻的性能。这些模型基于多头自我关注机制,可以灵活地参加一系列图像修补程序以编码上下文提示。一个重要问题是在给定贴片上参加图像范围内的上下文的这种灵活性是如何促进在自然图像中处理滋扰,例如,严重的闭塞,域移位,空间置换,对抗和天然扰动。我们通过广泛的一组实验来系统地研究了这个问题,包括三个vit家族和具有高性能卷积神经网络(CNN)的比较。我们展示和分析了vit的以下迷恋性质:(a)变压器对严重闭塞,扰动和域移位高度稳健,例如,即使在随机堵塞80%的图像之后,也可以在想象中保持高达60%的前1个精度。内容。 (b)与局部纹理的偏置有抗闭锁的强大性能,与CNN相比,VITS对纹理的偏置显着偏差。当受到适当训练以编码基于形状的特征时,VITS展示与人类视觉系统相当的形状识别能力,以前在文献中无与伦比。 (c)使用VIT来编码形状表示导致准确的语义分割而没有像素级监控的有趣后果。 (d)可以组合从单VIT模型的现成功能,以创建一个功能集合,导致传统和几枪学习范例的一系列分类数据集中的高精度率。我们显示VIT的有效特征是由于自我关注机制可以实现灵活和动态的接受领域。
translated by 谷歌翻译
自我监督学习的进步带来了强大的一般图像表示学习方法。到目前为止,它主要集中在图像级学习上。反过来,诸如无监督图像细分之类的任务并没有从这种趋势中受益,因为它们需要空间多样性的表示。但是,学习密集的表示具有挑战性,因为在无监督的环境中,尚不清楚如何指导模型学习与各种潜在对象类别相对应的表示形式。在本文中,我们认为对物体部分的自我监督学习是解决此问题的方法。对象部分是可以推广的:它们是独立于对象定义的先验性,但可以分组以形成对象后验。为此,我们利用最近提出的视觉变压器参与对象的能力,并将其与空间密集的聚类任务相结合,以微调空间令牌。我们的方法超过了三个语义分割基准的最新方法,提高了17%-3%,表明我们的表示在各种对象定义下都是用途广泛的。最后,我们将其扩展到完全无监督的分割 - 即使在测试时间也可以完全避免使用标签信息 - 并证明了一种基于社区检测的自动合并发现的对象零件的简单方法可产生可观的收益。
translated by 谷歌翻译
变形金刚和蒙版语言建模在计算机视觉中很快被视为视觉变压器和蒙版图像建模(MIM)。在这项工作中,我们认为由于图像中令牌的数量和相关性,图像令牌掩盖与文本中的令牌掩盖有所不同。特别是,为了为MIM产生具有挑战性的借口任务,我们主张从随机掩盖到知情掩盖的转变。我们在基于蒸馏的MIM的背景下开发并展示了这一想法,其中教师变压器编码器生成了一个注意力图,我们用它来指导学生为学生指导掩盖。因此,我们引入了一种新颖的掩蔽策略,称为注意引导蒙版(ATTMASK),我们证明了其对基于密集蒸馏的MIM以及基于普通蒸馏的自然剥离的自助力学习的有效性。我们确认ATTMASK可以加快学习过程,并提高各种下游任务的性能。我们在https://github.com/gkakogeorgiou/attmask上提供实现代码。
translated by 谷歌翻译
Transformer models have shown great success handling long-range interactions, making them a promising tool for modeling video. However they lack inductive biases and scale quadratically with input length. These limitations are further exacerbated when dealing with the high dimensionality introduced with the temporal dimension. While there are surveys analyzing the advances of Transformers for vision, none focus on an in-depth analysis of video-specific designs. In this survey we analyze main contributions and trends of works leveraging Transformers to model video. Specifically, we delve into how videos are handled as input-level first. Then, we study the architectural changes made to deal with video more efficiently, reduce redundancy, re-introduce useful inductive biases, and capture long-term temporal dynamics. In addition we provide an overview of different training regimes and explore effective self-supervised learning strategies for video. Finally, we conduct a performance comparison on the most common benchmark for Video Transformers (i.e., action classification), finding them to outperform 3D ConvNets even with less computational complexity.
translated by 谷歌翻译
由于其最近在减少监督学习的差距方面取得了成功,自我监督的学习方法正在增加计算机愿景的牵引力。在自然语言处理(NLP)中,自我监督的学习和变形金刚已经是选择的方法。最近的文献表明,变压器也在计算机愿景中越来越受欢迎。到目前为止,当使用大规模监督数据或某种共同监督时,视觉变压器已被证明可以很好地工作。在教师网络方面。这些监督的普试视觉变压器在下游任务中实现了非常好的变化,变化最小。在这项工作中,我们调查自我监督学习的预用图像/视觉变压器,然后使用它们进行下游分类任务的优点。我们提出了自我监督的视觉变压器(坐在)并讨论了几种自我监督的培训机制,以获得借口模型。静坐的架构灵活性允许我们将其用作自动统计器,并无缝地使用多个自我监控任务。我们表明,可以在小规模数据集上进行预训练,以便在小型数据集上进行下游分类任务,包括几千个图像而不是数百万的图像。使用公共协议对所提出的方法进行评估标准数据集。结果展示了变压器的强度及其对自我监督学习的适用性。我们通过大边缘表现出现有的自我监督学习方法。我们还观察到坐着很好,很少有镜头学习,并且还表明它通过简单地训练从坐的学到的学习功能的线性分类器来学习有用的表示。预先训练,FineTuning和评估代码将在以下:https://github.com/sara-ahmed/sit。
translated by 谷歌翻译
我们利用从预先训练的视觉变压器(VIT)提取的深度特征,如密集的视觉描述符。我们证明这些特征是当从自我监督的Vit模型(Dino-Vit)中提取时,表现出几种打击性质:(i)特征在高空间分辨率下编码强大的高级信息 - 即,捕获精细的语义对象部件空间粒度和(ii)编码的语义信息跨相关但不同的对象类别(即超级类别)共享。这些属性允许我们设计强大的密集Vit描述符,便于各种应用,包括共分割,部分共分割和通信 - 通过将轻量级方法应用于深度染色特征(例如,分布/聚类)来实现。我们将这些应用程序进一步接受级别任务的领域 - 展示相关类别的对象如何在显着的姿势和外观变化下常规分段为语义部分。我们的方法,在定性和定量地评估的方法,实现最先进的部分共分割结果,以及最近监督方法的竞争结果,专门针对共同分割和对应关系。
translated by 谷歌翻译
本文研究了两种技术,用于开发有效的自我监督视觉变压器(ESVIT)进行视觉表示学习。首先,我们通过一项全面的实证研究表明,具有稀疏自我生产的多阶段体系结构可以显着降低建模的复杂性,但具有失去捕获图像区域之间细粒度对应关系的能力的成本。其次,我们提出了一项新的区域匹配训练任务,该任务使模型可以捕获细粒的区域依赖性,因此显着提高了学习视觉表示的质量。我们的结果表明,ESVIT在ImageNet线性探针评估上结合两种技术,在ImageNet线性探针评估中获得了81.3%的TOP-1,优于先前的艺术,其较高吞吐量的顺序幅度约为较高。当转移到下游线性分类任务时,ESVIT在18个数据集中的17个中优于其受监督的对方。代码和模型可公开可用:https://github.com/microsoft/esvit
translated by 谷歌翻译
While the Transformer architecture has become the de-facto standard for natural language processing tasks, its applications to computer vision remain limited. In vision, attention is either applied in conjunction with convolutional networks, or used to replace certain components of convolutional networks while keeping their overall structure in place. We show that this reliance on CNNs is not necessary and a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks. When pre-trained on large amounts of data and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.), Vision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train. 1
translated by 谷歌翻译
Astounding results from Transformer models on natural language tasks have intrigued the vision community to study their application to computer vision problems. Among their salient benefits, Transformers enable modeling long dependencies between input sequence elements and support parallel processing of sequence as compared to recurrent networks e.g., Long short-term memory (LSTM). Different from convolutional networks, Transformers require minimal inductive biases for their design and are naturally suited as set-functions. Furthermore, the straightforward design of Transformers allows processing multiple modalities (e.g., images, videos, text and speech) using similar processing blocks and demonstrates excellent scalability to very large capacity networks and huge datasets. These strengths have led to exciting progress on a number of vision tasks using Transformer networks. This survey aims to provide a comprehensive overview of the Transformer models in the computer vision discipline. We start with an introduction to fundamental concepts behind the success of Transformers i.e., self-attention, large-scale pre-training, and bidirectional feature encoding. We then cover extensive applications of transformers in vision including popular recognition tasks (e.g., image classification, object detection, action recognition, and segmentation), generative modeling, multi-modal tasks (e.g., visual-question answering, visual reasoning, and visual grounding), video processing (e.g., activity recognition, video forecasting), low-level vision (e.g., image super-resolution, image enhancement, and colorization) and 3D analysis (e.g., point cloud classification and segmentation). We compare the respective advantages and limitations of popular techniques both in terms of architectural design and their experimental value. Finally, we provide an analysis on open research directions and possible future works. We hope this effort will ignite further interest in the community to solve current challenges towards the application of transformer models in computer vision.
translated by 谷歌翻译
本文显示屏蔽的自动化器(MAE)是可扩展的自我监督学习者,用于计算机愿景。我们的MAE方法很简单:我们掩盖输入图像的随机补丁并重建缺失像素。它基于两个核心设计。首先,我们开发一个不对称的编码器解码器架构,其中编码器仅在掩码的可见子集(没有掩码令牌)上,以及重量解码器,该重量解码器从潜像和掩码令牌重建原始图像。其次,我们发现掩蔽了高比例的输入图像,例如,75%,产生非凡和有意义的自我监督任务。耦合这两种设计使我们能够有效且有效地培训大型模型:我们加速培训(3倍或更多)并提高准确性。我们可扩展的方法允许学习概括的高容量模型:例如,Vanilla Vit-Maxim模型在使用Imagenet-1K数据的方法中实现最佳准确性(87.8%)。下游任务中的转移性能优于监督预培训并显示有前途的缩放行为。
translated by 谷歌翻译
在过去的几年中,基于自我注意力的变压器模型一直在主导许多计算机视觉任务。它们的出色模型质量在很大程度上取决于标记过多的图像数据集。为了减少对大型标记数据集的依赖,基于重建的掩盖自动编码器正在获得流行,这些自动编码器从未标记的图像中学习了高质量的可转移表示形式。出于同样的目的,最近弱监督的图像预处理方法探索了图像随附的文本字幕的语言监督。在这项工作中,我们提出了对语言辅助代表的预读图像,称为米兰。我们的预处理目标不是预测原始像素或低级别的特征,而是用使用字幕监督获得的大量语义信号来重建图像特征。此外,为了适应我们的重建目标,我们提出了更有效的促使解码器体系结构和语义意识到的掩码采样机制,从而进一步推进了预告片模型的传输性能。实验结果表明,米兰的精度比以前的工作更高。当掩盖的自动编码器在ImagEnet-1K数据集上进行了预估计并以224x224的输入分辨率进行了填充时,米兰在VITB/16上的前1位准确性达到了85.4%,使以前的先前最先前的艺术品达到1%。在下游的语义分割任务中,米兰在ADE20K数据集上使用VIT-B/16骨架达到52.7 MIOU,表现优于先前的蒙版预读结果4分。
translated by 谷歌翻译
自我监督的表示学习的最新趋势集中在消除训练管道中的归纳偏见。但是,当可用数据有限时,归纳偏差在设置中可能很有用,或者提供对基础数据分布的更多见解。我们提出了空间注意(SPAN),该框架利用未标记的图像数据集中使用一致的空间和语义结构来指导视觉变压器的注意。 SPAN通过将注意力面罩从单独的变压器头正规化,以跟随语义区域的各个先验。这些先验可以从数据统计数据或域专家提供的单个标记样本中得出。我们研究了几种详细的现实情况,包括医学图像分析和视觉质量保证。我们发现,所产生的注意力面膜比从域 - 不合义预审进的掩码更容易解​​释。 SPAN可为肺和心脏分割产生58.7的地图改进。我们还发现,与结构域 - 不合稳定的预处理相比,我们的方法在将验证的模型转移到下游胸部疾病分类任务时会产生2.2个MAUC的改善。最后,我们表明,与域 - 不可屈服的预处理相比,跨越预处理会导致低数据表格中的下游分类性能更高。
translated by 谷歌翻译
Vision transformers (ViTs) are quickly becoming the de-facto architecture for computer vision, yet we understand very little about why they work and what they learn. While existing studies visually analyze the mechanisms of convolutional neural networks, an analogous exploration of ViTs remains challenging. In this paper, we first address the obstacles to performing visualizations on ViTs. Assisted by these solutions, we observe that neurons in ViTs trained with language model supervision (e.g., CLIP) are activated by semantic concepts rather than visual features. We also explore the underlying differences between ViTs and CNNs, and we find that transformers detect image background features, just like their convolutional counterparts, but their predictions depend far less on high-frequency information. On the other hand, both architecture types behave similarly in the way features progress from abstract patterns in early layers to concrete objects in late layers. In addition, we show that ViTs maintain spatial information in all layers except the final layer. In contrast to previous works, we show that the last layer most likely discards the spatial information and behaves as a learned global pooling operation. Finally, we conduct large-scale visualizations on a wide range of ViT variants, including DeiT, CoaT, ConViT, PiT, Swin, and Twin, to validate the effectiveness of our method.
translated by 谷歌翻译
蒙面的自动编码器是可扩展的视觉学习者,因为Mae \ Cite {He2022masked}的标题表明,视觉中的自我监督学习(SSL)可能会采用与NLP中类似的轨迹。具体而言,具有蒙版预测(例如BERT)的生成借口任务已成为NLP中的事实上的标准SSL实践。相比之下,他们的歧视性对应物(例如对比度学习)掩埋了视力中的生成方法的早期尝试;但是,蒙版图像建模的成功已恢复了屏蔽自动编码器(过去通常被称为DeNosing AutoCoder)。作为在NLP中与Bert弥合差距的一个里程碑,蒙面自动编码器吸引了对SSL在视觉及其他方面的前所未有的关注。这项工作对蒙面自动编码器进行了全面的调查,以洞悉SSL的有希望的方向。作为第一个使用蒙版自动编码器审查SSL的人,这项工作通过讨论其历史发展,最新进度以及对不同应用的影响,重点介绍其在视觉中的应用。
translated by 谷歌翻译
Continual Learning (CL) is a field dedicated to devise algorithms able to achieve lifelong learning. Overcoming the knowledge disruption of previously acquired concepts, a drawback affecting deep learning models and that goes by the name of catastrophic forgetting, is a hard challenge. Currently, deep learning methods can attain impressive results when the data modeled does not undergo a considerable distributional shift in subsequent learning sessions, but whenever we expose such systems to this incremental setting, performance drop very quickly. Overcoming this limitation is fundamental as it would allow us to build truly intelligent systems showing stability and plasticity. Secondly, it would allow us to overcome the onerous limitation of retraining these architectures from scratch with the new updated data. In this thesis, we tackle the problem from multiple directions. In a first study, we show that in rehearsal-based techniques (systems that use memory buffer), the quantity of data stored in the rehearsal buffer is a more important factor over the quality of the data. Secondly, we propose one of the early works of incremental learning on ViTs architectures, comparing functional, weight and attention regularization approaches and propose effective novel a novel asymmetric loss. At the end we conclude with a study on pretraining and how it affects the performance in Continual Learning, raising some questions about the effective progression of the field. We then conclude with some future directions and closing remarks.
translated by 谷歌翻译
跨图像建立视觉对应是一项具有挑战性且必不可少的任务。最近,已经提出了大量的自我监督方法,以更好地学习视觉对应的表示。但是,我们发现这些方法通常无法利用语义信息,并且在低级功能的匹配方面过度融合。相反,人类的视觉能够将不同的物体区分为跟踪的借口。受此范式的启发,我们建议学习语义意识的细粒对应关系。首先,我们证明语义对应是通过一组丰富的图像级别自我监督方法隐式获得的。我们进一步设计了一个像素级的自我监督学习目标,该目标专门针对细粒的对应关系。对于下游任务,我们将这两种互补的对应表示形式融合在一起,表明它们是协同增强性能的。我们的方法超过了先前的最先进的自我监督方法,使用卷积网络在各种视觉通信任务上,包括视频对象分割,人姿势跟踪和人类部分跟踪。
translated by 谷歌翻译