已经发现基于混合的增强对于培训期间的概括模型有效,特别是对于视觉变压器(VITS),因为它们很容易过度装备。然而,先前的基于混合的方法具有潜在的先验知识,即目标的线性内插比应保持与输入插值中提出的比率相同。这可能导致一个奇怪的现象,有时由于增强中的随机过程,混合图像中没有有效对象,但标签空间仍然存在响应。为了弥合输入和标签空间之间的这种差距,我们提出了透明度,该差别将基于视觉变压器的注意图混合标签。如果受关注图的相应输入图像加权,则标签的置信度将会更大。传输令人尴尬地简单,可以在几行代码中实现,而不会在不引入任何额外的参数和拖鞋到基于Vit的模型。实验结果表明,我们的方法可以在想象集分类上一致地始终改善各种基于Vit的模型。在ImageNet上预先接受过扫描后,基于Vit的模型还展示了对语义分割,对象检测和实例分割的更好的可转换性。当在评估4个不同的基准时,传输展示展示更加强劲。代码将在https://github.com/beckschen/transmix上公开提供。
translated by 谷歌翻译
Modern deep networks can be better generalized when trained with noisy samples and regularization techniques. Mixup and CutMix have been proven to be effective for data augmentation to help avoid overfitting. Previous Mixup-based methods linearly combine images and labels to generate additional training data. However, this is problematic if the object does not occupy the whole image as we demonstrate in Figure 1. Correctly assigning the label weights is hard even for human beings and there is no clear criterion to measure it. To tackle this problem, in this paper, we propose LUMix, which models such uncertainty by adding label perturbation during training. LUMix is simple as it can be implemented in just a few lines of code and can be universally applied to any deep networks \eg CNNs and Vision Transformers, with minimal computational cost. Extensive experiments show that our LUMix can consistently boost the performance for networks with a wide range of diversity and capacity on ImageNet, \eg $+0.7\%$ for a small model DeiT-S and $+0.6\%$ for a large variant XCiT-L. We also demonstrate that LUMix can lead to better robustness when evaluated on ImageNet-O and ImageNet-A. The source code can be found \href{https://github.com/kevin-ssy/LUMix}{here}
translated by 谷歌翻译
Data mixing strategies (e.g., CutMix) have shown the ability to greatly improve the performance of convolutional neural networks (CNNs). They mix two images as inputs for training and assign them with a mixed label with the same ratio. While they are shown effective for vision transformers (ViTs), we identify a token fluctuation phenomenon that has suppressed the potential of data mixing strategies. We empirically observe that the contributions of input tokens fluctuate as forward propagating, which might induce a different mixing ratio in the output tokens. The training target computed by the original data mixing strategy can thus be inaccurate, resulting in less effective training. To address this, we propose a token-label alignment (TL-Align) method to trace the correspondence between transformed tokens and the original tokens to maintain a label for each token. We reuse the computed attention at each layer for efficient token-label alignment, introducing only negligible additional training costs. Extensive experiments demonstrate that our method improves the performance of ViTs on image classification, semantic segmentation, objective detection, and transfer learning tasks. Code is available at: https://github.com/Euphoria16/TL-Align.
translated by 谷歌翻译
CutMix is a vital augmentation strategy that determines the performance and generalization ability of vision transformers (ViTs). However, the inconsistency between the mixed images and the corresponding labels harms its efficacy. Existing CutMix variants tackle this problem by generating more consistent mixed images or more precise mixed labels, but inevitably introduce heavy training overhead or require extra information, undermining ease of use. To this end, we propose an efficient and effective Self-Motivated image Mixing method (SMMix), which motivates both image and label enhancement by the model under training itself. Specifically, we propose a max-min attention region mixing approach that enriches the attention-focused objects in the mixed images. Then, we introduce a fine-grained label assignment technique that co-trains the output tokens of mixed images with fine-grained supervision. Moreover, we devise a novel feature consistency constraint to align features from mixed and unmixed images. Due to the subtle designs of the self-motivated paradigm, our SMMix is significant in its smaller training overhead and better performance than other CutMix variants. In particular, SMMix improves the accuracy of DeiT-T/S, CaiT-XXS-24/36, and PVT-T/S/M/L by more than +1% on ImageNet-1k. The generalization capability of our method is also demonstrated on downstream tasks and out-of-distribution datasets. Code of this project is available at https://github.com/ChenMnZ/SMMix.
translated by 谷歌翻译
CutMix是一种流行的增强技术,通常用于训练现代卷积和变压器视觉网络。它最初旨在鼓励卷积神经网络(CNN)更多地关注图像的全球环境,而不是本地信息,从而大大提高了CNN的性能。但是,我们发现它对自然具有全球接收领域的基于变压器的体系结构的好处有限。在本文中,我们提出了一种新型的数据增强技术图,以提高视觉变压器的性能。 TokenMix通过将混合区分为多个分离的零件,将两个图像在令牌级别混合。此外,我们表明,Cutmix中的混合学习目标是一对地面真相标签的线性组合,可能是不准确的,有时是违反直觉的。为了获得更合适的目标,我们建议根据预先训练的教师模型的两个图像的基于内容的神经激活图分配目标得分,该图像不需要具有高性能。通过大量有关各种视觉变压器体系结构的实验,我们表明我们提出的TokenMix可以帮助视觉变形金刚专注于前景区域,以推断班级并增强其稳健性,以稳定的性能增长。值得注意的是,我们使用 +1%Imagenet TOP-1精度改善DEIT-T/S/B。此外,TokenMix的训练较长,在Imainet上获得了81.2%的TOP-1精度,而DEIT-S训练了400个时代。代码可从https://github.com/sense-x/tokenmix获得。
translated by 谷歌翻译
变压器最近在各种视觉任务上表现出卓越的性能。大型有时甚至全球,接收领域赋予变换器模型,并通过其CNN对应物具有更高的表示功率。然而,简单地扩大接收领域也产生了几个问题。一方面,使用致密的注意,例如,在VIT中,导致过度的记忆和计算成本,并且特征可以受到超出兴趣区域的无关紧要的影响。另一方面,PVT或SWIN变压器采用的稀疏注意是数据不可知论,可能会限制模拟长距离关系的能力。为了缓解这些问题,我们提出了一种新型可变形的自我关注模块,其中以数据相关的方式选择密钥和值对中的密钥和值对的位置。这种灵活的方案使自我关注模块能够专注于相关区域并捕获更多的信息性功能。在此基础上,我们呈现可变形的关注变压器,一般骨干模型,具有可变形关注的图像分类和密集预测任务。广泛的实验表明,我们的模型在综合基准上实现了一致的改善结果。代码可在https://github.com/leaplabthu/dat上获得。
translated by 谷歌翻译
Regional dropout strategies have been proposed to enhance the performance of convolutional neural network classifiers. They have proved to be effective for guiding the model to attend on less discriminative parts of objects (e.g. leg as opposed to head of a person), thereby letting the network generalize better and have better object localization capabilities. On the other hand, current methods for regional dropout remove informative pixels on training images by overlaying a patch of either black pixels or random noise. Such removal is not desirable because it leads to information loss and inefficiency during training. We therefore propose the CutMix augmentation strategy: patches are cut and pasted among training images where the ground truth labels are also mixed proportionally to the area of the patches. By making efficient use of training pixels and retaining the regularization effect of regional dropout, CutMix consistently outperforms the state-of-the-art augmentation strategies on CI-FAR and ImageNet classification tasks, as well as on the Im-ageNet weakly-supervised localization task. Moreover, unlike previous augmentation methods, our CutMix-trained ImageNet classifier, when used as a pretrained model, results in consistent performance gains in Pascal detection and MS-COCO image captioning benchmarks. We also show that CutMix improves the model robustness against input corruptions and its out-of-distribution detection performances. Source code and pretrained models are available at https://github.com/clovaai/CutMix-PyTorch.
translated by 谷歌翻译
事实证明,数据混合对提高深神经网络的概括能力是有效的。虽然早期方法通过手工制作的策略(例如线性插值)混合样品,但最新方法利用显着性信息通过复杂的离线优化来匹配混合样品和标签。但是,在精确的混合政策和优化复杂性之间进行了权衡。为了应对这一挑战,我们提出了一个新颖的自动混合(Automix)框架,其中混合策略被参数化并直接实现最终分类目标。具体而言,Automix将混合分类重新定义为两个子任务(即混合样品生成和混合分类)与相应的子网络,并在双层优化框架中求解它们。对于这一代,可学习的轻质混合发电机Mix Block旨在通过在相应混合标签的直接监督下对贴片的关系进行建模,以生成混合样品。为了防止双层优化的降解和不稳定性,我们进一步引入了动量管道以端到端的方式训练汽车。与在各种分类场景和下游任务中的最新图像相比,九个图像基准的广泛实验证明了汽车的优势。
translated by 谷歌翻译
变压器模型在处理各种视觉任务方面表现出了有希望的有效性。但是,与训练卷积神经网络(CNN)模型相比,训练视觉变压器(VIT)模型更加困难,并且依赖于大规模训练集。为了解释这一观察结果,我们做出了一个假设,即\ textit {vit模型在捕获图像的高频组件方面的有效性较小,而不是CNN模型},并通过频率分析对其进行验证。受这一发现的启发,我们首先研究了现有技术从新的频率角度改进VIT模型的影响,并发现某些技术(例如,randaugment)的成功可以归因于高频组件的更好使用。然后,为了补偿这种不足的VIT模型能力,我们提出了HAT,该HAT可以通过对抗训练直接增强图像的高频组成部分。我们表明,HAT可以始终如一地提高各种VIT模型的性能(例如VIT-B的 +1.2%,Swin-B的 +0.5%),尤其是提高了仅使用Imagenet-的高级模型Volo-D5至87.3% 1K数据,并且优势也可以维持在分发数据的数据上,并转移到下游任务。该代码可在以下网址获得:https://github.com/jiawangbai/hat。
translated by 谷歌翻译
Image segmentation is often ambiguous at the level of individual image patches and requires contextual information to reach label consensus. In this paper we introduce Segmenter, a transformer model for semantic segmentation. In contrast to convolution-based methods, our approach allows to model global context already at the first layer and throughout the network. We build on the recent Vision Transformer (ViT) and extend it to semantic segmentation. To do so, we rely on the output embeddings corresponding to image patches and obtain class labels from these embeddings with a point-wise linear decoder or a mask transformer decoder. We leverage models pre-trained for image classification and show that we can fine-tune them on moderate sized datasets available for semantic segmentation. The linear decoder allows to obtain excellent results already, but the performance can be further improved by a mask transformer generating class masks. We conduct an extensive ablation study to show the impact of the different parameters, in particular the performance is better for large models and small patch sizes. Segmenter attains excellent results for semantic segmentation. It outperforms the state of the art on both ADE20K and Pascal Context datasets and is competitive on Cityscapes.
translated by 谷歌翻译
在这项研究中,我们提出了混合图像建模(MixMim),这是一种适用于各种分层视觉变压器的简单但有效的MIM方法。现有的MIM方法用特殊的掩码符号替换输入令牌的随机子集,并旨在从损坏的图像中重建原始图像令牌。但是,我们发现,由于较大的掩蔽率(例如,Beit中的40%),使用蒙版符号会大大减慢训练并引起训练 - 不一致的不一致。相比之下,我们用另一个图像的可见令牌(即创建混合图像)代替一个图像的蒙版令牌。然后,我们进行双重重建以从混合输入中重建原始的两个图像,从而显着提高效率。虽然MixMim可以应用于各种体系结构,但本文探讨了更简单但更强的层次变压器,并使用MixMim -B,-L和-H缩放。经验结果表明,混合mim可以有效地学习高质量的视觉表示。值得注意的是,具有88M参数的MixMIM-B通过预处理600个时期的Imagenet-1k上的TOP-1精度达到了85.1%的TOP-1精度,在MIM方法中为具有可比模型尺寸(例如VIT-B)的神经网络创造了新的记录。此外,其在其他6个数据集上的传输性能显示MixMim比以前的MIM方法更好。代码可从https://github.com/sense-x/mixmim获得。
translated by 谷歌翻译
我们介绍克斯内变压器,一种高效且有效的变压器的骨干,用于通用视觉任务。变压器设计的具有挑战性的问题是,全球自我关注来计算成本昂贵,而局部自我关注经常限制每个令牌的相互作用。为了解决这个问题,我们开发了以平行的横向和垂直条纹在水平和垂直条纹中计算自我关注的交叉形窗口自我关注机制,通过将输入特征分成相等的条纹而获得的每个条纹宽度。我们提供了条纹宽度效果的数学分析,并改变变压器网络的不同层的条纹宽度,这在限制计算成本时实现了强大的建模能力。我们还介绍了本地增强的位置编码(LEPE),比现有的编码方案更好地处理本地位置信息。 LEPE自然支持任意输入分辨率,因此对下游任务特别有效和友好。 CSWIN变压器并入其具有这些设计和分层结构,展示了普通愿景任务的竞争性能。具体来说,它在ImageNet-1K上实现了85.4 \%Top-1精度,而无需任何额外的培训数据或标签,53.9盒AP和46.4掩模AP,ADE20K语义分割任务上的52.2 Miou,超过以前的状态 - 在类似的拖鞋设置下,艺术品+1.2,+2.0,+1.4和+2.0分别为+1.2,+2.0,+1.4和+2.0。通过在较大的数据集Imagenet-21k上进行前预先预订,我们在Ave20K上实现了87.5%的成像-1K和高分性能,55.7 miou。代码和模型可在https://github.com/microsoft/cswin-transformer中找到。
translated by 谷歌翻译
视觉变压器的最新进展在基于点产生自我注意的新空间建模机制驱动的各种任务中取得了巨大成功。在本文中,我们表明,视觉变压器背后的关键要素,即输入自适应,远程和高阶空间相互作用,也可以通过基于卷积的框架有效地实现。我们介绍了递归封闭式卷积($ \ textit {g}^\ textit {n} $ conv),该卷积{n} $ conv)与封闭的卷积和递归设计执行高阶空间交互。新操作是高度灵活和可定制的,它与卷积的各种变体兼容,并将自我注意的两阶相互作用扩展到任意订单,而无需引入大量额外的计算。 $ \ textit {g}^\ textit {n} $ conv可以用作插件模块,以改善各种视觉变压器和基于卷积的模型。根据该操作,我们构建了一个名为Hornet的新型通用视觉骨干家族。关于ImageNet分类,可可对象检测和ADE20K语义分割的广泛实验表明,大黄蜂的表现优于Swin变形金刚,并具有相似的整体体系结构和训练配置的明显边距。大黄蜂还显示出对更多训练数据和更大模型大小的有利可伸缩性。除了在视觉编码器中的有效性外,我们还可以将$ \ textit {g}^\ textit {n} $ conv应用于特定于任务的解码器,并始终通过较少的计算来提高密集的预测性能。我们的结果表明,$ \ textIt {g}^\ textit {n} $ conv可以成为视觉建模的新基本模块,可有效结合视觉变形金刚和CNN的优点。代码可从https://github.com/raoyongming/hornet获得
translated by 谷歌翻译
在本文中,我们通过利用视觉数据中的空间稀疏性提出了一种新的模型加速方法。我们观察到,视觉变压器中的最终预测仅基于最有用的令牌的子集,这足以使图像识别。基于此观察,我们提出了一个动态的令牌稀疏框架,以根据加速视觉变压器的输入逐渐和动态地修剪冗余令牌。具体而言,我们设计了一个轻量级预测模块,以估计给定当前功能的每个令牌的重要性得分。该模块被添加到不同的层中以层次修剪冗余令牌。尽管该框架的启发是我们观察到视觉变压器中稀疏注意力的启发,但我们发现自适应和不对称计算的想法可能是加速各种体系结构的一般解决方案。我们将我们的方法扩展到包括CNN和分层视觉变压器在内的层次模型,以及更复杂的密集预测任务,这些任务需要通过制定更通用的动态空间稀疏框架,并具有渐进性的稀疏性和非对称性计算,用于不同空间位置。通过将轻质快速路径应用于少量的特征,并使用更具表现力的慢速路径到更重要的位置,我们可以维护特征地图的结构,同时大大减少整体计算。广泛的实验证明了我们框架对各种现代体系结构和不同视觉识别任务的有效性。我们的结果清楚地表明,动态空间稀疏为模型加速提供了一个新的,更有效的维度。代码可从https://github.com/raoyongming/dynamicvit获得
translated by 谷歌翻译
视觉变压器由于能够捕获图像中的长期依赖性的能力而成功地应用于图像识别任务。但是,变压器与现有卷积神经网络(CNN)之间的性能和计算成本仍然存在差距。在本文中,我们旨在解决此问题,并开发一个网络,该网络不仅可以超越规范变压器,而且可以超越高性能卷积模型。我们通过利用变压器来捕获长期依赖性和CNN来建模本地特征,从而提出了一个新的基于变压器的混合网络。此外,我们将其扩展为获得一个称为CMT的模型家族,比以前的基于卷积和基于变压器的模型获得了更好的准确性和效率。特别是,我们的CMT-S在ImageNet上获得了83.5%的TOP-1精度,而在拖鞋上的拖曳率分别比现有的DEIT和EficitiveNet小14倍和2倍。拟议的CMT-S还可以很好地概括CIFAR10(99.2%),CIFAR100(91.7%),花(98.7%)以及其他具有挑战性的视觉数据集,例如可可(44.3%地图),计算成本较小。
translated by 谷歌翻译
Patch-based models, e.g., Vision Transformers (ViTs) and Mixers, have shown impressive results on various visual recognition tasks, alternating classic convolutional networks. While the initial patch-based models (ViTs) treated all patches equally, recent studies reveal that incorporating inductive bias like spatiality benefits the representations. However, most prior works solely focused on the location of patches, overlooking the scene structure of images. Thus, we aim to further guide the interaction of patches using the object information. Specifically, we propose OAMixer (object-aware mixing layer), which calibrates the patch mixing layers of patch-based models based on the object labels. Here, we obtain the object labels in unsupervised or weakly-supervised manners, i.e., no additional human-annotating cost is necessary. Using the object labels, OAMixer computes a reweighting mask with a learnable scale parameter that intensifies the interaction of patches containing similar objects and applies the mask to the patch mixing layers. By learning an object-centric representation, we demonstrate that OAMixer improves the classification accuracy and background robustness of various patch-based models, including ViTs, MLP-Mixers, and ConvMixers. Moreover, we show that OAMixer enhances various downstream tasks, including large-scale classification, self-supervised learning, and multi-object recognition, verifying the generic applicability of OAMixer
translated by 谷歌翻译
近期视觉变压器〜(VIT)模型在各种计算机视觉任务中展示了令人鼓舞的结果,因为他们的竞争力通过自我关注建模图像补丁或令牌的长距离依赖性。然而,这些模型通常指定每层中每个令牌特征的类似场景。这种约束不可避免地限制了每个自我注意层在捕获多尺度特征中的能力,从而导致处理具有不同尺度的多个对象的图像的性能下降。为了解决这个问题,我们提出了一种新颖和通用的策略,称为分流的自我关注〜(SSA),它允许VITS为每个关注层的混合秤的关注进行模拟。 SSA的关键概念是将异构接收领域的尺寸注入令牌:在计算自我注意矩阵之前,它选择性地合并令牌以表示较大的对象特征,同时保持某些令牌以保持细粒度的特征。这种新颖的合并方案能够自我注意,以了解具有不同大小的对象之间的关系,并同时降低令牌数字和计算成本。各种任务的广泛实验表明了SSA的优越性。具体而言,基于SSA的变压器实现了84.0 \%的前1个精度,并且在ImageNet上占据了最先进的焦距变压器,只有一半的模型尺寸和计算成本,并且在Coco上超过了焦点变压器1.3映射2.9 MIOU在ADE20K上类似参数和计算成本。代码已在https://github.com/oliverrensu/shunted-transformer发布。
translated by 谷歌翻译
弱监督的语义分割(WSSS)是具有挑战性的,特别是当使用图像级标签来监督像素级预测时。为了弥合它们的差距,通常生成一个类激活图(CAM)以提供像素级伪标签。卷积神经网络中的凸轮患有部分激活,即,仅激活最多的识别区域。另一方面,基于变压器的方法在探索具有长范围依赖性建模的全球背景下,非常有效,可能会减轻“部分激活”问题。在本文中,我们提出了基于第一变压器的WSSS方法,并介绍了梯度加权元素明智的变压器注意图(GetAn)。 GetaN显示所有特征映射元素的精确激活,跨越变压器层显示对象的不同部分。此外,我们提出了一种激活感知标签完成模块来生成高质量的伪标签。最后,我们将我们的方法纳入了使用双向向上传播的WSS的结束框架。 Pascal VOC和Coco的广泛实验表明,我们的结果通过显着的保证金击败了最先进的端到端方法,并且优于大多数多级方法.M大多数多级方法。
translated by 谷歌翻译
Transformer is a new kind of neural architecture which encodes the input data as powerful features via the attention mechanism. Basically, the visual transformers first divide the input images into several local patches and then calculate both representations and their relationship. Since natural images are of high complexity with abundant detail and color information, the granularity of the patch dividing is not fine enough for excavating features of objects in different scales and locations. In this paper, we point out that the attention inside these local patches are also essential for building visual transformers with high performance and we explore a new architecture, namely, Transformer iN Transformer (TNT). Specifically, we regard the local patches (e.g., 16×16) as "visual sentences" and present to further divide them into smaller patches (e.g., 4×4) as "visual words". The attention of each word will be calculated with other words in the given visual sentence with negligible computational costs. Features of both words and sentences will be aggregated to enhance the representation ability. Experiments on several benchmarks demonstrate the effectiveness of the proposed TNT architecture, e.g., we achieve an 81.5% top-1 accuracy on the ImageNet, which is about 1.7% higher than that of the state-of-the-art visual transformer with similar computational cost.
translated by 谷歌翻译
本文解决了由多头自我注意力(MHSA)中高计算/空间复杂性引起的视觉变压器的低效率缺陷。为此,我们提出了层次MHSA(H-MHSA),其表示以层次方式计算。具体而言,我们首先将输入图像分为通常完成的补丁,每个补丁都被视为令牌。然后,拟议的H-MHSA学习本地贴片中的令牌关系,作为局部关系建模。然后,将小贴片合并为较大的贴片,H-MHSA对少量合并令牌的全局依赖性建模。最后,汇总了本地和全球专注的功能,以获得具有强大表示能力的功能。由于我们仅在每个步骤中计算有限数量的令牌的注意力,因此大大减少了计算负载。因此,H-MHSA可以在不牺牲细粒度信息的情况下有效地模拟令牌之间的全局关系。使用H-MHSA模块合并,我们建立了一个基于层次的变压器网络的家族,即HAT-NET。为了证明在场景理解中HAT-NET的优越性,我们就基本视觉任务进行了广泛的实验,包括图像分类,语义分割,对象检测和实例细分。因此,HAT-NET为视觉变压器提供了新的视角。可以在https://github.com/yun-liu/hat-net上获得代码和预估计的模型。
translated by 谷歌翻译