As a powerful engine, vanilla convolution has promoted huge breakthroughs in various computer tasks. However, it often suffers from sample and content agnostic problems, which limits the representation capacities of the convolutional neural networks (CNNs). In this paper, we for the first time model the scene features as a combination of the local spatial-adaptive parts owned by the individual and the global shift-invariant parts shared to all individuals, and then propose a novel two-branch dual complementary dynamic convolution (DCDC) operator to flexibly deal with these two types of features. The DCDC operator overcomes the limitations of vanilla convolution and most existing dynamic convolutions who capture only spatial-adaptive features, and thus markedly boosts the representation capacities of CNNs. Experiments show that the DCDC operator based ResNets (DCDC-ResNets) significantly outperform vanilla ResNets and most state-of-the-art dynamic convolutional networks on image classification, as well as downstream tasks including object detection, instance and panoptic segmentation tasks, while with lower FLOPs and parameters.
translated by 谷歌翻译
现有的多尺度解决方案会导致仅增加接受场大小的风险,同时忽略小型接受场。因此,有效构建自适应神经网络以识别各种空间尺度对象是一个具有挑战性的问题。为了解决这个问题,我们首先引入一个新的注意力维度,即除了现有的注意力维度(例如渠道,空间和分支)之外,并提出了一个新颖的选择性深度注意网络,以对称地处理各种视觉中的多尺度对象任务。具体而言,在给定神经网络的每个阶段内的块,即重新连接,输出层次功能映射共享相同的分辨率但具有不同的接收场大小。基于此结构属性,我们设计了一个舞台建筑模块,即SDA,其中包括树干分支和类似SE的注意力分支。躯干分支的块输出融合在一起,以通过注意力分支指导其深度注意力分配。根据提出的注意机制,我们可以动态选择不同的深度特征,这有助于自适应调整可变大小输入对象的接收场大小。这样,跨块信息相互作用会导致沿深度方向的远距离依赖关系。与其他多尺度方法相比,我们的SDA方法结合了从以前的块到舞台输出的多个接受场,从而提供了更广泛,更丰富的有效接收场。此外,我们的方法可以用作其他多尺度网络以及注意力网络的可插入模块,并创造为SDA- $ x $ net。它们的组合进一步扩展了有效的接受场的范围,可以实现可解释的神经网络。我们的源代码可在\ url {https://github.com/qingbeiguo/sda-xnet.git}中获得。
translated by 谷歌翻译
虽然最初是为自然语言处理任务而设计的,但自我发挥的机制最近逐渐席卷了各种计算机视觉领域。但是,图像的2D性质带来了在计算机视觉中应用自我注意力的三个挑战。 (1)将图像作为1D序列忽略了其2D结构。 (2)对于高分辨率图像而言,二次复杂性太贵了。 (3)它仅捕获空间适应性,但忽略了通道适应性。在本文中,我们提出了一种新颖的线性注意力,名为“大核心注意”(LKA),以使自适应和远程相关性在自我注意力中避免其缺点。此外,我们提出了基于LKA的神经网络,即视觉注意力网络(VAN)。虽然非常简单,但范超过了相似的大小视觉变压器(VIT)和各种任务中的卷积神经网络(CNN),包括图像分类,对象检测,语义细分,泛型分割,姿势估计等。 ImageNet基准测试的精度为%,并为全景分割设置新的最先进性能(58.2 PQ)。此外,Van-B2超过Sw​​in-T 4%MIOU(50.1 vs. 46.1),用于ADE20K基准上的语义分割,2.6%AP(48.8 vs. 46.2)在COCO数据集上进行对象检测。它为社区提供了一种新颖的方法和简单而强大的基线。代码可从https://github.com/visual-crestention-network获得。
translated by 谷歌翻译
We propose focal modulation networks (FocalNets in short), where self-attention (SA) is completely replaced by a focal modulation mechanism for modeling token interactions in vision. Focal modulation comprises three components: (i) hierarchical contextualization, implemented using a stack of depth-wise convolutional layers, to encode visual contexts from short to long ranges, (ii) gated aggregation to selectively gather contexts for each query token based on its content, and (iii) element-wise modulation or affine transformation to inject the aggregated context into the query. Extensive experiments show FocalNets outperform the state-of-the-art SA counterparts (e.g., Swin and Focal Transformers) with similar computational costs on the tasks of image classification, object detection, and segmentation. Specifically, FocalNets with tiny and base size achieve 82.3% and 83.9% top-1 accuracy on ImageNet-1K. After pretrained on ImageNet-22K in 224 resolution, it attains 86.5% and 87.3% top-1 accuracy when finetuned with resolution 224 and 384, respectively. When transferred to downstream tasks, FocalNets exhibit clear superiority. For object detection with Mask R-CNN, FocalNet base trained with 1\times outperforms the Swin counterpart by 2.1 points and already surpasses Swin trained with 3\times schedule (49.0 v.s. 48.5). For semantic segmentation with UPerNet, FocalNet base at single-scale outperforms Swin by 2.4, and beats Swin at multi-scale (50.5 v.s. 49.7). Using large FocalNet and Mask2former, we achieve 58.5 mIoU for ADE20K semantic segmentation, and 57.9 PQ for COCO Panoptic Segmentation. Using huge FocalNet and DINO, we achieved 64.3 and 64.4 mAP on COCO minival and test-dev, respectively, establishing new SoTA on top of much larger attention-based models like Swinv2-G and BEIT-3. Code and checkpoints are available at https://github.com/microsoft/FocalNet.
translated by 谷歌翻译
Representing features at multiple scales is of great importance for numerous vision tasks. Recent advances in backbone convolutional neural networks (CNNs) continually demonstrate stronger multi-scale representation ability, leading to consistent performance gains on a wide range of applications. However, most existing methods represent the multi-scale features in a layerwise manner. In this paper, we propose a novel building block for CNNs, namely Res2Net, by constructing hierarchical residual-like connections within one single residual block. The Res2Net represents multi-scale features at a granular level and increases the range of receptive fields for each network layer. The proposed Res2Net block can be plugged into the state-of-the-art backbone CNN models, e.g., ResNet, ResNeXt, and DLA. We evaluate the Res2Net block on all these models and demonstrate consistent performance gains over baseline models on widely-used datasets, e.g., CIFAR-100 and ImageNet. Further ablation studies and experimental results on representative computer vision tasks, i.e., object detection, class activation mapping, and salient object detection, further verify the superiority of the Res2Net over the state-of-the-art baseline methods. The source code and trained models are available on https://mmcheng.net/res2net/.
translated by 谷歌翻译
建模长期依赖关系对于理解计算机视觉中的任务至关重要。尽管卷积神经网络(CNN)在许多视觉任务中都表现出色,但由于它们通常由当地核层组成,因此它们仍然限制捕获长期结构化关系。但是,完全连接的图(例如变形金刚中的自我发项操作)对这种建模是有益的,但是,其计算开销非常有用。在本文中,我们提出了一个动态图形消息传递网络,与建模完全连接的图形相比,该网络大大降低了计算复杂性。这是通过在图表中自适应采样节点(以输入为条件)来实现的,以传递消息传递。基于采样节点,我们动态预测节点依赖性滤波器权重和亲和力矩阵,以在它们之间传播信息。这种公式使我们能够设计一个自我发挥的模块,更重要的是,我们将基于变压器的新骨干网络用于图像分类预处理,并用于解决各种下游任务(对象检测,实例和语义细分)。使用此模型,我们在四个不同任务上的强,最先进的基线方面显示出显着改进。我们的方法还优于完全连接的图形,同时使用较少的浮点操作和参数。代码和型号将在https://github.com/fudan-zvg/dgmn2上公开提供。
translated by 谷歌翻译
Since the recent success of Vision Transformers (ViTs), explorations toward transformer-style architectures have triggered the resurgence of modern ConvNets. In this work, we explore the representation ability of DNNs through the lens of interaction complexities. We empirically show that interaction complexity is an overlooked but essential indicator for visual recognition. Accordingly, a new family of efficient ConvNets, named MogaNet, is presented to pursue informative context mining in pure ConvNet-based models, with preferable complexity-performance trade-offs. In MogaNet, interactions across multiple complexities are facilitated and contextualized by leveraging two specially designed aggregation blocks in both spatial and channel interaction spaces. Extensive studies are conducted on ImageNet classification, COCO object detection, and ADE20K semantic segmentation tasks. The results demonstrate that our MogaNet establishes new state-of-the-art over other popular methods in mainstream scenarios and all model scales. Typically, the lightweight MogaNet-T achieves 80.0\% top-1 accuracy with only 1.44G FLOPs using a refined training setup on ImageNet-1K, surpassing ParC-Net-S by 1.4\% accuracy but saving 59\% (2.04G) FLOPs.
translated by 谷歌翻译
本文解决了由多头自我注意力(MHSA)中高计算/空间复杂性引起的视觉变压器的低效率缺陷。为此,我们提出了层次MHSA(H-MHSA),其表示以层次方式计算。具体而言,我们首先将输入图像分为通常完成的补丁,每个补丁都被视为令牌。然后,拟议的H-MHSA学习本地贴片中的令牌关系,作为局部关系建模。然后,将小贴片合并为较大的贴片,H-MHSA对少量合并令牌的全局依赖性建模。最后,汇总了本地和全球专注的功能,以获得具有强大表示能力的功能。由于我们仅在每个步骤中计算有限数量的令牌的注意力,因此大大减少了计算负载。因此,H-MHSA可以在不牺牲细粒度信息的情况下有效地模拟令牌之间的全局关系。使用H-MHSA模块合并,我们建立了一个基于层次的变压器网络的家族,即HAT-NET。为了证明在场景理解中HAT-NET的优越性,我们就基本视觉任务进行了广泛的实验,包括图像分类,语义分割,对象检测和实例细分。因此,HAT-NET为视觉变压器提供了新的视角。可以在https://github.com/yun-liu/hat-net上获得代码和预估计的模型。
translated by 谷歌翻译
视觉变压器的最新进展在基于点产生自我注意的新空间建模机制驱动的各种任务中取得了巨大成功。在本文中,我们表明,视觉变压器背后的关键要素,即输入自适应,远程和高阶空间相互作用,也可以通过基于卷积的框架有效地实现。我们介绍了递归封闭式卷积($ \ textit {g}^\ textit {n} $ conv),该卷积{n} $ conv)与封闭的卷积和递归设计执行高阶空间交互。新操作是高度灵活和可定制的,它与卷积的各种变体兼容,并将自我注意的两阶相互作用扩展到任意订单,而无需引入大量额外的计算。 $ \ textit {g}^\ textit {n} $ conv可以用作插件模块,以改善各种视觉变压器和基于卷积的模型。根据该操作,我们构建了一个名为Hornet的新型通用视觉骨干家族。关于ImageNet分类,可可对象检测和ADE20K语义分割的广泛实验表明,大黄蜂的表现优于Swin变形金刚,并具有相似的整体体系结构和训练配置的明显边距。大黄蜂还显示出对更多训练数据和更大模型大小的有利可伸缩性。除了在视觉编码器中的有效性外,我们还可以将$ \ textit {g}^\ textit {n} $ conv应用于特定于任务的解码器,并始终通过较少的计算来提高密集的预测性能。我们的结果表明,$ \ textIt {g}^\ textit {n} $ conv可以成为视觉建模的新基本模块,可有效结合视觉变形金刚和CNN的优点。代码可从https://github.com/raoyongming/hornet获得
translated by 谷歌翻译
Channel and spatial attention mechanism has proven to provide an evident performance boost of deep convolution neural networks (CNNs). Most existing methods focus on one or run them parallel (series), neglecting the collaboration between the two attentions. In order to better establish the feature interaction between the two types of attention, we propose a plug-and-play attention module, which we term "CAT"-activating the Collaboration between spatial and channel Attentions based on learned Traits. Specifically, we represent traits as trainable coefficients (i.e., colla-factors) to adaptively combine contributions of different attention modules to fit different image hierarchies and tasks better. Moreover, we propose the global entropy pooling (GEP) apart from global average pooling (GAP) and global maximum pooling (GMP) operators, an effective component in suppressing noise signals by measuring the information disorder of feature maps. We introduce a three-way pooling operation into attention modules and apply the adaptive mechanism to fuse their outcomes. Extensive experiments on MS COCO, Pascal-VOC, Cifar-100, and ImageNet show that our CAT outperforms existing state-of-the-art attention mechanisms in object detection, instance segmentation, and image classification. The model and code will be released soon.
translated by 谷歌翻译
最近,Vision Transformer通过推动各种视觉任务的最新技术取得了巨大的成功。视觉变压器中最具挑战性的问题之一是,图像令牌的较大序列长度会导致高计算成本(二次复杂性)。解决此问题的一个流行解决方案是使用单个合并操作来减少序列长度。本文考虑如何改善现有的视觉变压器,在这种变压器中,单个合并操作提取的合并功能似乎不太强大。为此,我们注意到,由于其在上下文抽象中的强大能力,金字塔池在各种视觉任务中已被证明是有效的。但是,在骨干网络设计中尚未探索金字塔池。为了弥合这一差距,我们建议在视觉变压器中将金字塔池汇总到多头自我注意力(MHSA)中,同时降低了序列长度并捕获强大的上下文特征。我们插入了基于池的MHSA,我们构建了一个通用视觉变压器主链,称为金字塔池变压器(P2T)。广泛的实验表明,与先前的基于CNN-和基于变压器的网络相比,当将P2T用作骨干网络时,它在各种视觉任务中显示出很大的优势。该代码将在https://github.com/yuhuan-wu/p2t上发布。
translated by 谷歌翻译
建模长期依赖关系对于理解计算机视觉中的任务至关重要。尽管CNN在许多视觉任务中都表现出色,但由于它们通常由当地核层组成,因此它们仍然受到捕获长期结构关系的限制。完全连接的图对这种建模是有益的,但是,其计算开销是高度的。我们提出了一个动态图形消息传递网络,与建模完全连接的图形相比,该网络大大降低了计算复杂性。这是通过在图表中自适应采样节点(以输入为条件)来实现的,以传递消息传递。基于采样节点,我们动态预测节点依赖性滤波器权重和亲和力矩阵,以在它们之间传播信息。使用此模型,我们在三个不同的任务和骨干体系结构方面显示了强大的,最先进的基线的显着改进。我们的方法还优于完全连接的图形,同时使用较少的浮点操作和参数。项目网站是http://www.robots.ox.ac.uk/~lz/dgmn/
translated by 谷歌翻译
The three existing dominant network families, i.e., CNNs, Transformers, and MLPs, differ from each other mainly in the ways of fusing spatial contextual information, leaving designing more effective token-mixing mechanisms at the core of backbone architecture development. In this work, we propose an innovative token-mixer, dubbed Active Token Mixer (ATM), to actively incorporate flexible contextual information distributed across different channels from other tokens into the given query token. This fundamental operator actively predicts where to capture useful contexts and learns how to fuse the captured contexts with the query token at channel level. In this way, the spatial range of token-mixing can be expanded to a global scope with limited computational complexity, where the way of token-mixing is reformed. We take ATM as the primary operator and assemble ATMs into a cascade architecture, dubbed ATMNet. Extensive experiments demonstrate that ATMNet is generally applicable and comprehensively surpasses different families of SOTA vision backbones by a clear margin on a broad range of vision tasks, including visual recognition and dense prediction tasks. Code is available at https://github.com/microsoft/ActiveMLP.
translated by 谷歌翻译
诸如对象检测和分割等密集的计算机视觉任务需要有效的多尺度特征表示,用于检测或分类具有不同大小的对象或区域。虽然卷积神经网络(CNNS)是这种任务的主导架构,但最近引入了视觉变压器(VITS)的目标是将它们替换为骨干。类似于CNN,VITS构建一个简单的多级结构(即,细致粗略),用于使用单尺度补丁进行多尺度表示。在这项工作中,通过从现有变压器的不同角度来看,我们探索了多尺度补丁嵌入和多路径结构,构建了多路径视觉变压器(MPVIT)。 MPVIT通过使用重叠的卷积贴片嵌入,将相同尺寸〜(即,序列长度,序列长度,序列长度的序列长度)嵌入不同尺度的斑块。然后,通过多个路径独立地将不同尺度的令牌独立地馈送到变压器编码器,并且可以聚合产生的特征,使得能够在相同特征级别的精细和粗糙的特征表示。由于多样化,多尺寸特征表示,我们的MPVits从微小〜(5m)缩放到基础〜(73米)一直在想象成分,对象检测,实例分段上的最先进的视觉变压器来实现卓越的性能,和语义细分。这些广泛的结果表明,MPVIT可以作为各种视觉任务的多功能骨干网。代码将在\ url {https://git.io/mpvit}上公开可用。
translated by 谷歌翻译
In this paper, we aim to design an efficient real-time object detector that exceeds the YOLO series and is easily extensible for many object recognition tasks such as instance segmentation and rotated object detection. To obtain a more efficient model architecture, we explore an architecture that has compatible capacities in the backbone and neck, constructed by a basic building block that consists of large-kernel depth-wise convolutions. We further introduce soft labels when calculating matching costs in the dynamic label assignment to improve accuracy. Together with better training techniques, the resulting object detector, named RTMDet, achieves 52.8% AP on COCO with 300+ FPS on an NVIDIA 3090 GPU, outperforming the current mainstream industrial detectors. RTMDet achieves the best parameter-accuracy trade-off with tiny/small/medium/large/extra-large model sizes for various application scenarios, and obtains new state-of-the-art performance on real-time instance segmentation and rotated object detection. We hope the experimental results can provide new insights into designing versatile real-time object detectors for many object recognition tasks. Code and models are released at https://github.com/open-mmlab/mmdetection/tree/3.x/configs/rtmdet.
translated by 谷歌翻译
In standard Convolutional Neural Networks (CNNs), the receptive fields of artificial neurons in each layer are designed to share the same size. It is well-known in the neuroscience community that the receptive field size of visual cortical neurons are modulated by the stimulus, which has been rarely considered in constructing CNNs. We propose a dynamic selection mechanism in CNNs that allows each neuron to adaptively adjust its receptive field size based on multiple scales of input information. A building block called Selective Kernel (SK) unit is designed, in which multiple branches with different kernel sizes are fused using softmax attention that is guided by the information in these branches. Different attentions on these branches yield different sizes of the effective receptive fields of neurons in the fusion layer. Multiple SK units are stacked to a deep network termed Selective Kernel Networks (SKNets). On the ImageNet and CIFAR benchmarks, we empirically show that SKNet outperforms the existing state-of-the-art architectures with lower model complexity. Detailed analyses show that the neurons in SKNet can capture target objects with different scales, which verifies the capability of neurons for adaptively adjusting their receptive field sizes according to the input. The code and models are available at https://github.com/implus/SKNet.
translated by 谷歌翻译
卷积神经网络(CNN)不仅被广泛普及,而且在包括图像分类,恢复和生成在内的许多应用中都取得了明显的结果。尽管卷积的重量共享特性使它们在各种任务中被广泛采用,但其内容不足的特征也可以视为主要缺点。为了解决这个问题,在本文中,我们提出了一个新型操作,称为Pixel自适应核(PAKA)。 Paka通过从可学习的功能中乘以空间变化的注意力来提供对滤波器重量的方向性。所提出的方法会沿通道和空间方向分别渗入像素自适应的注意图,以使用较少的参数来解决分解模型。我们的方法可以以端到端的方式训练,并且适用于任何基于CNN的模型。此外,我们建议使用PAKA改进的信息聚合模块,称为层次PAKA模块(HPM)。与常规信息聚合模块相比,我们通过在语义细分方面提出最先进的性能来证明HPM的优势。我们通过其他消融研究来验证提出的方法,并可视化PAKA的效果,从而为卷积的权重提供了方向性。我们还通过将其应用于多模式任务,尤其是颜色引导的深度图超分辨率来显示该方法的普遍性。
translated by 谷歌翻译
模型的时间/空间接受场在顺序/空间任务中起重要作用。大型接受场有助于长期关系,而小型接受场有助于捕获当地的细节。现有方法构建具有手工设计的接收场的模型。我们可以有效地搜索接收场合组合以取代手工设计的模式吗?为了回答这个问题,我们建议通过全球到本地搜索方案找到更好的接受现场组合。我们的搜索方案利用了全局搜索以找到粗糙的组合和本地搜索,以进一步获得精致的接收场组合。全球搜索发现除了人类设计的模式以外的其他可能的粗糙组合。除全球搜索外,我们提出了一种期望引导的迭代局部搜索方案,以有效地完善组合。我们的RF-NEXT模型,将接受现场搜索插入各种模型,提高许多任务的性能,例如时间动作分割,对象检测,实例分割和语音综合。源代码可在http://mmcheng.net/rfnext上公开获得。
translated by 谷歌翻译
大多数现有的深神经网络都是静态的,这意味着它们只能以固定的复杂性推断。但资源预算可以大幅度不同。即使在一个设备上,实惠预算也可以用不同的场景改变,并且对每个所需预算的反复培训网络是非常昂贵的。因此,在这项工作中,我们提出了一种称为Mutualnet的一般方法,以训练可以以各种资源约束运行的单个网络。我们的方法列举了具有各种网络宽度和输入分辨率的模型配置队列。这种相互学习方案不仅允许模型以不同的宽度分辨率配置运行,而且还可以在这些配置之间传输独特的知识,帮助模型来学习更强大的表示。 Mutualnet是一般的培训方法,可以应用于各种网络结构(例如,2D网络:MobileNets,Reset,3D网络:速度,X3D)和各种任务(例如,图像分类,对象检测,分段和动作识别),并证明了实现各种数据集的一致性改进。由于我们只培训了这一模型,它对独立培训多种型号而言,它也大大降低了培训成本。令人惊讶的是,如果动态资源约束不是一个问题,则可以使用Mutualnet来显着提高单个网络的性能。总之,Mutualnet是静态和自适应,2D和3D网络的统一方法。代码和预先训练的模型可用于\ url {https://github.com/tayang1122/mutualnet}。
translated by 谷歌翻译
在每个卷积层中学习一个静态卷积内核是现代卷积神经网络(CNN)的常见训练范式。取而代之的是,动态卷积的最新研究表明,学习$ n $卷积核与输入依赖性注意的线性组合可以显着提高轻重量CNN的准确性,同时保持有效的推断。但是,我们观察到现有的作品endow卷积内核具有通过一个维度(关于卷积内核编号)的动态属性(关于内核空间的卷积内核编号),但其他三个维度(关于空间大小,输入通道号和输出通道编号和输出通道号,每个卷积内核)被忽略。受到这一点的启发,我们提出了Omni维动态卷积(ODCONV),这是一种更普遍而优雅的动态卷积设计,以推进这一研究。 ODCONV利用了一种新型的多维注意机制,采用平行策略来学习沿着任何卷积层的内核空间的所有四个维度学习卷积内核的互补关注。作为定期卷积的倒数替换,可以将ODCONV插入许多CNN架构中。 ImageNet和MS-Coco数据集的广泛实验表明,ODCONV为包括轻量重量和大型的各种盛行的CNN主链带来了可靠的准确性提升,例如3.77%〜5.71%| 1.86%〜3.72%〜3.72%的绝对1个绝对1改进至ImabivLenetV2 | ImageNet数据集上的重新连接家族。有趣的是,由于其功能学习能力的提高,即使具有一个单个内核的ODCONV也可以与具有多个内核的现有动态卷积对应物竞争或超越现有的动态卷积对应物,从而大大降低了额外的参数。此外,ODCONV也优于其他注意模块,用于调节输出特征或卷积重量。
translated by 谷歌翻译