在混合完成的多任务,多域和多模式数据上进行预训练仍然是视力感知预训练的开放挑战。在本文中,我们提出了GPPF,这是一个普遍的感知预训练框架,预先培训任务级的动态网络,该网络是由在标签的多任务和多域数据集上的各层知识“乐高”组成的。通过检查人类在复杂环境中学习的先天能力,我们识别并将三个关键要素转移到深网上:(1)同时暴露于每个批次中的各种交叉任务和跨域信息。 (2)由知识共享驱动的单独的乐高单元中的分区知识存储。 (3)用于训练和下游任务的乐高单元子集的稀疏激活。值得注意的是,由于其在输入形状,损失功能,输出格式,数据分布等方面的差异,不同视觉任务的联合培训是不平凡的。因此,我们创新地开发了插件的多任务培训算法,该培训算法是支持单个迭代多个任务(SIMT)同时培训。 Simt用大型多任务多任务数据集为预训练的基础奠定了基础,并且被证明对于我们的GPPF实验中的稳定培训至关重要。令人兴奋的是,详尽的实验表明,我们的GPPF-R50型号在GPPF-15M中的8个预训练预培训任务的强大基线上取得了显着改善,并在22个下游任务中收获了一系列SOTA,并具有相似的计算预算。我们还验证了GPPF对SOTA视觉变压器的概括能力,并具有一致的改进。这些可靠的实验结果充分证明了我们新颖的GPPF框架提供的有效的知识学习,存储,共享和转移。
translated by 谷歌翻译
To date, most existing self-supervised learning methods are designed and optimized for image classification. These pre-trained models can be sub-optimal for dense prediction tasks due to the discrepancy between image-level prediction and pixel-level prediction. To fill this gap, we aim to design an effective, dense self-supervised learning method that directly works at the level of pixels (or local features) by taking into account the correspondence between local features. We present dense contrastive learning (DenseCL), which implements self-supervised learning by optimizing a pairwise contrastive (dis)similarity loss at the pixel level between two views of input images.Compared to the baseline method MoCo-v2, our method introduces negligible computation overhead (only <1% slower), but demonstrates consistently superior performance when transferring to downstream dense prediction tasks including object detection, semantic segmentation and instance segmentation; and outperforms the state-of-the-art methods by a large margin. Specifically, over the strong MoCo-v2 baseline, our method achieves significant improvements of 2.0% AP on PASCAL VOC object detection, 1.1% AP on COCO object detection, 0.9% AP on COCO instance segmentation, 3.0% mIoU on PASCAL VOC semantic segmentation and 1.8% mIoU on Cityscapes semantic segmentation.
translated by 谷歌翻译
We present Momentum Contrast (MoCo) for unsupervised visual representation learning. From a perspective on contrastive learning [29] as dictionary look-up, we build a dynamic dictionary with a queue and a moving-averaged encoder. This enables building a large and consistent dictionary on-the-fly that facilitates contrastive unsupervised learning. MoCo provides competitive results under the common linear protocol on ImageNet classification. More importantly, the representations learned by MoCo transfer well to downstream tasks. MoCo can outperform its supervised pre-training counterpart in 7 detection/segmentation tasks on PASCAL VOC, COCO, and other datasets, sometimes surpassing it by large margins. This suggests that the gap between unsupervised and supervised representation learning has been largely closed in many vision tasks.
translated by 谷歌翻译
Building instance segmentation models that are dataefficient and can handle rare object categories is an important challenge in computer vision. Leveraging data augmentations is a promising direction towards addressing this challenge. Here, we perform a systematic study of the Copy-Paste augmentation (e.g., [13,12]) for instance segmentation where we randomly paste objects onto an image. Prior studies on Copy-Paste relied on modeling the surrounding visual context for pasting the objects. However, we find that the simple mechanism of pasting objects randomly is good enough and can provide solid gains on top of strong baselines. Furthermore, we show Copy-Paste is additive with semi-supervised methods that leverage extra data through pseudo labeling (e.g. self-training). On COCO instance segmentation, we achieve 49.1 mask AP and 57.3 box AP, an improvement of +0.6 mask AP and +1.5 box AP over the previous state-of-the-art. We further demonstrate that Copy-Paste can lead to significant improvements on the LVIS benchmark. Our baseline model outperforms the LVIS 2020 Challenge winning entry by +3.6 mask AP on rare categories.
translated by 谷歌翻译
最近的进展表明,使用对比图像文本对的大规模预训练可以是从自然语言监督的高质量视觉表演学习的有前途的替代方案。从更广泛的监督来源受益,这种新的范例展示了对下游分类任务和数据集的令人印象深刻的可转移性。然而,从图像文本对中学习的知识转移到更复杂的密集预测任务的问题几乎没有访问过。在这项工作中,我们通过隐式和明确地利用来自剪辑的预先训练的知识来提出了一种新的密集预测框架。具体地,我们将剪辑中的原始图像文本匹配问题转换为像素文本匹配问题,并使用像素文本分数图来指导致密预测模型的学习。通过进一步使用图像中的上下文信息来提示语言模型,我们能够促进我们的模型来更好地利用预先接受训练的知识。我们的方法是模型 - 不可行的,它可以应用于任意密集的预测系统和各种预先训练的视觉底座,包括夹模型和想象成预先训练的模型。广泛的实验证明了我们对语义分割,对象检测和实例分段任务的方法的卓越性能。代码可在https://github.com/raoyongming/denseclip获得
translated by 谷歌翻译
We propose a new neural network design paradigm Reversible Column Network (RevCol). The main body of RevCol is composed of multiple copies of subnetworks, named columns respectively, between which multi-level reversible connections are employed. Such architectural scheme attributes RevCol very different behavior from conventional networks: during forward propagation, features in RevCol are learned to be gradually disentangled when passing through each column, whose total information is maintained rather than compressed or discarded as other network does. Our experiments suggest that CNN-style RevCol models can achieve very competitive performances on multiple computer vision tasks such as image classification, object detection and semantic segmentation, especially with large parameter budget and large dataset. For example, after ImageNet-22K pre-training, RevCol-XL obtains 88.2% ImageNet-1K accuracy. Given more pre-training data, our largest model RevCol-H reaches 90.0% on ImageNet-1K, 63.8% APbox on COCO detection minival set, 61.0% mIoU on ADE20k segmentation. To our knowledge, it is the best COCO detection and ADE20k segmentation result among pure (static) CNN models. Moreover, as a general macro architecture fashion, RevCol can also be introduced into transformers or other neural networks, which is demonstrated to improve the performances in both computer vision and NLP tasks. We release code and models at https://github.com/megvii-research/RevCol
translated by 谷歌翻译
我们提出了一种适用于半全球任务的自学学习(SSL)方法,例如对象检测和语义分割。我们通过在训练过程中最大程度地减少像素级局部对比度(LC)损失,代表了同一图像转换版本的相应图像位置之间的局部一致性。可以将LC-LOSS添加到以最小开销的现有自我监督学习方法中。我们使用可可,Pascal VOC和CityScapes数据集评估了两个下游任务的SSL方法 - 对象检测和语义细分。我们的方法的表现优于现有的最新SSL方法可可对象检测的方法1.9%,Pascal VOC检测1.4%,而CityScapes Sementation则为0.6%。
translated by 谷歌翻译
The pretrain-finetune paradigm in modern computer vision facilitates the success of self-supervised learning, which tends to achieve better transferability than supervised learning. However, with the availability of massive labeled data, a natural question emerges: how to train a better model with both self and full supervision signals? In this paper, we propose Omni-suPErvised Representation leArning with hierarchical supervisions (OPERA) as a solution. We provide a unified perspective of supervisions from labeled and unlabeled data and propose a unified framework of fully supervised and self-supervised learning. We extract a set of hierarchical proxy representations for each image and impose self and full supervisions on the corresponding proxy representations. Extensive experiments on both convolutional neural networks and vision transformers demonstrate the superiority of OPERA in image classification, segmentation, and object detection. Code is available at: https://github.com/wangck20/OPERA.
translated by 谷歌翻译
Frozen pretrained models have become a viable alternative to the pretraining-then-finetuning paradigm for transfer learning. However, with frozen models there are relatively few parameters available for adapting to downstream tasks, which is problematic in computer vision where tasks vary significantly in input/output format and the type of information that is of value. In this paper, we present a study of frozen pretrained models when applied to diverse and representative computer vision tasks, including object detection, semantic segmentation and video action recognition. From this empirical analysis, our work answers the questions of what pretraining task fits best with this frozen setting, how to make the frozen setting more flexible to various downstream tasks, and the effect of larger model sizes. We additionally examine the upper bound of performance using a giant frozen pretrained model with 3 billion parameters (SwinV2-G) and find that it reaches competitive performance on a varied set of major benchmarks with only one shared frozen base network: 60.0 box mAP and 52.2 mask mAP on COCO object detection test-dev, 57.6 val mIoU on ADE20K semantic segmentation, and 81.7 top-1 accuracy on Kinetics-400 action recognition. With this work, we hope to bring greater attention to this promising path of freezing pretrained image models.
translated by 谷歌翻译
转移学习可以在源任务上重新使用知识来帮助学习目标任务。一种简单的转移学习形式在当前的最先进的计算机视觉模型中是常见的,即预先训练ILSVRC数据集上的图像分类模型,然后在任何目标任务上进行微调。然而,先前对转移学习的系统研究已经有限,并且预计工作的情况并不完全明白。在本文中,我们对跨越不同的图像域进行了广泛的转移学习实验探索(消费者照片,自主驾驶,空中图像,水下,室内场景,合成,特写镜头)和任务类型(语义分割,物体检测,深度估计,关键点检测)。重要的是,这些都是与现代计算机视觉应用相关的复杂的结构化的输出任务类型。总共执行超过2000年的转移学习实验,包括许多来源和目标来自不同的图像域,任务类型或两者。我们系统地分析了这些实验,了解图像域,任务类型和数据集大小对传输学习性能的影响。我们的研究导致了几个见解和具体建议:(1)对于大多数任务,存在一个显着优于ILSVRC'12预培训的来源; (2)图像领域是实现阳性转移的最重要因素; (3)源数据集应该\ \ emph {include}目标数据集的图像域以获得最佳结果; (4)与此同时,当源任务的图像域比目标的图像域时,我们只观察小的负面影响; (5)跨任务类型的转移可能是有益的,但其成功严重依赖于源和目标任务类型。
translated by 谷歌翻译
带有像素天标签的注释图像是耗时和昂贵的过程。最近,DataSetGan展示了有希望的替代方案 - 通过利用一小组手动标记的GaN生成的图像来通过生成的对抗网络(GAN)来综合大型标记数据集。在这里,我们将DataSetGan缩放到ImageNet类别的规模。我们从ImageNet上训练的类条件生成模型中拍摄图像样本,并为所有1K类手动注释每个类的5张图像。通过在Biggan之上培训有效的特征分割架构,我们将Bigan转换为标记的DataSet生成器。我们进一步表明,VQGan可以类似地用作数据集生成器,利用已经注释的数据。我们通过在各种设置中标记一组8K实图像并在各种设置中评估分段性能来创建一个新的想象因基准。通过广泛的消融研究,我们展示了利用大型生成的数据集来培训在像素 - 明智的任务上培训不同的监督和自我监督的骨干模型的大增益。此外,我们证明,使用我们的合成数据集进行预培训,以改善在几个下游数据集上的标准Imagenet预培训,例如Pascal-VOC,MS-Coco,Citycapes和Chink X射线以及任务(检测,细分)。我们的基准将公开并维护一个具有挑战性的任务的排行榜。项目页面:https://nv-tlabs.github.io/big-dataseTgan/
translated by 谷歌翻译
自动视觉解对我们多样化和开放的世界需要计算机视觉模型,以概括为特定任务的最小定制,类似于人类视力。计算机视觉基础型号培训,培训多样化,大型数据集,可以适应各种下游任务,对该任务来解决现实世界计算机视觉应用而言至关重要。虽然现有的视觉基础模型如剪辑,对齐和吴道2.0主要集中在映射图像和文本表示到跨模型共享表示,我们介绍了一台新的计算机视觉基础模型,佛罗伦萨,扩大粗糙的表示(现场)到精细(对象),从静态(图像)到动态(视频),以及从RGB到多个模态(标题,深度)。通过从Web级图像文本数据中纳入通用视觉语言表示,我们的佛罗伦萨模型可以很容易地适应各种计算机视觉任务,例如分类,检索,对象检测,VQA,图像标题,视频检索和动作识别。此外,佛罗伦萨在许多类型的转移学习中表现出出色的表现:全面采样的微调,线性探测,几次射击传输和用于新颖图像和物体的零拍摄传输。所有这些属性对于我们的视觉基础模型至关重要,以提供通用视觉任务。佛罗伦萨实现了新的最先进的导致44个代表性基准,例如Imagenet-1K零射击分类,最高1精度为83.74,最高5个精度为97.18,62.4地图上的Coco微调, 80.36在VQA上,动力学-600上的87.8。
translated by 谷歌翻译
自我监督的对比学习的最新进展产生了良好的图像级表示,这有利于分类任务,但通常会忽略像素级详细信息,从而导致转移性能不令人满意地转移到密集的预测任务,例如语义细分。在这项工作中,我们提出了一种称为CP2的像素对比度学习方法(拷贝性对比度预处理),该方法促进了图像和像素级表示学习,因此更适合下游密集的预测任务。详细说明,我们将随机的作物从图像(前景)复制到不同的背景图像,并为语义分割模型提供了以1)为目标的语义分割模型。共享相同的前景。表现出色表明CP2在下游语义分段中的表现强劲:通过对Pascal VOC 2012上的CP2预审计的模型,我们获得了78.6%MIOU,具有RESNET-50和79.5%的vit-s。
translated by 谷歌翻译
我们通过以端到端的方式对大规模未标记的数据集进行分类,呈现扭曲,简单和理论上可解释的自我监督的表示学习方法。我们使用Softmax操作终止的暹罗网络,以产生两个增强图像的双类分布。没有监督,我们强制执行不同增强的班级分布。但是,只需最小化增强之间的分歧将导致折叠解决方案,即,输出所有图像的相同类概率分布。在这种情况下,留下有关输入图像的信息。为了解决这个问题,我们建议最大化输入和课程预测之间的互信息。具体地,我们最小化每个样品的分布的熵,使每个样品的课程预测是对每个样品自信的预测,并最大化平均分布的熵,以使不同样品的预测变得不同。以这种方式,扭曲可以自然地避免没有特定设计的折叠解决方案,例如非对称网络,停止梯度操作或动量编码器。因此,扭曲优于各种任务的最先进的方法。特别是,在半监督学习中,扭曲令人惊讶地表现出令人惊讶的是,使用Reset-50作为骨干的1%ImageNet标签实现61.2%的顶级精度,以前的最佳结果为6.2%。代码和预先训练的模型是给出的:https://github.com/byteDance/twist
translated by 谷歌翻译
Driven by improved architectures and better representation learning frameworks, the field of visual recognition has enjoyed rapid modernization and performance boost in the early 2020s. For example, modern ConvNets, represented by ConvNeXt, have demonstrated strong performance in various scenarios. While these models were originally designed for supervised learning with ImageNet labels, they can also potentially benefit from self-supervised learning techniques such as masked autoencoders (MAE). However, we found that simply combining these two approaches leads to subpar performance. In this paper, we propose a fully convolutional masked autoencoder framework and a new Global Response Normalization (GRN) layer that can be added to the ConvNeXt architecture to enhance inter-channel feature competition. This co-design of self-supervised learning techniques and architectural improvement results in a new model family called ConvNeXt V2, which significantly improves the performance of pure ConvNets on various recognition benchmarks, including ImageNet classification, COCO detection, and ADE20K segmentation. We also provide pre-trained ConvNeXt V2 models of various sizes, ranging from an efficient 3.7M-parameter Atto model with 76.7% top-1 accuracy on ImageNet, to a 650M Huge model that achieves a state-of-the-art 88.9% accuracy using only public training data.
translated by 谷歌翻译
为了同时朝着对多个下游任务的整体理解,需要提取具有更好可传递性的功能。尽管许多最新的自我监管的预训练方法在普遍的预处理前范式下在各种视觉任务上取得了令人印象深刻的表现,但它们对多任务学习方案的概括能力尚待探索。在本文中,我们在三个下游任务上进行了广泛研究各种类型的自我监督方法的转移性能,例如Moco和Simc​​lr,包括语义细分,可驱动的区域细分和交通对象检测,在大规模驾驶数据集中BDD100K。我们出人意料地发现,他们的表现是最佳的甚至落后于单任务基线的滞后,这可能是由于训练目标和建筑设计的区别在于预处理范式。为了克服这一难题,并避免重新设计资源密集的预培训阶段,我们提出了一种简单而有效的预处理 - 适应性 - 赛范围,用于一般的多任务培训,可以有效地适应现行预审预周态的模型没有增加培训开销。在自适应阶段,我们利用可学习的多尺度适配器来动态调整由多任务目标监督的预验证的模型权重,同时使经过预告片的知识未经触及。此外,我们将视觉语言预训练模型剪辑视为对预处理 - 适应 - 最终范式的强烈补充,并提出了一个名为LV-Adapter的新型适配器,该适配器通过任务特定的提示将语言先验纳入了多任务的模型中和视觉和文本特征之间的对齐。
translated by 谷歌翻译
We launch EVA, a vision-centric foundation model to explore the limits of visual representation at scale using only publicly accessible data. EVA is a vanilla ViT pre-trained to reconstruct the masked out image-text aligned vision features conditioned on visible image patches. Via this pretext task, we can efficiently scale up EVA to one billion parameters, and sets new records on a broad range of representative vision downstream tasks, such as image recognition, video action recognition, object detection, instance segmentation and semantic segmentation without heavy supervised training. Moreover, we observe quantitative changes in scaling EVA result in qualitative changes in transfer learning performance that are not present in other models. For instance, EVA takes a great leap in the challenging large vocabulary instance segmentation task: our model achieves almost the same state-of-the-art performance on LVISv1.0 dataset with over a thousand categories and COCO dataset with only eighty categories. Beyond a pure vision encoder, EVA can also serve as a vision-centric, multi-modal pivot to connect images and text. We find initializing the vision tower of a giant CLIP from EVA can greatly stabilize the training and outperform the training from scratch counterpart with much fewer samples and less compute, providing a new direction for scaling up and accelerating the costly training of multi-modal foundation models. To facilitate future research, we release all the code and models at https://github.com/baaivision/EVA.
translated by 谷歌翻译
本文介绍了密集的暹罗网络(Denseiam),这是一个简单的无监督学习框架,用于密集的预测任务。它通过以两种类型的一致性(即像素一致性和区域一致性)之间最大化一个图像的两个视图之间的相似性来学习视觉表示。具体地,根据重叠区域中的确切位置对应关系,Denseiam首先最大化像素级的空间一致性。它还提取一批与重叠区域中某些子区域相对应的区域嵌入,以形成区域一致性。与以前需要负像素对,动量编码器或启发式面膜的方法相反,Denseiam受益于简单的暹罗网络,并优化了不同粒度的一致性。它还证明了简单的位置对应关系和相互作用的区域嵌入足以学习相似性。我们将Denseiam应用于ImageNet,并在各种下游任务上获得竞争性改进。我们还表明,只有在一些特定于任务的损失中,简单的框架才能直接执行密集的预测任务。在现有的无监督语义细分基准中,它以2.1 miou的速度超过了最新的细分方法,培训成本为28%。代码和型号在https://github.com/zwwwayne/densesiam上发布。
translated by 谷歌翻译
Jitendra Malik once said, "Supervision is the opium of the AI researcher". Most deep learning techniques heavily rely on extreme amounts of human labels to work effectively. In today's world, the rate of data creation greatly surpasses the rate of data annotation. Full reliance on human annotations is just a temporary means to solve current closed problems in AI. In reality, only a tiny fraction of data is annotated. Annotation Efficient Learning (AEL) is a study of algorithms to train models effectively with fewer annotations. To thrive in AEL environments, we need deep learning techniques that rely less on manual annotations (e.g., image, bounding-box, and per-pixel labels), but learn useful information from unlabeled data. In this thesis, we explore five different techniques for handling AEL.
translated by 谷歌翻译
自我监督学习(SSL)的承诺是利用大量未标记的数据来解决复杂的任务。尽管简单,图像级学习取得了出色的进步,但最新方法显示出包括图像结构知识的优势。但是,通过引入手工制作的图像分割来定义感兴趣的区域或专门的增强策略,这些方法牺牲了使SSL如此强大的简单性和通用性。取而代之的是,我们提出了一个自我监督的学习范式,该学习范式本身会发现这种图像结构。我们的方法,ODIN,夫妻对象发现和表示网络,以发现有意义的图像分割,而无需任何监督。由此产生的学习范式更简单,更易碎,更一般,并且取得了最先进的转移学习结果,以进行对象检测和实例对可可的细分,以及对Pascal和CityScapes的语义细分,同时超过监督的预先培训,用于戴维斯的视频细分。
translated by 谷歌翻译