Learning image representations using synthetic data allows training neural networks without some of the concerns associated with real images, such as privacy and bias. Existing work focuses on a handful of curated generative processes which require expert knowledge to design, making it hard to scale up. To overcome this, we propose training with a large dataset of twenty-one thousand programs, each one generating a diverse set of synthetic images. These programs are short code snippets, which are easy to modify and fast to execute using OpenGL. The proposed dataset can be used for both supervised and unsupervised representation learning, and reduces the gap between pre-training with real and procedurally generated images by 38%.
translated by 谷歌翻译
在Imagenet或其他大规模数据数据上的预培训模型导致计算机愿景的主要进步,尽管伴随着与策划成本,隐私,使用权和道德问题相关的缺点。在本文中,我们首次研究了基于由图形模拟器生成的合成数据到来自非常不同的域的下游任务的培训模型的可转换性。在使用此类合成数据进行预培训时,我们发现不同任务的下游性能受到不同配置的不同配置(例如,照明,对象姿势,背景等),并且没有单尺寸适合 - 所有解决方案。因此,更好地将合成的预训练数据量身定制到特定的下游任务,以获得最佳性能。我们介绍Task2SIM,一个统一的模型将下游任务表示映射到最佳模拟参数,以为它们生成合成的预训练数据。 Task2SIM通过培训学习此映射,以查找一组“看到”任务上的最佳参数集。曾经训练过,它可以用于预测一个新颖的“看不见”任务的最佳仿真参数,而无需额外的培训。鉴于每级图像数量的预算,我们具有20个不同的下游任务的广泛实验,显示了Task2SIM的任务 - 自适应预训练数据导致明显更好的下游性能,而不是在看见和看不见的任务上的非自适应选择模拟参数。它甚至是竞争对手的真实图像的竞争力。
translated by 谷歌翻译
大多数现有的工作在几次学习中,依赖于Meta-Learning网络在大型基础数据集上,该网络通常是与目标数据集相同的域。我们解决了跨域几秒钟的问题,其中基础和目标域之间存在大移位。与未标记的目标数据的跨域几秒识别问题在很大程度上在文献中毫无根据。启动是使用自我训练解决此问题的第一个方法。但是,它使用固定的老师在标记的基础数据集上返回,以为未标记的目标样本创建软标签。由于基本数据集和未标记的数据集来自不同的域,因此将基本数据集的类域中的目标图像投影,具有固定的预制模型可能是子最优的。我们提出了一种简单的动态蒸馏基方法,以方便来自新颖/基础数据集的未标记图像。我们通过从教师网络中的未标记图像的未标记版本的预测计算并将其与来自学生网络相同的相同图像的强大版本匹配来施加一致性正常化。教师网络的参数被更新为学生网络参数的指数移动平均值。我们表明所提出的网络了解可以轻松适应目标域的表示,即使它尚未在预先预测阶段的目标专用类别训练。我们的车型优于当前最先进的方法,在BSCD-FSL基准中的5次分类,3.6%的3.6%,并在传统的域名几枪学习任务中显示出竞争性能。
translated by 谷歌翻译
vision变压器(VIT)最近在图像分类上实现了对卷积神经网络(CNNS)的可比结果的强大能力。然而,Vanilla Vit只是直接从自然语言处理继承相同的架构,这通常不会针对视觉应用进行优化。在这篇文章的推动中,我们提出了一种采用金字塔结构的新架构,并在视觉变压器中采用新的区域到局部关注,而不是全球自我关注。更具体地,我们的模型首先从具有不同补丁大小的图像生成区域令牌和本地标记,其中每个区域令牌与基于空间位置的一组本地代币相关联。区域到当地的注意力包括两个步骤:第一,区域自我关注提取所有区域代币之间的全球信息,然后通过自我关注将局部自我关注与相关的本地代币之间的信息交换。因此,尽管局部自我关注限制了当地区域的范围,但它仍然可以接收全球信息。在四个视觉任务中进行广泛的实验,包括图像分类,对象和关键点检测,语义分割和动作识别,表明我们的方法优于或与最先进的Vit变体(包括许多并发作品)的差异。我们的源代码和模型可在https://github.com/ibm/regionvit上使用。
translated by 谷歌翻译
The recently developed vision transformer (ViT) has achieved promising results on image classification compared to convolutional neural networks. Inspired by this, in this paper, we study how to learn multi-scale feature representations in transformer models for image classification. To this end, we propose a dual-branch transformer to combine image patches (i.e., tokens in a transformer) of different sizes to produce stronger image features. Our approach processes small-patch and large-patch tokens with two separate branches of different computational complexity and these tokens are then fused purely by attention multiple times to complement each other. Furthermore, to reduce computation, we develop a simple yet effective token fusion module based on cross attention, which uses a single token for each branch as a query to exchange information with other branches. Our proposed cross-attention only requires linear time for both computational and memory complexity instead of quadratic time otherwise. Extensive experiments demonstrate that our approach performs better than or on par with several concurrent works on vision transformer, in addition to efficient CNN models. For example, on the ImageNet1K dataset, with some architectural changes, our approach outperforms the recent DeiT by a large margin of 2% with a small to moderate increase in FLOPs and model parameters. Our source codes and models are available at https://github.com/IBM/CrossViT.
translated by 谷歌翻译
To reduce the significant redundancy in deep Convolutional Neural Networks (CNNs), most existing methods prune neurons by only considering statistics of an individual layer or two consecutive layers (e.g., prune one layer to minimize the reconstruction error of the next layer), ignoring the effect of error propagation in deep networks. In contrast, we argue that it is essential to prune neurons in the entire neuron network jointly based on a unified goal: minimizing the reconstruction error of important responses in the "final response layer" (FRL), which is the secondto-last layer before classification, for a pruned network to retrain its predictive power. Specifically, we apply feature ranking techniques to measure the importance of each neuron in the FRL, and formulate network pruning as a binary integer optimization problem and derive a closed-form solution to it for pruning neurons in earlier layers. Based on our theoretical analysis, we propose the Neuron Importance Score Propagation (NISP) algorithm to propagate the importance scores of final responses to every neuron in the network. The CNN is pruned by removing neurons with least importance, and then fine-tuned to retain its predictive power. NISP is evaluated on several datasets with multiple CNN models and demonstrated to achieve significant acceleration and compression with negligible accuracy loss.
translated by 谷歌翻译
Deep learning models can achieve high accuracy when trained on large amounts of labeled data. However, real-world scenarios often involve several challenges: Training data may become available in installments, may originate from multiple different domains, and may not contain labels for training. Certain settings, for instance medical applications, often involve further restrictions that prohibit retention of previously seen data due to privacy regulations. In this work, to address such challenges, we study unsupervised segmentation in continual learning scenarios that involve domain shift. To that end, we introduce GarDA (Generative Appearance Replay for continual Domain Adaptation), a generative-replay based approach that can adapt a segmentation model sequentially to new domains with unlabeled data. In contrast to single-step unsupervised domain adaptation (UDA), continual adaptation to a sequence of domains enables leveraging and consolidation of information from multiple domains. Unlike previous approaches in incremental UDA, our method does not require access to previously seen data, making it applicable in many practical scenarios. We evaluate GarDA on two datasets with different organs and modalities, where it substantially outperforms existing techniques.
translated by 谷歌翻译
The development of social media user stance detection and bot detection methods rely heavily on large-scale and high-quality benchmarks. However, in addition to low annotation quality, existing benchmarks generally have incomplete user relationships, suppressing graph-based account detection research. To address these issues, we propose a Multi-Relational Graph-Based Twitter Account Detection Benchmark (MGTAB), the first standardized graph-based benchmark for account detection. To our knowledge, MGTAB was built based on the largest original data in the field, with over 1.55 million users and 130 million tweets. MGTAB contains 10,199 expert-annotated users and 7 types of relationships, ensuring high-quality annotation and diversified relations. In MGTAB, we extracted the 20 user property features with the greatest information gain and user tweet features as the user features. In addition, we performed a thorough evaluation of MGTAB and other public datasets. Our experiments found that graph-based approaches are generally more effective than feature-based approaches and perform better when introducing multiple relations. By analyzing experiment results, we identify effective approaches for account detection and provide potential future research directions in this field. Our benchmark and standardized evaluation procedures are freely available at: https://github.com/GraphDetec/MGTAB.
translated by 谷歌翻译
As one of the prevalent methods to achieve automation systems, Imitation Learning (IL) presents a promising performance in a wide range of domains. However, despite the considerable improvement in policy performance, the corresponding research on the explainability of IL models is still limited. Inspired by the recent approaches in explainable artificial intelligence methods, we proposed a model-agnostic explaining framework for IL models called R2RISE. R2RISE aims to explain the overall policy performance with respect to the frames in demonstrations. It iteratively retrains the black-box IL model from the randomized masked demonstrations and uses the conventional evaluation outcome environment returns as the coefficient to build an importance map. We also conducted experiments to investigate three major questions concerning frames' importance equality, the effectiveness of the importance map, and connections between importance maps from different IL models. The result shows that R2RISE successfully distinguishes important frames from the demonstrations.
translated by 谷歌翻译
Compressed videos often exhibit visually annoying artifacts, known as Perceivable Encoding Artifacts (PEAs), which dramatically degrade video visual quality. Subjective and objective measures capable of identifying and quantifying various types of PEAs are critical in improving visual quality. In this paper, we investigate the influence of four spatial PEAs (i.e. blurring, blocking, bleeding, and ringing) and two temporal PEAs (i.e. flickering and floating) on video quality. For spatial artifacts, we propose a visual saliency model with a low computational cost and higher consistency with human visual perception. In terms of temporal artifacts, self-attention based TimeSFormer is improved to detect temporal artifacts. Based on the six types of PEAs, a quality metric called Saliency-Aware Spatio-Temporal Artifacts Measurement (SSTAM) is proposed. Experimental results demonstrate that the proposed method outperforms state-of-the-art metrics. We believe that SSTAM will be beneficial for optimizing video coding techniques.
translated by 谷歌翻译