依靠变压器进行复杂的视觉功能学习,对象跟踪目睹了最先进的新标准(SOTA)。但是,这种进步伴随着更大的培训数据和更长的培训期,使跟踪越来越昂贵。在本文中,我们证明了变压器的依赖性不是必需的,并且在实现SOTA跟踪方面,纯粹的convnets仍然具有竞争力,甚至更经济和友好。我们的解决方案是释放多模式视觉语言(VL)跟踪的功能,只需使用Convnet。本质在于通过我们的模态混音器(Modamixer)和不对称Convnet搜索学习新颖的统一自适应VL表示。我们表明,我们的统一自适应VL表示形式纯粹是用Convnet学习的,是变压器视觉特征的简单而强大的替代方案,它令人难以置信地将基于CNN的基于CNN的Siamese Tracker提高了14.5%的SUC,在挑战性的Lasot方面(50.7%> 65.2%> 65.2%> 65.2% ),即使表现优于几个基于变压器的SOTA跟踪器。除经验结果外,我们理论上分析了我们的方法以证明其有效性。通过揭示VL代表的潜力,我们希望社区将更多的关注转移到VL跟踪上,并希望为超越变形金刚的未来跟踪开放更多的可能性。代码和模型将在https://github.com/judasdie/sots上发布。
translated by 谷歌翻译
我们介绍了一种新颖的骨干架构,提高特征表示的目标感知能力。具体地,已经观察到事实上框架简单地使用来自骨干网的输出来执行特征匹配,从备份目标本地化,没有从匹配模块到骨干网的直接反馈,尤其是浅层。更具体地,只有匹配模块可以直接访问目标信息(在参考帧中),而候选帧的表示学习对参考目标是盲目的。结果,浅级中的目标 - 无关干扰的累积效果可能降低更深层的特征质量。在本文中,我们通过在暹罗类似的骨干网(inbn)内进行多个分支 - 方面交互来从不同角度接近问题。在INBN的核心是一个通用交互建模器(GIM),其将参考图像的先前知识注入骨干网络的不同阶段,导致候选特征表示的更好的目标感知和鲁棒的牵引力,其计算成本具有可忽略的计算成本。所提出的GIM模块和INBN机制是一般的,适用于不同的骨干类型,包括CNN和变压器,以改进,如我们在多个基准上的广泛实验所证明的那样。特别是,CNN版本(基于Siamcar),分别在Lasot / TNL2K上改善了3.2 / 6.9的Suc绝对收益。变压器版本获取Lasot / TNL2K的SUC 25.7 / 52.0,与最近的艺术态度相提并论。代码和模型将被释放。
translated by 谷歌翻译
Siamese network based trackers formulate tracking as convolutional feature cross-correlation between a target template and a search region. However, Siamese trackers still have an accuracy gap compared with state-of-theart algorithms and they cannot take advantage of features from deep networks, such as ResNet-50 or deeper. In this work we prove the core reason comes from the lack of strict translation invariance. By comprehensive theoretical analysis and experimental validations, we break this restriction through a simple yet effective spatial aware sampling strategy and successfully train a ResNet-driven Siamese tracker with significant performance gain. Moreover, we propose a new model architecture to perform layer-wise and depthwise aggregations, which not only further improves the accuracy but also reduces the model size. We conduct extensive ablation studies to demonstrate the effectiveness of the proposed tracker, which obtains currently the best results on five large tracking benchmarks, including OTB2015, VOT2018, UAV123, LaSOT, and TrackingNet. Our model will be released to facilitate further researches.
translated by 谷歌翻译
我们介绍了一个基于仅用于跟踪的变压器的暹罗样的双分支网络。给定模板和搜索映像,我们将它们分成非重叠补丁,并基于其在注意窗口中的其他人的匹配结果提取每个补丁的特征向量。对于每个令牌,我们估计它是否包含目标对象和相应的大小。该方法的优点是,该特征从匹配中学到,最终匹配。因此,功能与目标跟踪任务对齐。该方法实现更好或比较的结果作为首先使用CNN提取特征的最佳性能,然后使用变压器熔断它们。它优于GOT-10K和VOT2020基准上的最先进的方法。此外,该方法在一个GPU上实现了实时推理速度(约为40美元的FPS)。代码和模型将被释放。
translated by 谷歌翻译
Astounding results from Transformer models on natural language tasks have intrigued the vision community to study their application to computer vision problems. Among their salient benefits, Transformers enable modeling long dependencies between input sequence elements and support parallel processing of sequence as compared to recurrent networks e.g., Long short-term memory (LSTM). Different from convolutional networks, Transformers require minimal inductive biases for their design and are naturally suited as set-functions. Furthermore, the straightforward design of Transformers allows processing multiple modalities (e.g., images, videos, text and speech) using similar processing blocks and demonstrates excellent scalability to very large capacity networks and huge datasets. These strengths have led to exciting progress on a number of vision tasks using Transformer networks. This survey aims to provide a comprehensive overview of the Transformer models in the computer vision discipline. We start with an introduction to fundamental concepts behind the success of Transformers i.e., self-attention, large-scale pre-training, and bidirectional feature encoding. We then cover extensive applications of transformers in vision including popular recognition tasks (e.g., image classification, object detection, action recognition, and segmentation), generative modeling, multi-modal tasks (e.g., visual-question answering, visual reasoning, and visual grounding), video processing (e.g., activity recognition, video forecasting), low-level vision (e.g., image super-resolution, image enhancement, and colorization) and 3D analysis (e.g., point cloud classification and segmentation). We compare the respective advantages and limitations of popular techniques both in terms of architectural design and their experimental value. Finally, we provide an analysis on open research directions and possible future works. We hope this effort will ignite further interest in the community to solve current challenges towards the application of transformer models in computer vision.
translated by 谷歌翻译
最近的跟踪器采用变压器来组合或替换广泛使用的重新NET作为其新的骨干网络。尽管他们的跟踪器在常规场景中运行良好,但是他们只是将2D功能弄平为序列,以更好地匹配变压器。我们认为这些操作忽略了目标对象的空间先验,这可能仅导致次优结果。此外,许多作品表明,自我注意力实际上是一个低通滤波器,它与输入功能或键/查询无关。也就是说,它可能会抑制输入功能的高频组成部分,并保留甚至放大低频信息。为了解决这些问题,在本文中,我们提出了一个统一的空间频率变压器,该变压器同时建模高斯空间先验和高频强调(GPHA)。具体而言,高斯空间先验是使用双重多层感知器(MLP)生成的,并注入了通过将查询和自我注意的关键特征乘产生的相似性矩阵。输出将被馈入软磁层,然后分解为两个组件,即直接信号和高频信号。低通和高通的分支被重新缩放并组合以实现全通,因此,高频特征将在堆叠的自发层中得到很好的保护。我们进一步将空间频率变压器整合到暹罗跟踪框架中,并提出一种新颖的跟踪算法,称为SFTRANST。基于跨级融合的SwintransFormer被用作骨干,还使用多头交叉意见模块来增强搜索和模板功能之间的相互作用。输出将被馈入跟踪头以进行目标定位。短期和长期跟踪基准的广泛实验都证明了我们提出的框架的有效性。
translated by 谷歌翻译
The current popular two-stream, two-stage tracking framework extracts the template and the search region features separately and then performs relation modeling, thus the extracted features lack the awareness of the target and have limited target-background discriminability. To tackle the above issue, we propose a novel one-stream tracking (OSTrack) framework that unifies feature learning and relation modeling by bridging the template-search image pairs with bidirectional information flows. In this way, discriminative target-oriented features can be dynamically extracted by mutual guidance. Since no extra heavy relation modeling module is needed and the implementation is highly parallelized, the proposed tracker runs at a fast speed. To further improve the inference efficiency, an in-network candidate early elimination module is proposed based on the strong similarity prior calculated in the one-stream framework. As a unified framework, OSTrack achieves state-of-the-art performance on multiple benchmarks, in particular, it shows impressive results on the one-shot tracking benchmark GOT-10k, i.e., achieving 73.7% AO, improving the existing best result (SwinTrack) by 4.3\%. Besides, our method maintains a good performance-speed trade-off and shows faster convergence. The code and models are available at https://github.com/botaoye/OSTrack.
translated by 谷歌翻译
基于无人机(UAV)基于无人机的视觉对象跟踪已实现了广泛的应用,并且由于其多功能性和有效性而引起了智能运输系统领域的越来越多的关注。作为深度学习革命性趋势的新兴力量,暹罗网络在基于无人机的对象跟踪中闪耀,其准确性,稳健性和速度有希望的平衡。由于开发了嵌入式处理器和深度神经网络的逐步优化,暹罗跟踪器获得了广泛的研究并实现了与无人机的初步组合。但是,由于无人机在板载计算资源和复杂的现实情况下,暹罗网络的空中跟踪仍然在许多方面都面临严重的障碍。为了进一步探索基于无人机的跟踪中暹罗网络的部署,这项工作对前沿暹罗跟踪器进行了全面的审查,以及使用典型的无人机板载处理器进行评估的详尽无人用分析。然后,进行板载测试以验证代表性暹罗跟踪器在现实世界无人机部署中的可行性和功效。此外,为了更好地促进跟踪社区的发展,这项工作分析了现有的暹罗跟踪器的局限性,并进行了以低弹片评估表示的其他实验。最后,深入讨论了基于无人机的智能运输系统的暹罗跟踪的前景。领先的暹罗跟踪器的统一框架,即代码库及其实验评估的结果,请访问https://github.com/vision4robotics/siamesetracking4uav。
translated by 谷歌翻译
We study joint learning of Convolutional Neural Network (CNN) and Transformer for vision-language pre-training (VLPT) which aims to learn cross-modal alignments from millions of image-text pairs. State-of-the-art approaches extract salient image regions and align regions with words step-by-step. As region-based visual features usually represent parts of an image, it is challenging for existing visionlanguage models to fully understand the semantics from paired natural languages. In this paper, we propose SOHO to "See Out of tHe bOx" that takes a whole image as input, and learns vision-language representation in an endto-end manner. SOHO does not require bounding box annotations which enables inference 10 times faster than regionbased approaches. In particular, SOHO learns to extract comprehensive yet compact image features through a visual dictionary (VD) that facilitates cross-modal understanding. VD is designed to represent consistent visual abstractions of similar semantics. It is updated on-the-fly and utilized in our proposed pre-training task Masked Visual Modeling (MVM). We conduct experiments on four well-established vision-language tasks by following standard VLPT settings. In particular, SOHO achieves absolute gains of 2.0% R@1 score on MSCOCO text retrieval 5k test split, 1.5% accuracy on NLVR 2 test-P split, 6.7% accuracy on SNLI-VE test split, respectively.
translated by 谷歌翻译
变压器是一种基于关注的编码器解码器架构,彻底改变了自然语言处理领域。灵感来自这一重大成就,最近在将变形式架构调整到计算机视觉(CV)领域的一些开创性作品,这已经证明了他们对各种简历任务的有效性。依靠竞争力的建模能力,与现代卷积神经网络相比在本文中,我们已经为三百不同的视觉变压器进行了全面的审查,用于三个基本的CV任务(分类,检测和分割),提出了根据其动机,结构和使用情况组织这些方法的分类。 。由于培训设置和面向任务的差异,我们还在不同的配置上进行了评估了这些方法,以便于易于和直观的比较而不是各种基准。此外,我们已经揭示了一系列必不可少的,但可能使变压器能够从众多架构中脱颖而出,例如松弛的高级语义嵌入,以弥合视觉和顺序变压器之间的差距。最后,提出了三个未来的未来研究方向进行进一步投资。
translated by 谷歌翻译
准确且强大的视觉对象跟踪是最具挑战性和最基本的计算机视觉问题之一。它需要在图像序列中估计目标的轨迹,仅给出其初始位置和分段,或者在边界框的形式中粗略近似。判别相关滤波器(DCF)和深度暹罗网络(SNS)被出现为主导跟踪范式,这导致了重大进展。在过去十年的视觉对象跟踪快速演变之后,该调查介绍了90多个DCFS和暹罗跟踪器的系统和彻底审查,基于九个跟踪基准。首先,我们介绍了DCF和暹罗跟踪核心配方的背景理论。然后,我们在这些跟踪范式中区分和全面地审查共享以及具体的开放研究挑战。此外,我们彻底分析了DCF和暹罗跟踪器对九个基准的性能,涵盖了视觉跟踪的不同实验方面:数据集,评估度量,性能和速度比较。通过提出根据我们的分析提出尊重开放挑战的建议和建议来完成调查。
translated by 谷歌翻译
视觉问题应答(VQA)任务利用视觉图像和语言分析来回回答图像的文本问题。它是一个流行的研究课题,在过去十年中越来越多的现实应用。本文介绍了我们最近对AliceMind-MMU的研究(阿里巴巴的编码器 - 解码器来自Damo Academy - 多媒体理解的机器智能实验室),其比人类在VQA上获得相似甚至略微更好的结果。这是通过系统地改善VQA流水线来实现的,包括:(1)具有全面的视觉和文本特征表示的预培训; (2)与学习参加的有效跨模型互动; (3)一个新颖的知识挖掘框架,具有专门的专业专家模块,适用于复杂的VQA任务。处理不同类型的视觉问题,需要具有相应的专业知识在提高我们的VQA架构的表现方面发挥着重要作用,这取决于人力水平。进行了广泛的实验和分析,以证明新的研究工作的有效性。
translated by 谷歌翻译
随着图像文本对的大量数据以及视觉和语言(V&L)任务的多样性,学者在该研究领域引入了大量的深度学习模型。此外,近年来,转移学习还显示出在计算机愿景中的巨大成功,例如图像分类,对象检测等以及在自然语言处理中以进行问答,机器翻译等的自然语言处理。继承转移学习的精神, V&L的研究工作已经在大规模数据集上设计了多种预训练技术,以增强下游任务的性能。本文的目的是提供当代V&L预审前模型的全面修订。特别是,我们对预处理的方法进行了分类和描述,以及最先进的视觉和语言预训练模型的摘要。此外,还提供了培训数据集和下游任务的列表,以进一步提高V&L预处理的观点。最后,我们决定采取进一步的一步,讨论众多未来研究的方向。
translated by 谷歌翻译
多模式变压器表现出高容量和灵活性,可将图像和文本对齐以进行视觉接地。然而,由于自我发挥操作的二次时间复杂性,仅编码的接地框架(例如,transvg)遭受了沉重的计算。为了解决这个问题,我们通过将整个接地过程解散为编码和解码阶段,提出了一种新的多模式变压器体系结构,以动态MDETR形成。关键观察是,图像中存在很高的空间冗余。因此,我们通过在加快视觉接地过程之前利用这种稀疏性来设计一种新的动态多模式变压器解码器。具体而言,我们的动态解码器由2D自适应采样模块和文本引导的解码模块组成。采样模块旨在通过预测参考点的偏移来选择这些信息补丁,而解码模块则可以通过在图像功能和文本功能之间执行交叉注意来提取接地对象信息。这两个模块也被堆叠起来,以逐渐弥合模态间隙,并迭代地完善接地对象的参考点,最终实现了视觉接地的目的。对五个基准测试的广泛实验表明,我们提出的动态MDETR实现了计算和准确性之间的竞争权衡。值得注意的是,在解码器中仅使用9%的特征点,我们可以降低〜44%的多模式变压器的GLOP,但仍然比仅编码器的对应物更高的精度。此外,为了验证其概括能力并扩展我们的动态MDETR,我们构建了第一个单级剪辑授权的视觉接地框架,并在这些基准测试中实现最先进的性能。
translated by 谷歌翻译
This paper presents a detailed study of improving visual representations for vision language (VL) tasks and develops an improved object detection model to provide object-centric representations of images. Compared to the most widely used bottom-up and top-down model [2], the new model is bigger, better-designed for VL tasks, and pre-trained on much larger training corpora that combine multiple public annotated object detection datasets. Therefore, it can generate representations of a richer collection of visual objects and concepts. While previous VL research focuses mainly on improving the vision-language fusion model and leaves the object detection model improvement untouched, we show that visual features matter significantly in VL models. In our experiments we feed the visual features generated by the new object detection model into a Transformer-based VL fusion model OSCAR [21], and utilize an improved approach OSCAR+ to pre-train the VL model and fine-tune it on a wide range of downstream VL tasks. Our results show that the new visual features significantly improve the performance across all VL tasks, creating new state-of-the-art results on seven public benchmarks. Code, models and pre-extracted features are released at https://github.com/pzzhang/VinVL. ♥ Microsoft Corporation♠ University of Washington † indicates equal contributions.
translated by 谷歌翻译
Scene text spotting is of great importance to the computer vision community due to its wide variety of applications. Recent methods attempt to introduce linguistic knowledge for challenging recognition rather than pure visual classification. However, how to effectively model the linguistic rules in end-to-end deep networks remains a research challenge. In this paper, we argue that the limited capacity of language models comes from 1) implicit language modeling; 2) unidirectional feature representation; and 3) language model with noise input. Correspondingly, we propose an autonomous, bidirectional and iterative ABINet++ for scene text spotting. Firstly, the autonomous suggests enforcing explicitly language modeling by decoupling the recognizer into vision model and language model and blocking gradient flow between both models. Secondly, a novel bidirectional cloze network (BCN) as the language model is proposed based on bidirectional feature representation. Thirdly, we propose an execution manner of iterative correction for the language model which can effectively alleviate the impact of noise input. Finally, to polish ABINet++ in long text recognition, we propose to aggregate horizontal features by embedding Transformer units inside a U-Net, and design a position and content attention module which integrates character order and content to attend to character features precisely. ABINet++ achieves state-of-the-art performance on both scene text recognition and scene text spotting benchmarks, which consistently demonstrates the superiority of our method in various environments especially on low-quality images. Besides, extensive experiments including in English and Chinese also prove that, a text spotter that incorporates our language modeling method can significantly improve its performance both in accuracy and speed compared with commonly used attention-based recognizers.
translated by 谷歌翻译
自我监督的视觉和语言预处理(VLP)旨在从大规模的图像文本数据中学习可转移的多模式表示形式,并在填充后在广泛的视觉范围内实现强大的表现。以前的主流VLP方法通常采用依靠外部对象检测器来编码多模式变压器框架中的图像的两步策略,该框架遭受了限制性对象概念空间,有限的图像上下文和效率低下的计算。在本文中,我们提出了一个对象感知的端到端VLP框架,该框架将来自CNN的图像网格特征直接馈送到变压器中,并共同学习多模式表示。更重要的是,我们建议执行对象知识蒸馏,以促进在不同语义级别的学习跨模式对齐。为了实现这一目标,我们通过将对象特征及其来自外部检测器的语义标签作为监督来设计两个新颖的借口任务:1。)对象引导的蒙版视觉建模任务的重点是在多模式变压器中强制执行对象感知的表示的学习; 2.)短语区域对准任务旨在通过利用语言空间中名词短语和对象标签之间的相似性来改善跨模式对齐。对各种视觉语言任务进行的广泛实验证明了我们提出的框架的功效,并且我们在现有的预科策略中实现了竞争性或优越的表现。
translated by 谷歌翻译
Video paragraph captioning aims to generate a multi-sentence description of an untrimmed video with several temporal event locations in coherent storytelling. Following the human perception process, where the scene is effectively understood by decomposing it into visual (e.g. human, animal) and non-visual components (e.g. action, relations) under the mutual influence of vision and language, we first propose a visual-linguistic (VL) feature. In the proposed VL feature, the scene is modeled by three modalities including (i) a global visual environment; (ii) local visual main agents; (iii) linguistic scene elements. We then introduce an autoregressive Transformer-in-Transformer (TinT) to simultaneously capture the semantic coherence of intra- and inter-event contents within a video. Finally, we present a new VL contrastive loss function to guarantee learnt embedding features are matched with the captions semantics. Comprehensive experiments and extensive ablation studies on ActivityNet Captions and YouCookII datasets show that the proposed Visual-Linguistic Transformer-in-Transform (VLTinT) outperforms prior state-of-the-art methods on accuracy and diversity.
translated by 谷歌翻译
在这项工作中,我们探索了用于视觉接地的整洁而有效的基于变压器的框架。先前的方法通常解决了视觉接地的核心问题,即具有手动设计的机制,即多模式融合和推理。这样的启发式设计不仅复杂化,而且使模型容易过度拟合特定的数据分布。为了避免这种情况,我们首先提出了TransVG,该TransVG通过变压器建立了多模式的对应关系,并通过直接回归框坐标来定位引用区域。我们从经验上表明,复杂的融合模块可以用具有更高性能的变压器编码层的简单堆栈代替。但是,TransVG中的核心融合变压器是针对Uni-Modal编码器的独立性,因此应在有限的视觉接地数据上从头开始训练,这使得很难优化并导致次优性能。为此,我们进一步介绍了TransVG ++以进行两倍的改进。一方面,我们通过利用Vision Transformer(VIT)进行视觉功能编码来将框架升级到一个纯粹的基于变压器的框架。对于另一个人来说,我们设计了语言有条件的视觉变压器,以去除外部融合模块,并重用Uni-Modal vit进行中间层的视觉融合。我们对五个普遍数据集进行了广泛的实验,并报告一系列最先进的记录。
translated by 谷歌翻译
我们提出了一种称为独角兽的统一方法,可以使用相同的模型参数同时使用单个网络解决四个跟踪问题(SOT,MOT,VOS,MOTS)。由于对象跟踪问题本身的定义零散,因此开发了大多数现有的跟踪器来解决任务的单个或一部分,并过分地对特定任务的特征进行了专业化。相比之下,Unicorn提供了一个统一的解决方案,在所有跟踪任务中采用相同的输入,骨干,嵌入和头部。我们第一次完成了跟踪网络体系结构和学习范式的巨大统一。Unicorn在8个跟踪数据集中的特定于任务特定的对应物(包括Lasot,TrackingNet,Mot17,BDD100K,Davis16-17,MOTS20和BDD100K MOT)在PAR上或更好的对应物。我们认为,独角兽将是朝着一般视觉模型迈出的坚实一步。代码可从https://github.com/masterbin-iiau/unicorn获得。
translated by 谷歌翻译