The architecture of transformers, which recently witness booming applications in vision tasks, has pivoted against the widespread convolutional paradigm. Relying on the tokenization process that splits inputs into multiple tokens, transformers are capable of extracting their pairwise relationships using self-attention. While being the stemming building block of transformers, what makes for a good tokenizer has not been well understood in computer vision. In this work, we investigate this uncharted problem from an information trade-off perspective. In addition to unifying and understanding existing structural modifications, our derivation leads to better design strategies for vision tokenizers. The proposed Modulation across Tokens (MoTo) incorporates inter-token modeling capability through normalization. Furthermore, a regularization objective TokenProp is embraced in the standard training regime. Through extensive experiments on various transformer architectures, we observe both improved performance and intriguing properties of these two plug-and-play designs with negligible computational overhead. These observations further indicate the importance of the commonly-omitted designs of tokenizers in vision transformer.
translated by 谷歌翻译
Generative adversarial networks (GANs) have made great success in image inpainting yet still have difficulties tackling large missing regions. In contrast, iterative algorithms, such as autoregressive and denoising diffusion models, have to be deployed with massive computing resources for decent effect. To overcome the respective limitations, we present a novel spatial diffusion model (SDM) that uses a few iterations to gradually deliver informative pixels to the entire image, largely enhancing the inference efficiency. Also, thanks to the proposed decoupled probabilistic modeling and spatial diffusion scheme, our method achieves high-quality large-hole completion. On multiple benchmarks, we achieve new state-of-the-art performance. Code is released at https://github.com/fenglinglwb/SDM.
translated by 谷歌翻译
The accurate detection and grasping of transparent objects are challenging but of significance to robots. Here, a visual-tactile fusion framework for transparent object grasping under complex backgrounds and variant light conditions is proposed, including the grasping position detection, tactile calibration, and visual-tactile fusion based classification. First, a multi-scene synthetic grasping dataset generation method with a Gaussian distribution based data annotation is proposed. Besides, a novel grasping network named TGCNN is proposed for grasping position detection, showing good results in both synthetic and real scenes. In tactile calibration, inspired by human grasping, a fully convolutional network based tactile feature extraction method and a central location based adaptive grasping strategy are designed, improving the success rate by 36.7% compared to direct grasping. Furthermore, a visual-tactile fusion method is proposed for transparent objects classification, which improves the classification accuracy by 34%. The proposed framework synergizes the advantages of vision and touch, and greatly improves the grasping efficiency of transparent objects.
translated by 谷歌翻译
In dense image segmentation tasks (e.g., semantic, panoptic), existing methods can hardly generalize well to unseen image domains, predefined classes, and image resolution & quality variations. Motivated by these observations, we construct a large-scale entity segmentation dataset to explore fine-grained entity segmentation, with a strong focus on open-world and high-quality dense segmentation. The dataset contains images spanning diverse image domains and resolutions, along with high-quality mask annotations for training and testing. Given the high-quality and -resolution nature of the dataset, we propose CropFormer for high-quality segmentation, which can improve mask prediction using high-res image crops that provide more fine-grained image details than the full image. CropFormer is the first query-based Transformer architecture that can effectively ensemble mask predictions from multiple image crops, by learning queries that can associate the same entities across the full image and its crop. With CropFormer, we achieve a significant AP gain of $1.9$ on the challenging fine-grained entity segmentation task. The dataset and code will be released at http://luqi.info/entityv2.github.io/.
translated by 谷歌翻译
Autonomous driving confronts great challenges in complex traffic scenarios, where the risk of Safety of the Intended Functionality (SOTIF) can be triggered by the dynamic operational environment and system insufficiencies. The SOTIF risk is reflected not only intuitively in the collision risk with objects outside the autonomous vehicles (AVs), but also inherently in the performance limitation risk of the implemented algorithms themselves. How to minimize the SOTIF risk for autonomous driving is currently a critical, difficult, and unresolved issue. Therefore, this paper proposes the "Self-Surveillance and Self-Adaption System" as a systematic approach to online minimize the SOTIF risk, which aims to provide a systematic solution for monitoring, quantification, and mitigation of inherent and external risks. The core of this system is the risk monitoring of the implemented artificial intelligence algorithms within the AV. As a demonstration of the Self-Surveillance and Self-Adaption System, the risk monitoring of the perception algorithm, i.e., YOLOv5 is highlighted. Moreover, the inherent perception algorithm risk and external collision risk are jointly quantified via SOTIF entropy, which is then propagated downstream to the decision-making module and mitigated. Finally, several challenging scenarios are demonstrated, and the Hardware-in-the-Loop experiments are conducted to verify the efficiency and effectiveness of the system. The results demonstrate that the Self-Surveillance and Self-Adaption System enables dependable online monitoring, quantification, and mitigation of SOTIF risk in real-time critical traffic environments.
translated by 谷歌翻译
Perception algorithms in autonomous driving systems confront great challenges in long-tail traffic scenarios, where the problems of Safety of the Intended Functionality (SOTIF) could be triggered by the algorithm performance insufficiencies and dynamic operational environment. However, such scenarios are not systematically included in current open-source datasets, and this paper fills the gap accordingly. Based on the analysis and enumeration of trigger conditions, a high-quality diverse dataset is released, including various long-tail traffic scenarios collected from multiple resources. Considering the development of probabilistic object detection (POD), this dataset marks trigger sources that may cause perception SOTIF problems in the scenarios as key objects. In addition, an evaluation protocol is suggested to verify the effectiveness of POD algorithms in identifying the key objects via uncertainty. The dataset never stops expanding, and the first batch of open-source data includes 1126 frames with an average of 2.27 key objects and 2.47 normal objects in each frame. To demonstrate how to use this dataset for SOTIF research, this paper further quantifies the perception SOTIF entropy to confirm whether a scenario is unknown and unsafe for a perception system. The experimental results show that the quantified entropy can effectively and efficiently reflect the failure of the perception algorithm.
translated by 谷歌翻译
交叉路口是自动驾驶任务最具挑战性的场景之一。由于复杂性和随机性,在相交处的基本应用(例如行为建模,运动预测,安全验证等)在很大程度上取决于数据驱动的技术。因此,交叉点中对流量参与者(TPS)的轨迹数据集的需求很大。目前,城市地区的大多数交叉路口都配备了交通信号灯。但是,尚无用于信号交叉点的大规模,高质量,公开可用的轨迹数据集。因此,在本文中,在中国天津选择了典型的两相信号交叉点。此外,管道旨在构建信号交叉数据集(SIND),其中包含7个小时的记录,其中包括13,000多种TPS,具有7种类型。然后,记录了信德的交通违规行为。此外,也将信德与其他类似作品进行比较。 SIND的特征可以概括如下:1)信德提供了更全面的信息,包括交通信号灯状态,运动参数,高清(HD)地图等。2)TPS的类别是多种多样和特征的,其中比例是脆弱的道路使用者(VRU)最高为62.6%3)显示了多次交通信号灯违反非电动车辆的行为。我们认为,Sind将是对现有数据集的有效补充,可以促进有关自动驾驶的相关研究。该数据集可通过以下方式在线获得:https://github.com/sotif-avlab/sind
translated by 谷歌翻译
最近,越来越多的努力集中在弱监督的场景图(WSSGG)上。 WSSGG的主流解决方案通常遵循相同的管道:它们首先将文本实体与弱图像级别的监督(例如,未定位的关系三胞胎或字幕)相结合,然后用图像区域对齐,然后以完全固定的实例训练SGG模型 - 级别的“伪”标签。但是,我们认为大多数现有的WSSGG仅专注于对象一致性,这意味着接地区域应具有与文本实体相同的对象类别标签。尽管他们忽略了理想对齐的另一个基本要求:相互作用,这意味着接地区域对应具有与文本实体对相同的相互作用(即视觉关系)。因此,在本文中,我们建议通过使用对象感知和互动感知知识来增强简单的接地模块,以获取更可靠的伪标签。为了更好地利用这两种类型的知识,我们将它们视为两位老师,并融合其生成的目标,以指导我们接地模块的训练过程。具体而言,我们设计了两种不同的策略,可以通过评估每个培训样本的可靠性来适应不同的教师。广泛的实验表明,我们的方法始终在各种弱监督下提高WSSGG性能。
translated by 谷歌翻译
随着移动设备的快速开发,现代使用的手机通常允许用户捕获4K分辨率(即超高定义)图像。然而,对于图像进行示范,在低级视觉中,一项艰巨的任务,现有作品通常是在低分辨率或合成图像上进行的。因此,这些方法对4K分辨率图像的有效性仍然未知。在本文中,我们探索了Moire模式的删除,以进行超高定义图像。为此,我们提出了第一个超高定义的演示数据集(UHDM),其中包含5,000个现实世界4K分辨率图像对,并对当前最新方法进行基准研究。此外,我们提出了一个有效的基线模型ESDNET来解决4K Moire图像,其中我们构建了一个语义对准的比例感知模块来解决Moire模式的尺度变化。广泛的实验表明了我们的方法的有效性,这可以超过最轻巧的优于最先进的方法。代码和数据集可在https://xinyu-andy.github.io/uhdm-page上找到。
translated by 谷歌翻译
现有的基于学习的框架插值算法从高速自然视频中提取连续帧以训练模型。与自然视频相比,卡通视频通常处于较低的框架速度。此外,连续卡通框架之间的运动通常是非线性,它破坏了插值算法的线性运动假设。因此,它不适合直接从卡通视频中生成训练集。为了更好地适应从自然视频到动画视频的框架插值算法,我们提出了Autofi,这是一种简单有效的方法,可以自动渲染训练数据,以进行深层动画视频插值。 Autofi采用分层体系结构来渲染合成数据,从而确保线性运动的假设。实验结果表明,Autofi在训练Dain和Anin方面表现出色。但是,大多数框架插值算法仍将在容易出错的区域(例如快速运动或大闭塞)中失败。除了Autofi外,我们还提出了一个名为SKTFI的基于插件的后处理后处理模块,以手动使用用户提供的草图来完善最终结果。借助Autofi和SKTFI,插值动画框架显示出很高的感知质量。
translated by 谷歌翻译