In contrast to fully supervised methods using pixel-wise mask labels, box-supervised instance segmentation takes advantage of simple box annotations, which has recently attracted increasing research attention. This paper presents a novel single-shot instance segmentation approach, namely Box2Mask, which integrates the classical level-set evolution model into deep neural network learning to achieve accurate mask prediction with only bounding box supervision. Specifically, both the input image and its deep features are employed to evolve the level-set curves implicitly, and a local consistency module based on a pixel affinity kernel is used to mine the local context and spatial relations. Two types of single-stage frameworks, i.e., CNN-based and transformer-based frameworks, are developed to empower the level-set evolution for box-supervised instance segmentation, and each framework consists of three essential components: instance-aware decoder, box-level matching assignment and level-set evolution. By minimizing the level-set energy function, the mask map of each instance can be iteratively optimized within its bounding box annotation. The experimental results on five challenging testbeds, covering general scenes, remote sensing, medical and scene text images, demonstrate the outstanding performance of our proposed Box2Mask approach for box-supervised instance segmentation. In particular, with the Swin-Transformer large backbone, our Box2Mask obtains 42.4% mask AP on COCO, which is on par with the recently developed fully mask-supervised methods. The code is available at: https://github.com/LiWentomng/boxlevelset.
translated by 谷歌翻译
We present a simple yet effective end-to-end Video-language Pre-training (VidLP) framework, Masked Contrastive Video-language Pretraining (MAC), for video-text retrieval tasks. Our MAC aims to reduce video representation's spatial and temporal redundancy in the VidLP model by a mask sampling mechanism to improve pre-training efficiency. Comparing conventional temporal sparse sampling, we propose to randomly mask a high ratio of spatial regions and only feed visible regions into the encoder as sparse spatial sampling. Similarly, we adopt the mask sampling technique for text inputs for consistency. Instead of blindly applying the mask-then-prediction paradigm from MAE, we propose a masked-then-alignment paradigm for efficient video-text alignment. The motivation is that video-text retrieval tasks rely on high-level alignment rather than low-level reconstruction, and multimodal alignment with masked modeling encourages the model to learn a robust and general multimodal representation from incomplete and unstable inputs. Coupling these designs enables efficient end-to-end pre-training: reduce FLOPs (60% off), accelerate pre-training (by 3x), and improve performance. Our MAC achieves state-of-the-art results on various video-text retrieval datasets, including MSR-VTT, DiDeMo, and ActivityNet. Our approach is omnivorous to input modalities. With minimal modifications, we achieve competitive results on image-text retrieval tasks.
translated by 谷歌翻译
与使用像素面罩标签的完全监督的方法相反,盒子监督实例细分利用了简单的盒子注释,该盒子注释最近吸引了许多研究注意力。在本文中,我们提出了一种新颖的单弹盒监督实例分割方法,该方法将经典级别设置模型与深度神经网络精致整合在一起。具体而言,我们提出的方法迭代地通过端到端的方式通过基于Chan-Vese的连续能量功能来学习一系列级别集。一个简单的掩码监督的SOLOV2模型可供选择,以预测实例感知的掩码映射为每个实例的级别设置。输入图像及其深度特征都被用作输入数据来发展级别集曲线,其中使用框投影函数来获得初始边界。通过最大程度地减少完全可分化的能量函数,在其相应的边界框注释中迭代优化了每个实例的级别设置。在四个具有挑战性的基准上的实验结果表明,在各种情况下,我们提出的强大实例分割方法的领先表现。该代码可在以下网址获得:https://github.com/liwentomng/boxlevelset。
translated by 谷歌翻译
在这项工作中,我们为基于视觉的不均衡的BEV表示学习提出了PolarBev。为了适应摄像机成像的预先处理效果,我们将BEV空间横向和辐射上栅格化,并引入极性嵌入分解,以模拟极性网格之间的关联。极性网格被重新排列到类似阵列的常规表示,以进行有效处理。此外,为了确定2到3D对应关系,我们根据假设平面迭代更新BEV表面,并采用基于高度的特征转换。PolarBev在单个2080TI GPU上保持实时推理速度,并且在BEV语义分割和BEV实例分割方面都优于其他方法。展示彻底消融以验证设计。该代码将在\ url {https://github.com/superz-liu/polarbev}上发布。
translated by 谷歌翻译
关于驾驶场景图像的语义细分对于自动驾驶至关重要。尽管在白天图像上已经实现了令人鼓舞的性能,但由于暴露不足和缺乏标记的数据,夜间图像的性能不那么令人满意。为了解决这些问题,我们提出了一个称为双图像自动学习过滤器(拨号过滤器)的附加模块,以改善夜间驾驶条件下的语义分割,旨在利用不同照明下驾驶场景图像的内在特征。拨盘滤波器由两个部分组成,包括图像自适应处理模块(IAPM)和可学习的引导过滤器(LGF)。使用拨号过滤器,我们设计了无监督和有监督的框架,用于夜间驾驶场景细分,可以以端到端的方式进行培训。具体而言,IAPM模块由一个带有一组可区分图像过滤器的小型卷积神经网络组成,可以自适应地增强每个图像,以更好地相对于不同的照明。 LGF用于增强分割网络的输出以获得最终的分割结果。拨号过滤器轻巧有效,可以在白天和夜间图像中轻松应用它们。我们的实验表明,Dail过滤器可以显着改善ACDC_Night和Nightcity数据集的监督细分性能,而它展示了有关无监督的夜间夜间语义细分的最新性能,在黑暗的苏黎世和夜间驾驶测试床上。
translated by 谷歌翻译
与特殊线性组和嵌入谎言代数结构具有基本关系。尽管谎言代数表示优雅,但很少有研究人员在同构估计与代数表达之间建立了联系。在本文中,我们提出了扭曲的卷积网络(WCN),以有效地估计SL(3)组和SL(3)代数的分组转换。为此,SL(3)组中的六个换向子组组成以形成一个跨摄影转换。对于每个子组,提出了一个翘曲函数,以将Lie代数结构桥接到其在断层扫描中的相应参数上。通过利用扭曲的卷积,同构估计得出了几个简单的伪翻译回归。通过沿着谎言拓扑行走,我们提出的WCN能够学习对构造转换不变的功能。它可以很容易地插入其他基于CNN的方法中。对POT基准和MNIST-PROJ数据集进行了广泛的实验表明,我们提出的方法对同型估计和分类都有效。
translated by 谷歌翻译
虽然基于深度学习的对象检测方法在传统的数据集上取得了有希望的结果,但它仍然具有挑战性,以从恶劣天气条件下捕获的低质量图像定位对象仍然具有挑战性。现有方法在平衡图像增强和对象检测的任务方面具有困难,或者通常忽略有利于检测的潜在信息。为了减轻这个问题,我们提出了一种新颖的图像自适应yolo(IA-YOLO)框架,其中可以适自动化的图像以获得更好的检测性能。具体地,提出了可视的图像处理(DIP)模块以考虑YOLO检测器的恶劣天气条件,其参数由小型卷积神经网络(CNN-PP)预测。我们以端到端的方式共同学习CNN-PP和YOLOV3,确保CNN-PP可以学习适当的DIP以以弱监督方式增强图像以进行检测。我们所提出的IA-Yolo方法可以在正常和恶劣天气条件下自适应地处理图像。实验结果非常令人鼓舞,展示了我们提出的IA-Yolo方法在雾和低光场景中的有效性。
translated by 谷歌翻译
盒子监督的实例分割最近吸引了大量的研究工作,而在空中图像域中则收到很少的关注。与通用物体集合相比,空中对象具有大型内部差异和阶级相似性与复杂的背景。此外,高分辨率卫星图像中存在许多微小的物体。这使得最近的一对亲和力建模方法不可避免地涉及具有劣势的噪声监督。为了解决这些问题,我们提出了一种新颖的空中实例分割方法,该方法驱动网络为空中对象的一系列级别设置功能,只有盒子注释以端到端的方式。具有精心设计的能量函数的级别集方法而不是学习成对亲和力将对象分段视为曲线演进,这能够准确地恢复对象的边界并防止来自无法区分的背景和类似对象的干扰。实验结果表明,所提出的方法优于最先进的盒子监督实例分段方法。源代码可在https://github.com/liwentomng/boxLevelset上获得。
translated by 谷歌翻译
恢复程序的呼叫图对于基于流程间分析任务和应用程序至关重要。核心挑战是识别间接呼叫的目标(即,间接分支机构)。由于二进制文件中的信息丢失,如果目标程序以二元形式为二元形式,则变得更具挑战性。二进制文件的现有间接Callee识别解决方案都具有高误报和负面,使呼叫图不准确。在本文中,我们提出了一种基于暹罗神经网络的新解决方案,受到质疑答案应用的进步的启发。关键洞察力是,神经网络可以学习通过理解其上下文,即附近呼叫和分支机构的指示是间接代表的潜在目标。在此洞察力之后,我们首先预处理目标二进制文件,以提取电话和分支的上下文。然后,我们构建适用于汇编语言的自定义自然语言处理(NLP)模型。此外,我们收集了丰富的呼叫和分支,并将其上下文与NLP模型嵌入,然后培训暹罗网络和分类器以回答电呼叫路上的问题。我们已经实施了Inclelee的原型,并在几组目标上进行了评估。评价结果表明,我们的解决方案可以将手段与F1措施相匹配93.7%,召回的93.8%,精度为93.5%,比最先进的解决方案好得多。为了展示其有用性,我们将iCallee应用于两个特定的应用 - 二进制代码相似性检测和二进制程序硬化,并发现它可以大大提高最先进的解决方案。
translated by 谷歌翻译
Passive monitoring of acoustic or radio sources has important applications in modern convenience, public safety, and surveillance. A key task in passive monitoring is multiobject tracking (MOT). This paper presents a Bayesian method for multisensor MOT for challenging tracking problems where the object states are high-dimensional, and the measurements follow a nonlinear model. Our method is developed in the framework of factor graphs and the sum-product algorithm (SPA). The multimodal probability density functions (pdfs) provided by the SPA are effectively represented by a Gaussian mixture model (GMM). To perform the operations of the SPA in high-dimensional spaces, we make use of Particle flow (PFL). Here, particles are migrated towards regions of high likelihood based on the solution of a partial differential equation. This makes it possible to obtain good object detection and tracking performance even in challenging multisensor MOT scenarios with single sensor measurements that have a lower dimension than the object positions. We perform a numerical evaluation in a passive acoustic monitoring scenario where multiple sources are tracked in 3-D from 1-D time-difference-of-arrival (TDOA) measurements provided by pairs of hydrophones. Our numerical results demonstrate favorable detection and estimation accuracy compared to state-of-the-art reference techniques.
translated by 谷歌翻译