In this paper, we introduce MINTIME, a video deepfake detection approach that captures spatial and temporal anomalies and handles instances of multiple people in the same video and variations in face sizes. Previous approaches disregard such information either by using simple a-posteriori aggregation schemes, i.e., average or max operation, or using only one identity for the inference, i.e., the largest one. On the contrary, the proposed approach builds on a Spatio-Temporal TimeSformer combined with a Convolutional Neural Network backbone to capture spatio-temporal anomalies from the face sequences of multiple identities depicted in a video. This is achieved through an Identity-aware Attention mechanism that attends to each face sequence independently based on a masking operation and facilitates video-level aggregation. In addition, two novel embeddings are employed: (i) the Temporal Coherent Positional Embedding that encodes each face sequence's temporal information and (ii) the Size Embedding that encodes the size of the faces as a ratio to the video frame size. These extensions allow our system to adapt particularly well in the wild by learning how to aggregate information of multiple identities, which is usually disregarded by other methods in the literature. It achieves state-of-the-art results on the ForgeryNet dataset with an improvement of up to 14% AUC in videos containing multiple people and demonstrates ample generalization capabilities in cross-forgery and cross-dataset settings. The code is publicly available at https://github.com/davide-coccomini/MINTIME-Multi-Identity-size-iNvariant-TIMEsformer-for-Video-Deepfake-Detection.
translated by 谷歌翻译
由于监视摄像头网络的无处不在,从图像中计算的自动人士最近引起了现代智能城市的城市监测的注意。当前的计算机视觉技术依赖于基于深度学习的算法,这些算法估算了静止图像中的行人密度。只有一堆作品利用视频序列中的时间一致性。在这项工作中,我们提出了一个时空的细心神经网络,以估计监视视频中的行人数量。通过利用连续帧之间的时间相关性,我们在广泛使用的FDST基准上将最新的计数误差降低了5%,定位误差降低了7.5%。
translated by 谷歌翻译
深度神经网络的学习算法通常基于有误后传播(BackProp)的监督端到端随机梯度下降(SGD)培训。 Backprop算法需要大量标记的训练样本才能获得高性能。但是,在许多现实的应用中,即使有很多图像样本,很少有标签被标记,并且必须使用半监督的样品培训策略。 Hebbian学习代表了一种可能采取样本培训的方法;但是,在当前解决方案中,它不能很好地扩展到大型数据集。在本文中,我们提出了FastheBB,这是HEBBIAN学习的有效且可扩展的解决方案,通过1)合并在一批输入上更新计算和聚集,以及2)利用有效的GPU上的有效矩阵乘法算法。在半监督的学习方案中,我们在不同的计算机视觉基准测试方面验证了我们的方法。 FastheBB在训练速度方面最多优于先前的解决方案,尤其是,我们首次能够将HEBBIAN算法带入ImageNet量表。
translated by 谷歌翻译
深层生成技术正在快速发展,使创建现实的操纵图像和视频并危及现代社会的宁静成为可能。新技术的持续出现带来了一个要面对的另一个问题,即DeepFake检测模型及时更新自己的能力,以便能够使用最新方法识别进行的操作。这是一个非常复杂的问题,因为训练一个模型需要大量数据,如果深层生成方法过于最近,这很难获得。此外,不断地重新训练网络是不可行的。在本文中,我们问自己,在各种深度学习技术中,是否有一个能够概括深层的概念,以至于它不会与培训中使用的一种或多种或多种特定的深层捕获方法息息相关。放。我们将视觉变压器与基于伪造网络数据集的跨性别环境中的有效NETV2进行了比较。从我们的实验中,有效的NETV2具有更大的专业趋势,通常会在训练方法上获得更好的结果,而视觉变压器具有卓越的概括能力,即使在使用新方法生成的图像上也使它们更有能力。
translated by 谷歌翻译
虽然卷积神经网络(CNNS)在许多愿景任务中显示出显着的结果,但它们仍然是通过简单但具有挑战性的视觉推理问题所紧张的。在计算机视觉中最近的变压器网络成功的启发,在本文中,我们介绍了经常性视觉变压器(RVIT)模型。由于经常性连接和空间注意在推理任务中的影响,该网络实现了来自SVRT数据集的同样不同视觉推理问题的竞争结果。空间和深度尺寸中的重量共享正规化模型,允许它使用较少的自由参数学习,仅使用28K培训样本。全面的消融研究证实了混合CNN +变压器架构的重要性和反馈连接的作用,其迭代地细化内部表示直到获得稳定的预测。最后,本研究可以更深入地了解对求解视觉抽象推理任务的注意力和经常性联系的作用。
translated by 谷歌翻译
太空探索一直是人类灵感的来源,并且由于现代望远镜,现在可以观察远离我们的天体。在网络上越来越多的空间的现实和虚构的图像,并利用现代深层学习架构,如生成的对抗网络,现在可以生成新的空间表示。在这项研究中,使用轻量级GaN,从网络获得的图像数据集,以及Galaxy动物园数据集,我们已经产生了数千个新的天体,星系,最后,最后的宇宙视图。。复制我们的结果的代码在https://github.com/davide-ccomini/ganiverse上公开提供,并且可以在https://davide-ccomini.github.io/goccomiverse/中探索生成的图像。
translated by 谷歌翻译
There is a dramatic shortage of skilled labor for modern vineyards. The Vinum project is developing a mobile robotic solution to autonomously navigate through vineyards for winter grapevine pruning. This necessitates an autonomous navigation stack for the robot pruning a vineyard. The Vinum project is using the quadruped robot HyQReal. This paper introduces an architecture for a quadruped robot to autonomously move through a vineyard by identifying and approaching grapevines for pruning. The higher level control is a state machine switching between searching for destination positions, autonomously navigating towards those locations, and stopping for the robot to complete a task. The destination points are determined by identifying grapevine trunks using instance segmentation from a Mask Region-Based Convolutional Neural Network (Mask-RCNN). These detections are sent through a filter to avoid redundancy and remove noisy detections. The combination of these features is the basis for the proposed architecture.
translated by 谷歌翻译
Explainability is a vibrant research topic in the artificial intelligence community, with growing interest across methods and domains. Much has been written about the topic, yet explainability still lacks shared terminology and a framework capable of providing structural soundness to explanations. In our work, we address these issues by proposing a novel definition of explanation that is a synthesis of what can be found in the literature. We recognize that explanations are not atomic but the product of evidence stemming from the model and its input-output and the human interpretation of this evidence. Furthermore, we fit explanations into the properties of faithfulness (i.e., the explanation being a true description of the model's decision-making) and plausibility (i.e., how much the explanation looks convincing to the user). Using our proposed theoretical framework simplifies how these properties are ope rationalized and provide new insight into common explanation methods that we analyze as case studies.
translated by 谷歌翻译
Objective: Accurate visual classification of bladder tissue during Trans-Urethral Resection of Bladder Tumor (TURBT) procedures is essential to improve early cancer diagnosis and treatment. During TURBT interventions, White Light Imaging (WLI) and Narrow Band Imaging (NBI) techniques are used for lesion detection. Each imaging technique provides diverse visual information that allows clinicians to identify and classify cancerous lesions. Computer vision methods that use both imaging techniques could improve endoscopic diagnosis. We address the challenge of tissue classification when annotations are available only in one domain, in our case WLI, and the endoscopic images correspond to an unpaired dataset, i.e. there is no exact equivalent for every image in both NBI and WLI domains. Method: We propose a semi-surprised Generative Adversarial Network (GAN)-based method composed of three main components: a teacher network trained on the labeled WLI data; a cycle-consistency GAN to perform unpaired image-to-image translation, and a multi-input student network. To ensure the quality of the synthetic images generated by the proposed GAN we perform a detailed quantitative, and qualitative analysis with the help of specialists. Conclusion: The overall average classification accuracy, precision, and recall obtained with the proposed method for tissue classification are 0.90, 0.88, and 0.89 respectively, while the same metrics obtained in the unlabeled domain (NBI) are 0.92, 0.64, and 0.94 respectively. The quality of the generated images is reliable enough to deceive specialists. Significance: This study shows the potential of using semi-supervised GAN-based classification to improve bladder tissue classification when annotations are limited in multi-domain data.
translated by 谷歌翻译
In the era of digital healthcare, the huge volumes of textual information generated every day in hospitals constitute an essential but underused asset that could be exploited with task-specific, fine-tuned biomedical language representation models, improving patient care and management. For such specialized domains, previous research has shown that fine-tuning models stemming from broad-coverage checkpoints can largely benefit additional training rounds over large-scale in-domain resources. However, these resources are often unreachable for less-resourced languages like Italian, preventing local medical institutions to employ in-domain adaptation. In order to reduce this gap, our work investigates two accessible approaches to derive biomedical language models in languages other than English, taking Italian as a concrete use-case: one based on neural machine translation of English resources, favoring quantity over quality; the other based on a high-grade, narrow-scoped corpus natively written in Italian, thus preferring quality over quantity. Our study shows that data quantity is a harder constraint than data quality for biomedical adaptation, but the concatenation of high-quality data can improve model performance even when dealing with relatively size-limited corpora. The models published from our investigations have the potential to unlock important research opportunities for Italian hospitals and academia. Finally, the set of lessons learned from the study constitutes valuable insights towards a solution to build biomedical language models that are generalizable to other less-resourced languages and different domain settings.
translated by 谷歌翻译