Video synthesis methods rapidly improved in recent years, allowing easy creation of synthetic humans. This poses a problem, especially in the era of social media, as synthetic videos of speaking humans can be used to spread misinformation in a convincing manner. Thus, there is a pressing need for accurate and robust deepfake detection methods, that can detect forgery techniques not seen during training. In this work, we explore whether this can be done by leveraging a multi-modal, out-of-domain backbone trained in a self-supervised manner, adapted to the video deepfake domain. We propose FakeOut; a novel approach that relies on multi-modal data throughout both the pre-training phase and the adaption phase. We demonstrate the efficacy and robustness of FakeOut in detecting various types of deepfakes, especially manipulations which were not seen during training. Our method achieves state-of-the-art results in cross-manipulation and cross-dataset generalization. This study shows that, perhaps surprisingly, training on out-of-domain videos (i.e., videos with no speaking humans), can lead to better deepfake detection systems. Code is available on GitHub.

由于滥用了深层，检测伪造视频是非常可取的。现有的检测方法有助于探索DeepFake视频中的特定工件，并且非常适合某些数据。但是，这些人工制品的不断增长的技术一直在挑战传统的深泡探测器的鲁棒性。结果，这些方法的普遍性的发展已达到阻塞。为了解决这个问题，鉴于经验结果是，深层视频中经常在声音和面部背后的身份不匹配，并且声音和面孔在某种程度上具有同质性，在本文中，我们建议从未开发的语音中执行深层检测 - 面对匹配视图。为此，设计了一种语音匹配方法来测量这两个方法的匹配度。然而，对特定的深泡数据集进行培训使模型过于拟合深层算法的某些特征。相反，我们提倡一种迅速适应未开发的伪造方法的方法，然后进行预训练，然后进行微调范式。具体而言，我们首先在通用音频视频数据集上预先培训该模型，然后在下游深板数据上进行微调。我们对三个广泛利用的DeepFake数据集进行了广泛的实验-DFDC，Fakeavceleb和DeepFaketimit。与其他最先进的竞争对手相比，我们的方法获得了显着的性能增长。还值得注意的是，我们的方法在有限的DeepFake数据上进行了微调时已经取得了竞争性结果。

Videos are a rich source of multi-modal supervision. In this work, we learn representations using self-supervision by leveraging three modalities naturally present in videos: visual, audio and language streams. To this end, we introduce the notion of a multimodal versatile network -a network that can ingest multiple modalities and whose representations enable downstream tasks in multiple modalities. In particular, we explore how best to combine the modalities, such that fine-grained representations of the visual and audio modalities can be maintained, whilst also integrating text into a common embedding. Driven by versatility, we also introduce a novel process of deflation, so that the networks can be effortlessly applied to the visual data in the form of video or a static image. We demonstrate how such networks trained on large collections of unlabelled video data can be applied on video, video-text, image and audio tasks. Equipped with these representations, we obtain state-of-the-art performance on multiple challenging benchmarks including UCF101, HMDB51, Kinetics600, Audioset and ESC-50 when compared to previous self-supervised work. Our models are publicly available [1, 2, 3]. * Equal contribution. † Work done during an internship at DeepMind. 34th Conference on Neural Information Processing Systems (NeurIPS 2020),

Online media data, in the forms of images and videos, are becoming mainstream communication channels. However, recent advances in deep learning, particularly deep generative models, open the doors for producing perceptually convincing images and videos at a low cost, which not only poses a serious threat to the trustworthiness of digital information but also has severe societal implications. This motivates a growing interest of research in media tampering detection, i.e., using deep learning techniques to examine whether media data have been maliciously manipulated. Depending on the content of the targeted images, media forgery could be divided into image tampering and Deepfake techniques. The former typically moves or erases the visual elements in ordinary images, while the latter manipulates the expressions and even the identity of human faces. Accordingly, the means of defense include image tampering detection and Deepfake detection, which share a wide variety of properties. In this paper, we provide a comprehensive review of the current media tampering detection approaches, and discuss the challenges and trends in this field for future research.

Deep learning has enabled realistic face manipulation (i.e., deepfake), which poses significant concerns over the integrity of the media in circulation. Most existing deep learning techniques for deepfake detection can achieve promising performance in the intra-dataset evaluation setting (i.e., training and testing on the same dataset), but are unable to perform satisfactorily in the inter-dataset evaluation setting (i.e., training on one dataset and testing on another). Most of the previous methods use the backbone network to extract global features for making predictions and only employ binary supervision (i.e., indicating whether the training instances are fake or authentic) to train the network. Classification merely based on the learning of global features leads often leads to weak generalizability to unseen manipulation methods. In addition, the reconstruction task can improve the learned representations. In this paper, we introduce a novel approach for deepfake detection, which considers the reconstruction and classification tasks simultaneously to address these problems. This method shares the information learned by one task with the other, which focuses on a different aspect other existing works rarely consider and hence boosts the overall performance. In particular, we design a two-branch Convolutional AutoEncoder (CAE), in which the Convolutional Encoder used to compress the feature map into the latent representation is shared by both branches. Then the latent representation of the input data is fed to a simple classifier and the unsupervised reconstruction component simultaneously. Our network is trained end-to-end. Experiments demonstrate that our method achieves state-of-the-art performance on three commonly-used datasets, particularly in the cross-dataset evaluation setting.

Face manipulation technology is advancing very rapidly, and new methods are being proposed day by day. The aim of this work is to propose a deepfake detector that can cope with the wide variety of manipulation methods and scenarios encountered in the real world. Our key insight is that each person has specific biometric characteristics that a synthetic generator cannot likely reproduce. Accordingly, we extract high-level audio-visual biometric features which characterize the identity of a person, and use them to create a person-of-interest (POI) deepfake detector. We leverage a contrastive learning paradigm to learn the moving-face and audio segment embeddings that are most discriminative for each identity. As a result, when the video and/or audio of a person is manipulated, its representation in the embedding space becomes inconsistent with the real identity, allowing reliable detection. Training is carried out exclusively on real talking-face videos, thus the detector does not depend on any specific manipulation method and yields the highest generalization ability. In addition, our method can detect both single-modality (audio-only, video-only) and multi-modality (audio-video) attacks, and is robust to low-quality or corrupted videos by building only on high-level semantic features. Experiments on a wide variety of datasets confirm that our method ensures a SOTA performance, with an average improvement in terms of AUC of around 3%, 10%, and 4% for high-quality, low quality, and attacked videos, respectively. https://github.com/grip-unina/poi-forensics

As ultra-realistic face forgery techniques emerge, deepfake detection has attracted increasing attention due to security concerns. Many detectors cannot achieve accurate results when detecting unseen manipulations despite excellent performance on known forgeries. In this paper, we are motivated by the observation that the discrepancies between real and fake videos are extremely subtle and localized, and inconsistencies or irregularities can exist in some critical facial regions across various information domains. To this end, we propose a novel pipeline, Cross-Domain Local Forensics (XDLF), for more general deepfake video detection. In the proposed pipeline, a specialized framework is presented to simultaneously exploit local forgery patterns from space, frequency, and time domains, thus learning cross-domain features to detect forgeries. Moreover, the framework leverages four high-level forgery-sensitive local regions of a human face to guide the model to enhance subtle artifacts and localize potential anomalies. Extensive experiments on several benchmark datasets demonstrate the impressive performance of our method, and we achieve superiority over several state-of-the-art methods on cross-dataset generalization. We also examined the factors that contribute to its performance through ablations, which suggests that exploiting cross-domain local characteristics is a noteworthy direction for developing more general deepfake detectors.

The remarkable success of deep learning in various domains relies on the availability of large-scale annotated datasets. However, obtaining annotations is expensive and requires great effort, which is especially challenging for videos. Moreover, the use of human-generated annotations leads to models with biased learning and poor domain generalization and robustness. As an alternative, self-supervised learning provides a way for representation learning which does not require annotations and has shown promise in both image and video domains. Different from the image domain, learning video representations are more challenging due to the temporal dimension, bringing in motion and other environmental dynamics. This also provides opportunities for video-exclusive ideas that advance self-supervised learning in the video and multimodal domain. In this survey, we provide a review of existing approaches on self-supervised learning focusing on the video domain. We summarize these methods into four different categories based on their learning objectives: 1) pretext tasks, 2) generative learning, 3) contrastive learning, and 4) cross-modal agreement. We further introduce the commonly used datasets, downstream evaluation tasks, insights into the limitations of existing works, and the potential future directions in this area.

我们使用无卷积的变压器架构提出了一种从未标记数据学习多式式表示的框架。具体而言，我们的视频音频文本变压器（Vatt）将原始信号作为输入提取，提取丰富的多式化表示，以使各种下游任务受益。我们使用多模式对比损失从头划线训练Vatt端到端，并通过视频动作识别，音频事件分类，图像分类和文本到视频检索的下游任务评估其性能。此外，我们通过共享三种方式之间的重量来研究模型 - 无话的单骨架变压器。我们表明，无卷积VATT优于下游任务中的最先进的Convnet架构。特别是，Vatt的视觉变压器在动力学-400上实现82.1％的高精度82.1％，在动力学-600，72.7％的动力学-700上的72.7％，以及时间的时间，新的记录，在避免受监督的预训练时，新的记录。通过从头划伤训练相同的变压器，转移到图像分类导致图像分类导致78.7％的ImageNet精度为64.7％，尽管视频和图像之间的域间差距，我们的模型概括了我们的模型。 Vatt的音雅音频变压器还通过在没有任何监督的预训练的情况下在Audioset上实现39.4％的地图来设置基于波形的音频事件识别的新记录。 Vatt的源代码是公开的。

强大的深度学习技术的发展为社会和个人带来了一些负面影响。一个这样的问题是假媒体的出现。为了解决这个问题，我们组织了可信赖的媒体挑战（TMC）来探讨人工智能（AI）如何利用如何打击假媒体。我们与挑战一起发布了一个挑战数据集，由4,380张假和2,563个真实视频组成。所有这些视频都伴随着Audios，采用不同的视频和/或音频操作方法来生产不同类型的假媒体。数据集中的视频具有各种持续时间，背景，照明，最小分辨率为360p，并且可能包含模拟传输误差和不良压缩的扰动。我们还开展了用户学习，以展示所作数据集的质量。结果表明，我们的数据集具有有希望的质量，可以在许多情况下欺骗人类参与者。

随着面部伪造技术的快速发展，DeepFake视频在数字媒体上引起了广泛的关注。肇事者大量利用这些视频来传播虚假信息并发表误导性陈述。大多数现有的DeepFake检测方法主要集中于纹理特征，纹理特征可能会受到外部波动（例如照明和噪声）的影响。此外，基于面部地标的检测方法对外部变量更强大，但缺乏足够的细节。因此，如何在空间，时间和频域中有效地挖掘独特的特征，并将其与面部地标融合以进行伪造视频检测仍然是一个悬而未决的问题。为此，我们提出了一个基于多种模式的信息和面部地标的几何特征，提出了地标增强的多模式图神经网络（LEM-GNN）。具体而言，在框架级别上，我们设计了一种融合机制来挖掘空间和频域元素的联合表示，同时引入几何面部特征以增强模型的鲁棒性。在视频级别，我们首先将视频中的每个帧视为图中的节点，然后将时间信息编码到图表的边缘。然后，通过应用图形神经网络（GNN）的消息传递机制，将有效合并多模式特征，以获得视频伪造的全面表示。广泛的实验表明，我们的方法始终优于广泛使用的基准上的最先进（SOTA）。

AI-synthesized face-swapping videos, commonly known as DeepFakes, is an emerging problem threatening the trustworthiness of online information. The need to develop and evaluate DeepFake detection algorithms calls for large-scale datasets. However, current DeepFake datasets suffer from low visual quality and do not resemble Deep-Fake videos circulated on the Internet. We present a new large-scale challenging DeepFake video dataset, Celeb-DF, which contains 5, 639 high-quality DeepFake videos of celebrities generated using improved synthesis process. We conduct a comprehensive evaluation of DeepFake detection methods and datasets to demonstrate the escalated level of challenges posed by Celeb-DF.

近年来，随着面部编辑和发电的迅速发展，越来越多的虚假视频正在社交媒体上流传，这引起了极端公众的关注。基于频域的现有面部伪造方法发现，与真实图像相比，GAN锻造图像在频谱中具有明显的网格视觉伪像。但是对于综合视频，这些方法仅局限于单个帧，几乎不关注不同框架之间最歧视的部分和时间频率线索。为了充分利用视频序列中丰富的信息，本文对空间和时间频域进行了视频伪造检测，并提出了一个离散的基于余弦转换的伪造线索增强网络（FCAN-DCT），以实现更全面的时空功能表示。 FCAN-DCT由一个骨干网络和两个分支组成：紧凑特征提取（CFE）模块和频率时间注意（FTA）模块。我们对两个可见光（VIS）数据集Wilddeepfake和Celeb-DF（V2）进行了彻底的实验评估，以及我们的自我构建的视频伪造数据集DeepFakenir，这是第一个近境模式的视频伪造数据集。实验结果证明了我们方法在VIS和NIR场景中检测伪造视频的有效性。

In this paper, we introduce MINTIME, a video deepfake detection approach that captures spatial and temporal anomalies and handles instances of multiple people in the same video and variations in face sizes. Previous approaches disregard such information either by using simple a-posteriori aggregation schemes, i.e., average or max operation, or using only one identity for the inference, i.e., the largest one. On the contrary, the proposed approach builds on a Spatio-Temporal TimeSformer combined with a Convolutional Neural Network backbone to capture spatio-temporal anomalies from the face sequences of multiple identities depicted in a video. This is achieved through an Identity-aware Attention mechanism that attends to each face sequence independently based on a masking operation and facilitates video-level aggregation. In addition, two novel embeddings are employed: (i) the Temporal Coherent Positional Embedding that encodes each face sequence's temporal information and (ii) the Size Embedding that encodes the size of the faces as a ratio to the video frame size. These extensions allow our system to adapt particularly well in the wild by learning how to aggregate information of multiple identities, which is usually disregarded by other methods in the literature. It achieves state-of-the-art results on the ForgeryNet dataset with an improvement of up to 14% AUC in videos containing multiple people and demonstrates ample generalization capabilities in cross-forgery and cross-dataset settings. The code is publicly available at https://github.com/davide-coccomini/MINTIME-Multi-Identity-size-iNvariant-TIMEsformer-for-Video-Deepfake-Detection.

主动演讲者的检测和语音增强已成为视听场景中越来越有吸引力的主题。根据它们各自的特征，独立设计的体系结构方案已被广泛用于与每个任务的对应。这可能导致模型特定于任务所学的表示形式，并且不可避免地会导致基于多模式建模的功能缺乏概括能力。最近的研究表明，建立听觉和视觉流之间的跨模式关系是针对视听多任务学习挑战的有前途的解决方案。因此，作为弥合视听任务中多模式关联的动机，提出了一个统一的框架，以通过在本研究中通过联合学习视听模型来实现目标扬声器的检测和语音增强。

Figure 1: FaceForensics++ is a dataset of facial forgeries that enables researchers to train deep-learning-based approaches in a supervised fashion. The dataset contains manipulations created with four state-of-the-art methods, namely, Face2Face, FaceSwap, DeepFakes, and NeuralTextures.

情绪识别涉及几个现实世界应用。随着可用方式的增加，对情绪的自动理解正在更准确地进行。多模式情感识别（MER）的成功主要依赖于监督的学习范式。但是，数据注释昂贵，耗时，并且由于情绪表达和感知取决于几个因素（例如，年龄，性别，文化），获得具有高可靠性的标签很难。由这些动机，我们专注于MER的无监督功能学习。我们考虑使用离散的情绪，并用作模式文本，音频和视觉。我们的方法是基于成对方式之间的对比损失，是MER文献中的第一次尝试。与现有的MER方法相比，我们的端到端特征学习方法具有几种差异（和优势）：i）无监督，因此学习缺乏数据标记成本； ii）它不需要数据空间增强，模态对准，大量批量大小或时期； iii）它仅在推理时应用数据融合； iv）它不需要对情绪识别任务进行预训练的骨干。基准数据集上的实验表明，我们的方法优于MER中应用的几种基线方法和无监督的学习方法。特别是，它甚至超过了一些有监督的MER最先进的。

这项工作的目的是通过利用视频中的音频和视觉流的自然共同发生来研究语音重建（视频到音频）对语音重建（视频到音频）的影响。我们提出了Lipsound2，其包括编码器 - 解码器架构和位置感知注意机制，可直接将面部图像序列映射到熔化谱图，而无需任何人类注释。提出的Lipsound2模型首先在$ 2400H的$ 2400h多语言（例如英语和德语）视听数据（VoxceleB2）上进行预先培训。为了验证所提出的方法的概括性，我们将在与以前的方法相比，微调在域特定数据集（网格，TCD-Timit）上进行预先训练的模型，以实现对语音质量和可懂度的显着提高扬声器依赖和依赖的设置。除了英语外，我们还在CMLR数据集上进行中文语音重建，以验证对转移性的影响。最后，我们通过微调在预先训练的语音识别系统上产生生成的音频并在英语和中文基准数据集中实现最先进的性能来培训级联唇读（视频到文本）系统。

我们在没有监督的情况下解决了学习对象探测器的问题。与弱监督的对象检测不同，我们不假设图像级类标签。取而代之的是，我们使用音频组件来“教”对象检测器，从视听数据中提取监督信号。尽管此问题与声音源本地化有关，但它更难，因为检测器必须按类型对对象进行分类，列举对象的每个实例，并且即使对象保持沉默，也可以这样做。我们通过首先设计一个自制的框架来解决这个问题，该框架具有一个对比目标，该目标共同学会了分类和本地化对象。然后，在不使用任何监督的情况下，我们只需使用这些自我监督的标签和盒子来训练基于图像的对象检测器。因此，对于对象检测和声音源定位的任务，我们优于先前的无监督和弱监督的检测器。我们还表明，我们可以将该探测器与每个伪级标签的标签保持一致，并展示我们的方法如何学习检测超出仪器（例如飞机和猫）的通用对象。

在今天的数字错误信息的时代，我们越来越受到视频伪造技术构成的新威胁。这种伪造的范围从Deepfakes（例如，复杂的AI媒体合成方法）的经济饼（例如，精致的AI媒体合成方法）从真实视频中无法区分。为了解决这一挑战，我们提出了一种多模态语义法医法，可以发现超出视觉质量差异的线索，从而处理更简单的便宜赌注和视觉上有说服力的德国。在这项工作中，我们的目标是验证视频中看到的据称人士确实是通过检测他们的面部运动与他们所说的词语之间的异常对应。我们利用归因的想法，以了解特定于人的生物识别模式，将给定发言者与他人区分开来。我们使用可解释的行动单位（AUS）来捕捉一个人的面部和头部运动，而不是深入的CNN视觉功能，我们是第一个使用字样的面部运动分析。与现有的人特定的方法不同，我们的方法也有效地对抗专注于唇部操纵的攻击。我们进一步展示了我们的方法在培训中没有看到的一系列假装的效率，包括未经视频操纵的培训，这在事先工作中没有解决。