Face Anti-Spoofing from the Perspective of Data Sampling

During the past decade, we have witnessed great advancements in face anti-spoofing methods thanks to the emergence of deep learning models. However, with the growth of facial recognition technology, face identity threats are evenly increasing at the same rate and the problem is still unsolved due to the difficulty in the design of discriminative features. Therefore, how to effectively detect spoofing attacks remains a critical problem for both practitioners and the research community. Existing learning-based approaches can be classified based on static and dynamic information. Compared to static or image-based face presentation attack detection (PAD) [1], video-based face anti-spoofing is more challenging because deep learning methods based on a 2D convolutional neural network (CNN) ignore the temporal dimension of the video and process each frame independently.

Although spatiotemporal feature learning based on optical flow [2], 3D CNN [3] or recurrent neural network (RNN) [4] have demonstrated their performance in the face anti-spoofing domain, there is a fundamental question that still needs to be addressed in the field -What are alternative approaches to improve the spatiotemporal representations for face anti-spoofing? Wang et al [5] argued that temporal depth difference between live and spoof faces along with a contrastive depth loss can make impressive progress in improving the PAD performance. Zitong et al [6] proposed a method based on central difference convolution (CDC) via aggregating both intensity and gradient information. Recently, self-supervised learning [7], meta-teacher learning, or single-shot face anti-spoofing [8] were introduced for improving PAD performance. However, there is no clear winner among spatiotemporal feature learning based-methods in terms of accuracy.

We argue that most of the existing video-based countermeasures employ a fixed frame selection and overlook other important factors such as the importance of data understanding. Moreover, within a given video, not all the frames are of equal importance for PAD detection, and action-related frames may occur sparsely in a few frames. Thus, the key-frame sampling [9] is crucial before the feature extraction step and it remains an unsolved problem for video-based PAD. Although our previous works [4, 7] demonstrate that the spoofed videos are consistent with motion cues associated with the artifacts, i.e., hand trembling or material reflection, such methods require complex feature engineering skills to generate the spatiotemporal representations.

As a result, we need to develop a simple strategy of selecting relevant frames according to their temporal variations. Thus, using simply the Gaussian Weighting Function [10], we propose a simple data sampling mechanism, which aims to accumulate information from video sequences into a single RGB image and generate the most discriminative frames. Specifically, we demonstrate that appropriate data sampling provides an alternative solution to enhance the existing deep learning-based face anti-spoofing. Our main objective is to provide the importance of data understanding, especially, i) how differently does CNN or RNN behave concerning spatial-temporal modeling of video data? and ii) conduct comprehensive experiments and analysis to demonstrate that our approach is extensible, which can be plugged into different deep learning-based face PAD models. In summary, the overall contributions of our work include:

1. We present a data-driven approach to capture the appearance and dynamics of video into a single RGB image.

2. Our analysis shows that the proposed temporal modeling can amplify important clues, e.g., hand movements, and surface edges, to improve the detection accuracy.

3. We provide an interpretation of the decisions made by the employed model. The model revealed that the motion cues are the most important factors for distinguishing whether an input image is spoofed or not.

4. Experiments on three benchmark datasets, consisting of CASIA-MFSD, Replay-Attack, and MSU-MFSD databases, show that our proposed method is significantly superior to the state-of-the-art generalization methods used now.

The proposed data sampling is based on the assumption that adjacent frames in videos complement the temporal motion which is an important and universal signal. This also motivates us to adaptively select frames from videos. To achieve this, Gaussian Weighting Function (GWF) is used to accumulate the video sequences into a fewer number of frames. Suppose that $\left\{S_n\right\}n\in N$ is an exhaustive non-overlapping sequence (pertaining to the full length of the video), which is given by

Where $\left\{S_k\right\}$ represents the $k^{th}$ sub-sequence of $\left\{S_n\right\}$ and $k<n$. Specifically, Gaussian Weighting Function G, for a sub-sequence $\left\{S_k\right\}$, is denoted as follows:

The function $G$ takes a sub-sequence $\left\{S_k\right\}$, and Gaussian weight vector $M$ as input, and accumulates the representation into a single RGB frame. In particular, $M_q$ depicts the $q^{th}$ element of the Gaussian weight vector $M$. $Z$ represents the size of the Gaussian weight vector. For instance, if the size of the Gaussian weight vector $Z$ is 30 and the sub-sequence $\left\{S_k\right\}$ has thirty frames of the video. Then, the vector $M$ is given by $M$ = $[1,2,3,...,30]$. A single RGB image can be acquired based on the weighted summation of the thirty frames associated with the sub-sequence $\left\{S_k\right\}$ as illustrated in equation (2). Thus, following equation (2), different frame sizes can be selected to be accumulated into a single RGB image based on the GWF. Likewise, the same procedure is applied for the subsequent thirty frames belonging to the next sub-sequence and so on. The main steps are illustrated in Fig.1. Intuitively, this data sampling method has at least three main advantages. First, the encoded Spatio-temporal information can be fed to any CNN architecture for a still image, where “still” captures long-range dynamic variations in videos. Second, the encoded images decrease the number of frames per video to be processed and force the learning-based classifiers to focus on discriminative clips. Third, our method accumulates raw video frames directly in comparison to the previous data preprocessing methods that use complex processing such as estimation of global motion [7] or optical flow [2].

We conduct experiments on three major databases: CASIA Face Anti-Spoofing [11], Idiap Replay-Attack [12], and MSU Mobile Face Spoofing [13]. In order to determine the performance, we report Half Total Error Rate (HTER), and Equal Error Rate (EER) by following the existing testing protocols used in the current works [27, 5]. A pre-trained CNN (ResNet-101 [14]) is utilized to extract off-the-shelf CNN features using the pooling layer without data augmentation. All the images are resized to $224\times 224$. The extracted features are then used as input to train the Bidirectional Long ShortTerm Memory Networks (BiLSTM) [15] for final detection. To train the BiLSTM, the Adam optimizer is utilized by fixing a learning rate of $0.001$, mini-batch size $32$, validation frequency $30$, and with $100$ hidden layer dimension for all the intra-database protocols. An early stopping function [16] is utilized to reduce the risk of over-fitting since we do not set the fixed number of epochs. For the cross-database or inter-database scenario, the same experimental settings are used except the learning rate which was increased up to $0.0001$ for the BiLSTM. Moreover, He initializer [17] is used for initializing recurrent weights that perform the best for all our experiments.

To investigate the influence of temporal variations in the videos, we thoroughly validate the performance of each sub-sequence size by performing an ablation study. The numerical results are reported in Table 1. The result shows that even capturing the small temporal variations leads to a competitive performance by using our proposed data sampling. It is worth mentioning that the model is trained on the training dataset of CASIA and uses the CASIA testing set as a validation set. Based on the EER of the CASIA testing set, we report HTER on a completely unseen Replay-Attack dataset. Similarly, the same experimental protocols are repeated for the Replay-Attack dataset. We note that the proposed data sampling provides consistent performance even with a sub-sequence size of 30. When the temporal length increases to 40, we achieved the best performance on the CASIA dataset and very competitive performance for the Replay-Attack dataset. Thus, we set the same sub-sequence size (40) for the intra-database evaluation. For further analysis, ROC curves are presented in Fig.2. The true positive rate (TPR) and false positive rate (FPR) allow a learning curve in the upper left corner in case the model provides a higher sensitivity rate. Through Fig.2, we can see that the model classifies robustly in both databases.

We compare the performance against the state-of-the-art methods in the intra-database scenario and report the results in Table 2. As it can be seen from Table 2, the model provides state-of-the-art results for the CASIA, REPLAY, and MSU datasets. Specifically, our method achieves $0.00$, EER for the Replay-Attack, CASIA, and MSU datasets, respectively. When we evaluated the most challenging scenario, i.e., cross-dataset in terms of HTER, a significant difference can be seen especially for the CASIA dataset in Table 3. It improves the performance by up to 8$\%$ more than the previous state-of-the-art method [7]. This is a remarkable improvement for the cross-database scenario. Furthermore, we also report the results of the ResNet-BiLSTM model without our proposed data sampling. One can see that the performance is improved up to 20 $\%$ on both datasets. To demonstrate that the proposed data sampling can enhance the performance of other deep learning models, we also report the performance of VGG19 architecture [28] using a sub-sequence size (40). The results confirm the effectiveness of our approach. Based on these experimental results, we demonstrate that sufficient data understanding is important in PAD detection. In addition, we provide heat map visualizations based on occlusion sensitivity maps [29] and local interpretable model-agnostic explanations (LIME) [30] to understand what patterns contribute to making a classification decision. The first column in Fig.3 (a) shows the images generated by the proposed data sampling scheme. The heat map in Fig.3 (b) based on occlusion sensitivity maps provides vital information for the model prediction. One can observe that the proposed frame accumulation contributes positively to distinguishing real and spoof faces. Specifically, images of photo and replay-attack in the first and second row show that hand movement provides salient information. For further evaluation, LIME interpretation in Fig.3 (c) is provided to demonstrate that the proposed data sampling helps the models to focus on the paper’s texture and hand rotation cues. The masked images in Fig.3 (d) represent the most important superpixels to view the model’s decision in a human-understandable way.

In this paper, we addressed the face PAD issue by proposing a simple data sampling strategy that requires a Gaussian weighting function to aggregate the video sub-sequences into a single image. Since the motion-based cues are naturally presented in the format of video streams, we explored the dynamic variations in video sub-sequences to design an effective frame selection scheme. Extensive experiments on three datasets demonstrate that our proposed method is robust in both intra-database and inter-database testing scenarios. Our future work includes utilizing video classification methods to represent videos in a more discriminative form.