Comprehensive Dataset of Face Manipulations for Development and Evaluation of Forensic Tools

We compiled a dataset of edited portrait-style images. The image data was sourced from a subset of the CelebA-HQ dataset [4]. The CelebA-HQ dataset is a high-quality subset of the Large-Scale CelebFaces Attributes (CelebA) dataset [5]. The CelebA-HQ dataset consists of 30,000 high-quality versions of images in the CelebA dataset. The images denote square-cropped faces from photographs captured in-the-wild and are saved at a resolution of 1024x1024 pixels (versions at 128x128 and 256x256 pixels also exist). In our subset, we only consider identities that appear at least twice (i.e., there are at least two images of a given identity) in the image data.

We created two partitions of image data for training and testing purposes. The training partition contains a total of 6,846 total images. Each sampled CelebA-HQ image in the training partition is manipulated in five (5) separate instances, in combination with the original (unedited) image. Each sampled CelebA-HQ image is also paired with a separate (unedited) image of the same face identity as a reference. The five manipulations consist of “smile” (smile added or enhanced), “not smile” (smile removed or reduced), “young” (face is modified to appear younger), “old” (face is modified to appear older), and “surprised” (face is modified to include a surprised expression). We applied the Pivotal Tuning approach by Roich et al. to create each manipulated image [8]. The testing partition contains a total of 7,644 images and includes the same types of manipulated images as in the training partition and an additional seven manipulated images for a total of twelve images per identity (plus the original and a reference). In the testing partition there are additional examples for “smile”, “not smile”, “young”, and “old”, where the edit magnitude is reduced. In addition, there are three novel manipulations not present in the training partition. These include “purple_hair” (hair is modified to have a purple color), “angry” (face is modified to depict an angry expression), and “Taylor Swift” (face shape and features modified to appear similar to Taylor Swift). Each manipulation type is summarized in Table 1. An example of a subject with the set of applied manipulations is illustrated in Figure 3. In this example, each of the fourteen images would denote one subject in the testing partition. Labels that are italicized would not appear for a given subject in the training partition.

We remark that some identities may appear more than once in a given partition (training or testing), however an identity appearing in the training set will not appear in the testing set (and vice-versa). Both partitions are available in .png and .jpg format.

The objective of the detection experiment is to identify whether a given image has been manipulated. For a given image in the testing partition return:

• (string) “$<$filename$>$” : Image filename

• (bool) [0,1] : Not edited or Edited

We measure balanced detection accuracy as the proportion of images that are correctly recognized as either edited or not edited.

The objective of the classification experiment is to classify the type of edit in a manipulated image. For a given image in the testing partition return:

• (string) “$<$filename$>$” : Image filename

• (string) “pristine” : if not edited; “$<$edit_type$>$” : if Edited

We measure classification accuracy as the proportion of edited images that are correctly recognized of being a given edit type.

We compiled a dataset of edited in-the-wild-style images. The image data was sourced from a subset of the Flickr-Faces-HQ (FFHQ) [3]. The FFHQ dataset consists of 70,000 high-quality in-the-wild images. The authors of the FFHQ datset posit that the FFHQ data is much more variant in terms of age, ethnicity, background, and presence of facial covariates (e.g., eyeglasses, headwear) compared to CelebA-HQ. A version of the dataset consists futher of 70,000 detected, aligned, and cropped faces, which are saved at a resolution of 1024x1024 (a version at 128x128 also exists), but we only consider the raw, full-scene images. Example in-the-wild images from the FFHQ dataset are illustrated in Figure 4.

Our edited in-the-wild dataset consists of a randomly sampled subset of the 70,000 raw in-the-wild FFHQ images. In our subset, we allow for the possibility that an image contains more than one person (face). This potentially adds an additional challenge in detecting and localizing edited faces. We created two partitions of image data for training and testing (validation) purposes. The training partition and test partition contain totals of 1,508 and 1,403 images, respectively. Within each partition, approximately 50% of the images are edited, while the remaining images are “pristine” (i.e., not edited). In the training partition, 759 images are edited and 750 are pristine. For the testing partition, 652 images are edited and 750 are pristine. All images are saved in .jpg format with a randomly chosen quality factor in the set $Q_f\in [75,80,85,90]$. Summary information for the in-the-wild dataset is listed in Table 3.

Unlike the portrait-style images, each edited image is only subject to a single edit type. In other words, there are not multiple copies of the same underlying image but with different edits applied. Images that are edited are subject to one of six possible manipulations. These include “smile”, “not_smile”, “young”, “old”, “male”, “female”. We adopt the approach from Tzaban et al. to inject edits to in-the-wild images [10].

For our in-the-wild face manipulation dataset, edits are localized to a region of the full-scene image. This is in contrast to the portrait-style face manipulation dataset, where images are fully synthesized from face-based GAN’s. For the images in the in-the-wild face manipulation dataset that are edited, we also provide a binary mask that captures the spatial image region where the edit was performed and transplanted back into the image. The edit region is identified using a modified BiSeNet for faces [13]. Figure 7 illustrates examples of binary edit masks associated with edited in-the-wild images with one and multiple persons. Additionally, Figure 8 reports the proportion of the edit region for edited images. Typically, the size of the edit region is between 8% and 20% of the spatial image area.

The objective of the detection experiment is to identify whether a given image has been manipulated. For a given image in the testing partition return:

• (string) “$<$filename$>$” : Image filename

• (bool) [0,1] : Not edited or Edited

The objective of the localization experiment is to identify the specific image-region where an edit exists. For a given image in the testing partition, users must generate a binary mask, $\hat{M}$, denoting the estimated edit region. The estimated mask is compared against the ground truth, $M$ using Matthews Correlation Coefficient (MCC). The MCC (phi coefficient, or mean-square contingency coefficient) is a measure of association for binary variables. MCC is computed from the confusion matrix of the pixel-based binary estimations. This is mathematically described in Equation (1), where TP denotes True Positive ($M_k={\hat{M}}_k=1$), TN denotes True Negative ($M_k={\hat{M}}_k=0$), FP denotes False Positive ($M_k=0$, ${\hat{M}}_k=1$), and FN denotes False Negative ($M_k=1$, ${\hat{M}}_k=0$).

The objective of the classification experiment is to classify the type of edit in a manipulated image. For a given image in the testing partition return:

• (string) “$<$filename$>$” : Image filename

• (string) “pristine” : if not edited; “$<$edit_type$>$” : if Edited

In our in-the-wild face manipulation dataset the types of edits that are present in the training partition are also represented in the testing partition. Similarly, the types of edits that are in the testing partition are also represented in the training partition. Thus, the classification problem for this dataset is closed-set. This is in contrast to the portrait-style data, where novel edit types exist in the testing partition. We encourage users utilizing this data and challenge problem to consider open-set solutions as the set of potential edit types is near-unlimited.