GAN-based generative modelling for dermatological applications – comparative study

In our experiments, the reference dataset for real images is based on the training set of the ISIC 2020 challenge [26] extended by malignant cases from previous years’ competitions [23]. The whole ISIC dataset collection is a leading standard open access database for medical image analysis using DL techniques.

Although the ISIC database contains, by the standards of medical data, an impressive amount of datapoints, there are some challenges related to the data quality and curation [8]. First of all, the dataset is highly unbalanced – only around 2% of its samples belong to histopathologically confirmed malignant melanoma cases. Moreover, the dataset has significant variation in terms of image acquisition set-up, colors, and sizes. Cassidy et al. [8] have identified a few artifacts which may represent a bias in the dataset, which further can affect the robustness of trained models. The artifacts can be divided into two categories – those coming from the patient (e.g., hair, skin tone) and those arising from the acquisition setup (e.g., dermatoscopic frames, clinical markings, stickers, physical rulers, pockets of air, and even removal of part of the lesion).

The considered ISIC database consists of the 37 648 images – the whole ISIC 2020 dataset, adding 4522 malignant samples from ISIC 2019 – where 20% were used for validation in first phase of central trainings. Later, we splitted the training subset based on patient ID attributes. To make the FL setup more appropriate, we ensured that the data from an individual patient would not be present on more than one client. For this setup, we created 3 clients and for them, data subsets with 2k, 12k, 20k images respectively. For each client the proportion of malignant and benign was roughly the same as in the whole dataset. In all experiments, we resized the input images to 256 × 256 pixels and normalized them using the same means and standard deviation per channel value as for the ImageNet dataset.

We investigated StyleGAN2-ADA performance using an original implementation from NVIDIA Research group²²2https://github.com/NVlabs/stylegan2-ada-pytorch. The ADA mechanism significantly stabilizes training in limited data regimes which we faced in the case of malignant melanoma samples. We trained StyleGAN2-ADA models with each of the two classes of training set as input, as well as in a conditional setting with and without augmentations. To select the best model, we considered both the Fréchet Inception Distance (FID) [12] and Kernel Inception Distance (KID) [7] metrics, along with training speed, similarly as proposed in [6]. At the end of this phase, we picked the best GAN to generate the corresponding dermatoscopy images for malignant and benign cases and continued the analysis of the generated synthetic data based on latent space exploration. For the projection of real images into the latent space of the generative model, we needed to optimize the latent code. For this purpose, we used VGG16 model as a feature extractor since this is the one used by default in the StyleGAN2 model by NVIDIA.

The classification task was performed using EfficientNet-B2 model [30], pretrained on ImageNet, with Ross Wightman’s implementation ³³3https://github.com/rwightman/efficientdet-pytorch. We chose this model based on the Kaggle competition scoreboard, and the model’s well-known performance on skin lesion analysis. As the main goal of the paper was examination of different GANs performance and the classification results on real ISIC2020 are publicly available on the Kaggle platform, the presented results are limited to one classifier. However, we employed for centralized scenario also GoogLeNet and ResNet-50 achieving slightly worse results. During training, we used the Adam method for optimizing the network weights with an adaptive learning rate initialized to 0.0005. We trained the models for a maximum of 20 epochs and an early stopping with a patience of 3 epochs. We applied standard data augmentation techniques, such as random rotation, horizontal and vertical flip, during the training phase for all experiments.

For the experiments in a FL setup, we used Flower framework [4]. We selected Flower framework to build our FL system on due to the simplicity of use, flexibility, ease of adaptation for custom datasets and the possibility to change from a simulated to a truly federated set-up. In our simulated setup, we created a network with 3 clients with different amounts of data and a server, where the weights of the trained model were exchanged every 100 iterations. We used the Federated Average (FedAvg) algorithm [20] as it is an effective and simple method that is commonly used for federated aggregation, and it is a built-in strategy in Flower.

There are different dimensions that one should take into account when evaluating GANs [1]. Firstly, fidelity as a measure of reliability, and diversity as a measure of fairness. Here, FID and KID scores evaluate these two characteristics, but rely on a preexisting classifier (InceptionNet) trained on ImageNet, and are insensitive to the global structure of the data distribution. In a similar way, we can look at Precision (P) and Recall (R) scores, where precision measures the fraction of synthetic samples that look realistic (fidelity), while recall measures the fraction of real samples that the generative model can synthesize (diversity). In order to evaluate the reliability and realistic look of the artificial images, a survey was carried out where experts assessed whether each image surveyed was from a real patient or generated artificially. For this type of subjective evaluation each participant was presented with 200 images roughly split between the classes. All images shown were downsampled to 256 × 256 pixels and generated from the cGAN model. On the other hand, Perceptual Path Length (PPL) [17] measures whether and how much the latent space is entangled or regularized, in the end, being capable of capturing the consistency of the images. Secondly, another dimension to look into is predictive performance. Although this is an immediate consequence of the above two concepts, it refers to the fact that samples should be just as useful as real data when used for the same predictive purpose. Here, we built a melanoma classifier to assess the predictive performance when using the synthetic data for training and the real data for testing.

As in the case of medical data, privacy is the most important factor, we evaluated the generalization or authenticity of the generative process [1]. It measures the rate at which the model invents new samples and is defined as the fraction of generated samples that are closer to the training set than other training data points. However, despite so many metrics, there is still no clear and gold standard protocol to evaluate the quality of the generated samples.

Finally, since the StyleGAN2-ADA framework learns a linearly separable latent space [16], it was tempting to explore if image editing is possible by manipulating the latent input of a trained unconditional GAN model on skin cancer images. Different methods were tested with the aim to see if we could obtain directions in the latent space where the influence of one feature could be controlled while keeping the rest of the image intact. Initially, we tried shifting a vector across the separating hyperplane between two features (e.g. dermascopic frame). This method showed qualitatively good results, however, it required using an auxiliary classifier as well as a large sample size of latent vectors. Later, we tried finding the principal directions of a large sample size and visually examined which directions yielded interpretable feature manipulation. Lastly, to mitigate the need to generate a large sample size of latent vectors, we used semantic factorization [29].

In the first phase of our experiments, we established the best model in terms of fidelity and diversity using well-known metrics such as KID, FID, P, R, and PPL (see Table 1). It is worth noting that the GAN responsible only for malignant melanoma generation (mal-GAN) had around 6 times less data than for benign cases (ben-GAN). Additionally, the vast majority of malignant melanoma examples in ISIC 2020 and ISIC 2019 exhibit a black frame of the dermatoscope, which highly biases the output. We discovered that using global explainability techniques [18], and it was also observed before by other researchers [21]. We used a conditional setting to provide the model with a wider variety of samples of dermatoscopy images, since there are a lot of characteristics that are common for both classes. We confirmed that this is beneficial for the minority class (malignant), achieving higher FID and KID scores. For this setting, we used ADA mechanism with and without (${}_{w/ocolor}$) color augmentation. The first experiment clearly led to leakage of color augmentation to the generated examples (i.e. a lot of skin samples had an unnatural red or violet hue). In addition, given the amount of data we tested whether in our particular case the augmentation could be detrimental for the training, so we repeated the experiments without any augmentation, obtaining significantly worse results.

We received four responses to the qualitative survey from two dermatologists and two deep learning experts. The overall average accuracy of the participants was 54%. There was no feature in any image that clearly suggested to the participants that the image was either real or synthetic. All experts were able to correctly label five of the 200 images as true positives, although they also falsely labeled seven of 200 images as false negatives. The participants expressed that they found it difficult to tell what image belonged to each class.

In the current use case, we have access to a large amount of data to generate realistic and diverse synthetic images. However, in a more realistic scenario, one single hospital can have a problem to gather enough amount of samples to work with GANs. To tackle this issue, we simulated a scenario where three hospitals with different amounts of data are connected in a FL setup (see section 3.1 for details). Using this simulated hospital scenario, we observed faster convergence and improved quality of the generated images mainly for the client with the smallest data resources - it converges around 1.6 times faster achieving a FID score similar to that of the central conditional GAN training. As the data distributions between different clients only differed in size, we put more emphasis on the classification task with centrally trained models.

After the evaluation with general metrics, we performed a study on predictive performance to measure how useful the synthetic data is for the subsequent task, i.e. malignant melanoma diagnosis (Table 2). As a baseline for the experiments, we first train the classifier on training subset of the real images of ISIC dataset, and then tested it on the validation set. Secondly, GAN-based augmentation was performed using two types of GANs models with two scenarios: training on balanced synthetic dataset with 55k images and testing on real (synt) validation subset (the same as in baseline experiment) and training on real images adding 22k synthetic melanoma samples (aug) to balance the dataset. The introduction of highly underrepresented malignant melanoma cases improves the classification accuracy roughly of few pp in both scenarios (Table 2). Overall GAN-based augmentation technique does not provide reliable improvements in case of classification using the whole ISIC 2020 and malignant samples from ISIC 2019.

To measure the authenticity and ensure that our generative model is not copying the real data we performed the following experiment. First, we projected 12 thousand samples from the real dataset into the latent space of the generator. This gave us the latent codes that caused our generator to synthesize the most similar output to the input image. To optimize for a latent code for the given input images, we used a VGG16 model as a feature extractor and computed the loss on the difference of the extracted features for both the target image and generated output and performed backpropagation.

Next, we extracted the features of both the real and their projected images using the last convolutional layer of our classifier trained on real and synthetic data. These embeddings were visualized in a 3D space using t-distributed stochastic neighbor embedding (t-SNE) method [19]. This allows visually exploring the closest near neighbors of each real image using cosine distances, and in this way examining how close the real images and their corresponding generated images are. Fig. 2 shows examples of real images projections in the latent space of the generator (with benign marked on red, malignant – blue) and projected embeddings of real and synthetic data. In both cases there is visible separation between two clusters created by two examined skin lesion classes. However, there are still plenty of the cases in the middle between two clusters, what is visible in Fig. 2(b), and even mixed with improper class (Fig. 2(a)). Additionally, we spotted some clusters inside classes, which are associated with instrumental bias, such as ruler and black dermatoscope frame.

For a more systematic inspection, we computed the cosine distances between the different pairs of real images and their projected samples. The mean distance was equal to 0.1444 and the median 0.00283 with only two projections being too close in terms of $Q1=0.013$ (range of 1e-5) to the real images. Only in these two cases the closest neighbor was the projection of the target image, meaning that the generative model could have memorized that sample. We treated this as a measure of the authenticity of the generated samples. We also spotted that some of the images were very distant from their projections (around 2) but still resembled the target image (Fig. 3).

We also scrutinized the images that lie in the boundary between the two clusters (Fig. 2), which correspond to edge cases, i.e. cases that are difficult to classify. These edge cases were studied using local explanations with the XRAI method [14], to identify why these samples were more difficult for the network. For images of malignant lesions that belong to the centroid of the embeddings, we found that the mole itself is the most important part of the image for the final prediction. In the sample image, the network focuses on boundary pixels which represent asymmetry in the mole, one of the main clues for detecting malignant melanomas. On the other hand, in edge cases the results are not as evident due to image distortions or poorly centred moles (Fig. 4). We selected all the misclassifications and edge cases and generated N neighbors using a distance of 0.1 to augment the dataset with more complex examples with the aim of making it more robust. First experiments showed an improvement in performance in those edge cases. However, in future a deeper analysis will be performed to confirm this statement.

Finally, we took care of explanation of trained GANs by providing the manipulations of latent input. The features whose influence we focused on reducing were the dermoscopic frames present in malignant melanoma images. We explored the semantic factorization (SeFa) method [29], which does not require a sample size of latent code or auxiliary classifiers. The latent $w$-vector corresponding to the image in Figure 5 was shifted along the second, forth and sixth eigenvectors, obtained using the SeFa framework.

In Figure-5 we see that applying SeFa image editing suggests less entangled features from visual inspection of the images across different directions. The eigenvectors displayed were chosen from a larger qualitative evaluation of 100 images along the first 10 largest eigenvectors. Visually, the second, fourth, and sixth eigenvectors showed the best result in removing the frames while leaving the other features intact. We observed a better result using this method of image editing compared to the other tested methods (for details, see Section 3.3).

To assess the quality of the edited images, we first generated a large sample size of images all containing frames. After acquiring the images we removed the frames by shifting the latent vectors along the direction where the presence of the dermoscopic frame was minimized. Finally, we trained a classifier on these images for the malignant melanoma with a training set of 10k images per class and a test set consisting of real images. The resulting test accuracy was $87\%$.