Deep Open-Set Recognition for Silicon Wafer Production Monitoring

During production, wafers get inspected to identify localized defects Sakata et al. (2005); Li and Tsai (2012); Cheon et al. (2019) and defect patterns in WDMs di Bella et al. (2019); Yu and Lu (2016); Fan et al. (2016); Chang et al. (2012); Nakata et al. (2017); Nakazawa and Kulkarni (2018); Yu et al. (2019); Saqlain et al. (2020), which is the problem we address in our work. Wafer Defect Maps are lists containing the coordinates at which inspection machines find defects, and these correspond to huge binary images (in our case $20,000\times 20,000$). This makes WDMs a challenging kind of data to handle, impossible to feed to a standard CNN. For this reason, the vast majority of existing solutions preprocess WDMs to reduce their size, usually building Wafer Bin Maps, which are smaller images (say $200\times 200$) where pixels are associated with small portions of the wafer, and the value of each pixel indicates whether the corresponding wafer portion contains defects or not.

The first supervised methods Yu and Lu (2016); Fan et al. (2016); Chang et al. (2012) employ hand-crafted features, including geometric regional features such as area, perimeter, or eccentricity of defect clusters Yu and Lu (2016), and density-based features such as the location of defect-dense areas Fan et al. (2016). Other informative features can be obtained by analyzing Wafer Bin Maps in a different domain using the Radon or Hough transforms to highlight specific patterns Chang et al. (2012). A few of these features are typically stacked in vectors and fed to a classifier such as a support vector machine (SVM) or a decision tree. However, hand-crafted features might not identify meaningful patterns in every condition, e.g., when patterns are rotated, shifted, or cover only part of the WDM. Moreover, hand-crafted features are usually defined to highlight defect patterns belonging to known classes. Thus, they might be meaningless for the detection of novel patterns.

Since Convolutional Neural Networks have achieved impressive results in image classification, the most recent methods Nakata et al. (2017); Nakazawa and Kulkarni (2018); Yu et al. (2019); Saqlain et al. (2020) employ Deep Learning models to classify Wafer Bin Maps. In particular, Nakata et al. (2017) addresses the simplified problem of distinguishing radial map patterns from non-radial ones. The solution presented in Nakazawa and Kulkarni (2018) proposes a specific preprocessing where the intensity value of each pixel represents the number of defects in the corresponding wafer portion. Our previous work di Bella et al. (2019), and others Yu et al. (2019); Saqlain et al. (2020) show the superiority of deep CNNs over traditional machine-learning methods based on hand-crafted features over the public dataset WM-811K Wu et al. (2015), where they achieve excellent classification performance. However, in these works, wafer monitoring is tackled as a closed-set classification problem, so these methods cannot detect novel defect patterns. Moreover, the WM-811K dataset contains small images representing Wafer Bin Maps, so we cannot use it to test the proposed solution, which takes the original WDMs as input.

Open-set recognition, introduced by Scheirer et al. Scheirer et al. (2014), is the problem of recognizing a certain number of known classes – for which a large annotated dataset is available – and detecting as novelties those samples that do not belong to any known class. Compared to the traditional closed-set classification, where all the classes occurring at test time are assumed to be known, the open-set recognition problem refers to a more realistic scenario, where only part of the classes have been already identified and included in the training set.

The first open-set recognition methods are based on traditional machine-learning algorithms. Scheirer et al. Scheirer et al. (2014) use modified Support Vector Machines with decision boundaries designed to reject unknown samples. Other methods detect novelties using the distance of test samples from the centroids of the known classes Bendale and Boult (2015), or the reconstruction error of sparse representations Zhang and Patel (2016).

More recently, deep open-set recognition methods started to gain more and more attention due to the outstanding results achieved by deep learning in most classification and pattern-recognition tasks. Bendale et al. Bendale and Boult (2016) propose the OpenMax function to replace SoftMax as the last layer of a CNN at test time. In particular, a standard CNN is trained on the known classes and, during testing, OpenMax evaluates the distance between the CNN score vectors and the mean activation vectors (MAVs), computed using the score vectors of training samples. Each MAV represents a known class, and a test sample is detected as a novelty when the likelihood of its distance from all the MAVs with respect to a Weibull distribution model fitted using the training set is below a certain threshold. Cevikalp et al. Cevikalp et al. (2021) propose deep classifiers using polyhedral conic boundaries to separate instances from different known classes, instead of the traditional linear boundaries. This makes the acceptance regions of the known classes more compact, thus easing novelty detection.

Another approach consists in applying an outlier-detection method to the latent representation of a deep classifier trained on known classes Zhu et al. (2018); Zhang et al. (2020). Instances from known classes are expected to be mapped in the same region of the latent space, following a multimodal distribution. Therefore, it is possible to detect novelties from their latent representations as outliers with respect to this distribution. To this purpose, Zhu et al. Zhu et al. (2018) employ a variant of Isolation Forest Liu et al. (2008), while Zhang et al. Zhang et al. (2020) define confidence intervals over each component of the latent space. In the context of wafer monitoring, Cheon et al. Cheon et al. (2019) follow the same approach to address open-set recognition on wafer surface defect images acquired by a scanning electron microscope (SEM). In particular, they apply a $k$-nearest neighbors outlier detector to the features extracted by a CNN trained on the known classes. However, this solution is designed to process traditional images of localized defects rather than WDMs, and thus it cannot be applied directly to the problem we address in this paper.

Socher et al. Socher et al. (2013) obtain a different latent representation by embedding the images into a semantic word space associated with the class labels. Then, they fit an isometric Gaussian model to represent each known class in the semantic space and use the likelihood as novelty score. More sophisticated deep methods for open-set recognition have been proposed, combining the reconstruction error of an autoencoder with classification features to detect novelties Yoshihashi et al. (2019); Oza and Patel (2019); Sun et al. (2020). However, these approaches based on autoencoders are designed for relatively small images and cannot be applied directly to WDMs.

The open-set recognition problem includes a traditional multi-class classification problem with a fixed set of known classes, i.e., those represented in the training set. Traditional Convolutional Neural Networks, which represent state of the art in image classification, cannot be directly applied to WDMs because they take as input relatively small images (e.g., VGG16 Simonyan and Zisserman (2015) and ResNet50 He et al. (2016) take as input $224\times 224$ RGB images), while images representing full-resolution WDMs are huge and result impossible to use to train and test CNNs.

To handle WDMs efficiently, we build a deep network based on Submanifold Sparse Convolution (SSC) Graham et al. (2018), a modified convolutional operator designed to process sparse images at arbitrary resolution. The output of an SSC is the same as that of a regular convolution, but only on the active sites of its receptive field, namely, the non-zero locations, i.e.:

where $\odot$ indicates the Hadamard product, $1$ the indicator function and $\text{supp}\left(w\right)$ the support of $w$, i.e., the set of its non-zero locations. The main advantage of SSC compared to traditional convolution is that it enables a very efficient processing of sparse images, regardless of their resolution. Thus, a WDM can be processed directly as the list of the coordinates where defects lie within the wafer, which correspond to the active sites. Moreover, (2) implies that SSC maintains the input sparsity – thus the shape of the defective patterns in WDMs – throughout the layers and does not increase the number of active sites, while regular convolutions reduce the sparsity of WDMs, as illustrated in Figure 2.

The architecture of the proposed SSCN is inspired by the convolutional part of the VGG16 Simonyan and Zisserman (2015), and is made of consecutive building blocks reducing the spatial resolution of the activation maps. Each block (Figure 3(a)) includes an SSC layer, with Batch Normalization (BN) and ReLu activations, followed by a Max-Pooling layer of stride 2. Thus, a single block reduces the resolution of the feature map by a factor 2 on each dimension. Our architecture, illustrated by Figure 3(b), is formed by 13 such blocks followed by a convolutional layer, and yields a 128-dimensional latent representation $L\left(w\right)$ of each WDM $w$, which we also employ to detect novelties (Section 4.2). Eventually, a fully-connected layer with SoftMax activations outputs a vector of $\#L$ scores, whose maximum determines the predicted class of the input WDM.

We remark that both our solution and those based on traditional CNNs extract a compact representation of the WDMs Nakazawa and Kulkarni (2018); Yu et al. (2019); Saqlain et al. (2020). The main difference is that traditional CNNs for WDM classification require a preliminary binning of the WDMs to reduce their resolution, which we believe might lead to a loss of information. In contrast, our solution is entirely data-driven and does not discard any piece of information contained in WDMs, thanks to Submanifold Sparse Convolutions. Our SSCN yields a spatial downsampling factor $2^{13}$ on each dimension, way larger than what is customarily obtained by CNNs for image classification: for instance, the CNN proposed in Saqlain et al. (2020) for Wafer Bin Maps only achieves a downsampling factor $2^5$. Although the network architecture made of convolutional and pooling layers recalls the VGG16 Simonyan and Zisserman (2015), our SSCN has a substantially fewer trainable parameters ($164,077$) due to the fact that the VGG16 has 3 large fully-connected layers on top of the convolutional part, which we do not include in our SSCN.

The second task in open-set recognition is detecting instances that do not belong to any known class, namely, novelties. This is particularly important in monitoring WDMs because a novel class of defect patterns must be detected as soon as possible and studied to determine which failure caused that unknown pattern. We follow the simple but effective idea illustrated in Figure 4 and apply an outlier detector to the latent representation of our classifier – i.e., the output of the penultimate layer of the network. This approach leverages the fact that a classifier maps instances belonging to the same class in the same area of the latent space so that the last, fully connected layer can separate the classes by linear boundaries Zhu et al. (2018); Zhang et al. (2020). Hence, the latent representation of the known classes can be described by a multimodal distribution $\varphi$, where each mode is associated with a known class. We expect novelties to be mapped in low-density regions of the latent space with respect to $\varphi$. For this reason, we fit a Gaussian Mixture Model (GMM) with $\#L$ components, namely the number of known classes, and obtain a density $\hat{\varphi }$ to describe the distribution of the latent representations $L\left(w\right)$ extracted by the SSCN trained on WDMs from known classes, i.e.:

where each Gaussian component $N({\hat{\mu }}_i,{\hat{\Sigma }}_i)$ for $i=1,\dots ,\#L$ represents a known class. Then, at test time, we employ as novelty score the negative log-likelihood of $\hat{\varphi }$, i.e., for each WDM $w$,

Eventually, a WDM $w$ is detected as Novel when $S\left(w\right)>\eta$, where $\eta$ is a threshold we set such that $P\left(S\right(w)\ge \eta |\ell \in L)=\alpha$, where $\ell$ is the label associated to $w$ and $\alpha$ is the target false positive probability.

To prevent $\hat{\varphi }$ from overfitting the latent representation of the training set, we use 90% of the data to train the SSCN, and compute the latent representation of another 5% of the training set. Then, we use these latent representations to fit $\hat{\varphi }$ by estimating its parameters using the well-known Expectation Maximization algorithm Dempster et al. (1977). Finally, we compute the novelty score $S\left(w\right)$ (4) of each instance $w$ of the remaining 5% of the training data, and set the threshold $\eta$ as the empirical $(1-\alpha )$-quantile of these novelty scores. Since we employ the same latent representation for recognizing known and novel patterns, novelty detection does not require additional computations other than those needed to compute $S\left(w\right)$, compared to closed-set classification.

The ST dataset is relatively small compared to the traditional datasets used for image classification and is characterized by an extreme class imbalance. Both these facts increase the risk of overfitting when training a deep classifier, and, in particular, the classification performance is likely to be poor on under-represented classes. To address this problem, we implement an ad-hoc data augmentation procedure based on a set of label-preserving transformations to be applied to the WDMs. Let $T^{\ell }$ denote the set of label-preserving transformations for WDMs belonging to the class $\ell$:

where $\theta$ indicates the parameters that define each transformation $T_{\theta }^{\ell }$, and ${\Theta }_{\ell }$ is the set of transformation parameters specific for each class $\ell \in L$. Each $T_{\theta }^{\ell }$ combines different transformations widely used for data augmentation on images, such as rotations around the center, horizontal flips, and small translations of the defective coordinates. Together with this customary set of transformations, we also apply two transformations that we specifically designed for WDMs:

We execute this data augmentation procedure in each training epoch to produce new augmented batches, and also when fitting $\hat{\varphi }$, to obtain a sufficient number of samples from under-represented classes. This is done by generating several versions of each original WDM $w$ using class-specific transformations $T_{\theta }^{\ell }\left(w\right)$, whose parameters $\theta$ are randomly sampled from ${\Theta }_{\ell }$.

To apply data augmentation when testing an open-set recognition method, it is important to verify that the transformations in $T$ preserve not only the known class labels $\ell \in L$, but also the Novel class label considered at test time. This is certainly guaranteed when $T$ is a group since any $T_{\theta }\in T$ that transforms a novelty into a sample of a known class would have an inverse $T_{\theta }^{-1}\in T$ that does not preserve the known class labels because it transforms an instance of a known class into a novelty. Here is a contradiction, therefore every $T_{\theta }\in T$ preserves the Novel class label.

In our case, $T$ is not a group because noise injection has no inverse transformation in $T$. However, noise injection naturally preserves every label, including Novel, since it simulates the noise affecting all the manufactured wafers. We can obtain the other transformations $T_{\theta }\in T$ by composing, in any order, a rotation around the center of an angle $\beta \in \{0^{\circ },{90}^{\circ },{180}^{\circ },{270}^{\circ }\}$, an optional horizontal flip and a translation in a random direction with maximum distance $\nu$. The inverse $T_{\theta }^{-1}$ of an element of $T$ can be obtained by composing, in the opposite order, the inverses of the rotation, horizontal flip, and translation that define $T_{\theta }$. This implies that also $T_{\theta }^{-1}\in T$, hence $T$ preserves the Novel class label by the same argument used above for the case in which $T$ is a group. Thus, we can safely employ the transformations in $T$ for data augmentation at test time without influencing the novelty-detection performance.

Here we summarize the proposed pipeline to classify a test WDM $w$: first, we generate a set of $N$ augmented maps

where each ${\theta }_i$ is randomly sampled from $\Theta$ (Section 4.3). Then, we feed all the WDMs in $A_w$ to the network, and average both the novelty score and the classification scores over all the augmented versions of $w$ contained in $A_w$ to stabilize the output, as in Simonyan and Zisserman (2015). Thus, output of our classifier $K$ is:

where $\eta$ is the threshold defined in Section 4.2 for the novelty score $S$ and $\text{SSCN}\left(w_i\right)$ indicates the classification scores of our SSCN (Section 4.1) obtained from $w_i\in A_w$. Thus, when a WDM is not detected as Novel, it is classified by taking the label $\ell \in L$ maximizing the traditional classification score, averaged over the $N$ augmented versions of $w$ contained in $A_w$.

Open-set recognition performance is typically assessed over models trained on datasets with numerous classes such as CIFAR-100 Krizhevsky et al. (2009), and ImageNet Russakovsky et al. (2015) so that a certain number of classes can be taken out during training and considered Novel at test time. However, in our industrial scenario, we have only 12 defect classes, and WDMs containing novel patterns have not been included in the dataset. For this reason, we follow a leave-one-out approach by training our model on all the known classes except a single defect class, which is considered Novel at test time. By doing so, we assess the capability of our model to recognize each defect class as novel when trained on the other classes. Since we want to detect novel defect patterns, we always consider the Normal class as known.

We employ 10-fold cross-validation and average all these figures of merit over the 10 test folds to provide an overall assessment of the classification performance. We also rank the considered methods (rank $=1$ for the best method, 2 for the second-best, and so on) according to the 1vsRest-AUC and the 1vs1-AUC, and compute their average rank over the 10 test folds to compare their classification performance, as suggested in Demšar (2006).

Detecting novel patterns is, in fact, a binary classification problem where novelties represent the positive class (Novel), and the known classes used for training are merged in the negative class (Known). Therefore, we can directly compute the AUC of these binary classification problems, which does not depend on class proportions, and this is crucial because novelties are rare. In contrast to other metrics such as F-scores, the AUC does not depend on how detection thresholds on the novelty scores are set, which allows to evaluate the effectiveness of different methods without setting the thresholds. To provide an overall assessment of the novelty-detection performance, we rank the considered methods according to the AUC and compute their average rank over the different novel classes as suggested in Demšar (2006).

In this experiment, we evaluate the classification performance of our solution over the thirteen classes identified so far by STMicroelectronics engineers. We train our SSCN using the Adam optimizer Kingma and Ba (2015) on an Nvidia Titan Xp GPU. Training the model for 100 epochs requires about 8 hours, while the average time to classify a WDM is $0.061\pm 0.055$ seconds. This time includes the generation and processing of $N=250$ augmented WDMs as in (7). We explain the large variability of the classification time by observing that the number of operations executed by SSC layers directly depends on the input WDM sparsity, which varies substantially from class to class.

We compare our SSCN to these models because fine-tuning CNNs for image classification is rather standard in wafer monitoring Wu et al. (2015), because it reduces the risk of overfitting. We do not employ custom architectures for wafer classification Yu et al. (2019); Saqlain et al. (2020) since the parameters of these models, pre-trained on the WM-811K dataset Wu et al. (2015), are not publicly available. Therefore, these deep models with several million parameters would risk overfitting when trained from scratch on the relatively small ST dataset. For a fair comparison with our SSCN, we train and test the VGG16 and ResNet50 using the same data augmentation described in Section 4.3 on the original WDMs, before reducing their resolution.

To assess the impact of data augmentation, we also evaluate the performance of our SSCN without data augmentation (both at training and test time), and with traditional augmentation, i.e., using only geometric transformations (translations, rotations, and flips). This experiment evaluates the importance of our custom transformations, namely noise injection and random mixing, in data augmentation.

First, we observe that methods trained and tested with our data augmentation procedure achieve consistently high accuracy, and that our SSCN outperforms the other methods on six out of thirteen classes. Overall, our SSCN is the best in terms of 1vs1-AUC on all the 10 test folds (avg. rank$=1$), and the Wilcoxon Signed-Rank test confirms that the differences are statistically significant (p-value $\le 0.05$). Notably, our SSCN outperforms the VGG16, which has a similar architecture to our SSCN, in both 1vsRest-AUC and 1vs1-AUC. The ResNet50 model is the best in terms of 1vsRest-AUC thanks to its high accuracy on the Normal class (approximately $96\%$), which is by far the most represented in the ST dataset (see Figure 1(b)), and the 1vsRest-AUC is sensitive to class proportions Provost and Domingos (2003). Our SSCN provides a better trade-off between the accuracy on the Normal class (over $93\%$) and accuracy on the other classes, as it outperforms the ResNet50 on nine out of twelve defect classes, often by a substantial margin. This explains the significant difference between the two in terms of 1vs1-AUC, which is not influenced by class proportions Hand and Till (2001).

To assess the impact of our data augmentation procedure on classification, we report in the fourth column of Table 1 the performance differences between the proposed solution, which leverages data augmentation both during training and testing, and the same SSCN architecture trained and tested without augmentation. Remarkably, training the VGG16 and ResNet50 without data augmentation leads to overfitting due to the fact that the ST dataset is relatively small and the considered CNNs have several million trainable parameters. In contrast, our SSCN can be effectively trained without augmentation since it has a substantially lower number of parameters. However, without augmentation, our SSCN achieves substantially lower classification accuracy on all classes except the Normal class, which is by far the most common in the ST dataset (see Figure 1(b)) and therefore can also be learned very accurately (approximately 96%) without augmentation. The SSCN without augmentation also achieves lower 1vsRest-AUC and 1vs1-AUC compared to our SSCN trained using our data augmentation procedure, and the differences are statistically significant (p-value $\le 0.05$) according to the Wilcoxon test.

Similarly, the last three columns of Table 1 report the performance differences between the models trained and tested with our augmentation procedure (which includes noise injection and random mixing) and the same models trained and tested using only traditional augmentation, namely geometric transformations. We observe that traditional augmentation yields lower accuracy on most classes, and lower 1vsRest-AUC and 1vs1-AUC compared to the same model with our augmentation. According to the Wilcoxon test, the difference in 1vsRest-AUC is significant (p-value $\le 0.05$) for the ResNet50. Remarkably, the differences in 1vs1-AUC are significant for all the considered models, which confirms that our augmentation procedure improves the robustness to class imbalance, since the 1vs1-AUC is not influenced by class proportions Hand and Till (2001).

In this experiment, we assess the performance of our solution in detecting WDMs containing novel defect patterns. After training our SSCN on the known classes, we fit a GMM on the latent representation and use the log-likelihood as novelty score, as illustrated in Section 4.2. In particular, we train twelve different models, each time taking out one of the defect classes, which we then consider Novel at test time. By doing so, we assess the open-set recognition performance of our solution in a realistic scenario where a new defective class unexpectedly occurs due to a production failure. Since some of the identified defect classes in the ST dataset (e.g., ClusterSmall and Incomplete) are very similar, we expect some novelties to be more difficult to detect than others.

• SoftMax is a widely employed open-set recognition baseline Hendrycks and Gimpel (2017). It is based on the intuition that novelties are typically classified with low confidence. Let v be the score vector associated to a WDM $w$, i.e., the output of the last fully-connected layer of our SSCN (or any neural network). Then, the SoftMax is defined as:

  $${\text{p}}_j=\text{SoftMax}(\text{v}{)}_j=\frac{\exp \left({\text{v}}_j\right)}{{\sum }_{i=1}^{\#L}\exp \left({\text{v}}_i\right)},$$

  (9)

  which guarantees that the scores are non-negative and sum to 1. The novelty score is simply $S\left(w\right)=-{max}_j{\text{p}}_j$, which is high when the posterior probability of the selected class is low.

• PreSoftMax is based on the same idea as SoftMax, but uses as novelty score the opposite of the maximum classification score before applying SoftMax, i.e., $S\left(w\right)=-{max}_j{\text{v}}_j$. As remarked in Hendrycks and Gimpel (2017), even if all the scores v of a novel instance $w$ are low, SoftMax might still assign $w$ to one of the known classes with extremely high confidence since posteriors are forced to sum to 1, thus preventing SoftMax to detect $w$ as Novel. The PreSoftMax scores are not subject to this constraint, so they might be more informative than the SoftMax scores in detecting novelties.

• OpenMax is a function designed in Bendale and Boult (2016) to replace SoftMax in open-set classifiers. This method employs as novelty score the likelihood of a Weibull distribution fitted on the distances between the score vectors v of a classifier and the Mean Activation Vectors (MAVs) of each known class, computed from the training set.

• We denote by SME the open-set recognition method based on SoftMax Entropy proposed in Geng et al. (2020). SME relies of the observation that the posterior probability vectors of novel instances tend to have a larger Shannon entropy compared to instances of known classes, thus the novelty score is defined as the Shannon entropy of the posterior probabilities obtained by our SSCN, i.e., $S\left(w\right)=-{\sum }_j{\text{p}}_j\log {\text{p}}_j$. This method is part of a broader solution for open-set recognition and one-shot detection Geng et al. (2020), but here we only consider the proposed open-set recognition solution.

• We denote by IFOR the open-set recognition method proposed in Zhu et al. (2018) that applies the Isolation Forest Liu et al. (2008) outlier detector to the latent representation of the WDMs, i.e., to $L\left(w\right)$. The method presented in Zhu et al. (2018) considers a latent representation including the features extracted by a classifier and an embedding of the class labels to a semantic word space. However, in our industrial setting, there is no meaningful embedding of the labels to a semantic space. Hence we only consider $L\left(w\right)$ as input.

• We denote by CI the open-set recognition method proposed in Zhang et al. (2020) that applies an outlier detector based on Confidence Intervals to the latent representation of the classifier, treating each component of $L\left(w\right)$ as an independent variable. This method computes a confidence interval $\hat{\mu }\pm \lambda \hat{\sigma }$ for each component, where $\hat{\mu }$ and $\hat{\sigma }$ are estimated from the training set, and its novelty score is the number of components of $L\left(w\right)$ that exceed their confidence intervals.

Each of these methods produces a novelty score and, similarly to ours, requires a decision threshold computed on a small part of the dataset, as illustrated in Section 4.2. However, we evaluate the novelty-detection performance by the AUC, which does not depend on the threshold, nor, most importantly, on class proportions. SoftMax, PreSoftMax and SME do not require training other than that of the SSCN. In OpenMax, we follow the procedure presented in Bendale and Boult (2016) to compute the MAVs and fit the Weibull model from the entire dataset used to train the SSCN. For a fair comparison with our solution, we adopt in IFOR and CI the same procedure illustrated in Section 4.2, so we fit the Isolation Forest and compute the parameters $\hat{\mu }$, $\hat{\sigma }$ from a small portion of the dataset (5%) that was not used to train the SSCN.

We remark that open-set recognition solutions based on the reconstruction error of autoencoders Yoshihashi et al. (2019); Oza and Patel (2019); Sun et al. (2020) cannot be directly used on WDMs, hence we have not considered them. Instead, we have focused on open-set recognition methods built on top of a pre-trained SSCN, which can accurately classify WDMs from known classes. This enables a fair comparison between the novelty detection power of methods that achieve the same classification performance over known classes, which is of paramount importance in our industrial scenario.

Table 2 indicates that the proposed GMM is the best method in detecting eight out of the twelve novel classes, five of which with statistical significance (p-value $\le 0.05$) according to the Mann-Whitney test DeLong et al. (1988). As we note in our first experiment, some classes are very similar and might be easily confused by the classifier (e.g., ClusterSmall and Incomplete), therefore they are more difficult to detect as novelties than others, as is apparent from Table 2. Most importantly, our solution achieves the best average rank among the considered methods, and the p-values of the Wilcoxon test show that there is enough statistical evidence to claim that GMM performs significantly better (p-value $\le 0.05$) than all the alternative methods.

The AUC values achieved by IFOR follow a trend similar to those of GMM, even though inferior in most cases. Since we expect that $L\left(w\right)$ follows a multimodal distribution due to the presence of different known classes, the explicitly multimodal GMM outperforms IFOR, which is completely nonparametric. In contrast, SME and CI achieve relatively good results only on few specific novel classes, such as ClusterSmall. SoftMax performs worse than most of the other methods: as expected, novelties are often classified with high confidence due to the SoftMax operation. OpenMax performs better than SoftMax according to the average ranks but worse than GMM. Perhaps surprisingly, PreSoftMax performs very well, and represents the best method to detect ClusterSmall and Incomplete. This is probably due to the fact that these two classes are difficult to distinguish, hence the SSCN often classifies them with low confidence.

The fact that PreSoftMax outperforms OpenMax and SoftMax outperforms SME shows that the highest classification score (before and after applying SoftMax) is the most informative for novelty detection in WDMs, confirming the intuition that novelties are likely to be classified with low confidence. By also considering the other scores, OpenMax and SME search for outliers in a 12-dimensional space where most dimensions are not informative for novelty detection, while PreSoftMax and SoftMax operate on a 1-dimensional space since they only consider the highest score. For this reason, we speculate that OpenMax and SME might suffer from an effect similar to detectability loss Alippi et al. (2016) i.e., the higher the dimensionality, the harder it is to detect a distribution change.