Probing Contextual Diversity for Dense Out-of-Distribution Detection

Consider a semantic segmentation decoder based on the popular spatial pyramid architecture [zhao2017pyramid, chen2017deeplab, chen2017rethinking, cheng2020panoptic], i.e. containing either a Spatial Pyramid Pooling (SPP [yoo2015multi]) or Atrous Spatial Pyramid Pooling (ASPP [chen2017deeplab]) structure. Such structures include a series of pooling or dilated convolutional operations applied in parallel to a set of feature embeddings. We refer to these operations as spatial pyramid modules. Each spatial pyramid module has a unique pooling scale or receptive field, allowing for scale invariance and providing the decoder with various degrees of context: the larger the receptive field the more global context at the cost of a loss in detail. The outputs of the spatial pyramid modules are concatenated and fed to a segmentation head that produces a single prediction.

In this section we propose a method for extracting the contextual diversity between the representations produced by the different spatial pyramid modules. By exploiting said diversity we are able to improve OoD segmentation performance and network calibration without the need of expensive ensembling of multiple models. Our method is non-invasive with regard to the main semantic segmentation task and lightweight in terms of computational cost, making it suitable for real-world applications without affecting segmentation performance. In addition, since our approach improves the uncertainty scores of an existing model, it can be easily combined with other state of the art approaches.

We start from a generic semantic segmentation architecture featuring $K$ spatial pyramid modules $F=\{f_{c_1},...,f_{c_K}\}$, each with a different context size $c_k$, as in Figure 2. The encoder features $\theta$ are fed to the pyramid modules producing a set of contextual embeddings $\Phi =\{{\varphi }_{c_1},...,{\varphi }_{c_K}\}$, where ${\varphi }_{c_k}=f_{c_k}\left(\theta \right)$. The segmentation output is produced by the global head $h_g$, which takes all the concatenated features $\Phi$. We denote the output logits of the global head as $z_g:=h_g\left(\Phi \right)$.

Our method consists of a simple addition to this generic architecture, namely the introduction of probes to extract context-dependent information from the main model. We pair each spatial pyramid module $f_{c_k}$ with a contextual prediction head $h_{c_k}$, which is trained to produce segmentation logits $z_{c_k}=h_{c_k}\left({\varphi }_{c_k}\right)$ from the context features of size $c_k$ (and the global pooling features, see [chen2017rethinking]). Given the input image $x$, we denote the prediction distributions as:

for the global and contextual heads respectively.

We train all the heads using a standard Cross Entropy loss given $N$ classes and the ground truth segmentation $y$ (we drop the mean operation over the spatial dimensions for simplicity):

although any semantic segmentation objective could be used.

The contextual heads are designed to act as probes and extract information from the context-specific representations. For this reason, we do not back-propagate the gradients coming from the contextual heads to the rest of the network, and only update the weights of the heads themselves. The spatial pyramid modules are distinct operations, each with its specific scope depending on its context size. By stopping the gradients before the spatial pyramid modules we force each head to solve the same segmentation task but using different features, preserving prediction diversity. As a byproduct, our architectural modifications do not interfere with the rest of the network and the main segmentation task.

A model for dense OoD detection should assign to each location in the input image an anomaly score. To obtain per-pixel OoD scores we test three scoring functions, applied to the outputs of the segmentation heads: maximum softmax probability (MSP) [hendrycks17baseline], prediction entropy (H) [smith2018understanding] and maximum logit (ML) [a_benchmark]. We adapt each scoring function to work with predictions from multiple heads. The maximum softmax probability is computed on the average predicted distribution over all the heads, including the global head:

Similarly, for the entropy we compute the entropy of the expected output distribution:

where $H$ denotes the information entropy. For maximum logit we average the logits over the different heads and compute their negated maximum:

All scores should be directly proportional to the model’s belief of a pixel belonging to an anomalous object, therefore for MSP and ML the negatives are taken.

StreetHazards [a_benchmark] is a synthetic dataset for semantic segmentation and OoD detection. It features street scenes in diverse settings, created with the CARLA simulation environment [dosovitskiy2017carla]. The 1500 test samples feature instances from 250 different anomalous objects, diverse in appearance, location, and size.

The BDD-Anomaly [a_benchmark] dataset is derived from the BDD100K [yu2018bdd100k] semantic segmentation dataset by removing the samples containing instances of the motorcycle, bicycle, and train classes, and using them as a test set for OoD segmentation, yielding a 6280/910/810 training/validation/test split. BDD-Anomaly and StreetHazards constitute the CAOS benchmark [a_benchmark].

Fishyscapes - LostAndFound [pinggera2016lost, blum2019fishyscapes] is a dataset for road obstacle detection, designed to be used in combination with the Cityscapes [Cordts2016Cityscapes] driving dataset. Its test split contains 1203 images of real street scenes featuring road obstacles, whose presence is marked in the segmentation ground truth.

RoadAnomaly [lis2019detecting] consists of 60 real world images of diverse anomalous objects in driving environments, collected from the internet. The images come with pixel-wise annotations of the anomalous objects, making them suitable for testing models trained on driving datasets.

Results for the anomaly track of the SegmentMeIfYouCan [segmentmeifyoucan2021] benchmark can be found in the supplementary material.

We evaluate OoD detection performance using the area under the precision-recall curve (AUPR), and the false positive rate at 95% true positive rate (FPR${}_{95}$). As is customary, anomalous pixels are considered positives. Other works include results for the area under the ROC curve (AUROC), however the AUPR is to be preferred to this metric in the presence of heavy class imbalance, which is the case for anomaly segmentation [davis2006relationship].

MOoSe relies on semantic segmentation models to perform dense OoD detection. For the experiments on StreetHazards and BDD-Anomaly we report results for two convolutional architectures, (DeepLabV3 [chen2017deeplab] and PSPNet [zhao2017pyramid], each with ResNet50 and ResNet101 backbones) and one transformer based (Lawin [Lawin], see supplementary material). For the experiments on LostAndFound and RoadAnomaly we use DeepLabV3+ [deeplabv3plus2018] with a ResNet101 backbone, trained on Cityscapes¹¹1Parameters available at:
https://github.com/NVIDIA/semantic-segmentation or BDD100k [yu2018bdd100k] respectively.

In this section we compare MOoSe with the single prediction baseline (global head) and with two types of ensembles. Deep ensembles [deep_ensembles] (DeepEns) consist of sets of independent segmentation networks, each trained on a different random subset of $67\%$ of the original data, starting from a different random parameter initialization [ilg2018uncertainty]. Similarly, multi-head ensembles (MH-Ens) are trained on random data subsets, but share the same encoder and only feature diverse prediction heads, for increased efficiency.

We compare to ensembles with 5 members/heads to match the number of heads in our method. Additionally, we pick the ensemble member with the median AUPR performance to serve both as the single model baseline and as initialization for MOoSe. The shared backbone of the multi-head ensembles also comes from the same model.

Table 1 shows results for the CAOS benchmark (StreetHazards and BDD-Anomaly) using DeepLabV3 and PSPNet as base architectures, with ResNet50 [he2016deep] backbones. We report results for the three OoD scoring functions described in Section 3.3; results for MOoSe are averaged over 3 runs, standard deviations are available in the supplementary material. For all datasets, architectures, scoring functions, and metrics, MOoSe consistently outperforms its respective global head. Similarly, MOoSe outperforms multi-head ensembles, as well as deep ensemble in most cases, while having a smaller computational cost than both.

In accordance with what observed in other works [a_benchmark], the maximum-logit scoring function tends to outperform entropy, most notably in terms of FPR${}_{95}$ and on the BDD-Anomaly dataset. Both scoring functions consistently outperform maximum-softmax probability. Moreover, maximum-logit appears to combine well with MOoSe by effectively reducing false positives. Results for models using the ResNet101 backbone are available in the supplementary material.

Figure 3 shows an example for OoD segmentation on a driving scene. The top row compares the entropy obtained using the global head (2(c)) and our multi-head approach (2(d)). The probes of MOoSe disagree on the nature of the anomalous objects in the image, and its aggregated entropy score is able to outline the anomalous objects more clearly than the global head. However, prediction disagreement also produces false positives for smaller inlier objects, such as the street sign on the right, highlighting a possible failure mode of our approach.

Here we compare MOoSe with the best approaches for dense OoD detection that do not require negative training data (see Section 4.5.3).

We are interested in comparing MOoSe to the closely related ensembles in terms of prediction diversity, of which a simple metric is variance. We compare the average variance of the output distributions of MOoSe and ensembles on StreetHazards and BDD-Anomaly validation, as reported in Table 4 (left). On both datasets our method’s predictions have higher variance than both ensembles.

Variance, however, gives us no insights on what the predictions disagree upon, and is therefore of limited interest. From the literature on Bayesian networks we can borrow a more informative metric: the mutual information (MI) between the model distribution and the output distribution [malinin2019ensemble]. Consider an ensemble of $K$ networks, or a multi-head model with $K$ heads. Each model or head produces a prediction $p(\hat{y}|x,k)$. We can compute the mutual information between the distribution of the models $k$ and the distribution of their predictions as:

which is the entropy of the expected output distribution minus the average entropy of the output distributions. MI is high for a sample $x$ if the predictions are individually confident but also in disagreement with each other. This tells us how much additional information the diversity brings to the overall model: if all the predictions are equally uncertain about the same samples they disagree on, then aggregating them will not affect the aggregated uncertainty estimate.

In Table 4 (left) we report the average MI on StreetHazards and BDD-Anomaly validation, comparing again MOoSe and ensembles. Similarly to variance, our method’s predictions have higher MI than both ensemble types, indicating that contextual probing not only produces more diversity in absolute terms, but also that this diversity adds more information to the model’s predictive distribution.

Finally, in Table 4 (left) we report the Expected Calibration Error [guo2017calibration] of all methods, to show that even if ensembles are better calibrated than the baseline, it is MOoSe that performs the best at uncertainty estimation overall.

In the previous section we showed that our approach produces highly diverse predictions. In this section we investigate the source of this diversity: our hypothesis is that each head relies differently on contextual information depending on the dilation rate of their respective spatial pyramid module, resulting in diverse predictive behaviors.

We test this hypothesis by evaluating the ability of each head to perform semantic segmentation when only contextual information is available. We corrupt the pixels of the foreground classes in BDD-Anomaly²²2Pole, traffic light, traffic sign, person, car, truck, bus. with random uniform noise while leaving the background pixels unchanged, then we evaluate how well each head can still classify the corrupted foreground pixels by relying on the context. An example of the process can be seen in Figure 5 (left). Figure 5 (right) shows the mIoU on the noisy foreground as a percentage of the foreground mIoU on the original clean image. We can observe that dilation rate and robustness to foreground corruption are proportional to each other at multiple noise levels, as further illustrated by the qualitative example in the figure. The different result quality for different dilation rates confirm the validity of contextual aggregation as a source of prediction diversity, as anticipated by the comparison with regular (non-contextual) ensembles on variance and mutual information in Section 5.1.

The results presented in Section 4 already show that contextual probing improves performance on the task. Moreover, results obtained from the application of MOoSe to transformer-based models (7.3% average AUPR increase on StreetHazards across scoring functions), which are available in the supplementary material, indicate that the principle is applicable across architectures and its gains are not an artifact of CNNs.

The last point to address is the contribution of contextual diversity to out-of-distribution detection. To quantify this contribution, we performed an ablation study removing receptive field diversity from DeepLabV3 by using the same dilation rate for all the convolutions in the spatial pyramid module. We train several versions of this single-dilation (SD) MOoSe, each with a different dilation rate, and present the comparison with standard MOoSe in Table 4 (right). Firstly, all single dilation models have lower prediction variance than regular MOoSe. Secondly, although the single-dilation models still outperform their global head, MOoSe yields larger gains than all SD models, both in absolute and relative terms. While these results confirm that contextual diversity is crucial for the success of our method, they also show that there are more contributing factors, compatibly with the known benefits of ensembles.