Detecting the unknown in Object Detection

The goal of open-set object detection is to detect objects that have not been seen during the training phase [dhamija2020overlooked]. To perform this task, the model is provided with a training dataset $T_{train}=\left\{\right(x,y\left)\right\}$, where $x$ is an image and $y$ is its corresponding ground-truth label. As in standard object detection, $y$ contains bounding box annotations for a set of classes, that we indicate as known classes $Y$. We note that, in object detection, the training images contains multiple objects, but not all of them are annotated. We denote all the objects not annotated as unknown objects. During testing, we are provided with a dataset $T_{test}$ containing objects of both $Y$ and never seen categories (unknown).

Focusing on R-CNN architectures [girshick2015fast, ren2015faster], our aim is to learn a function $F$ mapping an image $x$ to its known and unknown predictions at bounding box level. We consider $F$ as built upon four key components. The first one is the feature extractor $F_{FE}$ that is responsible for producing a feature map for each image $x$. The second one is a region proposal network $F_{RPN}$ that is fed with the feature map of the input image and produces a set of rectangular object proposals associated with a binary objectness score. In particular, it firstly projects the input feature map into a lower dimensional space by means of an intermediate projection layer and then the projected features are fed into two separate convolutional layers, one responsible for regressing the bounding box and the other responsible to output an objectness score $\omega$. The set of object proposals is then applied to the feature map and pooled, producing $R$ regions of interest (RoIs). The third component is the classification head and aims to classify the objects and regress the correct bounding box. It is composed by two functions: a class-scoring function $F_{\psi }:R\to {IR}^{R\times |Y|}$ that maps $R$ RoIs to a box-level class scores, and a function $F_{\rho }$ that regresses, for each RoIs and class, the precise bounding box coordinates, i.e. $F_{\rho }:R\to {IR}^{R\times 4|Y|}$. Finally, the last component is the unknown detection function $F_{\varphi }:[0,1{]}^{R\times |K_{known}|}\to {IR}^R$ that a binary score indicating if the RoI is unknown ($F_{\varphi }=1$) or not ($F_{\varphi }=0$).

In this work we take inspiration from out-of-distribution detection approaches [lee2018simple, hendrycks2018deep, vyas2018out, dhamija2018reducing, li2020background] that leverage external OOD data as extra supervision to learn richer decision boundaries between known and unknown samples. As collecting external unknown datasets is not always feasible, and most of the time human intervention is required to at least partially annotate them, our intuition is that we can leverage as OOD data a portion of the objects labelled as background. This is uniquely possible due to the object detection framework, that provides labels only for the classes of interest in $Y$, leaving all the other objects that are not meant to be learned not annotated.

Towards this end, we train our $F$ detector module in a four alternate steps strategy, adapting the methodology proposed in Faster R-CNN [ren2015faster] and proposing UNKAD (UNKnown Aware Detection), that extends the model classification ability towards unknown objects. In particular, in the first and the third steps the $F_{RPN}$ learns to extracts class-agnostic RoIs, while in the second and the fourth the detector $F$ learns to classify known classes $Y$ and the unknown. An illustration of UNKAD four steps training strategy is reported in Fig. 2.

Discovering the unknown. The intuition behind UNKAD is that unknown objects are already present in the training images and we can learn to recognize them by using the RPN class-agnostic ability to detect objects. In particular, the RPN exploits the ground truth labels to find and predict the RoI, i.e. n region where is likely to find an object. During preliminary experiments (see Section 4.3), we found that the RPN was able to detect, other than the annotated objects, the objects in the image background with high confidence. Arguing that its class-agnostic nature helps to find any object in the image, we propose to pseudo-label the objects detected in the background as unknown objects. In order to generate pseudo-labels during training, we propose to use a simple thresholding mechanism. In particular, we define a threshold to establish whether a RoI is a region containing an object (known or unknown) or not. The threshold ${\tau }_{obj}$ is derived from data, and it is computed as:

where

and $\mu$ and $\sigma$ represent respectively the mean and the standard deviation of foreground-layer activations, $R_{FG}$ is the set of RoIs that either (i) has the highest IoU, or (ii) has an IoU overlap with with any ground-truth box higher than $0.7$. We recall that ${\omega }_i$ represents the objectness score for the $i^{th}$ region fed to $F_{RPN}$. $\lambda$ is hyperparameter set to 1.

To obtain the pseudo-labels, we first select all the RoIs having an objectness score greater than ${\tau }_{obj}$. Then, we select all the RoIs that do not match with a ground truth annotation (i.e. they do not overlap more than 0.3 IoU with it) ad we associate them to the unknown class $u$.

Once we obtain pseudo-labels on the unknown objects that are present in the training images, we exploit them to train the classification head to detect every unknown object.

Learning to predict unknown As our primary goal is to learn an unknown detector through pseudo-supervision, we add to the final classification layer $F_{\varphi }$ an additional class that leverages the unknown pseudo-labels generated by $F_{RPN}$. More precisely, following the alternate training strategy of Faster R-CNN [ren2015faster, girshick2015fast], at the end of the first and the third steps, we perform the pseudo-labelling phase. In the second and the fourth steps, $F_{\rho }$ and $F_{\varphi }$ are learned, leveraging both annotated and pseudo-annotated data. By using the pseudo-annotations as ground truth, $F$ learns to separate at training time the unknown category from all the known classes.

In this section we present different strategies to predict a RoI as an unknown object, i.e. different implementation of the $F_{\varphi }$ function. In particular, we assume that, at inference time, we obtain $R$ RoIs from the RPN, which are passed to the classification head to obtain the final classification scores $s$ using $F_{\psi }$.

Direct prediction. The standard approach of localizing and classifying objects in closed-set scenarios is to localize the object and assign it the class with the highest class probability. Since UNKAD extends the classification logits also to the unknown class, we may apply the same principle and predict a sample as unknown only if has the unknown class has the highest score. Formally, given an image and a RoI $r$, we obtain the set $s$ of class scores, including also the score for the $u$ class, and we compute $F_{\varphi }$ as:

where $\hat{y}={argmax}_{c\in Y\cup u}{s}_c$ is the class with highest score, ${\omega }_r$ is the RPN objetness score for $r$, and $b$ indicates the background class. We note that we added the second case to avoid missing potential unknown objects.

Maximum Softmax Probability (MSP). Maximum Softmax Probability [hendrycks2016baseline] leverages the highest probability value assigned to any known class in $Y$ as a measure of uncertainty. For an image, given a RoI $r$ and the class scores $s$ for it, MSP computes the value for the unknown class $F_{\varphi }\left(r\right)$ as:

where ${\tau }_{MSP}$ is a user defined threshold that we set to $0.5$ and $b$ indicates the background class. Intuitively, if a class or background is predicted with a probability inferior to ${\tau }_{MSP}$, MSP identifies the RoI as an unknown object.

Energy-based classifier (Energy). The energy-based scoring function [liu2020energy] adopts a completely different perspective from traditional classifiers. Instead of computing class probabilities, it maps each samples to a scalar value, i.e. the energy. Given an RoI $r$ of a image, $F_{\varphi }\left(r\right)$ is computed as:

where T is the temperature hyperparameter, ${\tau }_{EN}$ is a user defined threshold set to $-3$, and $s_c$ is the classification score for class $c$. To align with the convention introduced by [liu2020energy], we keep the negative notation for the energy score, which is higher for known classes and lower for unknown ones.

ODIN. To enhance MSP performances, ODIN [liang2018enhancing] adopted temperature scaling and input perturbation. In detail, before feeding the scores to the softmax function, ODIN scales each class scores by a temperature parameter $T$. In addition, it also pre-processes each input image $x$ by introducing a small perturbation, that we adapt in the context of object detection as:

where $ϵ$ is the perturbation magnitude, ${\nabla }_x$ is the gradient vector with respect to $x$, and $p$ is the softmax of the scores $s$, i.e. $p=\text{softmax}\left(s\right)$.

As for MSP [hendrycks2016baseline], given a perturbed image $\tilde{x}$ and the class scores $s$ computed on the RoI $r$, ODIN computes $F_{\varphi }\left(r\right)$ as:

${\tau }_{ODIN}$ is a user defined threshold set to $0.4$.

Datasets. We conduct our experiments on the popular Pascal VOC 2007 [Everingham2009ThePV] and MS-COCO [lin2014microsoft] datasets. Pascal VOC 2007 is a widely used benchmark that includes 20 foreground object classes and consists in 5K images for training, considering both training and the validation splits, and 5K for testing. MS-COCO is a large scale dataset that provide 80K images for training, 20K for validation and for the test. It contains annotation for 80 object classes of which 20 are in common with Pascal VOC 2007. For the open-set evaluation, we follow the split defined as WR${}_1$ in [dhamija2020overlooked]. In particular, the test set of Pascal VOC 2007 is used for the evaluation on the $Y$ classes, while 4952 images from MS-COCO training set that do not contain any Pascal VOC class are selected for the evaluation on the $u$ class (see. Figure 3), resulting in a total of nearly 10k test images.

Baseline. We compare our proposed training strategy with the 4-steps Faster R-CNN [ren2015faster] training procedure. We implement both strategies upon multiple open-set and out-of-distribution detection strategies. In particular, we implement the training strategy described in Sec. 3.3: direct prediction, MSP [hendrycks2016baseline], Energy [liu2020energy] and ODIN[liang2018enhancing]. We note that it is not possible to use direct prediction on the standard Faster R-CNN training since it does not provide the unknown class in the classification head.

Architectures. Each of the evaluated method share the same Faster R-CNN architecture with a ResNet-50 backbone initialized with ImageNet pretrained weights [deng2009imagenet]. We train the network using SGD optimizer with batch size equal to $4$, learning rate set to ${10}^{-3}$ and decreased after $75\%$ of iterations by a factor of $0.1$. The weight decay is set equal to ${10}^{-4}$ and SGD momentum to $0.9$. The number of training iterations is set to $40k$ for both the $1^{st}$ and the $2^{nd}$ training step, while it is decreased down to $10k$ in the $3^{rd}$ and $4^{th}$ step. We resize images to $800\times 1333$ in both training and testing, while during training we also performed random crop and random horizontal and vertical flip operations.

Metrics. To assess the impact of the unknown objects on the performance of standard object detection models, [dhamija2020overlooked] introduced the metric called Wilderness Impact (WI), defined as:

where $T{P}_c,F{P}_c$, indicate the true positives and false positives of known classes, while $T{P}_o,F{P}_o$ the true and false positive on unkowns. However, in [dhamija2020overlooked] the metric was simplified since it did not considered the rejection option, i.e. the models cannot predict the unknown class, thus $T{P}_o=0$. They considered the following metric:

We note that $W{I}_{no\_rej}$ does not consider the detection performance on unknown classes, but it considers only the performance on the known classes. In particular, classifying all the unknown as background objects would get the optimal score since $F{P}_o=0$. Despite the second metric has been used by recent works [joseph2021towards, dhamija2020overlooked], we consider more suited to our task the $WI$ metric since it took also into account the true positive on unknown objects.

Moreover, since in our work we are explicitly interested in evaluating the models on unknown objects, we report the recall, precision and F1 metrics considering only the unknown class. We define them as:

The more the model tends to predict each sample as belonging to the unknown category, the higher the recall will be, but the lower the precision will be as the number of false positive tends to increase. For this reason, we introduce the $U_{F1}$ metric which summarizes both $U_{Recall}$ and $U_{Precision}$. $U_{F1}$ is maximized if and only if both $U_{Recall}$ and $U_{Precision}$ are maximized. It is defined as:

Finally, in addition to the open-set metrics, we also report the mAP to evaluate the model performances on $Y$ classes, defined as:

where $A{P}_c$ is the average precision for the class $c$ computed at different recall levels.

Tab. 1 reports the comparison among the standard Faster R-CNN training and UNKAD, considering multiple unknown detection strategies. As the table shows, our approach improves detection of both known and unknown objects. Starting from the former, standard Faster R-CNN architecture with no rejection capability benefits from our approach, achieving up to to 67.75% mAP. This behaviour is also confirmed by the WI improvement from 1.63 to 1.5 (the lower the better), indicating that UNKAD distinguishes better known and unknown objects. The same behavior on the known classes is evident when employing a rejection strategy: considering the WI metric, MSP improves from -17.52 to -19.22 and ODIN from -20.43 to -21.56, while Energy obtain similar results (-30.39 vs -29.82). We note that, while the Energy approach is the best on the WI and WI${}_{no\_rej}$ metric, it obtains very low mAP. We ascribe this behavior to the high unknown score that Energy assigns to samples, leading to the rejection of most of the known objects. Moreover, the contrast among the WI and mAP metrics reveals that WI is not well-suited to evaluate methods on the open-set task, since it does not consider the overall model performance but only the ratio among closed- and open-set precision.

While improving the results on known classes is important, our goal is mainly detect unknown objects. Considering the $U$-$F1$ score, we note that UNKAD increases the results of MSP from $3.10\%$ to $3.78\%$ and from $2.02\%$ to $2.43\%$ for ODIN [liang2018enhancing]. This remarks the impressive ability of our training strategy, that improves the detection ability of the model without introducing additional costs, pseudo-labeling unknown objects in the background of the images. We acknowledge that our training procedure slightly hamper the Energy performance. However, we remark that Energy already assigns a very high score for the objects, as demonstrated by the very low $U_{Precision}$ achieved by it. We also note that only Energy shows this behavior, while both MSP and ODIN, when using UNKAD, improve the results on both $U_{Precision}$ and $U_{Recall}$ metrics. Finally, comparing the simple direct prediction approach with the others, we see that it achieves comparable or even better performance. In particular, it achieves the best performance on $U_{Recall}$ without suffering performance drop, as indicated by the mAP performance. However, the $U_{Precision}$ is lower than other approaches (-1.74% w.r.t. MSP).

Direct prediction rejection strategy. In Table 2 we report a detailed analysis on the direct prediction rejection strategy available when using our UNKAD approach. It is composed by two key components: the additional unknown class on the final classification layer, able to predict the unknown class for each RoI; and the objectness threshold ${\tau }_{obj}$. As we can see from the second row, the additional unknown class allows to maintain the same $67.75\%$ mAP performance over the known classes of the standard Faster R-CNN with no rejection capability; while it also achieves 3.46% of $U$-$F1$. In the third row, we reported instead the results achieved by directly applying ${\tau }_{obj}$ to background RoIs. In particular, for each RoI predicted as background with the highest probability among all classes, if the $F_{RPN}$ emits an objectness score higher than ${\tau }_{obj}$ it is then considered unknown. Combining the two strategies together allows to achieve the best results, as shown in the last row of the table. The overall rejection procedure maintains the highest mAP on known classes, while also increases the $U$-$F1$ up to $3.67\%$ and decreases the WI down to $1.48$.

RPN unknown detection. Although $F_{RPN}$ is a class-agnostic detector by design [ren2015faster, girshick2015fast], an open question is whether it is able to recognize even objects on which it has not been explicitly trained on, as it is essential for the unknown pseudo-labeling procedure. To this end, we formulated the $AV{G}_{obj}$ metric as a quantitative measure of the $F_{RPN}$ ability to identify objects within an image in both closed- and open-set scenarios. It is formulated as follows:

where $P_{fg}$ is the set of ground truth foreground proposals and $f_{fp}\left(p\right)$ is the probability that the proposal $p$ is actually considered foreground.

We report in Tab. 3 the evaluation of standard training procedure and UNKAD  under the $AV{G}_{obj}$ computed on both known and unknown objects. Our approach achieves up to $0.99$ on known and $0.98$ on unknown (being 1 the upper bound), surpassing the standard procedure in both the evaluations. Achieving comparable performance in both closed- and open-set scenarios proves our intuition that $F_{RPN}$ is able to precisely detect objects despite their belonging to the training distribution or not. It is worth noting that UNKAD increases the confidence on considering proposals as foreground ones on the known classes, as shown in the comparison between the first row and the third one.