Few-shot Adaptive Object Detection
with Cross-Domain CutMix

Most object detection methods have been studied for RGB images [26, 18, 41]. These methods can be roughly divided into two-stage and one-stage detection methods. R-CNN and its extended technologies [10, 11, 32] represent the two-stage methods. YOLO [31], SSD (Single Shot Multi-box Detector) [27], and their extended technologies represent the one-stage methods. Additionally, object detection techniques that use transformer have been proposed [6, 3, 52, 44, 28], and they are expected that it will be applied to various environments.

There is research on applications in the real environment such as robots [19, 36], drones [13, 42], object detection for in-vehicle cameras [23, 49], and license plate detection [43]. Many datasets [24, 7, 22, 34, 20, 4] that can be used to train object detection are available to the public. The night scenes on these datasets are fewer than daytime scenes. Furthermore, visibility is poor because the pixel value of the subject in the RGB image captured at night is small. Thus, it is difficult to achieve high detection performance using RGB images both day and night.

Highly accurate object detection with in-vehicle cameras and outdoor drones is required for both day and night. Some methods for detecting objects using spectra information other than RGB images were proposed to detect objects with high accuracy on both day and night. Lu et al. [47] proposed object detection using RGB and infrared images in the framework of weakly supervised learning. This method focuses on the use of multispectral information, and it detects objects using a roughly aligned RGB image and an infrared image. Liu et al. [25] and Konig et al. [21] proposed methods for inputting RGB and thermal infrared images into a deep learning model and fusing their features. Highly accurate object detection is possible under various lighting conditions using RGB and infrared images simultaneously. These methods are algorithms that assume that there are numerous RGB and infrared images. Thus, they can be used if the RGB and the infrared images can be photographed in large quantities and annotated in the same environment as the inference scene. However, the cost of collecting data and annotating in each application environment is high in reality.

To reduce the cost of preparing infrared images and training high accurate object detection models, domain adaptation and transfer learning use a pre-trained model with a small number of labeled infrared and RGB images.

Akkaya et al. [1] proposed unsupervised domain adaptation between a model taken from numerous RGB and thermal infrared images for image classification. Vibashan et al. [38] used paired RGB and thermal infrared images to perform domain adaptation for object detection. When only the recognized object is displayed as in the image classification, the domains of both the difference in the sensor and shooting scene can be applied. However, when the background area without objects occupies a large area in object detection, it is difficult to adapt the domain of both the sensor difference and the shooting scene difference without using a pair of datasets. Thus, it is still a problem to adapt between different sensors and scenes for object detection.

There is a knowledge transfer method, which is by fine-tuning infrared images, using a model trained with RGB images as pre-trained model. For fine-tuning to be effective, the feature of training data between source domain and target domain must be similar. RGB and infrared images have extremely different spectrum to be imaged, and they look very different even if they have the same object and color. Negative transfer [30, 40] occurs because of the difference in the distribution of these data. Therefore, knowledge transfer using a small number of labeled infrared images is difficult. In both domain adaptation and fine-tuning, the key to improving performance is transferring knowledge while making the differences between domains closer.

In this paper, we propose high-accuracy object detection on infrared images using a large number of labeled RGB images and a small number of labeled infrared images. RGB and infrared images receive different spectra; thus, there is a significant difference in appearance, which is a large gap between domains. Therefore, we not only align the gaps between domains at the feature level using methods such as adversarial learning but also explicitly reduce the gaps between domains at the image level. This improves the accuracy of domain adaptation by converting the input image to conditions that make it adapt the domain easier.

We propose Object aware Cross-Domain CutMix (OCDC) and OCDC-based Discriminator Label (OCDCDL) based on the domain for each location. Figure 1 shows our proposed framework. We explain the outline of the proposed method using the model of the domain adaptation method based on adversarial learning proposed by Han-Kai et al. [14] as an example. This method trais using the loss of both object detection and adversarial learning to reduce the difference between domains. This proposed method is simple and easy to incorporate into the type of domain adaptation that uses adversarial learning in object detection problems, which uses few labeled images.

OCDC (Fig. 1 (a)) is a method for cutting out an object area and pasting a part of the image between domains to reduce the gap between the source and the target domains. Zhou et al. showed that there is a domain generalization effect by mixing images with different domains in a batch [50]. Inspired by that study, we focused on mixing object units, which is important for object detection. When the entire image is mixed, it is trained to reduce the distance between the background domains that occupy most of the image. In the proposed method, the distance between domains is emphasized rather than the background features, so the object detection performance is expected to improve.

OCDCDL (Fig. 1 (b)) is a method for adaptively changing the domain label of the discriminator based on the pasting position of other domain images. By converting the input image using OCDC, the domain identification label is no longer one value. The output feature using the input image includes information from multiple domains by pasting an image of another domain on the input image using OCDC. Thus, the conventional single identification label cannot correctly discriminate the domain. By adaptively changing the label based on the OCDC, the discriminator makes it possible to discriminate the domain even if there is information on different domains in the image.

For domain adaptation to data with a large gap between domains such as RGB and infrared images, we propose a method of aligning domains at the image level in addition to the conventional method of aligning domains using features. A method based on data augmentation called DomainMix augmentation [39] was proposed: as a method for reducing the domain gap at the image level. This method reduces the difference in appearance between images by simply connecting different domains without using a deep generative model. Mixup [46], BC-learning [37], CutMix [45], and CutDepth [16] are data augmentation methods to be mixed at the image level. These studies mention that mixing images can generate features that interpolate two images. By mixing them at the image level, the features between the domains in the input image can be brought closer to each other.

However, the background area without objects occupies most of the image, and the objects that should align the domains are only part of the image. Thus, in Mixup or CutMix, which uses the entire image or mixes random regions, respectively, the gap in the background domain is small, but the gap in the domain of the object to be detected is not always small. Therefore, we propose an OCDC to cut out the object area existing in one domain and paste it onto the other domain. The domain gap is reduced at the image level by cutting out the object area from the source and target and pasting them together.

The pasting process is performed for each training iteration. The images of the source domain and that of the target domain included in the batch are used. The object to be detected is cut out from those images based on the ground truth coordinates. The object image cut out from the image of the source domain is pasted onto the image of the target domain selected at random. On the other hand, the image cut out from the object image of the target domain is pasted onto the image of the source domain selected at random. The ground truth labels for detection loss are updated by adding the ground truth label of the pasted image of another domain to that of the image where the image of another domain is pasted.

If the object images are pasted while the objects already exist, the originally existing objects will be hidden. The loss of object detection becomes large because it is difficult to detect such hidden objects. Therefore, we decide the pasting position of the image based on the overlap between the image to be pasted and the object of the image to which the object image is pasted. If the area where the object is hidden increases more than a certain percentage after pasting the image, the pasting position is reselected when deciding the pasting position. This prevents the original object from being hidden after pasting.

Additionally, to train the position and label of an object, the object detection model pastes an object image to a real-world location. For example, if the domain is adapted between the images of the in-vehicle camera, which is installed at almost the same position, the coordinates before and after pasting do not change significantly. Alternatively, the object image is pasted at the same position as the position before pasting or at a slightly shifted position.

The domain adaptation for object detection that has been used so far has insufficient consideration for object detection. On the other hand, our method using cutting out the position of objects is a new point of view that the domain gap of the object to be recognized can be reduced. In addition, we argue that it causes problems in adversarial learning and propose a solution. In the following subsections, each proposed method is explained concretely.

In domain adaptation using adversarial learning such as a method [14], features calculated from the source or target domains are input into the discriminator. In our proposed OCDC, the information of another domain is included in a part of the feature because the image of another domain is pasted on the part of the input image. Therefore, when one value is used for the domain identification label of the discriminator as in the conventional method, the loss of the area where the image of another domain is pasted cannot be calculated correctly.

The discriminator is trained so that neither of the two domains can be discriminated. We append the discriminator $D$ after the backbone $F$. The input image of the source domain and the target domain are $I_S$ and $I_T$, respectively. $D$ outptus a domain prediction map of each pixel $D(\cdot {)}_{h,w}$. The source domain identification label and target domain identification label are $d=0$ and $d=1$, respectively. Eq. 1 is a adversarial loss $L_{adv}$ and Eq. 2 is an overall loss $L$.

where $L_{det}(\cdot )$ is the detection loss, and ${\lambda }_{adv}$ is a weight that determines the loss balance. We set ${\lambda }_{disc}$ is 0.1 in our experiments.

However, because the features near the boundary of the area where the object of another domain is pasted using OCDC are a mixture of the features of the two domains, it is difficult to distinguish which domain is near this boundary. Thus, the loss near the boundary of the trained discriminator is smaller than that in other regions. However, information from different domains is rarely mixed at a position far from the boundary of the pasted object. Therefore, the loss of discriminator in the feature is large in that area.

Furthermore, we adaptively determine the domain identification label based on the position and domain of the image pasted by OCDC. After the pasting process, the RGB domain label is replaced with the domain identification label corresponding to the infrared image region, and the infrared domain label is replaced with the domain identification label corresponding to the RGB image region.

We explain the comparison method used for each experimental setting. In our experiment, images of the target domain are used for evaluation. Source-only and target-only labels in the tables of experimental results show the evaluation results when the model is trained using only the image of the source or target domains, respectively. The fine-tuning label shows the results using a pre-trained model and fine-tuning using target domain data. The target samples label shows the number of target domain data using fine-tuning. The Domain-Adversarial Training of Neural Networks (DANN) [9], one of the adversarial learning methods, is used as the baseline of the adversarial learning method. DANN label shows the detection results using domain adaptation with DANN. We use Faster R-CNN [32] as the detection network and VGG16 [35] for the backbone. In domain adaptation, the model parameters are pre-trained in the source domain. The height size is 600 of the image resolution, but if the maximum width size is more than 1,000, we set it to 1,000 while maintaining the aspect ratio. Ours label shows the result using the proposed method, which uses the OCDC and OCDCDL. The optimizer is SGD; the learning rate, the weight decay, and the momentum are 0.001, 0.0005, and 0.9, respectively. The batch size is one. The evaluation metrics are the average precision at an intersection over union (IoU), which threshold is 0.5. The front of the arrow indicates the source domain, and the tip of the arrow indicates the target domain.

BDD100k $\to$ FLIR: The BDD100k dataset [20] is collected based on six types of weather conditions, six different scenes, and three categories of time of data; the number of images is 100,000. This dataset is annotated in ten categories. FLIR ADAS dataset [12] is an image captured by a FLIR Tau2 camera, and the number of images is 10,228. Only thermal infrared images from this dataset are used. In our experiment, the training data includes 36,728 images labeled as daytime from the BDD100k dataset as the source domain data and 8,862 thermal infrared images used as training splits from the FLIR ADAS dataset. The categories person, bicycle, and car, which are common categories.

Table 1 shows the evaluation results. The detection accuracy of source-only is the lowest because this does not use knowledge of the target domain. In target-only, mAP is 72.1 % using all target data named Full. However, performances considerably deteriorates when the amount of data decreased. In fine-tuning, performances are higher than the performances of target-only because fine-tuning models had a knowledge that performance improves somewhat even in different domains. The performances of DANN are higher than fine-tuning because of the effects of domain adaptation. There is no significant performance degradation due to negative transfer, but the effect of domain adaptation can be confirmed. The performances of the proposed method tend to improve overall, but in particular, the performance of the person label outperformed DANN.

When the target samples are Full, the number of each object is large, so even if the object areas are small, there are enough numbers to improve the performance by domain adaptation. However, in the case of Full, there is only a difference of 0.3 points, so the conventional domain adaptation does not have a difference drastically. In particular, in this experimental result, it should be noted that the target samples, which are our targets, are smaller than 1/2, rather than the detection result of Full. Under these conditions, the our method has higher performance than the conventional method in almost all cases. We confirmed about 4 point improvements over DANN even if there is a few data. For example, if we consider person label, this is because the percentage of people in the dataset was high, and the overall percentage of pasting person’s images onto another domain based on the input image was high.

Caltech $\to$ KAIST: The Caltech Pedestrian dataset [5] is a dataset that contains labeled images of pedestrians captured using an in-vehicle camera. 42,782 images from this dataset are used for the images. The KAIST Multispectral Pedestrian dataset [15] is a dataset captured using the FLIR A35 microbolometer LWIR camera and contains 95,000 images labeled for pedestrians. The thermal infrared image of this dataset is used. In KAIST, 7,688 thermal infrared images are used for training, and 2,252 thermal infrared images are used for testing. This is based on the procedure in the paper [48]. In this experiment, only person is used as the category.

Table 2 shows the evaluation results. Source-only detection accuracy is extremely low because the appearance of RGB and thermal infrared images differs significantly. Target-only and fine-tuning detection accuracy deteriorated as the target sample decreased, as demonstrated in the FLIR experiment. Particularly, fine-tuning detection accuracy was lower than target-only detection accuracy by pre-training in the source domain. However, the performance degradation is suppressed when the target samples decrease. DANN and the proposed method perform better than target-only when there are many target samples. Performance is higher than fine-tuning when the number of samples decreased. The proposed method showed more than two points higher performance than DANN when target samples are 1/32 and 1/64. This experiment shows that the proposed method is more effective in single-class object detection than in multi-class object detection. In single-class object detection, when an image of another domain is pasted on an image of another domain, the image is rarely pasted to the detected object. Therefore, few occlusion problems due to pasting occurred in the BDD100k $\to$ FLIR experiment. Since the proposed method has a remarkable effect when the number of targets is small, it is expected to be effective in applications that are often used, such as pedestrian detection.

SIM10K $\to$ Cityscapes: The SIM10K dataset [17] is a composite of 10,000 images generated by the Grand Theft Auto (GTA) game engine and is annotated with cars and other similar images. The Cityscapes dataset [4] consists of real images captured in multiple urban areas and segmentation labels. We used the circumscribed rectangle of the object segmentation label as the bounding box for evaluation in the car categories. Furthermore, we used 10,000 composite images from SIM10K as a training set. Note that 2,975 images, which are training splits from Cityscapes, are used as training data, and 500 images, which are validation splits, are used as evaluation data. The evaluation is performed using the common category of car.

Table 2 shows the evaluation results. Similar to the previous results, the source-only detection accuracy is the lowest. If the target domain detection accuracy has a large amount of data, the target-only detection performance is high, but when the amount of data is small, the performance is significantly reduced. Fine-tuning detection accuracy reduces performance degradation when the amount of data is small. However, when the amount of data is small, the performance of DANN improves, so the effect of domain adaptation can be confirmed. When the target samples were 1/8, the proposed method and DANN had the same accuracy. Cityscapes are in-vehicle camera images, and the size of the car in the image is much larger than that of a person. Therefore, even if the number of data is reduced a little, the vehicle area can be used for domain adaptation in the same area as the background, so the accuracy is not so decrease. In this experiment, we confirmed that the proposed method is effective not only for domain adaptation between RGB and infrared images but also for conventional problem settings. This experiment showed that the proposed method is a general-purpose technique that can be used in various source and target data domains.

This section shows the comparison results under different experimental conditions. In either case, a comparison is made under the evaluation conditions of BDD $\to$ FLIR.

Contribution of Components: We evaluate the effect of OCDC and OCDCDL. Table 3 shows the results. For Person, the method using both OCDC and OCDCDL had high performance. On the other hand, Bicycle and Car have high performance even with OCDC alone. Our experiment do not consider labels near the boundaries of objects. Thus, even if a person with a small area makes a mistake in the discriminator label near the object boundary, the effect on detection accuracy is small. However, Bicycle and Car have a large area. Therefore, the performance decrease if the discriminator label near the object boundary is mistaken. However, in comparison with mAP, our proposed methods have the highest accuracy and effectiveness.

Region Selection Strategies: We compare the accuracy of whether the pasting position and scale are the same before and after pasting in OCDC. Table 4 shows the experimental results. A fixed label indicates position or scale are the same before and after pasting, and a random label indicates that the position and scale are set randomly. The detection accuracy in many cases is higher if the same position is maintained before and after pasting. For example, in the case of an in-vehicle camera, objects are concentrated on the lower side of the image. The object detection model trains a set of the position and the class. To train the relationship between the position and the class, which is unlikely to occur, does not have a positive effect on the inference result. In our experiment, by making the pasting position and size the same, the detection model is able to train the positions and scales that are likely to occur during inference.

Figure 2 is object detection results of FLIR, KAIST, and Cityscapes after domain adaptation performed in subsection 4.1. At the top of each dataset is the result when target samples is 1/16, and at the bottom of each dataset is the result when it is 1/64. The comparison methods are (a) fine-tuning, (b) DANN, and (c) Ours, respectively, and (d) Ground Truth. The car detection result is shown in magenta, and the person detection result is shown in cyan.

In the FLIR results, there is no difference in the car detection results, but there is a difference in the person detection results. In fine-tuning and DANN, even people with similar reflection intensities in thermal infrared images are not detected. On the other hand, the proposed method detects objects with similar reflection intensities, even if they are people far away. The proposed method adapted the domain to information about the reflection intensity of persons, which is a small area in the image. On the other hand, the conventional methods did not fully adapt the domain, so some objects could not be detected.

In the KAIST results, the conventional methods did not detect some small persons. In domain adaptation, it is difficult to adapt information on small objects because even if the information in a small object is ignored, the loss of the detection model decreases. However, the proposed method makes it easier to detect even small objects by explicitly giving information from other domains to the input image.

In the Cityscapes results, in both 1/16 and 1/64, the farthest car on the left side was not detected by conventional methods, but the proposed method could detect the car. This is because it is difficult to adapt the domain of a small object, which is the same reason as in the case of KAIST. In this experiment, we clarified the importance of explicitly giving information of other domains to the input image in domain adaptation of small objects as in the proposed method.