AWADA: Attention-Weighted Adversarial Domain Adaptation for Object Detection

Domain adaptation in deep learning object detection started in 2018 by introducing global and local gradient reversal layers (GRL [5]) into a Faster-RCNN object detector network [2, 19]. Recent work utilizes image and instance full alignment (iFAN) to align multi-level features between two related data domains [37]. The authors in [18] propose a similarity-based domain alignment by matching only features of objects belonging to the same cluster. Furthermore, normalizing features of two domains before applying GRL is discussed by Su et al. [23] and suppression of gradients is applied by Wang et al. [26]. In the direction of feature learning, there are several other publications [27, 1, 24].
Another research direction e.g. by the authors of [9], [12] and [31] leverage iterative pseudo labeling of the target domain. Work by Soviany et al. uses curriculum learning [22] by first creating pseudo labels for easy instances and iteratively takes more difficult instances into account. In [4], the authors applied Unbiased Mean Teacher, which uses knowledge distillation to align two domains in a student-teacher setup.
With DA-DETR [33], transformers were applied in domain adaptation for the first time. Using hybrid self-attention modules, they improve object detection performance while also providing a strong transformer baseline across different automotive benchmarks. Another research direction is using Graph Neural Networks (GNN) to close the domain gap by structuring the domain distributions using graphs [28, 13].
Contrary to other approaches, we do not modify the Faster-RCNN object detector from Ren et al. [17]. Therefore our work is orthogonal to other’s work [2, 8, 37], as we keep the detector network untouched and apply style-transfer to align the data distributions.

In semantic segmentation, the first approach suing style-transfer for domain adaptation was Cycle-GAN [36], introducing cycle-consistency to enforce consistent generation with reducing visual artifacts. CyCADA [7] introduced semantic consistency into the Cycle-GAN framework. MADAN [34] extended CyCADA by utilizing multiple source domains in their generation framework. Other orthogonal style-transfer frameworks as UNIT [15] and Pix2PixHD [25] have been released as well.
Style-transfer has also been applied to object detection by a wide range of other publications [9, 31, 22]. They mostly rely on a fixed Cycle-GAN style-transfer network to transform the image style of the source domain. However, the authors of [16] propose to reduce the patch size in training and adapt CyCADA from semantic segmentation to object detection. They have shown, that including semantic consistency in style transfer also improves object detection performance.
We extend this approach by introducing object detection-guided attention into the transformation cycle to modify the style-transfer network specifically for object detection, which has not been considered in related work before.

Leveraging foreground object attention mechanisms in domain adaptation is not new anymore. Zheng et al. propose a method guiding the GRL-based domain transfer by attention maps drawn from an object detector [35]. Specific for anchor-less object detection models [8] utilizes center and objectiveness scores to align intermediate feature representations. Yang et al. weight multi-level GRL predictions with hard-attention maps constructed from object proposals [29]. By utilizing pseudo attention maps acquired from box-level semantic segmentation, the authors of [32] propose a course domain alignment framework. Using self-attention on multi-level feature maps, ILLUME [11] guides cross-domain feature alignment with self-attentive gradient reversal layers.
Previous methods try to perform selective feature alignment based on attention mechanisms. In contrast, we combine attention with style-transfer by creating a feedback loop from object detection to the style-transfer network. We note that previous research using style-transfer does not connect the downstream task to the style-transfer network. Therefore we propose to use attention maps drawn from object proposals to re-weight the loss functions of style-transfer networks.

We assume a labeled source domain $D_S$ containing source images $X_S=\{x_S^i{\}}_{i=1}^{N_S}$ and corresponding bounding box labels $Y_S=\{y_S^i{\}}_{i=1}^{N_S}$. We further assume an unlabeled target domain $D_T$ with target images $X_T=\{x_T^i{\}}_{i=1}^{N_T}$ and no labels $Y_T=\left\{\right\}$. The number of samples in each domain is represented by $N_S$ for the source domain and $N_T$ for the target domain respectively. Our setting is unsupervised in the target domain, as we do not use any labels from it except for evaluation purposes.
With unsupervised domain adaptation, our aim is to extract predictions ${\hat{Y}}_T=\{{{\hat{y}}_T}^i{\}}_{i=1}^{N_T}$ from an object detector trained without target domain labels. The two domains should be related to each other regarding class occurrences.

Our proposed AWADA framework in Figure 2 is based on CyCADA* [16]. We extend on CyCADA* by adding our Attention-Weighting Modules (AWMs) to each generator and discriminator loss to focus the style-transfer network training on foreground objects. AWADA consists of a generator transforming source images to target-style images. Additionally, there is a generator transforming target images to source-like images. Our framework also consists of two discriminators that predict the root domain of each image.
Furthermore we define a novel training sequence for domain adaptive object detection. First, we train the CyCADA* style-transfer network without foreground-background distinction. Afterwards we train a Faster-RCNN [17] object detector on CyCADA* stylized source images $X_{ST}$ to construct attention maps for the target domain. We also train an object detector on the source images to construct attention maps for the source domain. Finally, we re-weight adversarial loss functions with the attention maps using our proposed AWMs and train AWADA to focus the transformation on foreground object regions. Afterwards we freeze the source-to-target generator of AWADA to transfer source images to the target style. Using these stylized source images, we train our final Faster-RCNN object detector and evaluate on the target domain.
By utilizing attention maps constructed from pre-trained object detector proposals, we are the first to connect the downstream task with the style-transfer network. Therefore, our proposed AWADA framework constructs a feedback loop between the object detection downstream task and the style-transfer network. Following we explain each individual components of our AWADA framework in more detail.

We follow the authors of [16] and adapt CyCADA [7] from semantic segmentation to object detection, and name their approach CyCADA*. In this section we describe some details of the latter, since our proposed AWADA framework is based on it.
Style-transfer networks based on Cycle-GAN [36] rely on two generators and two discriminators. The generators should generate high-quality fake samples, which the discriminators classify as such. A generator $G_{ST}$ transforms source image patches into target-like ones. Contrary, a second generator $G_{TS}$ transforms target images into source-like ones. The discriminators $D_S$ and $D_T$ use adversarial GAN loss $L_{GAN}^{ST}$ and $L_{GAN}^{TS}$ respectively to predict the root domain:

In addition, Cycle-GAN [36] introduced a cycle consistency loss $L_{CYC}$. Cycle consistency enforces the style-transfer network to remain the image content without hallucinating new objects. Using a L1 reconstruction loss, cycle consistency is calculated as shown in Equation 2 for both transformation directions:

Introduced by CyCADA [7], semantic consistency is used to retain semantic information through the style-transformation. Cycle-GAN often aligns data distributions by e.g. generating trees in sky areas, which leads to decreased performance of downstream tasks on stylized images. The authors of [16] observe that semantic consistency is generally a well-suited regularization mechanism for style-transfer tasks, independent of the downstream task. Using a pre-trained semantic segmentation network $F$ trained on the source domain Equation 3 shows the Kullback-Leibler divergence between predictions of the semantic consistency network on source and stylized source samples.

For training the full style-transfer model, all loss functions, including GAN losses of both generator and discriminator pairs, cycle-consistency loss, and semantic consistency loss, are weighted by factors $a_1$ to $a_4$ we derive from CyCADA [7]. We sum all weighted losses for the final loss as shown in Equation 4:

In this Section we describe how we construct our attention maps for training AWADA. Object detector models typically create object proposals separating foreground and background regions. We sample proposals for source domain images from a pre-trained object detector trained on the source domain. We sample proposals from an object detector trained on CyCADA* stylized source images for the target domain to create more accurate attention maps for the target domain than by training on the source domain directly. Compared to perfect predictions, proposals additionally introduce some context and hard negatives into training, which we found to be in fact beneficial in preliminary experiments. From object proposals, we create attention maps, highlighting foreground regions and masking out background regions.

Attention maps are created on input image resolution using the hard accumulation method $f$. Our method receives a set $S^c(u,v)$ of proposal confidences $\ge c$ for proposals at position (u,v) and outputs the attention map value based on Equation 5. For the source domain, we construct $N_S$ attention maps $A_{f,S}=\{a_{f,S}^i{\}}_{i=1}^{N_S}$. For the target domain, we construct $N_T$ attention maps $A_{f,T}=\{a_{f,T}^i{\}}_{i=1}^{N_T}$ the same size as the original patch.

As seen in Figure 2, we use our proposed Attention-Weighting Modules (AWM) to re-weight the style-transfer network loss functions in AWADA.
Typically, loss functions are averaged over all spatial loss pixels. In our AWMs we move this averaging behind the weighting of the per-pixel loss $L_{pix}$ with the corresponding attention map $a_{f,d}$ of domain $d$.

From the attention map associated with the input image, we crop the same region we used to crop a random input image patch. We then resize the cropped attention map to be the same size as the pixel-wise loss map.
We multiply the per-pixel loss $L_{pix}$ with the associated cropped and resized attention map $a_{f,d}$. Thus, only regions highlighted in $a_{f,d}$ remain in the loss and background is multiplied with zero. Afterwards we calculate the mean of the weighted loss map of size $W$x$H$ to form the final weighted mean loss $L_w$. This weighted loss is used in training to perform the model weight update.
Our generators should produce high quality foreground object regions, as we mask out background regions which are less relevant for the final detection task. We do not weight the cycle and semantic constsiency losses, as they are responsible for regularizing the whole GAN training. Further analyses are performed on this in the ablation studies.

We evaluate AWADA on three automotive benchmarks for object detection: Synthetic-to-Real, Adverse Weather and Cross-Camera domain..

Our experiments first compare AWADA to the current state-of-the-art methods in unsupervised domain adaptation. Table 1 shows results of detectors trained on an AWADA stylized Cityscapes dataset and evaluated on the Foggy-Cityscapes dataset for adverse weather and on BDD100k dataset for cross-camera adaptation. AWADA outperforms all current SOTA methods on both benchmarks. Both benchmarks are highly complex and show that AWADA is capable of improving detection performance on different automotive relevant classes. As we do not modify the object detector network, AWADA can be combined with approaches such as ILLUME or CGD to likely further improve object detection performance.
In Table 2, we report results of AWADA on synthetic-to-real adaptation and show, that AWADA is on paar with other methods on the synthetic-to-real benchmark. From the repetition of our experiments we see the increased stability of AWADA compared to CyCADA*. Unfortunately we cannot conduct such a comparison to other methods, since the corresponding results have not been reported.
AWADA requires a total of 88 hours trained on a single V100 GPU. As we pre-compute the attention maps, the remaining AWADA training requires 40 GPU hours, which is only about 10 hours longer than a CyCADA* training.

In this section, we conduct ablation studies on our proposed AWADA framework. We first show experiments on different types of loss functions we apply our Attention-Weighting Modules (AWM) to. Afterwards we show the influence of our attention map creation strategy on the object detection performance. At the end we analyze the total runtime of AWADA.