Data-free Dense Depth Distillation

Monocular depth estimation (MDE) aims to predict scene depths from only a single image. Deep learning-based approaches have dominated recent progress [laina2016deeper, ma2017sparse, fu2018deep, huynh2020guiding, jiang2021plnet] in which the advanced performances are attributed to modeling and estimating depth using large and complex CNNs with data-driven learning.

On the other hand, deploying MDE algorithms into real-world applications often faces practical challenges, such as limited hardware resources and inefficient computation. Therefore, an emerging requirement of MDE is to develop lightweight models to meet the above demands. This problem has been specifically considered in previous studies [Mancini2016FastRM, Nekrasov2019RealTimeJS, Wofk2019FastDepthFM, hu2021boosting] where several different lightweight networks have been designed.

However, lightweight networks inevitably degrade their MDE performance due to the trade-off between the model complexity and accuracy. Hence, it remains an open question: how to reduce the model complexity while maintaining high accuracy. One potential solution to this problem is knowledge distillation (KD) which transfers the knowledge from a cumbersome teacher network to a compact student network with descent accuracy improvement. However, KD requires the original training dataset for implementation. Currently, there are no existing solutions in data-free scenarios for MDE.

Knowledge distillation [Hinton2015DistillingTK] was initially introduced in image recognition where either the soft label or the one-hot label predicted by the teacher is used to supervise the student learning. Existing methods can be generally categorized into two classes considering whether they can access the original training set: 1) the standard data-aware KD, and 2) data-free KD.

For data-aware KD, its effectiveness has also been demonstrated on various vision tasks, such as image recognition [Hinton2015DistillingTK], semantic segmentation [Liu2020StructuredKD], object detection [chen2017learning], and depth estimation [Pilzer2019RefineAD, Wang2021KnowledgeDF], etc. In addition to the conventional setup, researchers have proposed to improve KD via distilling intermediate features [Huang2017LikeWY, Liu2020StructuredKD], distilling from multiple teachers [Tarvainen2017MeanTA, Liu2019KnowledgeFI], employing an additional assistant network [Mirzadeh2020ImprovedKD], and adversarial distillation [Chung2020FeaturemaplevelOA, Shen2019MEALME].

For data-free KD, researchers resorted to synthesize the training set from random noises [Lopes2017DataFreeKD, yin2020dreaming, Fang2019DataFreeAD, Fang2021UpT1, Yoo2019KnowledgeEW] or employ other large scale data from different domains [Chen2021LearningSN, Xu2019PositiveUnlabeledCO, Fang2021MosaickingTD, nayak2021effectiveness]. However, existing methods are most effective for classification tasks due to the natural property of deep classification models and cannot be applied to MDE. We will elaborate these observations in Sec. III-A. In this paper, we propose the first method of data-free distillation for MDE. Our method leverages data from simulated environments to distill a model trained on a real-world dataset.

We especially clarify the differences between data-free KD and domain adaptation (DA), e.g., sim-to-real adaptation, since they may cause some misunderstandings. Data-free KD essentially differs from DA in two aspects. First, DA transfers a model from the original domain to a different domain, while data-free KD aims at preserving the model accuracy in the original domain. Second, both RGB images in the original and the new domains are usually considered prior information and can be freely accessed in DA. In contrast, training images in the original domain are unknown for data-free KD.

Image Mixing is a common technique used for applying data augmentation in semi-supervised learning. One can generate new training pairs in data-scarce scenarios by linearly blending two images and their respective labels (or pseudo-labels). Then, those mixed images and labels are utilized for model training. Classical methods of image mixing include MixUp [zhang2018mixup], MixMatch [Berthelot2019MixMatchAH] for pixel-wise blending, and CutMix [Yun2019CutMixRS], ClassMix [Olsson2021ClassMixSD] for mask-based mixing, i.e., exchanging parts of patches or objects between two images. Among them, the former three methods are applicable to classification, and the ClassMix is tailored to image segmentation.

In our method, we apply object-wise mixing to cover the distributed objects in the original training dataset. Unlike the above methods, we only mix RGB images because mixing depth maps will destroy the geometric relations among objects and yield wrong target labels. In a nutshell, instead of generating training pairs of the mixed images and the mixed labels, we use pairs of the mixed images and predictions from them as additional training samples.

Given the above difficulties, data-free KD for MDE seems intractable, since no correct depth maps are available. A plausible way is to use some OOD data if we can decipher the essential requirements for $X^{\prime }$. Here, we consider that three factors are essential for selecting $X^{\prime }$: i) scene structure similarity to $X$, ii) data-scale for performing KD, and iii) domain gap between $X^{\prime }$ and $X$.

We conducted preliminary experiments to analyze how a network reacts to different types of OOD data. Specifically, we let a model trained on the NYU-v2 [NYUv2] dataset as the teacher and apply KD with several different OOD datasets with the same number of randomly sampled images.

Fig. 3 shows the histogram of depths predicted by the teacher for different datasets, where Fig. 3 (a) denotes the histogram of the target domain. First, it is observed that the teacher yields similar depth histograms for those datasets as all exhibit a long-tail distribution. Second, although the teacher tended to produce smaller depths for OOD data, the outputted depths are still constrained in the target distribution. Based on these observations, we can say that the teacher is able to inherently produce depths in the target depth distribution even for data sampled from different domains. However, this constraint is not sufficient to ensure KD, as shown in Table I where random Gaussian noises led to the lowest performance even though they yield similar depth histograms.

Then, we analyze the effect of the above three factors by comparing target scenarios, data scale, and data domain in Table I. Except for $\left(g\right)$, all datasets have 50,000 (50K) images and $\left(b\right)=\left(f\right)>\left(e\right)>\left(c\right)>\left(d\right)$, in terms of scene similarity to the original data, i.e., $\left(a\right)$. The ${\delta }_1$ accuracy is consistent with the scene similarity in input space. In addition, $\left(g\right)$ is an augmented version of $\left(f\right)$ obtained by increasing the data scale. Not surprisingly, high scene structure similarity and small domain gap in images, and training models using large-scale datasets are beneficial for performance boost.

However, it is challenging to satisfy all these three conditions simultaneously. Considering the difficulties of data collection in the real world applications, we prefer to apply data-free KD for MDE with simulated images.

Despite a significant domain gap, we observe only an $11.9\%$ accuracy drop from $\left(a\right)$ to $\left(f\right)$. A reasonable interpretation is that some monocular cues, such as lines, and object boundaries, are essential to activate the related neurons and form a meaningful internal representation inside the teacher model. It can be validated in Fig. 4 that the depth maps inferred from the simulated images are perceptually correct despite being wrong in absolute scales.

We have shown that the trained model would map the out-of-distribution data to the depth histogram distribution in the target space. Admittedly, Eq. (3) is equivalent to Eq. (2) if $P_{X^{\prime }}=P_X$. However, due to the domain gap, there is a significant discrepancy between $X$ and $X^{\prime }$. Thus, we wish to mitigate this domain gap and accordingly improve KD. Since the original trained data is unavailable, we leverage the running average statistics captured in batch normalization (BN) as DeepInversion [yin2020dreaming] to regularize $x^{\prime }$. Specifically, assuming that feature statistics follow the Gaussian distribution and can be defined by mean $\mu$ and variance ${\sigma }^2$, then, $x^{\prime }$ is optimized through the following loss

where $u_l\left(x^{\prime }\right)$ and ${\sigma }_l^2\left(x^{\prime }\right)$ are the batch-wise mean and variance of feature maps of the $l$-th convolutional layer of $N_t$, respectively. ${\overline{u}}_l$ and ${\overline{\sigma }}_l^2$ are the running mean and variance of the $l$-th BN layer of $N_t$, respectively. Eq.(5) allows regularizing $x^{\prime }$ to approach the feature distribution of the teacher model. However, this optimization requires thousands of iterations¹¹13000 iterations in DeepInversion. for a single batch and is highly time-consuming. To tackle this problem, we use an additional network $G$ for data transformation. Then, Eq.(5) can be rewritten as

It is essential to ensure the fidelity of the original scenes to avoid arbitrarily meaningless transformation. Thus, we adopt an image reconstruction loss. Finally, the transformation of $x^{\prime }$ is formulated by

where ${\ell }_{rec}=\parallel x^{\prime }-G\left(x^{\prime }\right){\parallel }_1$ is the reconstruction error that penalizes the ${\ell }_1$ norm of image difference, and $\alpha$ and $\beta$ are weighting coefficients.

Since we have no clues about training data, including objects, textures, scene structures, etc., we naturally consider applying data augmentation to maximally cover the distributed patterns of objects in the target domain. We randomly change half of objects between two simulated images to obtain a new image with the help of semantic maps collected from the simulator. More formally, for two images $x_i^{\prime }$ and $x_j^{\prime }$ where $x_i^{\prime }\in X^{\prime }$, $x_j^{\prime }\in X^{\prime }$, we generate a new mixed image ${\hat{x}}^{\prime }$ by

where $m$ is a binary mask obtained from the semantic map of $x_i^{\prime }$, and randomly selects half of the classes from $x_i^{\prime }$.

This object-wise mixing operation will lead to significant artifacts around object boundaries. In order to remove those noises, $G$ is applied to the augmented images instead, then, Eq. (7) is rewritten as

where ${\hat{X}}^{\prime }$ denotes the augmented set.

We formally describe the distillation framework to enable data-free student learning. The learning objective consists of two loss terms. The first loss term adopts the plain distillation with the initial simulated images to ensure a lower bound performance. The second loss term penalizes depth differences between the teacher and the student models using images obtained from the transformation network.

The optimization objective of depth distillation from the teacher model to the student model is defined by

where $L$ is a loss function used for measuring the depth errors. We employ the loss function proposed in [Hu2019RevisitingSI] that penalizes losses of depth, gradient, and normal. The details of our method are given in Algorithm 1 where $random\_shuffle$ denotes the operation of randomizing images.

Fig. 6 shows some examples of the input and output images of the transformation network as well as their corresponding predictions. In the figure, $x_i^{\prime }$ and $x_j^{\prime }$ denote two images randomly selected from the simulated set, and ${\hat{x}}^{\prime }$ is the image generated by applying object-wise mixing between $x_i^{\prime }$ and $x_j^{\prime }$. $G\left({\hat{x}}^{\prime }\right)$ denotes the transformed image, i.e., the output of $G$. By visually comparing ${\hat{x}}^{\prime }$ and $G\left({\hat{x}}^{\prime }\right)$, we observe that $G$ tends to reduce artifacts around object boundaries such that $G$ can produce more realistic images. It can be validated by $|{\hat{x}}^{\prime }-G\left({\hat{x}}^{\prime }\right)|$ (Fig. 6. (g)) where differences at object boundaries are highlighted. Furthermore, Fig. 6. (d) and (f) shows the predicted depth maps for ${\hat{x}}^{\prime }$ and $G\left({\hat{x}}^{\prime }\right)$, respectively. They demonstrate a clear difference, as observed in Fig. 6. (h). We quantify these differences by evaluating the whole set $X_1^{\prime }$. As a result, the ${\ell }_1$-norm of the image and depth difference is 0.156 and 0.227, respectively.

We conduct several ablation studies to analyze our approach and provide additional results on the NYU-v2 dataset. Table V gives the results. Specifically, we perform several experiments as follows:

Without using $l_{rec}$: In our original method, we impose the reconstruction consistency between ${\hat{x}}^{\prime }$ and $G\left({\hat{x}}^{\prime }\right)$ to suppress undesirable noises while training the transformation model. We relax this constraint and observe that the REL and ${\delta }_1$ dropped to 0.172 and 0.735, respectively.

Without using $G$: We also test the performance while removing the transformation network in the pipeline. We directly perform distillation using $x^{\prime }$ and mixed images ${\hat{x}}^{\prime }$. As a result, the REL and ${\delta }_1$ dropped to 0.175 and 0.722, respectively.

Without using image mixing: We evaluate the effect without utilizing the object-wise image mixing. We feed the images $x^{\prime }$ to $G$ and apply distillation with both $x^{\prime }$ and $G\left(x^{\prime }\right)$. We find that the REL and ${\delta }_1$ dropped to 0.171 and 0.724, respectively.

With $G\left(x^{\prime }\right)$: Our method performs KD with the initial data $x^{\prime }$ and transformed data $G\left({\hat{x}}^{\prime }\right)$. We also conduct an experiment to investigate the performance of applying $G$ to $x^{\prime }$ by performing KD with both $G\left(x^{\prime }\right)$ and $G\left({\hat{x}}^{\prime }\right)$. We gain a slight performance boost as ${\delta }_1$ is improved from 0.742 to 0.748.

We argue that the scene structure of the alternative OOD data is critical for successfully applying KD. To verify this, we conduct additional experiments adding adversarial perturbations to simulated images to undermine data distribution. Note that even those adversarial perturbations are imperceptible to human vision; they will generate non-robust features [Ilyas2019AdversarialEA] and lead a depth estimator to a malfunction. We generate a set of adversarial images and predict depth maps from them by applying adversarial attacks. We then perform KD with those newly generated RGB and depth pairs. Following [hu2019analysis], we adopt IFGSM attack [Kurakin2017AdversarialEI] to the ResNet-34 based teacher model [hu2021boosting] with different perturbation bounds $ϵ$. The results provided by the student model are given in Fig. 7 where $ϵ=0$ denotes the result of no attack. It is seen that the accuracy of depth distillation will gradually deteriorate as $ϵ$ increases. It indicates that KD is vulnerable to adversarial attacks.