LiteDepth: Digging into Fast and Accurate
Depth Estimation on Mobile Devices

As shown in Fig. 1, our proposed network consists of an encoder and a lightweight decoder with skip connections. We sequentially introduce each component and design detail.

Encoder plays a crucial role in extracting features from input images for depth estimation. To achieve a trade-off between fidelity and inference speed, we choose MobileNet-v3 [howard2017mobilenets] as our encoder. It is worth noticing that MobileNet contains a dimension-increasing layer (1$\times$1 convolution with an input dimension of 96 and output dimension of 960) to facilitate training for a classification task. We remove this layer to improve the inference speed and reduce the number of model parameters. Following [ignatov2021fast], we adopt the Fast Downsampling Strategy in which a resize layer is inserted at the beginning of the encoder to resize the high-resolution input image from 480 × 640 to 128 × 160. As a result, the encoder can quickly downsample the resolution of feature maps, significantly shorten the inference time. Typically, input images are normalized to align with the pre-training setting. We discern the vanilla image normalization is time-consuming (19$ms$ of the image normalization v.s. 37$ms$ of the whole model) on the target device (i.e., Raspberry Pi 4). Therefore, we propose to merge the image normalization into the first convolution layer in a post-process manner so that we can avoid the redundant overhead without bells and whistles. Consider the image normalization and the first convolution layer:

where $I_n$ and $I_r\in R^{3\times H\times W}$ are normalized and raw input images. $m\in R^3$ and $s\in R^3$ are the mean and standard deviation used in the image normalization. $f\in R^{C\times H_f\times W_f}$ is the output feature map with $C$ channels of the first convolution in our network. $W\in R^{3\times C\times k^2}$ and $b\in R^C$ are the trained weight and bias of the first $k\times k$ convolution. $\ast$ denotes the convolution operation. Given a trained model with parameters $W$ and $b$ of the first convolution, we update the them based on the mean and standard deviation used in the image normalization during the training stage:

where the $W^{\prime }$ and $b^{\prime }$ are the updated weight and bias of the first $k\times k$ convolution. $Concat$ is the element-wise concatenation. $d$ is the index of RGB dimension. Consequently, we discard the image normalization and apply the first convolution directly on input images as:

As a result, the trained network can directly recieve the raw input images without the time-consuming image normalization.

Decoder is adopted to recover the spatial details by fusing the multi-level deep and shallow features. Unlike previous works [wofk2019fastdepth, ignatov2021fast, wang2021knowledge] that utilize the symmetrical encoder and decoder, we drop out the last decoder layer to further accelerate the model inference. Hence, the resolution of outputs is 4$\times$ downsampled (i.e., 32$\times$64). At each decoding stage, we apply a simple feature fusion module to aggregate the decoded and skip-connected features, which consists of a concatenation operation and a convolution layer (with ReLU as the activation function). To achieve the best trade-off between fidelity and speed, we utilize the 1$\times$1 and 3$\times$3 convolution for deep and shallow features, respectively. The final feature map is projected to the predicted depth map via the 1$\times$1 convolution, which is then passed by a ReLU function to suppress the plural prediction. Finally, we insert a resize block at the end of the decoder to upsample the predicted depth map to the raw resolution 480$\times$640. We highlight the lightweight design of the decoder that only consists of five convolution layers but achieves satisfactory fidelity.

Data augmentation is crucial to training models with better performance. Typically, the sequence of data augmentation for monocular depth estimation includes random rotation, random flip, random crop, and random color enhancement [lidepthtoolbox2022]. We propose the more effective crop strategy R${}^2$ crop, in which we randomly select the size of crop patches and the cropped locations. We highlight the discrepancy with other commonly used crop methods in Fig. 2. It increases the diversity of the scenes and effectively avoids overfitting the training set.

Previous depth estimation methods [lee2019bts, bhat2021adabins, li2022depthformer, li2022binsformer] only adopt the silog loss to train the neural network:

where $e_i=\log {\hat{d}}_i-\log d_i$ with the ground truth depth $d_i$ and predicted depth ${\hat{d}}_i$. $N$ denotes the number of pixels having valid ground truth values. Since we discover that the lightweight model supervised by this simple single loss lacks representation capability and is easily stuck in local optimal, we adopt diverse loss terms to provide various targets for sufficient model training.

Motivated by [sitzmann2020implicit], we first propose a gradience loss $L_{grad}$ formulated as:

where $\nabla$ is the gradience calculation operation. Since the gradience loss is calculated in a dislocation subtraction manner and there are tremendous invalid depth GT in the dataset as shown in Fig. 3, as presented in Fig. 4, simply applying gradience calculation will blemish the information of invalid pixels and introduce outlier values when calculating the loss term. Hence, it is necessary to carefully design a strategy to calculate masks $M$ to filter these invalid pixels in $L_{grad}$. To solve this issue, we first replace the invalid value with NaN and then calculate the GT for gradience loss. Thanks to the numeral property of NaN and Inf, invalid information can be reserved. Consequently, we can filter the NaN and Inf when calculating the gradience loss.

Moreover, we also adopt the virtual norm loss $L_{vnl}$ [yin2019enforcing], and robust loss $L_{robust}$ [barron2019general]. We formulate them as follows:

where $n$ is the virtual norm. We refer more details in the original paper [yin2019enforcing]. Unlike the original implementation, we sample points from reconstructed point clouds and adopt constraints on predictions to filter invalid samples instead of ground truth. It helps the model convergence at the beginning of training.

where $e_i={\hat{d}}_i-d_i$. We experimentally set $\alpha =1$ and $c=2$. In fact, the loss reduces to a simple $L_2$ loss, but which is proven to be more effective compared with the proposed adaptive version in our task. More experiments can be conducted to decide a better choice for $\alpha$ and $c$.

Finally, we adopt a combination of these loss terms to train our network. The total depth loss is

We set $w_1=1$, $w_2=0.25$, $w_3=2.5$, and $w_4=0.6$ based on tremendous experiments. Then, we apply a dynamic re-weight strategy in which the loss weights $w$ are set as model parameters and are automatically fine-tuned during the training stage. Experimental results indicate that this strategy can achieve similar results as tuning the weights by hand.

We apply the structure-aware distillation strategy [liu2020structured, wang2021knowledge] to further boost model performance. For the teacher model, we choose Swin Transformer [liu2021swin] as the encoder and adopt a similar lightweight decoder to recover featrue resolution and predict depth maps. We present the network architecture in Fig. 5. The teacher model is trained via the supervision of $L_{depth}$ and is then fixed when distilling the student model. During the distillation, multi-level distlling losses are adopted to provide supervisons on immediate features as shown in Fig. 6. The distillation loss is formulated as

where $a$ is the affinity map calculated via inner-product of $L_2$ normalized features. We refer to [liu2020structured, wang2021knowledge] for more details. $s$ and $t$ indicate the features are from the student and teacher model, respectively. We choose three level ($L=3$) features for distll, which are DeFeat2, DeFeat3, and DeFeat4 in Fig. 1.

Consequently, the student model is trained via the total loss $L$:

where $w_d=10$ in our experiments. Notably, unlike previous work [liu2020structured, wang2021knowledge], we adopt a two-stage training paradigm. During the first stage, the student model is only trained via $L_{depth}$. In the second stage, we adopt the teacher model and utilize the total loss $L$ to further boost the performance of the student model.

We implement the proposed model via the monocular depth estimation toolbox [lidepthtoolbox2022], which is based on the open-source machine learning library Pytorch. The model is converted to TFLite [lite2019deploy] after training. We use Adam optimizer with betas = (0.9, 0.999) and eps=1e-3. A poly schedule is adopted where the base learning rate is 4e${}^{-3}$ and the power is 0.9. The total number of epochs is 600 with batch size = 32 on two RTX3090 GPUs, which takes around 4 hours to train a model. The encoder of our network is pretrained on ImageNet, and the decoder part is trained from scratch.

As shown in Tab. 1, our proposed method achieves a score of 232.04 on the challenge test set and ranks second place. Our solution achieves 0.311 si-RMSE with 37$ms$ on the Raspberry Pi 4. Notably, our runtime is lower than the other methods and the performance is comparable.

We visualize the prediction results of our proposed methods as shown in Fig. 7, which demonstrates that our methods can achieve reasonable depth estimation results. However, the predicted depth maps are very rough around the edges due to the excessive down-sampling.

In this section, we verify that our method can achieve high-throughput monocular depth estimation on mobile devices. We convert our model to TensorFlow-Lite and test the inference time on various mobile devices, including smartphones with Kirin 980 and Snapdragon 7 Gen 1. We test the model using AI Benchmark [ignatov2018ai, ignatov2019ai]. Following the challenge requirements, the resolution of input and output images is 640$\times$480. The data type is set to float (32 bit). As presented in Tab 2, our network can obtain extremely high-throughput inference. It achieve 162FPS on smart phones with Snapdragon 7 Gen 1 processor. Interestingly, we can observe that the model is CPU-friendly, with an even faster inference on CPU than GPU on mobile devices.