Uncertainty-Guided Depth Fusion for Spike Camera

Monocular and stereo methods are two parallel mainstreams in the development of depth estimation algorithms. One of the earliest works that inspired recent trends for monocular depth estimation was introduced by Eigen et at.[11]. This work proposed a kind of novel architecture that includes coarse-scale and fine-scale two steps, defining depth estimation as a pixel-wise regression problem. Similar to semantic segmentation task, one popular design for monocular depth estimation is encoder-decoder structure with CNNs[43, 30, 23, 31, 12, 14] or transformers[33, 44]. In the encoding stage, the encoder captures context information and learns a global representation. And in the decoding stage, the network tends to establish a coupling connection between context texture and depth information with ground-truth full-supervision or self-supervision[14, 15, 25]. Innovation has also been made in regression style[12, 4, 35], for more efficient representation of depth information. A recent study shows great potential in combining monocular depth estimation as auxiliary tasks for semantic segmentation[20, 18].

Stereo depth estimation shows quite a different design strategy from monocular ways. Early works concentrate mainly on stereo matching for left and right stereo pairs[1, 3]. After deep learning was applied to this task, a stereo depth estimation pipeline contains three main steps: (1) feature extraction (2) cost aggregation, and (3) disparity/depth regression. Thanks to 3D convolution and the proposal of 3D cost-volume[28, 21], the whole pipeline can be constructed end-to-end[22, 5]. PSMNet[6] concatenates left-fight features to cost volume and perform hourglass-shaped 3D convolution to make aggregation. GwcNet[16] uses correlation formulation to divide cost volume into groups, which decreases computation while improving prediction results. In addition, self-supervised methods also gain competitive performance[52].

Recent study shows a new trend for rethinking the connection between monocular and stereo depth estimation[39, 9]. Besides, left-right consistency has been the main cue for unsupervised depth estimation works of monocular task[14].

Compared to the standard frame-based cameras, biologically-inspired event-based sensors capture visual information with low latency and minimal redundancy. Dynamic Vision Sensors (DVS) is a kind of representative event-based camera. Compared with frame-based camera, DVS is capable of capturing motional objects. Zhu et al.[55] provides a 100Hz DVS dataset containing depth ground-truth. They also propose time-synchronized event disparity volumes in [54] to handle DVS data for stereo matching. Similar research[56] uses discretized event volume to supervise monocular optical flow and depth estimation without labels. Another work[32] adopts a spiking neural network to estimate event depth.

Spike camera is also a kind of novel bio-inspired event camera. Distinct from the frame-based cameras and dynamic vision sensors, spike camera mimics the retina to record the nature scenes by continuous-time spikes[45, 57]. [48] develops a new image reconstruction approach for potential retina-inspired spike camera to recover high-speed motion scenes. [50, 49, 51, 59]use spiking or regular convolutional neural networks to reconstruct high quality and high-speed images like a human vision from spike streams. Spike vision shows obvious advantages in capturing high-speed moving objects or scenes, so it provides new solutions for some long-standing problems in the field of machine vision.

Different from the RGB cameras and dynamic vision sensors, spike camera mimics the retina to record natural scenes by continuous-time spikes [45, 57]. [48] develop a new image reconstruction approach for the spike camera to recover high-speed motion scenes. [50, 49, 51, 59] use spike or regular CNNs to reconstruct high quality and high-speed images from spike streams. Spike vision shows obvious advantages in capturing high-speed moving objects or scenes, so it provides new solutions to some long-standing problems in the field of computer vision. In this paper, we propose a novel method for high-quality spike depth estimation by fusing monocular and stereo depth estimation.

For spike camera, natural lights are captured by photoreceptors and converted to voltage under the integration of time series $t$. Once the voltage at a certain sensing unit reaches a threshold $\Theta$, a one-bit spike is fired and the voltage is reset to zero at the same time [48].

The above formula reveals the basic working pipeline of the spike camera, where $I(i,j)$ represents the luminance of pixel $(i,j)$, and $t{0}_{i,j}^{pre}$ represents the time that fires the last spike at pixel$(i,j)$. An ideal Analog to Digital Conversion(ADC) process is by continuous time, but such circumstances do not exist due to the inherent limitations of the digital circus. Even so, the spike camera is still able to generate much more dense frames than RGB models like streams, at a maximum frequency of 40000Hz[47, 58, 49]. Suppose we have $H\times W$ receptive field, the camera would output a $H\times W$ binary spike frame at a certain moment, and as time goes on, high-frequency spike frames are produced. However, directly performing depth estimation on spike frames remains challenging. On one hand, high contrast between 1-bit spike data makes it more difficult to distinguish local context information. On the other hand, different light intensities in the scene cause different frequencies of spike generation. So in practice, we make spike data in a fixed size time window to be a multi-channel tensor. For example, we take spike frames from continuous 100 time-steps frames and concatenate them at time dimension as $100\times H\times W$ voxels, which then become inputs to our designed networks.

We propose a simple yet efficient network that includes a shared spike-encoder and a spike-decoder with two branches. First, we build a neuromorphic encoding module to extract spiking features in both time domain and frequency domain. The spike voxel $V\in Z^{100\times H\times W}$ is split into spike sequences $V_s=\{v_1,v_2,...,v_s\}$ by a fixed length of time window n, where s = $|100/n{|}_Z$ and $v_s\in Z^{n\times H\times W}$ . Then, the spike sequences are fed into a ConvRNN to extract temporal connections. Meanwhile, an FFT operation is performed on spike voxel $V$ to extract global information in frequency domain.

we use a shared deep encoder to learn a representation to build stereo cost volume and monocular depth regression, as shown in part (a) of Figure 3. We adopt MobileNetV3[17] as our encoder to make a trade-off between computation cost and model performance. The encoder contains 3 downsampling stage and the final feature map size of coding is $B\times 256\times \frac{H}{8}\times \frac{W}{8}$.

As shown in the part (b1) and (b2) of Figure 3, two parallel branches stretch away for monocular and stereo depth estimation and serves as the spike-decoder. Different from any other previous works, we fuse monocular and depth estimation in one workflow at a multi-task level. In the stereo branch, we found a common ground for monocular and stereo tasks to learn a global context representation. So we take advantage of the encoder module, and concatenate the unary features(obtained coding from spike encoders) to build a 4D cost volume($256\times \text{Max-Disp.}\times \frac{H}{8}\times \frac{W}{8}$) for stereo disparity regression, where Max-Disp. represents the maximum disparity level to regression. Then inspired by [6], we perform a 3D hourglass-shaped convolution to aggregate the disparity dimension feature of 4D cost volume. We stack three 3D hourglasses, each of which contains two blocks with $3\times 3\times 3$ kernel size and 2-stride 3D convolution, and two blocks with $3\times 3\times 3$ kernel size and 2-stride 3D transposed convolution. The disparity map is regressed at the final 3D convolution stage via a soft-argmin operation[21]:

where $\gamma (\cdot )$ represents soft-max operation at disparity dimension, $c_d$ is predicted costs for disparity $d$, and $Dis{p}_{max}^{\ast }$ means length of disparity dimension of the output features. The final disparity is weighed by a normalized probability. In the training phase, three 3D hourglass disparity outputs are all involved in building loss function. And the last output of three 3D hourglasses is used for the evaluation process.

In the monocular branch, the right unary features(coding) are then sent to a decoding block for depth estimation. The decoder consists of three upsample blocks, each block contains one bilinear interpolation module and two convolutional layers along with batch normalization and Mish activation. The output layer is a $1\times 1$ convolution which squeeze the features to two channels, one of which is used for depth estimation and the other one used for uncertainty allocation, as described in the next subsection.

Inspired by SUB-Depth[53], we assume the distribution over the output of either branch can be modeled as exponential family distribution such as Laplace’s distribution or Gaussian distribution. The stereo branch and monocular branch adopt the same approach while we use the monocular branch as an illustration example. Given a dataset with left and right spike frames and corresponding depth ground truth $(x_l,x_r,y_l,y_r)$, we let our monocular branch output the mean $\hat{y}$ and variance $\sigma$ of the posterior probability distribution $p(y_r|\widehat{y_r},x_r)$. We use Laplace’s distribution as:

We can convert the above distribution to a log-likelihood formula like:

So according to max-posterior probability estimation, an uncertainty loss can be formulated in the form of:

We can minimize this loss function to obtain a max-posterior probability distribution over estimated monocular depth $\hat{y}$. The uncertainty coefficient ${\sigma }_m$ and predicted depth $\hat{y}$, which are the outputs of the part (b1), are regressed from the same decoder at the same time, so ${\sigma }_m$ can be seen as prediction uncertainty for monocular depth estimation task. Similar to the monocular branch, a lite CNN is added behind the probability map of the stereo branch, and regresses uncertainty coefficient ${\sigma }_s$ after a sigmoid activation.

We notice that the monocular branch outperforms the stereo branch in farther regions and the stereo branch is good at predicting closer regions. So we hope the fusion style may combine both monocular and stereo advantages. So we define a distance threshold:

With estimated uncertainty, an uncertainty-guided fusion mask $F$ can be defined as:

where i represents i-th element of the uncertainty map, and ${\sigma }_{dis}$ represents an uncertainty threshold to make fusion. To take advantage of both monocular and stereo branches, we use monocular depth prediction results ${\hat{D}}_{mono.}$ and stereo depth prediction results ${\hat{D}}_{ster.}$ to make further fusion, exploiting complementary advantages for monocular and stereo models. Instead of directly performing linear addition between two kinds of outputs, we fuse them in a more efficient uncertainty-guided way. And the uncertain-guided fusion is given:

We present training strategies for baseline network without fusion and UGDF with fusion. The training loss of baseline network consists of monocular depth estimation $los{s}_{disp.}$ and stereo disparity regression $los{s}_{depth}$, which use smooth-L1 loss during the training phase under the supervision of depth ground-truth and generated disparity labels. The baseline $los{s}_{base}$ is shown as below:

in which $los{s}_{disp.}$ and $los{s}_{depth.}$ are shown as below:

In which $\eta =0.1$ and $c_i=log\left(D_i\right)-log\left(\widehat{D_i}\right)$. And N is the size of data and M=3 is the number of stacked 3D hourglasses, $\{{\alpha }_1,{\alpha }_2,{\alpha }_3\}$ = $\{0.5,0.7,1.0\}$. $\hat{D}$ and $D$ represent predicted depth and depth ground truth respectively. Similarly, $\hat{d}$ means predicted disparity and $d^{\ast }$ is generated disparity ground truth.

The training loss of UGDF consists of five losses, including monocular depth estimation $los{s}_{disp.}$, stereo disparity regression $los{s}_{depth.}$, monocular branch uncertainty $los{s}_{mono\_unc.}$, stereo branch uncertainty $los{s}_{ster\_unc.}$ and fusion $los{s}_{fu.}$. The whole UGDF $los{s}_{ugdf.}$ is shown as:

where $los{s}_{mono-unc.}$ and $los{s}_{ster-unc.}$ follow Eq. 6 while $\widehat{D_f}$ denotes the fusion predicted depth from two branches. To be mentioned, the depth of the stereo branch is converted from disparity under intrinsic parameters of the camera. The training details of baseline and UGDF are the same. Further training details are presented in Section 5.

We train our proposed UGDF network on spike-depth pairs, including stereo spike frames and right depth-ground-truth. The whole training phase contains 200 epochs and takes about 16 hours with the batch size of 4 on two NVIDIA-Tesla P100 GPUs, for $256\times 512$ resolution spiking frames.

We utilize 24-bit 0-1000m absolute depth ground-truth to supervise training for the monocular branch. We normalize depth ground truth $D$ to $D^{\ast }\in (0,1)$, with the function $D^{\ast }=D/1000$. Meanwhile, the disparity is transformed from depth with camera intrinsics.

As for optimization, we use Adam optimizer with $({\beta }_1,{\beta }_2)=(0.9,0.999)$. We set an initial learning rate of 1e-3 and decay to 0.33e-3 at epoch 35 for the sake of a more smooth optimizing process.

In this section, we compare UGDF against the state-of-the-art and classic depth estimation methods on CitySpike20K dataset.

Data processing Our proposed dataset contains 20K frames of spike-depth pairs. We split 7 out of 10 total sequences for training, 2 sequences for testing and 1 sequence for validating. All the data used for training and validating is sampled every 100 time-stamps to form a spike voxel. So we obtain 140 training pieces and 40, 20 for testing and validating our framework. To emphasize the advantage of spike data, we use blurred 30fps RGB frames in our dataset to train the RGB-based baseline methods. So there are 571 training pieces, 142 testing pieces and 111 validation pieces for baseline methods.

Baseline methods To demonstrate the effectiveness of UGDF, we compare it with some state-of-the-art and classic depth estimation methods which have shown remarkable performance on our proposed CitySpike20k dataset. For monocular methods, we choose classical UNet [34] and DPT[33]. UNet has been demonstrated a successful design on semantic segmentation[34, 2] and image reconstruction[8, 7]. we adopt its proposed structure and evaluate it on our spike-depth estimation task. In DPT, we use Vit-b16 as the backbone and 224x224 as input resolution. PSMNet[6] uses subtract and concatenation method to build a 3D cost volume. GwcNet[16] proposes group-wise correlation to reduce computation while conducting 3D convolution. CFNet[36] employs a variance-based uncertainty estimation to adaptively search disparity space.

Main Results and Analysis for CS20K Table 1 and Table 2 show quantitative results with the comparison of the RGB methods. We experiment with classic monocular depth estimation works, as well as stereo depth estimation methods. We can see that under the uncertainty-guided fusion, our result gets the top performance among all the methods. Compared with the best monocular method, UGDF reduces $4.93\%$, $2.688$ error in terms of AbsRel.and RMSE metric respectively. For stereo methods, we also gained improvements on all metrics. We also show the qualitative comparison in Figure 5. As can be seen, our method achieves better depth estimation compared with blurred RGB-based methods. Other visualization results are in our supplement.

Evaluation on Spike-Real Set We also train and evaluate our network on a dataset captured by a stereo real-world Vidar in a series of outdoor scenes. The dataset contains 40 sequences of outdoor scenes and we split 33 sequences for training and 7 sequences for testing. Table 4 and Figure 6 shows results evaluated on its test set.

We carry out ablation experiments from two aspects. The first of those is to explore the effect of different choices of time-window widths, and the other is to verify the effectiveness of uncertainty-guided fusion design.

Effectiveness of Uncertainty Guided Fusion We conduct experiments to verify the effectiveness of monocular and stereo uncertainty jointly guided fusion. In order to demonstrate the benefits of joint-guided fusion, we first compare it with a linear additive ensemble fusion manner. To be specific, we make a uniform linear addition of monocular and stereo estimation results, denoted as E. Fusion in Table 4 As can be seen, the linear additive fusion manner is inferior to other fusion methods. In addition, we visualize the improvement gap between fusion results and the other two branches. We can see the advantages of our UGDF framework. Firstly it combines both advantages of stereo and monocular estimation. And secondly, it brings substantial improvements, rather than the compromising fusion of ensemble style.

Time window Width of Neuromorphic Encoding In our framework, we apply a kind of neuromorphic encoding method to effectively extract the feature of spike data. As we have described in Section 3.2, we chunk the spike voxel into spike sequences by the time window of 24 to obtain better local representations. Then the sequences are sent into the Conv-RNN to extract temporal connections between different sequences. Theoretically, applying a smaller time window is beneficial to extract local connections between spike sequences, yet increases the convergence and inference time on hardware. So we set different widths of encoding time-window and train the whole network for 200 epochs. We make evaluations on the test set of CitySpike20K. The results are shown in Table 5, and we can see that 24 turns out to be the optimal choice of time-window widths for shorter inference time and better precision.

We propose CitySpike20K, a spike-depth dataset to help explore the depth estimation algorithms for spike camera. The dataset is generated by Unity3D and contains 10 sequences, 5 of which are day scenes and 5 others are night scenes. In the dataset, the frequency of the spike data and corresponding depth GTs is 1000Hz. Besides, we supply 30Hz RGB images for each scenes as well as 1000Hz RGB images that aligned with spike data.

To fully simulate the city environments, we add moving automobiles and dynamic traffic lights. We set 5-10 moving automobiles including buses, cars, vans and trucks for each scene. Figure 6 gives a visualization of CitySpike20K which contains RGB frames, spike data and depth maps. Specifically, we split scene03, scene07 for testing, scene09 for validation and others for training.

We conducted to evaluate the effectiveness of supervised depth estimation model on CitySpike20K. Our evaluation metrics for depth estimation is described as follows:

Given an estimated depth map $\hat{D}$, and its corresponding ground truth $D$, $N=H\times W$, $Abs\_Rel$ is quantified as:

and RMSE defined:

we also introduce $RMSE\_log$ metric:

and Sq_Rel metric as here:

Above metrics measure output errors from different statistic aspect, weighting the distance between predictions and ground-truth labels, where lower values mean better model performance. Below metrics are for evaluation of whether predictions are accurate within certain range of ground-truth, and higher values mean better performance. Note that $j\in \{1,2,3\}$

As we have described in our submitted paper, we also evaluate our framework on a real-recorded dataset by a spike camera. The dataset contains 40 sequences data and each of which includes 3-6 $[400\times 250\times 400]$ spike voxels in the format of $[T\times H\times W]$. We split 33 sequences for training and 7 for testing.

To demonstrate that our UGDF framework still works in real-world scenes, we carry out experiment on a spike-kitti dataset. To convert Kitti[13] from RGB modality to spike modality, we first make frame interpolation using XVFI[37] by 128 times. Then we use a Simulated-Vidar code script to generate spike data from RGB Kitti images to form spike voxels in the format $(128\times 375\times 1242)$ , where 128 represents the time dimension and $(375\times 1242)$ is the original size of Kitti RGB images. We maintain the same way to operate neuromorphic encoding as what we design for CitySpike20K dataset in our submitted paper. As mentioned above, we set this experiment to further explore the effectiveness of our fusion strategy. We train our framework for 50 epochs on 4 RTX-2080Ti GPUs. Figure 9 provides a group of visualization of the output of our results on validation set.

In addition to 10 sequences of 1000Hz spike data we provide in the CitySpike20K dataset, we still supply a 40000Hz demo to simulate real spike as possible as we could. The demo contains 60K paired data and records a 1.5 seconds video of a fast-driving car in the city street. Different from our submitted papers, we use this demo to evaluate the performance of models loaded with spike data. Considering existing methods for monocular or stereo depth estimation are mostly based on RGB 3-channel data, we change the input channel of the models to the time-window width of applied spike sequences, i.e. 32 as we adopted. And we use the first half of the demo for training and the second half for testing.