Scatter Points in Space: 3D Detection from Multi-view Monocular Images

Without 3D prior of the scene, it is harder to build anchor points other than volume grid points. Therefore, we resolve to choose monocular depth estimation as a proxy for learning 3D anchor points. For depth estimation, we use ordinal regression instead of direct regression following the advanced monocular depth estimation method [6]. We simplify the network structure in [6] to reduce the amount of parameters since it works as a submodule in our end-to-end framework. We remove the Scene Understanding Layer the decoder from [6], and downsample the output resolution of depth map by factor 4. As the depth range of our major dataset ScanNet is relatively small, we apply linear discretization instead of spacing-increasing scheme for simplicity. However, the output depth value from ordinal regression is discrete. To obtain the continuous depth from ordinal depth regression, we propose a conditional depth residual estimation layer. The depth residual layer takes image feature $F$ and coarse depth estimation $D$ as input, and infer the residual $D^{{}^{\prime }}$ between ordinal depth regression and ground-truth:

Hence, combined with the average of pixelwise ordinal loss $L_{ordinal}$ defined in [6] over the entire image domain, our final loss function for monocular depth estimation is:

where $D^{\ast }$ stands for continuous ground-truth depth map.

In this section, we propose to build a 3D detector on top of the final multi-view features augmented points. Different from traditional point cloud 3D detection, the pseudo point cloud is generated by previous stages and associated with multi-view features. First, to extract 3D features, we voxelize the augmented scattered points to feed them into a sparse convolution U-Net backbone. Then, a VoteNet-based 3D detector is applied to infer 3D object oriented bounding boxes in the scene. Following convention, 3D bounding box is represented as $[x,y,z,w,h,d,r_z]$. $[x,y,z]$ is the offset from proposal center to ground-truth object center, $[w,h,d]$ indicates object size, $r_z$ is the rotation around $z$-axis (assuming zero roll and pitch). In addition, object category is predicted by another classification head. The detection losses are the same as in [14].

As a bonus, a reconstruction head can be added in parallel with the detection heads. The reconstruction head shares the same center proposal feature. In order to (1) decouple object pose from shape and (2) get complete object mesh with partial observation, the reconstruction head is only required to estimate a shape code of a pretrained 3D mesh decoder. Here we choose [13] pretrained on ShapeNet[3] as the mesh decoder. The ground-truth shape code is generated offline by the mesh decoder for every instance in our training data, which helps the reconstruction network learn faster. During training, the discrepancy between GT shape code and prediction is minimized. Besides the shape code loss $L_{code}=||\hat{c}-c|{|}^2$, chamfer distance between target and prediction as well as several regularization terms are also considered. Thus, the overall reconstruction loss is defined as: $L_{recon}=L_{code}+L_{error}+L_{CD}+L_{edge}+L_{normal}+L_{smoothness}$. For the detailed definition, please refer to the original paper[13].

We conduct experiments on ScanNet(V2) [5] with Scan2CAD [1] annotations to demonstrate effectiveness of the proposed method. ScanNet is an indoor dataset with ground-truth image poses, mesh reconstruction and 3D instance segmentation labels, however no 3D box annotation is available. Following [17], the orientated box annotations in [1] are used in our experiments. Other data such as camera intrinsic and extrinsic remain the same. We follow the standard training/validation splits(1201 and 312 scans respectively) provided by the benchmark.

Mean Average Precision (mAP) is commonly used for evaluating 3D object detection. mAP is computed by averaging AP(Average Precision) through semantic classes. AP of each semantic class is adopted as our evaluation metrics. Recall of each class will also be reported.

Chamfer Distance and F-Score is applied as reconstruction metrics. $G$ and $R$ denote ground-truth point set and reconstructed point set. Chamfer Distance is formulated as,

And F-Score is defined as,

Although our method can be trained end-to-end, we observed that it takes longer time to converge due to the mutual adaptation of different modules. Therefore, the training is divided into three stages: (1) Train Monocular Depth Estimation; (2) Train Depth Estimation and Surface Filter Module from pretrained model of stage-1; (3) Train the whole network with pretrained weights from stage-2. As it is redundant to feed all frames to the network, keyframes are chosen based on a strategy that prefers frames with 2D detection and adequate ego-motion. The number of keyframes are 32 and 50 for training and validation respectively. Adam optimizer with learning rate $5\times {10}^{-4}$ is used for all experiments.

As far as we know, [17][7] are the only two multi-frame methods that report metrics of the same Scan2CAD labels. To compare with more other existing methods in the same setting, we extend two single-frame schemes [12][8] to fuse multi-frame predictions. During inference, we estimate 3D detection for every single frame, and then assemble all detection in the same scene coordinate. After that, non-maximum suppression (NMS) is applied with an IoU threshold of 0.01. To align evaluation metrics with FroDO[17] and MOLTR[7], the 11-points AP is calculated in all of our experiments.

As shown in Table 1, our method outperforms FroDO[17] by a margin of 0.24 AP on chair class, without time consuming non-linear optimization used in FroDO. Due to the limitation of single frame FoV, MOLTR[7] and other methods that combine 3D detection from each individual frame perform worse than our method, especially those large objects such as tables. Meanwhile, we train on all of 9 categories with a single model, and evaluation results are shown in Table 2.

To measure reconstruction performance of our pipeline, (1) only those boxes that has IoU$\text{textgreater}0.25$ with any ground-truth box are kept. (2) 2048 points are randomly sampled from both predicted and ground-truth mesh. The ground-truth meshes are taken from [13], where only chair category is available. For F-score, distance threshold is set to 0.004. The results are shown in Table 4. From 4 we find that when we train multiple categories within a single model, the corresponding AP score for most categories are slightly lower than those in Table 1 where we train a model for each separate category, but still higher than other methods we compare with.

Some qualitative results for both 3D detection and reconstruction can be found in Figure 4 and 5 respectively. As shown by the pictures, the reconstructed meshes are correctly located and akin to the ground-truth objects.

In order to validate the effectiveness of components in our method, we do several ablation experiments on ScanNet. All reported metrics are evaluated on chair class with AP@IoU=0.5.