Implicit and Efficient Point Cloud Completion for 3D Single Object Tracking

3D SOT task aims to search the identical object with accuracy in 3D location, size, and rotation. One most common scheme is a matching-based method, which conducts the matching procedure between template and search features by measuring feature similarity. The first 3D point cloud single object tracker was proposed in SC3D [13], which generates the target proposals and matches the proposals to the target by cosine similarity. P2B [22] is the first end-to-end 3DSOT model, which generates the predicted target localization by Hough Voting in VoteNet [6]. BAT [36] encodes more rich information with valuable points within the box, it achieves better performance by embedding the box feature to target and search proposals. PTT [23] and PTTR [38] employ an attention mechanism to structure the matching procedure between target and proposals, which leverages implicit similarity operation for better matching. Although the above matching manners achieve remarkable performance, they still fail to tackle the phenomenon of sparse point returns. Therefore, the stream of motion-centric methods try to utilize point clouds in multi frames. By merging the point clouds within identical target regions, MM-Track [37] alleviates the sparse issue and structures a strong template feature for matching. However, it exposes an inference speed bottleneck since it needs to cost a heavy overhead to extract robust template features again. In this paper, we introduce an efficient implicit point cloud completion method, TKT, which only brings about slight overhead.

Attention mechanism [27] is extensively applied in NLP [5], vision [7, 2], and point cloud [10] tasks. Attention module is adept at capturing long-range feature information by measuring similarity. Due to the discreteness of the point cloud, the attention mechanism is very suitable for the 3D modal. Different from the origin attention module, PCT [15] found an offset operation better in enriching features. Point Transformer [35] designed a Point Transformer layer that could keep the permutation invariant of a point cloud. In this paper, we leverage the self-attention module to structure discriminative features and carry out implicit similarity measurement for better result prediction.

Due to occlusion, and sensor quality, the point cloud is often sparse. Therefore, it often lacks intact information, which would be disastrous for several downstream tasks. A natural solution is to conduct point cloud completion to make up for more complete information [9, 3, 14, 17, 25, 28]. AtlasNet [14] utilizes the powerful 3D convolution to process raw point cloud. However, these methods lose important detailed information and suffer from heavy computational consumption. PointNet++ [21] and its variants could capture hierarchical detailed features from a point cloud, which inspires some researchers to apply it in downstream tasks. PCN [33] applies the encoder-decoder framework to process raw data and then maps the 2D points onto a 3D surface gradually by using the FoldingNet [32]. Delving into the 3D SOT task, MM-Track [37] employs the technology to structure a strong template feature for matching. In this paper, we only use the point cloud combinations at the training stage. Then we design a novel TKT module to effectively transfer the valuable information from the feature of the merged point cloud to template feature, which implicitly and efficiently carries out the point cloud completion procedure and only produces slight overhead at the inference stage.

Following most previous methods [23, 38], we utilize the common PointNet++ [21] to extract feature. Given a single point cloud frame, we sample 1024, and 512 key points for search area and template, respectively. In particular, we leverage the re-sampling strategy to fill the scale of point as facing sparse points returns (i.e., the number of points is less than the requirement). To generate template and search seeds, we employ the farthest point sampling (FPS) strategy [21] to select M and N point features and then conduct set abstraction (SA) layer [21] to fetch final features. The template and search areas are encoded as compact feature sets $F_T\in R^{M\times C}$ and $F_S\in R^{N\times C}$, where C is the dimension of each feature.

Searching the optimal corresponding template and search features is the main target for SOT task. The cruxes of it are generating the more discriminative features and executing a better matching mechanism. Recent works often adopt cosine similarity [13, 22, 36] to establish the matching relationship for template and search features. Inspire by [8, 27, 38], we instead leverage a versatile attention mechanism to implement the above two keys. To this end, we first introduce an augmentor to enhance both template and search features themselves, which employs a self-attention module to enrich each feature representation and focuses more on the discriminative factors. This is significantly important for subsequent matching. Afterward, we further introduce a cross-attention based augmentation for enhancing template features with target-specific clues, which enables it to implicitly build alignment for matching and to aggregate effective information for the subsequent prediction.

We adopt the attention mechanism (i.e., self-attention and cross-attention) as in [27] for mentioned two keys. Specifically, given the features of template or search as $F=\{f_1,\dots ,f_n\},f_n\in R^{1\times C}$, linear projection layers are used to generate the vectors query Q, key K and value V. Then the cosine distances between Q and K are calculated following by normalization with Softmax operation, which forms the weight map. Different from common attention modules, our cross-attention module utilizes the offset operation to generate the Q followed by [38]. Finally, Q needs to be transformed by a simple linear layer and ReLU operation. The attention module could be formulated as:

where the W indicates the output attention weights, $W_q$, $W_k$, $W_v$ are the projection layer of “Query”, “Key”, “Value”, respectively.

where the Q, K, and V represent the previous “Query”, “Key”, and “Value” respectively, W indicates the attention weight map, $W_v$ is the projection layer of “Value”, $\varphi$ means the linear layer and ReLU operation.

The feature augmentation and matching parts are shown in Fig. 2, we first employ self-attention for target and search features enhancement, it helps generate more discriminative features for matching. Then we conduct an implicit matching procedure by applying cross attention. Especially, we produce the “Query” vector from the search features $F_S$ and the “Key”, and the “Value” vector from the template features $F_T$. By applying a 3 MLP layers, we use the ARP module (it will be introduced next section) to predict the coarse result.

Contemporary methods most regard the prediction with the maximum predicted score as the best result. However, we find that it exposes a misalignment issue between predicted scores and localization accuracy. As shown in Fig. 3, the prediction result with the maximum score is not the one with the highest localization accuracy. Thus, selecting the maximum score candidate will bring about a sub-optimal result, making it fall into a performance bottleneck. We argue that all predictions may carry valuable information for target since they are all supervised to forecast identical objects. From this perspective, we aim to gather all the predicted clues, and generate only one robust result. To this end, we introduce the ARP module to realize the motivation.

Intuitively, the boxes with better prediction qualities should contribute more to the final result. Based on this insight, we propose to predict the logits distance between the predicted box center and ground-truth box center, which implicitly measures the prediction quality. We leverage farthest point sample operation [21, 6] to select fixed N candidate point features, and aggregate neighbor point’s feature. Then, a set abstraction layer [21] is adopted to further encode the candidate features. Different from other SOT methods to only predict four offsets of center (x, y, z) and rotation $\theta$, and classification score. As shown in Fig. 4, we first get the sum of the logits of prediction score (feature map before softmax) and logits distance. Then we apply a small MLP layer followed by Softmax operation to project them to the final weights. Finally, the weights are aggregated with four offsets by multiplication operation to calculate the final result. The ARP mechanism will adaptively aggregate all prediction results by virtue of both logits of score and distance. The formula of ARP procedure is:

where ${off}_{N\times 4}$ represents the originally predicted offsets, the $R_{1\times 4}$ indicates the refined results after weighting, and the dis means prediction distance. During training, it is natural to conduct supervision for logits distance [31], but we find that an unsupervised manner performs better, which actually simplifies the training process.

Although the above cross attention augmentation (Sec. III-B) can enhance the template features, it still lacks more complete information due to the sparse point returns. As shown in Fig. 1, it reflects that merging point clouds by multiple frames shows more detailed tracking clues (e.g., shape and pose). Therefore, it naturally motivates us to build template features with more intact points for matching procedures during tracking. To do this, a naive way is to merge previous and current (coarse prediction) point cloud like [37]. This template’s construction pattern is effective, but it is prone to encounter inference speed bottleneck since it needs to encode the merging template again. To fill this gap, we introduce an efficient module, i.e., TKT, to transfer the valuable knowledge from more intact template features to refine template features during training. It is worth noting that it only brings about negligible latency during inference process.

During the training stage, our PCET is optimized in three stages: training coarse prediction, training source feature of TKT, and refined prediction with TKT procedure. All the predictions contain a regression component $Y_{reg}$, a classification component $Y_{cls}$. $Y_{reg}$ consists of the predicted offsets x, y, z and an angle offset $\theta$. Our classification loss $L_{cls}$ is defined by binary cross-entropy, and regression loss $L_{reg}$ is computed by mean square error. Especially, for the TKT module, we use the KL-divergence $D_{KL}$ to measure the difference between the target feature and the predicted feature. In a word, the optimal object is formulated as follow in three stages:

where ${\lambda }_1$, ${\lambda }_2$, and ${\lambda }_3$ indicate the weight parameters. $L_{KL}$ indicates the KL-Divergence loss in Eq. 4, $F_{Dev}$, $F_{Src}$ represent destination feature and source feature, respectively. $L_{coarse}$, $L_{source}$, and $L_{refine}$ correspond to the above three stages training procedures, respectively.

In this section, we conduct ablation experiments to verify the effectiveness of TKT and ARP modules. We report the detailed results of all categories on the KITTI dataset and WOD.

To better demonstrate the effectiveness of our method, we report some tracking results on the KITTI dataset. Obviously, our model PCET could achieve a better localization accuracy as shown in Fig. 8. In the sparse and occlusion environment, PTTR [38] fails to carry out accurate localization. In the contrast, our PCET still achieves accurate tracking due to our proposed ARP and TKT modules.