Real-time 3D Single Object Tracking
with Transformer

In tasks involving point clouds and deep learning, tracking is naturally different from detection and segmentation due to its spatial-temporal continuity. Since the tracking target is a sequence, which makes two frames similar in spatial and temporal terms. Thus, the dominant tracking algorithms utilize the spatial and temporal prior information to initialize the search area. In spite of its effectiveness in reducing computational complexity, bad tracking results will lead to large tracking error in challenging scenarios. This is because the consecutive tracking predictions will accumulate over the historical error and propagate it to the next frame. And when error accumulates enough, the tracker will fails.

The Siamese network from VOT is generally adopted in existing algorithms for point cloud object tracking. However, the image data is usually dense, while point cloud is naturally sparse. This causes gaps in the effect of applying Siamese network in image and point cloud. Sparse point cloud data makes it difficult for the backbone network to extract robust point cloud features. Hence, the existing LiDAR-based tracking methods are sensitive to the sparsity of point cloud data.

The sparsity of point clouds limits networks on modeling interactions among point patches located in different geometric positions and capturing contextual information, which makes the features extracted by the models ambiguous. The points at locations where the object surface have a low-dimensional structure, such as a plane, contribute ambiguous features[18]. The feature ambiguity makes the tracker hard to classify the fore-background points and regress the box center, and finally leads to bad tracking results.

These challenges motivate us to propose new algorithm to deal with those problems. To this end, we propose our PTT (Point-Track-Transformer) module and PTT-Net. PTT module can handle the sparsity of points with the help of the attention mechanism in Transformer, where the contribution of each point is automatically learned in the network training. Besides, the ambiguity of features will be suppressed through self-attention mechanism because of the role of attention is to refine features and guide the network to focus on more representative features of the tracking target. Finally, our PTT-Net embedded with PTT module could capture sparse dependencies even from a few points, thus reducing the accumulation of errors.

The original feature embedding module in natural language processing is to map each word in the input sequence to a high-dimensional vector. In this work, we use the linear layer to complete the feature embedding operation, and map the input point cloud feature dimension from $D$ To $M$: $R^D\to R^M$, which can place the feature closer in the embedding space when the semantics are more similar and make the network have a stronger fitting ability.

Position encoding module plays a crucial role in transformer, which allows operators to adapt to the local structure of the input data [21]. 3D point coordinates are valuable features indicating the local structures. Therefore, we utilize the coordinates directly as the input of the position encoding module. Compared to the techniques used in natural language processing, we use a simple and yet efficient approach by mapping the coordinates of each point to the feature dimension and the resulting position encoding is added to the attention matrix. We adopt the relative coordinates to make the network better capture the spatial correlation between points and local geometric shape information. Since the feature $f_i$ extracted by [59] can provide the local context information, we obtain the position encoding features $P={\left\{p_i\right\}}_{i=1}^N$ with function $\eta$. For input point set $S$ including N points, the position encoding feature for each point is:

where $c_i$ is the coordinate of the i-th point in $S$. $c_j$ is the j-th coordinate in local neighborhood region of $c_i$. $\eta$ is an MLP with two linear layers and one ReLU non-linearity. Here, we use $K$NN to capture the local context and set $k$ = 16 by inheriting the experimental results in [50].

As Fig. 2 shows, self-attention module computes three vectors for each point: $Q$, $K$, $V$ through $\alpha$, $\beta$, $\gamma$, where $\alpha$, $\beta$, $\gamma$ are all shared linear layers. It is worth noting that $K$ and $V$ are aggregated from the features of the $k$ neighborhood points, which aims to encode more local context information. Here, $Q\in R^M$, $K\in R^{k\times M}$, and $V\in R^{k\times M}$. For relation function $\sigma$, we use $\sigma (Q,K)$ to obtain point-wise attention weights, the detail implementation of $\sigma (Q,K)$ will be introduced in next part. And an MLP layer $\gamma$ is used to introduce additional trainable transformations and match the output dimension. Then, we add the position encoding features $P$ to both the attention vector $\sigma$ and the transformed features $K$. Finally, the residual features recorded as $A$ are defined as the weighted sum of the attention weights with all $V$ vectors. The formula is as follows:

where $\rho$ is a normalization function (softmax) and $\gamma$ is a non-linear mapping function (MLP) that includes two linear layers and one ReLU non-linearity. $\sigma$ is the relation function between $Q$ and $K$. $A$ is output attention features.

The relation function $\sigma (Q,K)$ is the core of self-attention. Different ways of obtaining the relation between Query vector and Key vector could construct different types of attention modules. As mentioned in [50], self-attention operators can be classified into two types: scalar attention [21] and vector attention [49]. In scalar attention, the relation function $\sigma (Q,K)$ can be expressed in the form of Eq. 2, which computes the scalar product between Query vector and Key vector. The vector attention obtains vector attention weights by using channel-wise subtraction operation. Besides, it has been confirmed in [50] that vector attention is a natural fit for point cloud than scalar attention since it supports adaptive modulation of individual feature channels, not just whole feature vectors. Therefore, we use the vector attention structure. The relation function $\sigma (Q,K)$ is a subtraction operation. The formula is as follows:

Here, we first discuss the impact of different sampling strategies on tracking task. The purpose of sampling is to extract key points and ensure that the number of points in the template or search point cloud is aligned with the input dimension of network. However, different sampling methods will lead to different degrees of target information loss. At the same time, we find that in the existing tracking pipeline, the more foreground points left in the search area after sampling, the more accurate the regression results of the network. In contrast, due to the unbalanced distribution of foreground and background points, the classification accuracy of the network will decrease. Therefore, a suitable sampling strategy can not only achieve input alignment, but also improve the tracking performance of the tracking network.

Common 3D point cloud sampling methods include Random Sampling (RS), Farthest Point Sampling (FPS), and Farthest Point Sampling in Feature dimension (Feat-FPS). RS can achieve high sampling efficiency, but the sampling result depends on the distribution density of point cloud. FPS can better retain the geometric information of the original point cloud, and is a more balanced sampling method. In addition, Feat-FPS proposed in 3D-SSD [62], which can sample in feature space, has better sampling results for targets of different semantic categories. The visualization of the three different sampling methods is shown in Fig. 4. We find that FPS has more uniform sampling results, while Feat-FPS pays more attention to points on the target. P2B uses RS, although it has a higher computational efficiency, it leads to the loss of some key information, which limits their tracking accuracy.

In summary, for the sampling strategy of tracking task, we adopted FPS which can retain the geometric information of the original point cloud and make the foreground and background points balanced. A demonstration of its good tracking performance will be shown in Sec. IV-D.

The ability of transformer to learn self-attention weights inspires us to try it on 3D SOT task. We formulate the problem of focusing on the differences of features as self-attention weighting. In order to verify the effect of our method, we embed our PTT module into the previous open-source state-of-the-art (SOTA) LiDAR-based 3D SOT work P2B [17]. More specifically, the PTT modules are inserted in seeds voting stage and proposals generation stage of P2B respectively. In the seeds voting stage, P2B generates votes by utilizing the augmented features, which are obtained from the backbone network (in Fig. 3). We notice that [17] ignores the differences among different point cloud features in the search area, and gives no preference to the points in different locations when generating votes. However, it is important to focus on the points which contain more geometric information. Therefore, we apply PTT module to weigh the augmented features and model interactions among point patches to learn the context-dependent feature (in Fig. 5(a)(b)).

In proposals generation stage, P2B generates proposals based on local context features. However, their method ignores the global semantic features of the targets, so that they could not distinguish similar objects (e.g., two pedestrians, in Fig. 5(c)(d)). Therefore, we use the PTT module to further weigh the target-wise context features obtained by the aggregation network in P2B for tracking deeper-level target clues to capture the contextual information between object and background.

As shown in Fig. 3, we embed our PTT module in the open-source SOTA method P2B [17] to build PTT-Net. We add PTT module to the seeds voting and proposal generation stages, and weigh the augmented features and cluster features respectively. Experiments show that our PTT-Net outperforms the SOTA method with remarkable margins.

The PTT module is trained with the other sub-networks in [17]. Therefore, we follow [17] to design our loss function. The overall loss consists of two parts as follows:

where ${\lambda }_1$, ${\lambda }_2$, ${\lambda }_3$ represent the weighting coefficient of each loss. Classification loss includes voting classification loss $L_{cv}$ and proposal box classification loss $L_{cb}$. The regression loss includes the voting loss $L_{rv}$ and the proposal box regression loss $L_{rb}$.

We used the training set of KITTI and NuScenes dataset to train and evaluate our method. For KITTI dataset, there are more than 20,000 manually labeled 3D objects using Velodyne HDL-64E 3D lidar (10HZ). Following [16, 17, 19, 44], we split the dataset as follows: 0-16 for training, 17-18 for validation and 19-20 for testing. Specially, we first extract each frame label from every scene label of KITTI tracking benchmark. Then, we further extract each ID label from every frame label, and finally concatenate the labels of the same ID to obtain each tracklet label from its first frame to its final frame. By this way, we convert the KITTI MOT label to SOT label. Furthermore, for each tracklet, only the first frame includes the 3D ground truth bounding box (bbox) in the testing phase. Therefore, we initialize our tracker with the first frame 3D ground truth bbox during tracking, and then track the object in the sequence frames by our tracker. In order to better illustrate that the point cloud in KITTI is challenging for tracking task, we choose the Car category and count the number of foreground points in each frame, then calculate the percentage referred to all frames from 00-20 sequence in Fig. 6. The total number of frames is 27292. The number of frames that are less than 20 points is 7123, accounting for 26.10% of the total. The number of frames between 20-100 is 8650, accounting for 31.69% of the total. In addition, the number of 100-500 and more than 500 frames are 6810 and 4709, respectively, accounting for 24.95% and 17.25% of the total. This fully shows that about half of the frames in KITTI are sparse scenes. Hence the KITTI dataset is challenging for tracking task. This will be the bottleneck of limiting the tracking accuracy as mentioned in Sec. III-B. Furthermore, we exemplify the visualization results of point clouds at various number intervals in Fig. 6.

For NuScenes dataset, there are 1000 driving scenes and 23 object categories, which make it is more challenging because of more complex scenes. We follow the same settings with the BAT [64] to obtain a fair comparison, and directly refer to the results reported in BAT [64] for comparison.

Following previous work [16, 17, 19, 44], we report Success and Precision metrics defined by One Pass Evaluation (OPE) [65], which represent the overlap and error Area Under the Curve (AUC) respectively.

We use FPS in our implementation instead of RS used in original P2B [17] for point cloud sampling. In the training stage, we use the Adam optimizer and set the initial learning rate to 0.001 and decrease by 5 times after 12 epochs. The batch size is 48 and the training epoch is 60. Besides, we extend the offset from (x, y, $\theta$) to (x, y, z, $\theta$) when generating more template samples during data augmentation in [17]. In the testing stage, we also add $Z$ axis offset to generate the predicted box. Other parameters are consistent with settings of [17]. Meanwhile, we also follow the tracking setting of P2B [17]. Specially, we initialize the template point cloud with the point cloud of the first frame ground truth and update the template point cloud by fusing the point cloud of first frame ground truth with previous result. The search point cloud is updated based on the point cloud of the previous result, which can better meet the requirement of real scenes.

We compared the performance of our PTT-Net with the existing methods on the KITTI dataset and reported the results for 3D car tracking in Tab. I. To meet the requirement of real scenarios, we generate the search area centered on the previous tracking result. The results show that our PTT-Net has achieved SOTA performance in all evaluation metrics with remarkable margins. Compared with the baseline algorithm P2B [17], our performance has been greatly improved by $\sim$11% in 3D Success. Besides, compared with the previous SOTA method 3D-SiamRPN [19], our method performs better by a margin of $\sim$9% and $\sim$5% in 3D Success and 3D Precision respectively. It verifies the superiority of our method, and shows that PTT-Net could work better in challenging scenarios like sparse or occlusion scenarios, while other trackers often fail to track in these scenes. Additionally, compared with [63] and [44] which both use RGB+LIDAR fusion information, the Success/Precision results of PTT-Net outperform them 4.7%/12.1% and 30.7%/31.2% respectively. We think that this is because our method can capture important feature representations on tracking targets even if it is only based on raw point cloud data. More importantly, compared with other methods, our proposed method not only has a good performance, but also could run in real time with 40 FPS.

For four categories in KITTI tracking dataset, the Car category is the rigid object. To evaluate the tracking performance in more scenes and especially containing non-rigid objects, we further compared our method with previous methods on Pedestrian, Van, and Cyclist (Tab. II). Except for F-Siamese [44], which fuses image and point cloud information, all other methods adopt the same experimental settings.

As shown in Tab. II, the average performance of PTT-Net outperforms SC3D [16], P2B [17] and 3D-SiamRPN [19] by $\sim$24%, $\sim$13% and $\sim$9% respectively. It is worth noting that the Success/Precision results of PTT-Net show an improvement (9.7%/15.8%) on non-rigid object (Pedestrian) tracking. The result also verifies that our PTT module can help the network understand and learn the important features of the target better. Additionally, we notice that there are performance gaps between our method and the best method F-Siamese [44] in Van and Cyclist categories. We believe there are two reasons. First, the cyclist has the least training samples (only 1529 samples for training), which may limit the performance of the transformer. And we did not do any extra data augmentation for cyclist because we would like to use a fair setting among all categories. Second, F-Siamese firstly utilizes a 2D siamese tracker in the front end only using dense image data, and generates a search area based on the results of the 2D siamese tracker, which provides prior information for subsequent 3D SOT. We believe that the data fusion in F-Siamese may result in the better performance. Additionally, we also notice that the Success results of P2B achieves 28.7%/32.1% in pedestrian and cyclist respectively, and our method brings +16.2%/+5.2% gains in the two categories. Because our method is based on P2B, we believe the results also verify the effectiveness of our method. Besides, our performance is much higher than that of F-Siamese method in the categories with more abundant data, such as vehicles and pedestrians. The Success/Precision results of PTT-Net outperform F-siamese by 30.7%/31.2% and 28.7%/39.8% in vehicles and pedestrian category respectively.

Besides, we also evaluated our PTT-Net on NuScenes dataset to further confirm the effectiveness of our method. Although KITTI dataset is commonly used by previous 3D SOT methods, its scale is too small and this may limit the performance of proposed network. Recently, BAT [64] reports the results of the dominant methods on NuScenes dataset. Hence we follow the settings of BAT and evaluate our PTT-Net on NuScenes, and report the results in Tab II. As shown in Tab II, our PTT-Net outperforms BAT in all categories. This indicates that our PTT-Net could address the challenging scenes in NuScenes more effectively.

We further show the comparisons with previous trackers in terms of speed, 3D Success and 3D Precision on Car category. As shown in Fig. 7, our PTT-Net achieves SOTA performance on both 3D Success and 3D Precision. Meanwhile, our method has less computational burden. In other words, our method has both high accuracy and fast running speed.

Here, we choose the Car category and divide the tracking sequence according to the number of points in the first frame, which is used to initialize the template point cloud and plot the corresponding tracking performance curve (Fig. 8). As mentioned in Sec. III-B, the sparsity of point cloud limits the performance of 3D SOT trackers. Generally, in extremely sparse scenes (the number of points on the target is less than 50), most existing trackers often track off or fail. However, as shown in Fig. 8, the performance of our method outperforms P2B in both Success and Precision with a large margin when points number is less than 50. This shows that our method could deal with sparse scenes better and has more robust performance.

To show the effects of PTT modules in PTT-Net, we exemplified the attention points and scores when PTT-Net utilizes PTT Module or not in Fig. 9. For attention points and scores, we set the voting scores in the voting stage as attention scores. The higher the score, the more attention the corresponding points have been focused on. As shown in Fig. 9, from the first row to the fourth row, the point cloud of tracking target becomes more sparse, which makes the tracking difficulty higher. However, the results show that our PTT module can achieve robust tracking in all these different scenarios. And the attention scores tend to be higher in the location where the target features are rich. Meanwhile, we also observe that our PTT module helps network filter the background noise better and focus on the tracking target. Especially, from the visualization results in the fourth row of Fig. 9, even if there are few points, our PTT module could still help the network focus more on the foreground points, which once again proves the power of our PTT module.

We visualized our advantageous cases over P2B and SC3D in Fig. 10. We can observe from Fig. 10 (a) that in the sparse scenarios (less than 50 points) where both SC3D and P2B tracked off course or even failed, our PTT-Net still tracks the target robustly. In Fig. 10 (b), even though P2B can track the target, their position estimation still has a large deviation in z axis. This also shows the effectiveness of our method. We could not only track the target robustly in sparse scenes, but also estimate the location information of the target more accurately.

The Fig. 10 shows that PTT-Net can work well in most of scenes compared to SC3D and P2B. However, the sparsity of points still influences the performance of PTT-Net. Our method tends to fail when the points are extremely sparse. To show the failure case of our method in more detail, we visualized the failed tracking result when the points are less than 20. As shown in Fig. 11, our PTT-Net could not learn effective object features since there are almost no points in the initial search area. In addition, due to the sparse point cloud, the feature ambiguity also leads to the inaccurate estimation of the bounding box, which causes the propagation of errors and failure case finally.

The first ablation study presented is designed to support our claim that FPS could benefit the classification task. As mentioned in Sec. III-E1, different sampling methods will lead to different degrees of target information loss. Here, we compared the effects of three different down-sampling methods: Random Sampling (RS), Farthest Point Sampling (FPS), and Feat-FPS proposed by [62] (Tab. V) on the performance of our method. We also showed the visualizations of three sampling methods (Fig. 4). We found that RS had the worst performance, and FPS could obtain the best performance, which was $\sim$7% higher than RS. Besides, Feat-FPS also had good tracking performance, which was only $\sim$1% lower than the FPS. We attribute this result to the fact that FPS can obtain seeds which belong to the foreground and background points uniformly. Meanwhile, FPS could keep the distribution probability of the original input point cloud to the greatest extent while reducing the dimension of the input, which will be beneficial to the classification and regression tasks of the tracking network.

Our method is consistent with P2B when generating template point clouds. Therefore, we explored the different way of template point cloud generation, including the first ground truth, previous result, the fusion of the first ground truth and previous result, and all previous results. Specially, the fusion between first frame ground truth and previous results means the fusion of two point clouds within the two 3D bounding boxes respectively. First, we extract the points in the box from the point cloud according to the first frame ground-truth box and the predicted box in the previous frame. Second, according to the angle of the box and the coordinates of the center point, the two frames of point clouds are normalized to the same coordinate system by rotating and translating respectively. Finally, the updated template point clouds could be obtained by concatenating two point clouds directly. We reported the results in Tab. III. We found that our PTT-Net outperformed 3D-SiamRPN, P2B and SC3D in all settings. Besides, the proposed method has the best performance by fusing the first ground truth and the previous result. We believe that this is because our method has more robust tracking results in sparse scenes. Therefore, if we fuse the first ground truth and the previous result to update the template points, it could further enhance the target information and improve the algorithm performance.

The generation strategy of search area in object tracking task directly determines the feature scale and quality that the network can learn. In previous work SC3D [16], P2B [17], there are performance comparisons in different search area situations. To further explore the performance of our method, we also conducted experiments on the search area generation strategies, and compared them with SC3D [16] and P2B [17]. The experimental results are shown in Tab. IV. Specifically, we compared three different search area generation methods: 1) centered on previous result; 2) centered on previous ground truth; 3) centered on current ground truth. The results show that the performance of the three methods has been greatly improved with search area generated by ground-truth. The reason is that each frame uses the ground-truth result, which can effectively avoid the accumulation of errors caused over time. However, it is worth noting that our method PTT is slightly lower than P2B after using the ground-truth, but the performance is still at a similar level.

To verify our design in Sec. III-E of positions where PTT modules are embedded, we tried different schemes (Tab. VI). The results show that embedding PTT module in both stages of [17] can obtain the best improvement. Comparing (a) with (b) in Fig. 5, PTT-Net has better point cloud classification results which focus on foreground points. Comparing (c) with (d), PTT-Net could still track the target pedestrian robustly when more proposal centers are generated from another pedestrian. Besides, as shown in Fig. 9, PTT-Net can focus on foreground points with the help of PTT module. This result effectively shows that the transformer can learn more target-wise information.

Here we discuss the details of our PTT module, including the number of heads and the number of attention layers, as shown in Tab. VII. For the number of heads, we observe that $head=1$ and $layer=1$ achieves the best performance, and stacking more heads or layers could not bring in performance improvement but more parameters and lower speed. We believe that since our PTT module is directly applied on the fusion feature, which already has the similarity representations, more heads or layers in PTT may make the feature focus on other unimportant features, thus distracting the already fused similarity features. Therefore, we set both the heads and layers to 1.