Learning 6D Pose Estimation from Synthetic RGBD Images
for Robotic Applications

In this work, the synthetic data is generated in Blender [6] by leveraging its state-of-the-art raycasting rendering functionality. To render RGBD images, a textured 3D model of the object is required, which can be derived from CAD data or collected by 3D scanning.

Image Generation Given a set of objects, we generate a separate dataset for each object of interest, with the other objects and additional unrelated objects acting as distractors. For each scene to be rendered, we randomly place the objects in the camera’s view. In order to avoid overfitting on the color during training, we recolor 25% of the distracting objects with the dominant color of the main object. Moreover, the distractors’ optical properties, such as surface roughness and reflectivity, are varied to further increase the variety of generated images. During simulation, the randomly placed distracting objects can severly occlude the main object, which makes the main object not clearly visible, resulting in invalid training data. To avoid this, we check whether the centroid of the main object is occluded, in which case, we move the occluding objects to the back of the main object.

We sample images from SUN2012 [36] to use as backgrounds in Blender. Instead of adding the backgrounds to the images after rendering, we follow the idea of image-based lighting, where the backgrounds images are physically rendered as infinite spheres around the scene and emit light. Therefore, the backdrop images affect reflections and lighting conditions in the scene. Furthermore, a random number of point lights are added to the scene with arbitrary power and position. Once the scene is constructed, Blender starts rendering to generate RGB and depth images.

Labels The segmentation mask can be directly rendered using the object ID feature in Blender. The position and rotation of all objects and the camera is known, and the ground truth transformation matrix $Rt$ can be derived accordingly. The labels for each image are then saved to separate JSON files for each image.

After rendering the images, we apply several augmentation techniques to mitigate the reality-gap. Transformations that would change the object’s position in the image would invalidate the ground truth labels, the exception being rotations around the central axis of the imaging sensor. Therefore, an efficient method to increase the number of training data is to rotate each image around the central axis and adjust the labels accordingly. We can efficiently multiply our training data by this method and apply the following techniques separately to each rotated image.

RGB data augmentation The synthetic RGB images are augmented by randomizing saturation, brightness, hue and contrast, sharpening and blurring. Moreover, we add Gaussian and smooth 2D Perlin noise as in [25] to each color channel to cover different environments and sensors.

Depth data augmentation The synthetic depth images rendered from simulations are noiseless and almost perfect, which is not the case for images obtained from a real depth camera, where the depth values are often inconsistent and incomplete [22]. To approximate inconsistent depth values, we introduce Gaussian noise and Perlin noise to augment synthetic depth images. Similar to [31], pixel-level Gaussian noise is added to the synthetic depth images resembling a blurring effect. Smooth Perlin noise has been shown to significantly increase performance when learning from synthetic depth data [30]. We create Perlin noise with random frequency and amplitude and add it directly to the depth channel. The introduced Perlin noise shifts each depth point along the perceived Z-axis, resulting in a warped point cloud, similar to the observed point clouds of real depth cameras. In real RGBD images, a misalignment can be observed between depth and RGB images. Similar to [38], we use Perlin noise again to additionally warp the depth image in the image plane. Instead of using a 3D vector field to warp the entire depth image, we restrict warping to the edges of the objects. We apply a Sobel filter to detect the edges and obtain edge masks. We then shift the pixels on the edges using a 2D vector field generated using Perlin noise.

The rendered depth images have no depth information where there is no 3D model, resulting in large empty areas between objects. However, it is also very important to simulate plausible depth values for the background [30].

The background depth is based on a randomly tilted plane, to which we add a random Gaussian noise. The noise is sampled on a grid over the image and then interpolated. An additional Gaussian noise is sampled from a second grid and again interpolated. Due to the random and independent choices of grid sizes and interpolation for the two grids, we can achieve a wide variety of depth backgrounds. By adding an appropriate offset, we guarantee that the artificial background is in close proximity to the main object; hence, making object segmentation from the background more difficult. The artificial depth background then replaces empty depth pixels in the original synthetic depth image.

In the real depth images, some regions might miss the depth values and are observed as holes due to strong reflections of the object or other limitations of the depth sensor [22]. To simulate the missing depth problem, we first generate a random 2D Perlin noise map, which is converted to a binary masking map based on a threshold. This binary masking map is then used to create missing regions in the synthetic depth image. While this method is not an accurate simulation, we found this approximation, in combination with the other augmentation strategies, useful to improve the accuracy of the neural network.

The goal of 6D pose estimation is to estimate the homogeneous matrix $Rt\in SE\left(3\right)$, which transforms the object from its coordinate system to the camera’s coordinate system. This transformation matrix consists of a rotation $R\in SO\left(3\right)$ and the translation $t\in R^3$ of the target object. In this work, we use PVN3D [11] to infer the homogeneous matrix $Rt$ on the cropped region of interest (ROI) identified by a YOLO-V4-tiny [3] object detector. This two-stage approach is shown in Figure 1.

The RGB image is processed at the first stage using YOLO-V4-tiny, which provides several candidate bounding boxes and confidence scores. The bounding box with the highest confidence score for a specific object determines the ROI. Given the ROI, the cropped area is the smallest square centered on the ROI and including it, that is a multiple of the PVN3D input size (e.g. 80 x 80, 160 x 160, …). The square cropped images are then resized to 80 x 80 using nearest neighbor interpolation.

Following PVN3D[11] and PointNet++[26], the point cloud is enriched by appending point-wise R, G, B values and surface normals. We estimate the surface normal vectors by calculating the depth image’s gradients and the pixel-wise normals geometrically as in [14]. Differently from the original PVN3D [11] implementation, where the nearest neighbor approach is used to compute the normals from unstructured point clouds, calculating normals from structured depth image is more computationally efficient [23]. This also allows us to use a GPU-based gradient filter in TensorFlow. The resulting point cloud is then randomly subsampled to increase computational efficiency.

In the second stage, PVN3D is used for the pose estimation, with PSPNet [40] and PointNet++ [26] as backbones to extract RGB and point cloud features separately. The extracted latent features are then fused by DenseFusionNet [33] at pixel level. Because of the resizing of the cropped RGB image, we map the resized features back to the nearest point in the point cloud. Shared MLPs are then used to regress to the point-wise segmentation and keypoints offsets $\left\{o{f}_i\right\}\in R^3$.

To obtain the final object pose, the point-wise segmentation filters out background points and the keypoint offset are added to the input point cloud to get keypoint candidates. In [11], keypoint candidates are clustered by using Mean-Shift clustering for the final voted keypoints $\left\{\widehat{k{p}_i}\right\}\in R^3$. However, the Mean-Shift algorithm works iteratively and this prevents an efficient GPU implementation with deterministic execution time. To make the keypoint voting temporally deterministic, we first select a fixed amount of point cloud points for each keypoint with the smallest predicted offset. Compared to random sampling, this selection method already removes those outliers that show a high offset. To eliminate any further outliers, we filter out any keypoint candidate whose distance to the mean prediction $\mu$ exceeds the standard deviation $\sigma$, i.e. the offsets $o{f}_i$ will be masked out if $|o{f}_i-\mu |>\sigma$. After removing outliers, we apply global averaging on $\{x,y,z\}$ axis to obtain the voted keypoints $\left\{\widehat{k{p}_i}\right\}\in R^3$. We use singular value decomposition (SVD) to find the SE(3) transformation matrix $Rt$ between the predicted keypoints $\left\{\widehat{k{p}_i}\right\}$ and the reference model keypoints $\left\{k{p}_i\right\}$.

The prediction accuracy is improved by cropping the image to the ROI, as only the relevant part of the data is processed. With the same number of sampling points, the sampled point cloud from the cropped image is denser, providing PointNet++ with richer geometric information for feature processing, which can also be observed in [19]. Given the cropped input, we could build the PVN3D with only about 8 millions parameters, which is approximately 15% of the original implementation [12]. In our test on the LineMOD dataset, the reduced PVN3D performs similarly to the original model. We refer to the reduced PVN3D model as PVN3D-tiny.

In this work, we are interested in the 6D pose estimation of a single object in cluttered environments. For each LineMOD object, we render scenes in Blender to generate 20k synthetic images using the provided 3D model. We then augment the rendered dataset by rotating each image around the center of the image 16 times, resulting in around 300k images. We discard the images where the object is out of view and pad the images with zeros for empty areas after rotation. Additionally, each image is augmented offline by applying the domain randomization techniques introduced in section 2. For the training of PVN3D-tiny, we first crop the RGBD to obtain the region of interest according to the ground truth bounding boxes. Furthermore, we generate the point-wise semantic masks and keypoints offsets from the cropped RGBD images using the ground truth poses and segmentation masks.

To quantify the reality-gap between synthetic and real data, we sample 50 RGBD images from the synthetic and real dataset and compare global statistics of these two subsets. For RGB images, we compute the average and the standard deviation for brightness and saturation, as we qualitatively observed that these two factors have a strong influence on the appearances of the generated data. By comparing the statistic of brightness on the synthetic and real subsets, we can optimize the average power and randomization of the point lights and the light-emitting background in Blender. Similarly, with the statistics of saturation, we can optimize the color management in the Blender. To study the statistics of depth images, we use the average power spectral density (PSD) and compare the average distribution on frequencies, as shown in Figure 3. Studying PSD on frequencies allows us to inspect the structures of the depth images. And we can accordingly adjust the frequency of the Perlin noise used for depth augmentation. It can be seen that the augmented depth images are closer in frequency distribution to the real images than the non-augmented ones. This is an indication that depth augmentation reduces the gap between the synthetic and real data.

The study of the global statistics for RGBD images is fairly cheap since no real annotations are required. The study also enables us to identify the reality-gap qualitatively and adjust the data generation parameters accordingly, for instance, brightness, depth frequencies, etc., to bring the synthetic data closer to the real data. Figure 2 shows some examples of real and synthetic images.

The synthetic data generation pipeline is implemented in Python using Blender’s API. The data randomization and preprocessing are implemented using TensorFlow, accelerating the processing with GPUs. As for the two-stage 6D pose estimation approach, we use the original Darknet implementation [3] of YOLO-V4-tiny for the object detection at the first stage and PVN3D-tiny, implemented in TensorFlow, in the second stage.

We separately train a binary YOLO-V4-tiny model and a PVN3D-tiny model for each object of the LineMOD dataset. The YOLO-V4-tiny model is trained using the Darknet framework [27], and PVN3D-tiny is trained in TensorFlow [1]. All deep neural networks are trained from scratch using only synthetic data without any pretrained models. After training, we build the two-stage 6D pose estimation pipeline by combining YOLO and PVN3D. We follow [11] to evaluate the 6D pose estimation performance on the annotated real images provided in LineMOD. The 6D pose estimation performance is measured by using ADD(S) metrics [13]. ADD measures the average distance between the ground truth point cloud and the point cloud transformed with predicted $R,t$, which can be defined as follows:

where $m$ is the number of the sampled points, $R^{\ast },t^{\ast }$ is the ground truth pose, and $v\in R^3$ denotes a vertex from the object $O$. Similarly, the ADDS metric measures the average minimum distance between two point clouds as:

Compared to ADD, ADDS measures the distance to the nearest point instead of correspondent mesh points. For symmetrical objects, ADDS is better suited because ADD yields low scores if the object’s pose is different to the ground truth, even if the pose corresponds to an invariant rotation. The success rates on test images are used to quantify the pose estimation performance. A threshold of $10\%$ of the object’s diameter is typically used to classify a prediction as successful or not.

We evaluate the performance of the proposed 6D pose estimation approach on all objects from the LineMOD dataset. We report the results with comparison to the state-of-the-art work in Table 1, in which the performance of PointFusion is from [19] and performance of SSD-6D [16] is from [29]. Our approach generally has a good performance on most of the objects without pose refinement. Specially, it performs well on small objects like “ape”, “duck”, on which the SSD-6D[16] and AAE [29] are less accurate. On the other hand, our model performs less than optimal on “Lamp” and “Holepuncher”. The reason could lie in the low-quality textures of the LineMOD models. Our approach, being trained end-to-end on RGBD data, could be more sensitive to less-detailed textures compared to refinement based approaches.

Compared to the related work that only uses synthetic data for training, our approach outperforms AAE [29] and has competitive performance compared to SSD-6D[16]. Furthermore, [9] proposes a 6D pose estimation algorithm based on DGCNN [34] and reaches 98% average accuracy. However, it relies heavily on pose refinement, and it takes approximately one second to detect a single object. The table also shows that most of the work trained using real data outperforms the synthetic-only trained methods.

The proposed two-stage approach is limited by the object detection performance at the first stage. When tested with ground truth bounding boxes, the performance of the 6D pose estimation at the second stage is improved by a large margin, achieving approximately 94% accuracy, as presented in Table 2. This result is comparable to the state-of-the-art methods trained using real data. One potential way to improve the object detection performance is to use RGBD images [8], where the object detector can learn the more robust features from both appearances features provided by RGB images and geometry features provided by depth images.

We evaluate the running time efficiency on a desktop of Xeon Silver-CPU(2.1GHz) and an NVIDIA Quadro RTX 8000 Graphics card. As shown in Table 3, the inference of the PVN3D-tiny takes the most of the running time and the other procedures run at similar speeds. Given 480x640 RGB and depth images, our approach takes approximately 46ms for a single object pose estimation. Compared to the other approaches listed in Table 1, our approach achieves a sufficient running time for real-time tasks and comparable or better accuracy. As outlined in the next section, we found the accuracy to be sufficient for robotic grasping tasks.

We demonstrate the proposed approach on a household object, a rubber duck, as shown in Figure 4. We first use the proposed data generation pipeline to generate synthetic training data, for which the 3D model of the object is obtained using a Shining3D Transcan C 3D scanner. We train the proposed two-stage estimation approach using the synthetic data and use it for a robotic grasping task. The goal of the task is to grasp the rubber duck from a cluttered environment. The robot first estimates the 6D pose of the duck from an RGBD image and then will attempt a grasp using one of the pre-registered grasp poses. If the object can be grasped and lifted without slipping, this grasp will be regarded as a success, a failure otherwise. After each grasp attempt, the object will be deposited back to the table at a random location. Should the object be dropped in a position that allows no grasps, we skip this attempt and reset the experiment. We create three scenarios with normal, unbalanced and dark lighting conditions to test the algorithm’s robustness to different lighting levels. We repeat the grasp attempts 20 times for each test scenario.

As shown in Table 4, the robotic arm has achieved 53 out of 60 successful attempts with an approximate 88% successful rate. The three scenarios show similar success rates, showing the algorithm’s robustness to different lighting levels. Notably, the proposed algorithm works well in low-lighting conditions. The reason could be attributed to two factors: the first is that the training on domain-randomized synthetic data makes the algorithm learn more robust features; secondly, the use of depth can compensate for the blindness of the RGB when the lighting condition is poor.