PCDNF: Revisiting Learning-based Point Cloud Denoising via Joint Normal Filtering

As a fundamental geometry processing problem, point cloud denoising has drawn great attention in the past decades. Due to the abundance of literature on denoising techniques, it is beyond our scope to review all existing work. We refer the interested reader to [52] for a comprehensive review. Here, we first mention relevant traditional methods, and then concentrate on the recent learning-based techniques.

Traditional methods. MLS-relevant (Moving Least Squares) methods [1, 2, 12, 29] project the input point set onto the approximated underlying surface iteratively, which have been designed for reconstructing noise-free surfaces originally. However, these classical methods assume piecewise smooth priors for the underlying surface, which inevitably causes geometric features being smoothed. Later on, Lipman et al. [20] proposed the pioneering method LOP (Locally Optimal Projection), which has been proven successful for point cloud consolidation. LOP and its variants [17, 18, 31, 24] aim at producing a point set to describe the underlying surface while enforcing a uniform distribution. Unfortunately, although LOP-related methods can robustly remove noise and yield uniformly sampled results, they are unable to preserve geometric features in the presence of large noise. Optimization-based methods formulate the denoising process as optimization problems with appropriate priors. Among them, sparse optimization methods, such as [3, 41, 27, 21], are more effective in preserving geometric features (especially for sharp features), based on the prior that geometric features are sparse over underlying surfaces. Recently, using the self-similarity characteristics of underlying surfaces, low-rank and dictionary learning techniques [11, 9, 23, 42, 40] have received attention for providing structural repetition preservation of underlying surfaces. Although optimization-based methods have advantages in preserving some kinds of geometric features, geometric priors may downgrade their performance in preserving other kinds of geometric features. In general, traditional methods have to tackle complex computation or optimization problems, and require a tedious trial-and-error process to produce satisfactory results.

Learning-based methods. Recently, with the development of neural networks[32, 33, 43], deep learning techniques have been introduced into point cloud denoising extensively and achieved impressive results. Roveri et al. [35] proposed PointProNet, a fully differentiable denoising architecture based on 2D CNN, which converts unordered points to regularly sampled height maps. EC-Net[45] and DMRDenoise[25] mainly focus on upsampling and consolidating techniques over point clouds. EC-Net designs an edge-aware consolidation network to denoise point clouds. This method preserves sharp geometric features but retain noise to some extent, especially in the presence of large noise. Luo and Hu proposed a downsample-upsample architecture, called DMRDenoise, to reconstruct noise-free point clouds by learning manifold structures of underlying surfaces. Although DMRDenoise can remove noise successfully during the downsampling stage, it may blur geometric features. TotalDenoising, developed by Hermosilla et al. [15], introduces a spatial prior that steers converge to underlying surfaces without supervision. However, as an unsupervised method, TotalDenoising is sensitive to large noise and may suffer shrinkage artifacts. GPDNet [30] uses the graph-convolutional neural network for denoising. Luo and Hu [26] proposed a novel paradigm of denoising by exploiting the distribution model of noisy point clouds. Most of the above learning-based methods cannot effectively preserve geometric features, especially sharp edges and corners. Moreover, the noise removal performance of these methods decreases evidently as the noise level increases.

More recently, some techniques belonging to the two-stage paradigm [22, 44], i.e., normal filtering followed by updating point coordinates, have been developed for feature-preserving denoising. Lu et al. [22] classified the noisy input as feature points and non-feature points and then predicted multi-normal on the feature points to preserve sharp features. Wei et al. [44] proposed a geometry-supporting dual convolutional neural network (GeoDualCNN) to filter normals, and then updated point coordinates to match the filtered normals. Although GeoDualCNN can produce promising results, the requirement of computing the extra homogeneous neighborhood for each point limits its end-to-end applicability. Compared to those two-stage methods, an alternative paradigm is to develop networks that directly predict displacements of the noisy point cloud and then apply the predicted displacements to reposition point coordinates. Following this paradigm, some methods have been developed [34, 46, 10]. PointCleanNet [34] first excludes outliers and then predicts displacement vectors for the remaining noisy points. This method may retain extra noise on the denoised results and tends to smooth sharp features in varying degrees. To address this issue, Pointfilter [46] introduces a feature-preserving loss function with an encoder-decoder framework. Although Pointfilter can preserve sharp features, it cannot recover small-scale geometric features well. Later, Chen et al. [10] proposed a feature-aware recurrent architecture to learn more representative features for recovering multiscale geometric features effectively. However, in the case of large noise, their method seems to be difficult to keep a balance between noisy removal and recovery of geometric features.

As an important signal which indicates the direction field of the scanned surface, normals of point clouds have been widely applied in various practical problems (e.g., surface reconstruction [19, 16], 3D descriptor[36], registration[48]). However, it is challenging to accurately estimate point normals, since the captured point clouds are inevitably corrupted by noise and outliers. To address this issue, normal filtering has been studied extensively over the past decades. Due to the considerable amount of literature on normal filtering, we only review those learning-based methods related to our work. For a more comprehensive review of normal filtering, readers are referred to [28, 49].

Since the pioneering work of Qi et al. [32], hundreds of work have extended and applied the PointNet architecture to point cloud processing problems. PCPNet [13] is the first to apply the PointNet architecture for estimating normals from noisy point clouds (the normal filtering task). Zhou et al. [50] proposed a plane constraint mechanism to divide neighborhood points into main plane points and error points, and then only used the learned features of the main plane points to regress normals. Their method is robust to noise and neighborhood scales. To overcome the oversmoothing artifacts, Nesti-Net [5] introduces mixtures of experts to predict the optimal neighborhood scale instead of simply concatenating multiple scales together. Their multiscale strategy can improve performance effectively but leads to evident time consumption. Zhou et al. [51] proposed a normal filter based on multipatch stitching. Thanks to their patch-level architecture, their method can reduce computational costs and improve the robustness of noise removal. Instead of directly predicting normals from the learned features, some methods [4, 54, 47] estimate the normal for a specific point by fitting a local underlying surface through its neighboring points and then compute the normal from the fitting surface. These methods are based on the weighted least squares surface fitting of the local geometric neighborhood, which can improve the generalization ability of their networks on real scanning data [54]. Cao et al. [6] learned a latent tangent space representation with a lightweight network, and then utilized a differentiable RANSAC to estimate normals of the underlying surface. Zhou et al. [49] deployed a multi-feature scheme to capture geometric information from multiple feature representations and then updated normals in a refinement system. Unlike these existing methods, we present a unified architecture for jointly learning point cloud denoising and normal filtering. The combined technique is able to apply the best properties of each of the two tasks, and try to overcome the weakness of both. Thus, it performs well in preserving geometric features and removing noise, and at the same time avoids the artifacts in the results.

Point cloud denoising is nontrivial due to the ill-posed nature of the problem. Many learning-based methods [34, 46, 10] cast the noise as pointwise residuals (i.e., a displacement vector per input point), and try to predict the residual vectors in order to smooth the input point cloud. However, given only the point positions, it is still challenging to achieve satisfactory denoising results while preserving geometric features. Note that, high-quality normals can improve denoising performance; conversely, accurate normals can be computed from high-quality point clouds. Thus, as an alternative to directly predict the displacement vectors from the noisy input, we propose to perform point cloud denoising by combining position correction and normal filtering, which is stated as follows

where $P$ and $\hat{P}$ are the noisy point cloud and its corresponding denoised point cloud, $\hat{D}$ denotes the predicted displacement vectors, $N$ denotes the raw normals of the noisy input and $\hat{N}$ denotes the corresponding predicted normals. Our method aims to learn a mapping $F:(P,N)\to (\hat{D},\hat{N})$ for predicting displacement vectors and filtered normals simultaneously. Then, the denoised point cloud can be derived from (1) straightforwardly.

Based on the problem statement (1), we design a multitask network, dubbed PCDNF, for joint point cloud denoising and normal filtering. Normal filtering can be seen as an auxiliary task that helps our network better recover noise-free point clouds. Fig. 1 shows the architecture of PCDNF. Our network consists of four modules: the multiscale feature extractor, shape-aware selector, feature refinement, and the decoder.

Specifically, given a noisy patch and its corresponding raw normals, the multiscale feature extractor embeds the inputs into the coarse representations of point and normal features. Then, the shape-aware selector selects the points highly related to the specific point in terms of geometric information and coarse representations. These selected points form a latent tangent space of the specific point, making noise removal more effective. The feature refinement module first encodes local spatial information to augment the similarity representations and then fuses the augmented representations for better geometric features preservation. Finally, the coordinate and filtered normal of the specific point can be predicted by the coordinate and normal regressors of the decoder.

Note that most previous methods either directly predict point displacements for denoising or separate the denoising task into normal filtering and point updating and process them separately. In contrast, we combine the highly related denoising and normal filtering tasks to make them benefit each other.

To train our network in an end-to-end manner, we design three types of losses as our optimization objectives: a point-denoise loss, a normal-filter loss and an orthogonality loss. The joint loss function is formulated as

where ${\lambda }_1$, ${\lambda }_2$, and ${\lambda }_3$ are three hyperparameters that balance the importance of each term. We empirically set ${\lambda }_1=100$, ${\lambda }_2=10$ and ${\lambda }_3=10$ for training.

Dataset. To train our end-to-end network for denoising and normal filtering tasks, we adopt the dataset provided by [46]. The training set contains 11 CAD and 11 non-CAD models (clean data without noise). The ground truth point clouds with normal information can be obtained by randomly sampling the clean models. The number of sample points for each model is set uniformly as 100K. For these groundtruth point clouds, we corrupt each of them by Gaussian noise with standard deviations of 0.25%, 0.5%, 1%, 1.5% and 2.5% to the diameter of the bounding box. Thus, the final training set contains 110 noisy point clouds (normals are estimated with PCA) and the corresponding 22 ground truth point clouds (with normals).

To conduct comparisons more effectively for both tasks, we construct a test set consisting of synthetic point clouds and raw scanned data. For synthetic data, we rely on the synthetic dataset released in [46] including three categories: simple, medium, and complicated, in which there are 7, 6, and 7 clean models, respectively. The raw scanned data will be introduced in the following experimental section. Again, the number of sampled points for each model is unified to 100K. Each of the clean point clouds is perturbed by Gaussian noise with standard deviations of 0.25%, 0.5%, 1% and 1.5% to the diagonal of the bounding box.

Network inference. Our multitask network can produce feature-preserving denoising results and satisfactory normal filtering results simultaneously. Our method can be iteratively applied to improve the results when the noise level is high. The denoised points and filtered normals produced by previous iterations can further serve as the inputs to our method for further refinement. Note that this iterative process is only performed during the inference phase.

Implementation details. We implement our network in PyTorch and train it on a single NVIDIA GTX 2080Ti GPU for 45 epochs with the SGD optimizer. The learning rate is decreased from 1e-4 to 1e-8.

We present visual and numerical comparisons between our method and state-of-the-art denoising methods, including WLOP [17], RIMLS [29], EC-Net [45], DMRDenoise (DMR) [25], PointCleanNet (PCN) [34] and Pointfilter (PF) [46]. For WLOP and RIMLS, we perform the code provided by their authors and carefully tune the parameters to produce the denoised results. For DMR and PCN, we use the code released by their authors to retrain new models over our training set. For EC-Net and Pointfilter, we use the pretrained models provided by the authors. Sometimes, we reconstruct the denoised results produced by the tested methods for enhancing visual effects via RIMLS (provided by Meshlab for feature-preserving reconstruction).

Synthetic data. Fig. 31 demonstrates comparisons of CAD surfaces including sharp features (sharp edges and corners) and smooth regions. The tested CAD surfaces are corrupted by a considerable amount of noise. As shown in Fig. 31, WLOP and EC-Net may retain excessive noise in the results when the noise level is high. Although DMR can remove the noise, it seriously distorts the overall shapes of the surfaces; see Fig. (e)e. RIMLS and PCN recover smooth regions well but blur sharp features to varying degrees; see Figs. (c)c and (f)f. In addition, at the tested noise level, PCN may induce some artifacts in the smooth regions of the results. In contrast, our method and PF effectively preserve sharp features. Compared to PF, our method recovers sharp features more accurately; see Figs. (g)g and (h)h. Thus, our method is superior in preserving sharp features and smooth regions simultaneously to the compared methods.

Fig. 40 gives comparisons of a non-CAD surface corrupted by considerable noise. WLOP and EC-Net fail to remove noise entirely in this example. Although DMR does a good job of noise removal, it causes shape distortion and induces many bumps in the result; see Fig. (e)e. RIMLS and PCN seem to have difficulty balancing the performance of noise removal and feature recovery when the noise level is high; see Figs. (c)c and (f)f. PF and our method can generate visually better results than the other tested methods However, from the zoomed views of (g)g and (h)h, we observe that the trunk of the elephant in the result produced by PF is swelling; in contrast, our result does not induce this artifact. Therefore, our method visually yields the best result for faithfully preserving geometric features.

Fig. 49 shows comparisons of a non-CAD surface with multiscale geometric features. The tested non-CAD surface is corrupt by moderate noise. Except for WLOP, all the other methods can remove noise effectively. DMR flattens medium- and small-scale features and induces distortion and swelling in the ear regions of the bunny surface; see Fig. (e)e. RIMLS and PF oversmooth small-scale features and fine details to varying degrees, and PF makes this situation even worse; see the reconstruction results in Figs. (c)c and (g)g. Furthermore, although EC-Net and PCN can preserve different levels of geometric features in a better manner, they induce some artifacts in the ear regions of the results. These artifacts degrade the visual quality of the denoised results and further affect the reconstruction results; see the zoomed views and reconstruction results of Figs. (d)d and (f)f. In contrast, our method outperforms the other methods in terms of recovering most geometric features and preventing inducing annoying artifacts.

Scanned data. To further verify the effectiveness of our method, we test it on real scanning data and do not retrain our method (by using any scanned data). Fig. 58 shows the results for Kinect scanning data. As we can see, for the CAD surfaces in the first and second rows, all the other methods retain additional bumps on the denoised results. In contrast, our method can effectively prevent visible artifacts and show its superior performance in preserving geometric features and recovering smooth regions. For the non-CAD surface in the third row of Fig. 58, RILMS, PF, and our method can generate more natural results than the other methods; see Figs. (c)c, (g)g, and (h)h. However, from the numerical metrics in Table I for the tested non-CAD surface (David), we observe that the error of our method is lower than those of RIMLS and PF, which shows that our method outperforms RIMLS and PF in dealing with the scanned surface containing complex surface characteristics.

Then, we further evaluate the effectiveness of our method on laser scanned data. As shown in Fig. 67, although all of the tested methods can remove noise and preserve geometric features to some extent, our method can yield more appealing results with neat structure features preserved and clean smooth regions recovered; see the zoomed views of Fig. 67. Furthermore, we test our method on raw outdoor scenes from the Paris-rue-Madame Database [38]. As shown in Fig. 77, compared with the other methods, our method can produce better denoised results with fewer outliers in terms of removing the large noise of the outdoor scene.

Robustness tests. To further verify the robustness of our method, we demonstrate the performances of our method against irregular sampling, different noise levels, and outliers in the following. We show the effectiveness of our method against nonuniform sampling in Fig. 86. Although the noisy surface suffers from irregular sampling, our method is noticeably better than all the other compared methods, which can produce a compelling result that preserves sharp edges and corners. Fig. 91 shows the robustness of our method against different noise levels. As we can see in Figs. (a)a, (b)b, and (c)c, our method can remove noise while preserving sharp features well when increasing the noise level. However, when the noise level is larger than the geometric feature sizes, our method may blur some geometric features and cannot produce satisfactory results; see Fig. (d)d. Since our method takes into account the handling of outliers, it can deal with the surface corrupted by a larger number of outliers effectively; see Fig. (a)a.

Quantitative evaluation. We observe from the aforementioned qualitative comparisons that our method can generate visually better denoised results than the competing methods. Here, we compare them numerically. We utilize the Chamfer Distance (CD) to measure the fidelity of the denoised result to the ground truth point cloud. The CD error metric is widely used in the work [10, 46, 34], and a lower CD value denotes a better denoised result. We first compare the examples in Figs. 31, 40, 49 and 58 and list the evaluation results in Table I. As we can see, our method outperforms the competing ones because our CD errors are significantly smaller than all the other compared methods. Then, we further compare our method to those deep learning methods (EC-Net, DMR, PCN, and PF) on the test set introduced in subsection 4.1. The evaluation results on the test set are shown in Fig. 96. When the noise level is low, moderate, and considerable (0.25%, 0.5%, and 1.0% noise), our method clearly outperforms the learning-based methods in terms of the averages of the CD error metric. Moreover, under the highest noise level (1.5% noise), PF and our method both show lower error values, indicating that these two methods can produce more compelling results than the other competing methods. Accordingly, the quantitative evaluations are consistent with the visual comparisons, both of which show the superiority of our method.

Computational time. We also record the CPU costs of all the tested methods in Table I. As seen, DMR is the fastest method, while PCN is the slowest one. Our method ranks fourth in speed among the learning-based methods. In particular, the traditional methods (WLOP, RIMLS) require trial-and-error efforts to tune parameters to produce satisfactory results in practice; thus, we only discuss the inference time of the deep learning methods (EC-Net, DMR, PCN, PF, and ours). DMR and EC-Net are faster than the other deep learning methods (PCN, PF, and ours). The reason is that DMR and EC-Net have an analogously upsampling process and only use a few patches of the noisy point cloud as input. In contrast, the PCN, PF, and our method use a pointwise manner to deal with the noisy input; thus, the inference times of these three methods are higher. Moreover, due to the network complexity, our method takes less time than the PCN and more time than the PF. In summary, although the inference time of our method seems to be slightly computationally intensive, it can generate more appealing results in terms of visual quality and CD error metrics in most examples.

The normal filtering task also plays an important role in our method. We compare our method with state-of-the-art normal filtering methods, including the traditional method of Jet [7] and the deep learning methods of PCPNet [13], DeepFit [4] and AdaFit [54].

Qualitative comparisons. Fig. 103 visualizes the error map for normals, where the error is defined as the angular deviations between the filtered normals and the ground truth normals. As we can see, Jet and PCPNet produce higher errors in geometric feature regions. DeepFit can deal with smoothly curved regions well, but it blurs small-scale features. AdaFit effectively recovers small-scale features and fine details but slightly oversmoothes sharp features. In contrast, our method produces the desired results in terms of preserving sharp and detailed features. To test the generalization capability of our method, we also present Fig. 110 for comparisons between our method and four other methods applied to real indoor scenes acquired by Kinect sensors (NYU Depth V2 dataset [39]). Note that we do not train all the tested methods on scanned scenes. For a fair comparison, we only use the filtered normals produced by our method instead of retaining denoised point coordinates. As we can see in Fig. 110, Jet retains considerable noise in the results. PCPNet can smooth noisy surfaces effectively but flatten medium- and small-scale geometric features. DeepFit and AdaFit preserve geometric features well even for small-scale and detailed features, although they induce some bumps in the results. The reason for inducing the bumpy artifacts may be as follows. Since both noise and geometric details are high frequency information, DeepFit and Adafit may erroneously restore some high level noise as geometric features. Compared to DeepFit and AdaFit, our method can better preserve structure features, although some geometric details are flattened slightly. Moreover, our method tends to produce visually cleaner results without noticeable artifacts.

Quantitative Comparisons. To further compare the quality of the normal filtering results, we adopted the root mean squared error (RMSE) of the angle difference to quantitatively evaluate the filtered normals, as suggested in the previous work [13], [54]. Lower RMSE values indicate better normal filtered results. As we can see in Fig. 103, the results of our method have the lowest RMSE values for all the tested examples. We also compare our method with the other four on the test set introduced in subsection 4.1. As we can see in Fig. 112, when the noise is low and moderate (0.25%, 0.5% noise), our results show the lowest average RMSE values. While the noise level is increasing (1.0%, 1.5% noise), AdaFit achieves the best results, but our results are comparable to those of AdaFit (our method produces the second-best results).

We verify the individual contributions of the major modules in our network by conducting the following three ablation studies on our test dataset:

• Removing feature selection and refinement modules (V1).

• Removing feature refinement module (V2).

• The normal filtering network branch is removed (V3).

Table II lists the denoising results of our method and three ablated variants. From the table, we have the following observations. All three variants show lower performance than our full pipeline. Each module of our pipeline is necessary to ensure high-quality denoising results. More specifically, by comparing V1 with V2, it can be seen that the feature selection module is necessary for removing noise by selecting those feature points with similar characteristics to the denoised point. By comparing V2 with our full pipeline, we can see that the additional feature refinement module positively impacts denoising accuracy. The feature refinement module consists of two units (feature augmentation and fusion). The roles of these two units are demonstrated in Figs. 18 and 22 and explained in subsection 3.2.3. We use a two-branch network structure for point cloud denoising and normal filtering tasks. To demonstrate the positive interaction between these two tasks, we design a variant (V3) that performs only the denoising task. Compare V3 with our full pipeline, we observe that the additional normal filtering branch in the multitask learning of our network helps our network generate better denoised results through the mutual promotion of these two tasks, which confirms our hypothesis.

Although our method can produce high-quality results in terms of removing noise and recovering geometric structure and detailed features, it is subject to a couple of limitations.

The results of our method depend on the quality of the input initial normals to some extent. To examine the influence of the input initial normals on our method, we demonstrate our (denoising and normal filtering) results produced with different initial normals as inputs. As Fig. 118 shows, using better initial normals (produced by PCPNet and AdaFit) as inputs, we can obtain more faithful denoising and normal filtering results simultaneously. However, it is a chicken-and-egg problem to obtain a good normal estimation from the noisy point cloud. Thus, to keep the simplicity and generality of our method, we choose to use the normals estimated by PCA as input.

Our method may generate unsatisfactory results when the sampling of the input point cloud is extremely sparse. As shown in Figs. (b)b and (c)c, when the sampling ratios of the input point cloud are 0.3 and 0.2, our method can still produce reliable results, showing the robustness of our method at these sampling rates. However, our method suffers from the shrinkage artifacts when the sample ratio decreases to 0.05; see Fig. (d)d. One possible reason is that, when the sampling rate is very low, our network cannot learn an effective representation from these few points in order to characterize the shape of the patch.

To demonstrate the effectiveness of our method in various applications, we take our denoised results as inputs for surface reconstruction [19], point cloud registration [53], and plane fitting [37]. Fig. 120 shows, the reconstruction results from the noisy point cloud suffer from severe artifacts, in contrast, the reconstruction results from our denoised input are of high-quality and feature-preserving. Fig. 129 shows that our denosing method can help with the point cloud registration task, as it is noticeable that the result produced after applying our method is more accurate than that from the raw input. We perform plane fitting [37] on an indoor scene point cloud. As Fig. 133 shows, using our denoised result as input, the RANSAC algorithm [37] can produce a more reliable plane segmentation result, which is closer to the segmentation generated from the corresponding clean point cloud.