Learning to Construct 3D Building Wireframes from 3D Line Clouds

There are roughly two groups of methods to extract a line cloud from multi-view images: (1) reconstruct 3D lines and estimate camera parameters simultaneously [PLVIO, lineBasedSLAM, salaun2017line]; (2) reconstruct 3D lines with fixed camera parameters estimated from standard structure-from-motion (SfM) methods [Line3Dpp, jain2010exploiting]. In the work, we follow the latter one to reconstruct a line cloud, which is also adopted in Line3Dpp [Line3Dpp], the current state-of-the-art line cloud abstraction method. Specifically, the camera parameters are estimated from the multi-view images using SfM. Correspondences between the 2D line segments detected from each image (using any existing line detector) are established based on epipolar constraints, which are then used to solve 3D line segments based on the camera parameters. We use the line cloud extractor as provided in [Line3Dpp]. Note that, the extracted line cloud is potentially dense, noisy, and incomplete. The inset figure shows the histogram of the length of the line segments in the line cloud shown in Fig. 1 (b). Around 85% of the extracted line segments has a shorter length than the average edge length of the underlying building (Fig. 1 (h)). This suggests that the extracted line clouds contain large portion of short (and potentially noisy in orientations) line segments, which makes it challenging to extract a clean wireframe.

Specifically, we first project the 3D ground-truth wireframe $W^{\text{gt}}$ to image planes using the corresponding camera parameters to get the ground-truth 2D wireframe for each image $I_i\in I$, which allows us to check if a 3D line $l_i\in L$ is part of the wireframe $W^{\text{gt}}$ or not. If the 2D line segments, that are used to reconstruct the 3D line $l_i$, are close enough to the ground-truth 2D wireframes, $l_i$ will be classified as part of $W^{\text{gt}}$ and be labeled as 1. For a line $l_i$ with label 1, we further associate it with two ground-truth junction vertices that are the endpoints of the corresponding edge in $W^{\text{gt}}$ that $l_i$ belongs to. In summary, each line $l_i$ has a 5-dimensional label: $(f,i_1,d_1,i_2,d_2)$ where $f$ is binary indicating if this line is part of the wireframe, $i_1,i_2$ are the junction index in $V^{\text{gt}}$ and $d_1,d_2$ are the distances from $l_i$ to the two ground-truth junctions respectively if $f=1$. Note that the label $f$ is used to supervise our junction classifier, and the remaining labels are used to supervise our junction regressor.

In our setting, a line patch is defined as a group of 3D line segments collected w.r.t. sampling points. Specifically, given an arbitrary point $x\in R^3$, the corresponding line patch $G\left(x\right)$ is defined as: $G\left(x\right)=\{l\in L|\text{dist}(x,l)\le ϵ\}$, where $\text{dist}(x,l)$ measures the point-to-line distance between a point $x$ and the 3D line where the line segment $l$ lies. We can similarly define the line patch w.r.t. a pair of sampling points as $G(x,y)=G\left(x\right)\cup G\left(y\right)$. We observe that the line patches encode sufficient information to predict junction positions and connectivity between junctions. Specifically, the line patch $G\left(x\right)$ of point $x$ can be used to estimate the probability of having a junction located around point $x$, while the line patch $G(x,y)$ can be used to estimate the probability of having an edge connecting the point $x$ and point $y$.

See Fig. 2 for an illustration: In example (a1) and (a2), the lines in the red patch have multiple dominant orientations, suggesting that the blue sampling point is indeed close to a wireframe junction. This aligns with the fact that a 3D junction is formed by the intersection of at least three planes, and the corresponding intersecting lines shape the contour of the underlying building, which would lead to dominant line clusters in images. The blue sampling point in example (a5) is located on a roof plane, where the roof texture (see Fig. 1) contains structural lines. In this case there exists only one dominant direction, which is not enough to support a junction. Example (a6) shows the line patch of an outlier point, where the lines in the patch are extremely unstructured. Similar for the examples in (b1-b6), we can see that if two sampling points are likely to be connected to each other, the corresponding line patch will reveal strong pattern (e.g., having duplicated lines) to support it. All these observations of line patches perfectly align with the properties of the wireframe models of planar objects: the junctions are formed by the intersection of planes with at least three dominant directions determined by the intersecting lines. As a comparison, other points such as corners in textures or noisy points do not have comparably strong signals.

Given a line patch $G\left(x\right)$ or $G(x,y)$, how can we tell if there exists a junction or an edge? We propose a line-patch transformer, LPT, to extract features from line patches, which can then be used to predict the junctions/edges. Specifically, a line patch $G\left(x\right)$ can be represented as a 2D tensor $(N,F^{\text{in}})$, that stores $N$ neighboring lines in $G\left(x\right)$, and each line has $F^{\text{in}}$ features including the coordinates of the two endpoints and the distance between the line and the

sampling point $x$. We then collect $G$ groups of line patches in a 3D tensor $(G,N,F^{\text{in}})$. LPT contains two transformers (see inset figure): (1) the first transformer attends to the $N$ neighbors for each line patch, to potentially find the most prominent lines for junction predictions; (2) the second transformer attends on the $G$ groups, to potentially attend to the junctions that are co-planar. We can similarly use LPT to process line patches $G(x,y)$, where the initial features can be obtained by concatenating the features of $G\left(x\right)$ an $G\left(y\right)$.

We sample $G$ points $\{x_k{\}}_{k=1}^G$ from all the line endpoints of the line cloud $L$ to construct line patches for junction prediction. Specifically, we first sample a smaller set of points (around 25%) according to the endpoint density and then sample the remaining points via Farthest Point Sampling (FPS) [fps]. We then obtain the corresponding line patch $G\left(x_k\right)$ for each sample $x_k$ as discussed in Sec. 4.1. The line patches $\left\{G\right(x_k\left)\right\}$ are fed into LPT to extract patch features, which are used to classify if there exist a junction close to $x_k$, and regress the potential junction position $p_k$. We then collect the predicted junctions $p_k$ in $V^{\text{pred}}$. The classifier can help to filter out junctions with a low confidence. During the training phase, we first draw samples that are close to the ground-truth junctions, then sample via density and FPS to get $G$ sampling points for constructing the line patches. This guarantees that we draw both positive line patches (containing a junction) and negative line patches. Specifically, a line patch is considered as a negative sample if there are more than half of the line segments in the patch are labeled as noise (introduced in Sec. 3.2). The loss function for the classifier is a binary cross-entropy $E_{\text{v-clf}}$. The loss function for the regression is L${}_2$ distance between the predicted position and the ground-truth position.

We first sample $G$ pairs of predicted junctions $(p_k,q_k)\in V^{\text{pred}}\times V^{\text{pred}}$ w.r.t. the predicted probability. We can then construct the line patches $\left\{G\right(p_k,q_k\left)\right\}$ and feed them into LPT to extract patch features, which is used to classify the junction pair $(p_k,q_k)$ into five groups: (A) labeled as -1 if at least one of $(p_k,q_k)$ is a false positive junction (i.e., does not belong to the underlying wireframe); (B) two vertices are true positive junctions and the pair is labeled by the graph distance in the underlying wireframe, i.e., (B.0) with graph distance 0 ($p_k$ is identical to $q_k$), (B.1) with graph distance 1 ($p_k$ is connected to $q_k$), (B.2) with graph distance 2 ($p_k,q_k$ are adjacent to the same vertex), or (B.3) having graph distance larger than 2. Note that the edge labels can help to further prune the false positive junctions besides the probability produced by the junction classifier. During training, we sample from $V^{\text{gt}}\times V^{\text{gt}}$ and $V^{\text{pred}}\times V^{\text{pred}}$ to learn junction connectivity. A vertex from $V^{\text{pred}}$ is regarded as a false positive junction if its distance to the nearest ground-truth junction is larger than a threshold $ϵ$. Any vertex pair that contains a false positive junction is labeled as -1. The rest vertex pairs is labeled according to the graph distance in the ground-truth wireframe, where for a pair of predicted junctions, we use the graph distance between their nearest ground-truth junctions. The loss function for the classifier is standard cross-entropy $E_{e\text{-clf}}$.

The total training loss for our complete networks is: $E_{\text{total}}=E_{v\text{-clf}}+{\lambda }_v{E}_{v\text{-reg}}+{\lambda }_e{E}_{e\text{-clf}}$, where ${\lambda }_v,{\lambda }_e$ are balancing weights. For post-processing, we first use non-maximum suppression (NMS) to remove duplicated vertices and redundant edges that are close to each other. We then use the connectivity predictor to further prune the predicted junctions that tend to be false positives. Specifically, if a vertex pair is categorized to be identical (i.e., with label 0), then the junction with a lower confidence will be removed. For two vertices with similar Hamming distance in adjacency and small Euclidean distance, the vertex with a lower confidence will be removed. We also remove the isolated edges from the final wireframe.

To the best of our knowledge, there is no existing baseline for reconstructing building wireframes from multi-view images directly. We therefore mainly compare to the state-of-the-art 3D line cloud abstraction method, line3Dpp [Line3Dpp], building reconstruction methods, Line2Surface [surfaceLine] and PolyFit [polyfit], and 3D wireframe reconstruction method PC2WF [PC2WF]. Specifically, line3Dpp [Line3Dpp] outputs an abstracted line cloud from a dense line cloud based on heuristics for line clustering. Line2Surface [surfaceLine] is an optimization-based method that extracts planes from a line cloud via RANSAC to form a building mesh. PolyFit [polyfit] is the state-of-the-art optimization-based method for building mesh reconstruction from a potentially noisy point cloud. PC2WF [PC2WF] is a novel learning-based method to reconstruct a 3D wireframe from a point cloud, which achieves plausible results on man-made objects such as mechanical objects and furniture. For evaluation, we follow PC2WF[PC2WF] to measure the precision and recall on both predicted junctions and wireframes, and the Wireframe Edit Distance(WED): (1) $v{\text{AP}}_{\eta }$ and $v{\text{Recall}}_{\eta }$ show the precision/recall on the predicted junctions. (2) $s{\text{AP}}_{\eta }$ and $s{\text{Recall}}_{\eta }$ report the structural quality of the predicted wireframes. Specifically, it checks if a predicted edge is a true positive or if a ground-truth edge is retrieved according to the distances between the edge endpoints. (3) WED reports the number of operations and the editing distances of adding/removing predicted junctions/edges that are needed to transform the graph structure of the predicted wireframe into the ground-truth wireframe.

We compare to baseline methods on 757 test buildings. The line clouds (for line3Dpp, line2Surf, and our method) and the point clouds (for PolyFit and PC2WF) are extracted using the same camera parameters. Note that we use a commercial software to extract high-quality point clouds (See Fig. 1 (k) and Fig. 3 (a) for some examples). Moreover, to make a fair comparison to line2Surface [surfaceLine] and PolyFit [polyfit], we post-process the output meshes into wireframes by merging co-planar faces and parallel adjacent edges, removing interior edges and isolated vertices, etc. For PC2WF we use the provided NMS for post-processing. We report the evaluations on the results after post-processing (Tab. 2).

Tab. 1(a) shows a fair comparison to line2Surf and PolyFit, where all the output meshes are post-processed into cleaner wireframes (see Fig. 3 for some examples of the post-processed results; see Fig. 1 (B2,B3) and supplementary materials for examples of direct outputs from different methods). we report the number of vertex/edges, and the precision/recall of the predicted junctions/wireframes on the results after post-processing. We do not compare to Line3Dpp in this case since it outputs a nonstructural line cloud. The results show that our method outperforms all the baselines on building wireframe reconstruction. Tab. 1(b) shows the wireframe edit distance for different methods, where our method achieves the least number of editing operations and the smallest editing distances.

Fig. 3 shows a qualitative comparison of different baselines. We can see that line3Dpp can indeed provide more abstracted line clouds, but they are still far from clean wireframe models. Line2Surf can robustly recover planes from the line cloud (from Line3Dpp), but it is not robust to the noise. PC2WF is trained on point clouds from mechanical objects and furniture, which are likely to have a large domain gap to the rooftop structures. Therefore, the wireframes constructed by PC2WF can only recover the walls in building point clouds. Moreover, we also observe that the point clouds stemming from our scenes are more noisy (though they are accurate enough) than the point clouds that PC2WF is trained on. This can also lead to the less satisfactory results that PC2WF obtains. PolyFit is a powerful method for building reconstruction that is robust to noisy point clouds. However, PolyFit can be computationally costly when the input point cloud is too dense since the algorithm involves integer linear programming. As a comparison, our method can achieve visually comparable and quantitatively better results in a much more efficient way. For example, on average it takes our method 0.9s to infer a building wireframe while it takes PolyFit 45.6s to optimize a building mesh (see Tab. 1). We show more results of our reconstructed wireframes in the supplementary materials.