Self-Supervised Point Set Local Descriptors for Point Cloud Registration

Abstract

Descriptors play an important role in point cloud registration. The current state-of-the-art resorts to the high regression capability of deep learning. However, recent deep learning-based descriptors require different levels of annotation and selection of patches, which make the model hard to migrate to new scenarios. In this work, we learn local registration descriptors for point clouds in a self-supervised manner. In each iteration of the training, the input of the network is merely one unlabeled point cloud. Thus, the whole training requires no manual annotation and manual selection of patches. In addition, we propose to involve keypoint sampling into the pipeline, which further improves the performance of our model. Our experiments demonstrate the capability of our self-supervised local descriptor to achieve even better performance than the supervised model, while being easier to train and requiring no data labeling.

1. Introduction

Point cloud registration (PCR) is an essential task in various applications, including 3D reconstruction and simultaneous localization and mapping (SLAM). Usually, the accuracy of the calculated transformation will dominate the performance of higher level tasks. Thus, researchers either make back-end optimization on the high level task, such as SLAM [1,2,3], or improve on the PCR side.

Various techniques have been invented for the point set registration problem. As discussed in [4], it is extremely hard to find the optimal transformation $\mathbf{T}$ and correspondence matrix $\mathbf{P}$ simultaneously. The problem is addressed in [4] by alternating the optimization of $\mathbf{T}$ and $\mathbf{P}$. In recent decades, a multitude of algorithms have been proposed on 3D registration. They are divided into rigid and non-rigid algorithms [5] and work either iteratively to solve for the transformation matrix with repeatedly matched points [6,7,8,9,10] or treat the problem as an optimization program that omits the necessity of computing correspondences [11,12,13].

For large datasets, global optimization is inefficient. Iterative methods, such as the Iterative Closest Point algorithm (ICP) [7], are more practical. To find the correspondences, using distance among descriptors rather than Euclidean distance between points promises improvements in point cloud registration, especially when no good initial guess is available. Then, handcrafted descriptors [10,14,15,16,17,18,19,20] and learned descriptors [21,22,23,24,25,26,27] have been proposed during the last decades.

Although the learned point descriptors score better, their supervising usually requires extra labor to label the data. Those algorithms either get the correspondences from the matched point clouds [21,22,23,25], which is costly, or they are labeling the inter-point cloud relation [26,27], which is inefficient to train. Moreover, the existing models either train on patches, which is not globally learned for the scene [21,22,24], or learn the scene as the training loss works globally for the whole point cloud but comes with a triplet siamese loss that is not directly related to the true transformation [26].

In this paper, we propose to learn a point cloud local descriptor for registration without any annotation and selection of patches. The input of the network is a raw point cloud for each iteration of training. In addition, our loss function is directly related to the solved true transformation of registration. To realize the self-supervision, we propose the Full Connection Form Solution (CF) to solve the PCR problem non-iteratively in one-step without correspondences. Then, it serves as a layer of a neural network in the end of the descriptor, the gradients are propagated back to the descriptor learner. Moreover, in our model, we use a keypoint detector to sample points in the layer of sampling and grouping [28] to avoid learning on non-interested points, which further improves the performance.

To summarize, the major contributions of this paper are:

• We propose a self-supervised method to learn point cloud descriptors requiring no manual annotation and selection during training.
• We propose a keypoint sampling manner during training, which can focus on interesting points and further boost the performance.
• Experiments show that our self-supervised learned local descriptor has better performance than the supervised 3DFeatNet.

Experiments on various datasets, i.e., on the Oxford [29] and KITTI [30] datasets, demonstrate the performance of our descriptor.

2. Related Work

This section first reviews the technical advances in point cloud registration which are related to our registration layer. Then, it describes handcrafted registration descriptors and learned models.

2.1. Registration Model

The Iterative Closest Point (ICP) algorithm is the most famous registration method. It has been widely applied to various representations of 3D shapes [7] and is able to align a set of range images into a 3D model [8]. The generalized-ICP [9] even puts point-to-point ICP and point-to-plane ICP into one probabilistic framework. ICP consists of two steps, correspondence search and solving the transformation.

However, in ICP and related methods, the correspondences have to be recomputed each iteration. To avoid this, the kernel correlation (KC) method [11] uses an objective function that fully connects the point clouds. In each term of the summation, a robust function, the Gaussian distance, has been utilized. Similar to Maximum Mean Discrepancy (MMD), KC evaluates the distance between two distributions. Thus, it shows better sensitivity to noise and is more robust than ICP-like methods. Some recent publications do not rely on correspondences. Myronenko and Song [12] represented point clouds with Gaussian mixture models and solve the transformation by aligning the model centroids. Zheng et al. [13] built a continuous distance field for a fixed model and aligned the other point set model to minimize the energy iteratively. Yang et al. [31] reformulated the registration as a truncated least squares estimation (TEASER++), which is thus robust to many wrong correspondences. Resorting to frequency domain, Huang et al. [32] decomposed the registration problem of seven DoFs into multiple subproblems, which they solved with a closed-form solution.

Those methods either require correspondences, needed in frequency domain, or are solved iteratively, which cannot be applied as a differentiable layer in deep neural networks to solve the transformation without pre-knowing the match. Thus, we propose a registration layer to fill in this requirement.

2.2. Descriptors

Point Feature Histograms (PFH) are known as the most typical local 3D descriptors. They encode the geometrical properties of the neighborhood with a multi-dimensional histogram [14]. For real time application, Fast Point Feature Histograms (FPFH) break the full interconnection of neighbors in PFH. Thus, they achieve a linear time complexity and gradually have become the most commonly used handcrafted 3D descriptor [10]. Apart from the descriptors from point set geometry, spin images (SI) [15] and unique shape context (USC) [16] split the spatial space into bins and count the number of points in each as a histogram descriptor. In addition, the authors of [17,18] transformed local scans into range images to extract features. Flint et al. [19] proposed to extend the 2D-SIFT onto 3D images. Wu et al. [20] introduced a SIFT-like descriptor on projected 3D patches.

However, the correspondence from features requires a good distinctiveness of the descriptor, but the performance of descriptors usually varies on different point sets. Therefore, data-driven descriptors come into the view. 3DMatch proposes to learn a volumetric patch descriptor from correspondence labels [21]. Based on PointNet [33], PPFNet introduces a local descriptor that is highly aware of global context [23]. It learns from the truth correspondence matrix. With a voxelized smoothed density value representation, 3DSmoothNet also trains the network with a triplet of anchor and positive and negative samples [22]. Without using correspondence labels, PPF-FoldNet uses an encoder–decoder network to reconstruct the local patch fed in [24]. D3Feat proposes a joint learning of keypoint detector and descriptor [25]. The D3Feat provides descriptors and keypoint scores globally for all points, which introduces extra cost during inference. Thus, this method is also unable to provide descriptors solely for interested local patch.

As the training loss merely works on pairs of patches, the point-wise supervised models are not learning globally for the entire scene within the dataset. We classify the feature learning models into two groups, point-wise and point cloud-wise supervised models, on whether they learn directly from the relation between point clouds. For point cloud-wise supervised models, the training loss works globally for an entire point cloud, which is more related to the registration application. This intuition directs us to learn our model with only raw point clouds.

Weakly supervised on the positive/negative relation between point cloud frames, 3DFeatNet learns descriptors without explicitly specifying the correspondences [26]. As a by-product of its attention-aware loss function, keypoints are extracted by applying non-maximum suppression on the all points attentions. To tackle the speed issue, RSKDD proposes to use random sampling to replace the Farthest Point Sampling (FPS) of 3DFeatNet [27]. In addition, it embeds chamfer loss and point-to-point loss from the keypoint detection model USIP [34] to co-learn the keypoints and detectors. Since its learned descriptor is not for the cluster center but for shifted point instead, the detector and descriptor modules are not able to be decoupled. Therefore, 3DFeatNet provides a good basis to feed in whole point cloud as we demand. In addition, our model does not require any annotation and the loss function is directly on the solved transformation.

3. Method

In this section, the registration layer and keypoint sampling are introduced. Then, we demonstrate the whole training pipeline to learn the descriptor model.

3.1. The Registration Layer

We intend to use both full connection and the least squares form in this module. However, just replacing the kernel of KC with the quadratic distance will not work due to the distant pairs that would dominate the loss. As discussed in [11], the gradient of the quadratic function is very sensitive to outliers, so a more robust function, the Gaussian kernel, has been utilized. However, with Gaussian kernel, a solution in one step is impossible.

Thus, instead, our formula is a summation of weighted square distances for each fully connected point pair, which has a closed form solution for registration. Assume we have two point clouds $\mathbf{P}$ and $\mathbf{Q}$ with ${\mathbf{p}}_i{\in \mathbf{P}|}_{i\in \{1,\cdots N\}}$ and ${\mathbf{q}}_j{\in \mathbf{Q}|}_{j\in \{1,\cdots M\}}$. ${\mathbf{p}}_i,{\mathbf{q}}_j\in {\mathbb{R}}^3$. N and M are the number of points in $\mathbf{P}$ and $\mathbf{Q}$, respectively. Then, the optimization task is

$$\begin{array}{c}\hfill \underset{\mathbf{R},\mathbf{t}}{min}\sum _{i=1}^N\sum _{j=1}^M{w}_{i,j}\left|\right|\mathbf{R}{\mathbf{p}}_i+\mathbf{t}-{\mathbf{q}}_j{\left|\right|}^2\end{array}$$

(1)

where $\mathbf{R}$, $\mathbf{t}$ are the rotation matrix and translation vector to transform $\mathbf{P}$ into the coordinate system of $\mathbf{Q}$. The weight $w_{i,j}$ in range $(0,1]$ will be assigned for each term.

The other problem of Gaussian kernel distances in the KC method is that $\sigma$ in the Gaussian kernel has to be properly set according to the scale of different data sources. We use the square distance, as it is invariant to scale [35].

For the weighted function (1), there is a full connection with quadratic distance between every point $\mathbf{p}\in \mathbf{P}$ and $\mathbf{q}\in \mathbf{Q}$. Then, Equation (1) is reformulated with full connection as correspondences. The new point sets $(\mathcal{X},\mathcal{Y})$ are of size $N\times M$ and each pair is a connection. Let $\mathcal{X}=\{{\mathbf{p}}_1^{\prime },\cdots {\mathbf{p}}_{NM}^{\prime }\}$, $\mathcal{Y}=\{{\mathbf{q}}_1^{\prime },\cdots {\mathbf{q}}_{NM}^{\prime }\}$; the problem is formulated as

$$\begin{array}{c}\hfill (\mathbf{R},\mathbf{t})=\underset{\mathbf{R}\in SO\left(d\right),\mathbf{t}\in {\mathbb{R}}^d}{argmin}\sum _{i=1}^{NM}{w}_i\left|\right|(\mathbf{R}{\mathbf{p}}_i^{\prime }+\mathbf{t})-{\mathbf{q}}_i^{\prime }{\left|\right|}^2\end{array}$$

(2)

with known weights $w_i>0$.

The optimal solution is obtained with any algorithm that computes the transformation. We choose the SVD [36], as also detailed by Sorkine [37]. To make the paper self-contained, we briefly discuss this in Appendix A. Following Sorkine [37], we obtain a closed form solution for above formula by using weighted SVD. However, to have the desired suppression effect of pairs, weights cannot be arbitrarily chosen. To determine the weights, we use $f_{\mathcal{X}}\left(\mathbf{x}\right)$ to denote a function that extracts a feature descriptor of the point $\mathbf{x}$ from the point cloud $\mathcal{X}$. Then, the similarity is obtained as

$$\begin{array}{c}\hfill w_i=e^{-\frac{1}{\beta }\left|\right|f_{\mathcal{X}}\left({\mathbf{p}}_i^{\prime }\right)-f_{\mathcal{Y}}\left({\mathbf{q}}_i^{\prime }\right){\left|\right|}^2}.\end{array}$$

(3)

The lower is the similarity, the lower is the weight of the pairs. Thus, the effect of the term on the objective function will be less. In this way, a pair of points with low similarity contributes only a little, as they have a large feature descriptor distance. The constant $\beta$ in Equation (3) scales the feature distance. It depends on the selected feature descriptor.

More details and testing about this CF registration is provided in the Appendix A and Appendix B.

3.2. Keypoint Sampling

To learn with a whole point cloud as input, subsampling is a standard operation for PointNet-like model. 3DFeatNet uses FPS to sample points that are evenly distributed on the scene. RSKDD-Net uses random sampling to speed up on a large-scale dataset.

However, both sampling methods may result in the selection of non-interesting points, e.g., points that are not distinctive and do not contribute to the registration success, which requires to devote an extra pattern of features to those ordinary points. In the matching step, only interesting points are involved. It means that they waste both training power and feature space for non-interesting points.

Thus, in this work, we propose to use keypoint detectors in the sampling and grouping layer. Since the descriptors are learned for a specific detector, during inference, with the same detector, our descriptor scores better compared to the version with non-interesting points included.

We use one handcrafted ISS keypointer and learned 3DFeatNet keypoints (3DF kpt) because ISS are widely used handcrafted keypoints and 3DF kpt specially distributes points on the wall in Figure 1.

3.3. Network Architecture

We demonstrate the pipeline of the training process in Figure 2. The DESC module in between is the f we want to extract.

The whole training process consists of four parts. In the first stage, with a point cloud PC1 as input, we apply a random transformation to generate PC2. For both PC1 and PC2, we sample k points from a specific keypoint detector as centers. Then, neighbors are grouped around each center to obtain clusters. Then, those clustered are fed into the descriptor network f. Each cluster is processed separately and outputs a descriptor vector for the cluster center. Next, in the registration layer, CF (Section 3.1) solves Equation (2) for the transformation of sampled points with sampled centers and their descriptors from the two point clouds using the distance between the descriptors as weights according to Equation (3). The Rotation Matrix Distance Module computes the error between the solved $\mathbf{R}$ and ${\mathbf{R}}_{\mathrm{gt}}$ considering the distance between the descriptors as weights, which is the loss function for our model.

Given the ground truth transformation ${\mathbf{R}}_{\mathrm{gt}}$, ${\mathbf{t}}_{\mathrm{gt}}$, the loss function is the deviation from the identity matrix [38] as follows

$$loss=\left|\right|\mathbf{I}-\mathbf{R}{\mathbf{R}}_{\mathrm{gt}}^T{\left|\right|}_F.$$

(4)

When training the network, we only supervise the rotation because also involving the translation as a loss would further introduce additional hyperparameters to tune the balance between the effect from rotation and translation. Furthermore, the rotation is more important when performing the registration task.

With the above four parts of network components, the system merely requires to feed in one raw point cloud to learn for each iteration. Since the whole pipeline is differentiable, the parameters in the descriptor network are updated with gradient back-propagation. Given a random rotation, we minimize its distance to the solved rotation by optimizing f.

We call our model a self-supervised learning model because we generate labels (${\mathbf{R}}_{\mathrm{random}}$) from nothing and train the unlabeled data in a supervised way. The model is learned from a raw point cloud itself.

4. Experiment

4.1. Datasets

The Oxford RobotCar dataset [29] was used for network training and testing. Additionally, the KITTI dataset [30] was also used for testing the model.

4.1.1. Oxford RobotCar Dataset

The Oxford dataset contains repeated traverses through the Oxford city center from May 2014 to December 2015 that were collected with the Oxford RobotCar platform. We used the pre-processed data from [26], which have 35 trajectories for training and another 5 trajectories for testing. The points were scanned from 2D LIDARs and are accumulated into 3D point clouds, using the GPS/INS poses. Those poses were refined with ICP. The training point clouds were then downsampled to about 50,000 ± 20,000 points and the test point clouds to exactly 16,384 points. In this way, 21,875 training and 828 testing point cloud sets were obtained.

4.1.2. KITTI Dataset

Additionally, we tested our model on the 11 training sequences from the KITTI dataset [30] and processed them in the above-mentioned manner. The parts of the KITTI dataset used in the experiments include Velodyne laser point clouds, ground truth poses, and calibration files. The point clouds were also downsampled with a grid size of 0.2 m. We obtained 2369 point clouds in the end.

4.2. Setting

Our implementation makes use of the open source release (https://github.com/yewzijian/3DFeatNet) of 3DFeatNet [26]. In our pipeline, the descriptor directly uses the descriptor body of 3DFeatNet. Since this descriptor only considers a z-axis rotation, our provided $R_{\mathrm{random}}$ is generated by rotating around z-axis with $\varphi \sim \mathcal{N}(0,{\sigma }_r^2)$. We used ${\sigma }_r=0.6$. In addition, we applied a 3D jitter with $\Delta p\sim \mathcal{N}(\mathbf{0},{\sigma }_p\mathbf{I})$ (${\sigma }_p=0.01$) for each point in PC1 and PC2.

During the training of our model, we set batch size 6, Adam optimizer, and 32-dimensional descriptor. The training point clouds were randomly subsampled to 4096 points before feeding into the pipeline. We used the ISS and 3DFeatNet detectors (3DF kpt) to provide the keypoints as cluster centers to train. The setting of the 3DF kpt, e.g., ${\beta }_{\mathrm{attention}}$ and $r_{\mathrm{nms}}$, followed Yew and Lee [26]. We chose 256 keypoints from the point cloud to align the batch. We also used FPS to sample points as comparison. The FPS samples 512 points, which is the same as 3DFeatNet. In the cluster, each point is of dimension d, which can be 3 (xyz), 6 (xyzrgb), etc. We used $d=3$, thus we only used the xyz location of the point.

3DFeatNet states that it is hard to train. It takes 2 epochs to pretrain 3DFeatNet descriptor and the whole model can be trained in 70 epochs with $lr={10}^{-5}$. In contrast, our network is easy to train: without any pre-training, our model is randomly initialized and saved at 10 or 20 epochs training with a learning rate $lr={10}^{-3}$.

We used a PCL implementation of ISS to provide the ISS kpt and the released Tensorflow [39] checkpoint to achieve the network weight of 3DFeatNet to provide keypoints and run evaluation. We compared our method with handcrafted descriptors FPFH, SI, USC, and CGF and learned descriptors 3DMatch and 3DFeatNet.

4.3. Precision Test

Using exhaustive search as in [26], this test searched for the nearest descriptor neighbor in the paired models for each keypoint. Then, the Euclidean distance between the neighbor location and ground truth location as computed. We show the plot in Figure 3. The x-axis is a threshold to consider a pair as correct and the y-axis is the correct proportion.

For both 3DfeatNet descriptor and our descriptor, the test with 3DF kpt works better than ISS kpt. Without using the keypoint sampling (with FPS instead), our proposed unsupervised model achieves a similar result to 3DFeatNet descriptor on 3DF kpt and a better result on ISS kpt. We used the $x=1 \mathrm{m}$ line as a cut. Both 3DFeatNet descriptor and our descriptor achieve around $15\%$ precision, which is close to the best score in the record of Yew and Lee [26].

While using the keypoint sampling, we learned our ISS descriptor and our 3DF descriptor. ISS kpt + our ISS descriptor scores similar to our descriptor that used FPS. Both of our descriptors are better than 3DFeatNet descriptor on ISS keypoints. However, using ISS keypoint sampling during training does not improve our learned descriptor in the precision test. As shown in Figure 1, the ISS keypoints are evenly distributed in Oxford point cloud, which may introduce similar points as FPS. On the $x=1 \mathrm{m}$ line, with 3DF kpt, our 3DF descriptor learned the pattern and scores best. It is around 2% higher than the supervised descriptor.

Overall, our proposed unsupervised method scores better than the 3DFeatNet from which we borrow its descriptor part of model.

4.4. Geometric Registration

With ISS keypoints and 3DFeatNet keypoints, we evaluated the descriptors on the geometric registration. The registration uses nearest neighbor matches RANSAC to estimate the transformation. RANSAC iterations were limited to 10,000 and adjusted with $99\%$ confidence. The Relative Rotation Error (RRE) and Related Translation Error (RTE), with respect to the ground truth, were computed to evaluate the accuracy of the registration. A success was decided when RTE < 2 m and RRE < 5 ${}^{\circ }$. The speed of converging was reflected by the average number of iterations. Since we used the same datasets (Oxford and KITTI) as 3DFeatNet experiment [26], we compared to the results from their table.

The evaluation on the Oxford data is demonstrated in

Table 1. Registration error on the Oxford dataset.

	RTE (m)	RRE (${}^{\circ }$)	Success Rate	Avg #Iter
ISS + FPFH	0.396	1.60	92.32%	7171
ISS + SI	0.415	1.61	87.45%	9888
ISS + USC	0.324	1.22	94.02%	7084
ISS + CGF	0.431	1.62	87.36%	9628
ISS + 3DMatch	0.494	1.78	69.06%	9131
ISS + PN++	0.511	1.88	48.86%	9904
ISS + 3DFeatNet desc	0.314	1.08	97.66%	7127
3DFeatNet kpt + 3DFeatNet desc	0.300	1.07	98.10%	2940
ISS + 3DFeatNet desc	0.314	1.08	97.66%	7126
ISS + our desc	0.311	1.01	98.10%	5648
ISS + our ISS desc	0.311	1.00	98.23%	5545
3DF kpt + 3DFeatNet desc	0.304	1.08	97.66%	3294
3DF kpt + our desc	0.310	1.08	97.05%	3650
3DF kpt + our 3DF desc	0.298	1.02	97.90%	2703

. The first eight rows are taken from [26] and the last six rows are from our own experiments.

We observe that, firstly, except for PN++, the handcrafted descriptors cannot exceed the learned descriptors. Secondly, our unsupervised learned descriptor achieves the best result on RRE and the success rate with ISS and best result on RTE and average iteration with 3DF kpt. Thirdly, training merely on interested points, our keypoint sampling indeed improves the performance.

An example of a registration is shown in Figure 4. We observe that our 3DF descriptor has more inlier correspondences compared to 3DFeatNet descriptor by using 3DF keypoints, hich is revealed by the denser connection of the red lines.

Then, the model was transferred to another outdoor dataset, the KITTI dataset. The registration results are shown in

Table 2. Registration Error on the KITTI Dataset.

	RTE (m)	RRE (${}^{\circ }$)	Success Rate	Avg #Iter
ISS + FPFH	0.325	1.08	58.59%	7462
ISS + SI	0.358	1.17	55.92%	9219
ISS + USC	0.262	0.83	78.24%	7873
ISS + CGF	0.233	0.69	87.81%	7442
ISS + 3DMatch	0.283	0.79	89.12%	7292
3DF kpt + 3DFeatNet desc	0.258	0.57	95.97%	3798
ISS + 3DFeatNet desc	0.246	0.627	93.50%	8311
3DF kpt + 3DFeatNet desc	0.264	0.599	95.58%	4394
ISS + our desc	0.215	0.510	93.50%	5960
ISS + our ISS desc	0.215	0.459	93.85%	4356
3DF kpt + our desc	0.258	0.570	95.44%	3732
3DF kpt + our 3DF kpt	0.244	0.501	95.87%	2631

. The first six rows of the results are taken from [26] and the last six rows are from our experiments.

In the table, we observe that, firstly, our unsupervised model exceeds the supervised model. Secondly, ISS+our ISS descriptor achieves best accuracy. Its RRE even decreases about 0.041 compared to the FPS version (ISS + our descriptor). Thirdly, with 3DF kpt, our 3DF descriptor also achieves better results. A further example of a registration is shown in Figure 5. One can see that our ISS descriptor achieves much denser matching comparing to 3DFeatNet descriptor.

Overall, without using keypoint sampling, our unsupervised model achieves similar or even better performance than the supervised 3DFeatNet that uses the same descriptor body. In addition, with only interest points to train, our keypoint sampling indeed helps the model to learn more representative descriptors.

5. Conclusions

In this paper, we propose a novel self-supervised learning model to learn local descriptors for registration. We realize this goal by using a registration layer in the end. Thus, we use for supervision the randomly generated rotation of single point cloud input. In addition, we use keyopint sampling to make our model focus on interest points, in order to learn more expressive descriptors. In our pipeline, borrowing the same descriptor body as 3DFeatNet, our model is much easier to train, because this self-supervised method does not require any manual effort on annotation, and, without any pre-training, it converges with a higher learning rate, requiring far fewer iterations. Moreover, the experimental evaluation shows that our descriptor achieves much better performance on precision and geometric registration than the supervised 3DFeatNet descriptor.

As future work, we want to embedded our model into a SLAM framework to enable a no-annotation used data-driven descriptor for arbitrary scenes.