Abstract
LiDAR is widely used to capture accurate 3D outdoor scene structures. However, LiDAR produces many undesirable noise points in snowy weather, which hamper analyzing meaningful 3D scene structures. Semantic segmentation with snow labels would be a straightforward solution for removing them, but it requires laborious point-wise annotation. To address this problem, we propose a novel self-supervised learning framework for snow points removal in LiDAR point clouds. Our method exploits the structural characteristic of the noise points: low spatial correlation with their neighbors. Our method consists of two deep neural networks: Point Reconstruction Network (PR-Net) reconstructs each point from its neighbors; Reconstruction Difficulty Network (RD-Net) predicts point-wise difficulty of the reconstruction by PR-Net, which we call reconstruction difficulty. With simple post-processing, our method effectively detects snow points without any label. Our method achieves the state-of-the-art performance among label-free approaches and is comparable to the fully-supervised method. Moreover, we demonstrate that our method can be exploited as a pretext task to improve label-efficiency of supervised training of de-snowing.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Batson, J., Royer, L.: Noise2self: blind denoising by self-supervision. In: International Conference on Machine Learning, pp. 524–533 (2019)
Berthelot, D., Carlini, N., Goodfellow, I., Papernot, N., Oliver, A., Raffel, C.A.: MixMatch: a holistic approach to semi-supervised learning. In: Advances in Neural Information Processing Systems, vol. 32 (2019)
Bijelic, M., et al.: Seeing through fog without seeing fog: deep multimodal sensor fusion in unseen adverse weather. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 11682–11692 (2020)
Caesar, H., et al.: nuScenes: a multimodal dataset for autonomous driving. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 11621–11631 (2020)
Charron, N., Phillips, S., Waslander, S.L.: De-noising of LiDAR point clouds corrupted by snowfall. In: Conference on Computer and Robot Vision, pp. 254–261 (2018)
Chen, X., He, K.: Exploring simple siamese representation learning. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 15750–15758 (2021)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Doersch, C., Gupta, A., Efros, A.A.: Unsupervised visual representation learning by context prediction. In: IEEE International Conference on Computer Vision, pp. 1422–1430 (2015)
Fan, L., Xiong, X., Wang, F., Wang, N., Zhang, Z.: RangeDet: in defense of range view for LiDAR-based 3D object detection. In: IEEE International Conference on Computer Vision, pp. 2918–2927 (2021)
Gao, B., Pan, Y., Li, C., Geng, S., Zhao, H.: Are we hungry for 3D LiDAR data for semantic segmentation? A survey of datasets and methods. IEEE Trans. Intell. Transp. Syst. (2021)
Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? The KITTI vision benchmark suite. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3354–3361 (2012)
Gidaris, S., Singh, P., Komodakis, N.: Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv:1803.07728 (2018)
Grill, J.B., et al.: Bootstrap your own latent-a new approach to self-supervised learning. In: Advances in Neural Information Processing Systems, vol. 33, pp. 21271–21284 (2020)
Gruber, T., Bijelic, M., Heide, F., Ritter, W., Dietmayer, K.: Pixel-accurate depth evaluation in realistic driving scenarios. In: International Conference on 3D Vision, pp. 95–105. IEEE (2019)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Heinzler, R., Piewak, F., Schindler, P., Stork, W.: CNN-based LiDAR point cloud de-noising in adverse weather. IEEE Robot. Autom. Lett. 5(2), 2514–2521 (2020)
Hermosilla, P., Ritschel, T., Ropinski, T.: Total denoising: Unsupervised learning of 3D point cloud cleaning. In: IEEE International Conference on Computer Vision, pp. 52–60 (2019)
Iscen, A., Tolias, G., Avrithis, Y., Chum, O.: Label propagation for deep semi-supervised learning. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 5070–5079 (2019)
Jarrett, K., Kavukcuoglu, K., Ranzato, M., LeCun, Y.: What is the best multi-stage architecture for object recognition? In: IEEE International Conference on Computer Vision, pp. 2146–2153. IEEE (2009)
Ke, Z., Wang, D., Yan, Q., Ren, J., Lau, R.W.: Dual student: breaking the limits of the teacher in semi-supervised learning. In: IEEE International Conference on Computer Vision, pp. 6728–6736 (2019)
Kendall, A., Gal, Y.: What uncertainties do we need in Bayesian deep learning for computer vision? In: Advances in Neural Information Processing Systems, pp. 5580–5590 (2017)
Kilic, V., et al.: LiDAR light scattering augmentation (LISA): physics-based simulation of adverse weather conditions for 3D object detection. arXiv preprint arXiv:2107.07004 (2021)
Kim, K., Ye, J.C.: Noise2score: tweedie’s approach to self-supervised image denoising without clean images. In: Advances in Neural Information Processing Systems, vol. 34 (2021)
Klodt, M., Vedaldi, A.: Supervising the new with the old: learning SFM from SFM. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11214, pp. 713–728. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01249-6_43
Krull, A., Buchholz, T.O., Jug, F.: Noise2Void-learning denoising from single noisy images. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2129–2137 (2019)
Laine, S., Aila, T.: Temporal ensembling for semi-supervised learning. arXiv preprint arXiv:1610.02242 (2016)
Landrieu, L., Simonovsky, M.: Large-scale point cloud semantic segmentation with superpoint graphs. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 4558–4567 (2018)
Le, Q.V.: Building high-level features using large scale unsupervised learning. In: International Conference on Acoustics, Speech and Signal Processing, pp. 8595–8598. IEEE (2013)
Lee, D.H., et al.: Pseudo-label: the simple and efficient semi-supervised learning method for deep neural networks. In: Workshop on Challenges in Representation Learning, ICML, vol. 3, p. 896 (2013)
Lee, S., Prakash, S.P.S., Cogswell, M., Ranjan, V., Crandall, D., Batra, D.: Stochastic multiple choice learning for training diverse deep ensembles. In: Advances in Neural Information Processing Systems, pp. 2119–2127 (2016)
Lehtinen, J., et al.: Noise2noise: learning image restoration without clean data. In: International Conference on Machine Learning, pp. 2965–2974 (2018)
Luo, H., et al.: Patch-based semantic labeling of road scene using colorized mobile LiDAR point clouds. IEEE Trans. Intell. Transp. Syst. 17(5), 1286–1297 (2015)
Luo, H., et al.: Semantic labeling of mobile LiDAR point clouds via active learning and higher order MRF. IEEE Trans. Geosci. Remote Sens. 56(7), 3631–3644 (2018)
Luo, S., Hu, W.: Differentiable manifold reconstruction for point cloud denoising. In: ACM International Conference on Multimedia, pp. 1330–1338 (2020)
Luo, S., Hu, W.: Score-based point cloud denoising. In: IEEE International Conference on Computer Vision, pp. 4583–4592 (2021)
Luo, Y., Zhu, J., Li, M., Ren, Y., Zhang, B.: Smooth neighbors on teacher graphs for semi-supervised learning. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 8896–8905 (2018)
Meyer, G.P., Laddha, A., Kee, E., Vallespi-Gonzalez, C., Wellington, C.K.: LaserNet: an efficient probabilistic 3D object detector for autonomous driving. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 12677–12686 (2019)
Milioto, A., Vizzo, I., Behley, J., Stachniss, C.: RangeNet++: fast and accurate LiDAR semantic segmentation. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4213–4220. IEEE (2019)
Noroozi, M., Favaro, P.: Unsupervised learning of visual representations by solving Jigsaw puzzles. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9910, pp. 69–84. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46466-4_5
Park, J.I., Park, J., Kim, K.S.: Fast and accurate desnowing algorithm for LiDAR point clouds. IEEE Access 8, 160202–160212 (2020)
Piewak, F., et al.: Boosting LiDAR-based semantic labeling by cross-modal training data generation. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018. LNCS, vol. 11134, pp. 497–513. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11024-6_39
Pitropov, M., et al.: Canadian adverse driving conditions dataset. Int. J. Robot. Res. (2020)
Qi, C.R., Su, H., Mo, K., Guibas, L.J.: PointNet: deep learning on point sets for 3D classification and segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 652–660 (2017)
Qi, C.R., Yi, L., Su, H., Guibas, L.J.: PointNet++: deep hierarchical feature learning on point sets in a metric space. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
Quan, Y., Chen, M., Pang, T., Ji, H.: Self2self with dropout: learning self-supervised denoising from single image. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1890–1898 (2020)
Roriz, R., Campos, A., Pinto, S., Gomes, T.: DIOR: a hardware-assisted weather denoising solution for LiDAR point clouds. IEEE Sens. J. (2021)
Rusu, R.B., Cousins, S.: 3D is here: point cloud library (PCL). In: IEEE International Conference on Robotics and Automation, pp. 1–4 (2011)
Sajjadi, M., Javanmardi, M., Tasdizen, T.: Regularization with stochastic transformations and perturbations for deep semi-supervised learning. In: Advances in Neural Information Processing Systems, vol. 29 (2016)
Shi, S., Wang, X., Li, H.: PointRCNN: 3D object proposal generation and detection from point cloud. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–779 (2019)
Shim, I., et al.: Vision system and depth processing for DRC-HUBO+. In: IEEE International Conference on Robotics and Automation, pp. 2456–2463 (2016)
Sohn, K., et al.: FixMatch: simplifying semi-supervised learning with consistency and confidence. In: Advances in Neural Information Processing Systems, vol. 33, pp. 596–608 (2020)
Sun, P., et al.: Scalability in perception for autonomous driving: Waymo open dataset. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2446–2454 (2020)
Tarvainen, A., Valpola, H.: Mean teachers are better role models: weight-averaged consistency targets improve semi-supervised deep learning results. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
Van Engelen, J.E., Hoos, H.H.: A survey on semi-supervised learning. Mach. Learn. 109(2), 373–440 (2020)
Xie, Q., Luong, M.T., Hovy, E., Le, Q.V.: Self-training with noisy student improves ImageNet classification. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 10687–10698 (2020)
Xie, S., Gu, J., Guo, D., Qi, C.R., Guibas, L., Litany, O.: PointContrast: unsupervised pre-training for 3D point cloud understanding. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12348, pp. 574–591. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58580-8_34
Yang, G., Hu, P., Ramanan, D.: Inferring distributions over depth from a single image. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 6090–6096 (2019)
Zhou, H., Chen, K., Zhang, W., Fang, H., Zhou, W., Yu, N.: DUP-Net: denoiser and upsampler network for 3D adversarial point clouds defense. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1961–1970 (2019)
Zhou, Y., Tuzel, O.: VoxelNet: end-to-end learning for point cloud based 3D object detection. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 4490–4499 (2018)
Zoph, B., et al.: Rethinking pre-training and self-training. In: Advances in Neural Information Processing Systems, vol. 33, pp. 3833–3845 (2020)
Acknowledgement
This work was supported by the Agency for Defense Development (ADD) and by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. NRF-2022R1F1A1073505).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Bae, G., Kim, B., Ahn, S., Min, J., Shim, I. (2022). SLiDE: Self-supervised LiDAR De-snowing Through Reconstruction Difficulty. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13699. Springer, Cham. https://doi.org/10.1007/978-3-031-19842-7_17
Download citation
DOI: https://doi.org/10.1007/978-3-031-19842-7_17
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-19841-0
Online ISBN: 978-3-031-19842-7
eBook Packages: Computer ScienceComputer Science (R0)