{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,11,19]],"date-time":"2024-11-19T17:59:05Z","timestamp":1732039145328},"reference-count":35,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2020,4,15]],"date-time":"2020-04-15T00:00:00Z","timestamp":1586908800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"The characterization of natural spaces by the precise observation of their material properties is highly demanded in remote sensing and computer vision. The production of novel sensors enables the collection of heterogeneous data to get a comprehensive knowledge of the living and non-living entities in the ecosystem. The high resolution of consumer-grade RGB cameras is frequently used for the geometric reconstruction of many types of environments. Nevertheless, the understanding of natural spaces is still challenging. The automatic segmentation of homogeneous materials in nature is a complex task because there are many overlapping structures and an indirect illumination, so the object recognition is difficult. In this paper, we propose a method based on fusing spatial and multispectral characteristics for the unsupervised classification of natural materials in a point cloud. A high-resolution camera and a multispectral sensor are mounted on a custom camera rig in order to simultaneously capture RGB and multispectral images. Our method is tested in a controlled scenario, where different natural objects coexist. Initially, the input RGB images are processed to generate a point cloud by applying the structure-from-motion (SfM) algorithm. Then, the multispectral images are mapped on the three-dimensional model to characterize the geometry with the reflectance captured from four narrow bands (green, red, red-edge and near-infrared). The reflectance, the visible colour and the spatial component are combined to extract key differences among all existing materials. For this purpose, a hierarchical cluster analysis is applied to pool the point cloud and identify the feature pattern for every material. As a result, the tree trunk, the leaves, different species of low plants, the ground and rocks can be clearly recognized in the scene. These results demonstrate the feasibility to perform a semantic segmentation by considering multispectral and spatial features with an unknown number of clusters to be detected on the point cloud. Moreover, our solution is compared to other method based on supervised learning in order to test the improvement of the proposed approach.<\/jats:p>","DOI":"10.3390\/s20082244","type":"journal-article","created":{"date-parts":[[2020,4,16]],"date-time":"2020-04-16T09:15:41Z","timestamp":1587028541000},"page":"2244","source":"Crossref","is-referenced-by-count":14,"title":["Semantic Segmentation of Natural Materials on a Point Cloud Using Spatial and Multispectral Features"],"prefix":"10.3390","volume":"20","author":[{"ORCID":"http:\/\/orcid.org\/0000-0002-8009-9033","authenticated-orcid":false,"given":"J. M.","family":"Jurado","sequence":"first","affiliation":[{"name":"Computer Graphics and Geomatics Group of Ja\u00e9n, University of Ja\u00e9n, 23071 Ja\u00e9n, Spain"}]},{"ORCID":"http:\/\/orcid.org\/0000-0002-0315-4094","authenticated-orcid":false,"given":"J. L.","family":"C\u00e1rdenas","sequence":"additional","affiliation":[{"name":"Computer Graphics and Geomatics Group of Ja\u00e9n, University of Ja\u00e9n, 23071 Ja\u00e9n, Spain"}]},{"ORCID":"http:\/\/orcid.org\/0000-0003-0958-990X","authenticated-orcid":false,"given":"C. J.","family":"Ogayar","sequence":"additional","affiliation":[{"name":"Computer Graphics and Geomatics Group of Ja\u00e9n, University of Ja\u00e9n, 23071 Ja\u00e9n, Spain"}]},{"ORCID":"http:\/\/orcid.org\/0000-0002-7320-7382","authenticated-orcid":false,"given":"L.","family":"Ortega","sequence":"additional","affiliation":[{"name":"Computer Graphics and Geomatics Group of Ja\u00e9n, University of Ja\u00e9n, 23071 Ja\u00e9n, Spain"}]},{"ORCID":"http:\/\/orcid.org\/0000-0001-8230-6529","authenticated-orcid":false,"given":"F. R.","family":"Feito","sequence":"additional","affiliation":[{"name":"Computer Graphics and Geomatics Group of Ja\u00e9n, University of Ja\u00e9n, 23071 Ja\u00e9n, Spain"}]}],"member":"1968","published-online":{"date-parts":[[2020,4,15]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Asrar, G.R. (2019). Advances in Quantitative Earth Remote Sensing: Past, Present and Future. Sensors, 19.","DOI":"10.3390\/s19245399"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"2249","DOI":"10.1109\/JSTARS.2016.2639043","article-title":"Radar Remote Sensing of Agricultural Canopies: A Review","volume":"10","author":"McNairn","year":"2017","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote. Sens."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Moreno, H., Valero, C., Bengochea-Guevara, J.M., Ribeiro, A., Garrido-Izard, M., and Andujar, D. (2020). On-Ground Vineyard Reconstruction Using a LiDAR-Based Automated System. Sensors, 20.","DOI":"10.3390\/s20041102"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"189","DOI":"10.1007\/s11263-007-0107-3","article-title":"Modeling the world from internet photo collections","volume":"80","author":"Snavely","year":"2008","journal-title":"Int. J. Comput. Vis."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1016\/j.isprsjprs.2015.10.006","article-title":"Sensor modelling and camera calibration for close-range photogrammetry","volume":"115","author":"Luhmann","year":"2016","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Mokro\u0161, M., Liang, X., Surov\u1ef3, P., Valent, P., \u010cer\u0148ava, J., Chud\u1ef3, F., Tun\u00e1k, D., Salo\u0148, \u0160., and Mergani\u010d, J. (2018). Evaluation of close-range photogrammetry image collection methods for estimating tree diameters. ISPRS Int. J. Geo Inf., 7.","DOI":"10.3390\/ijgi7030093"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Jurado, J., C\u00e1rdenas, J., Ogayar, C., Ortega, L., and Feito, F. (2019, January 25\u201327). Accurate plant modeling based on the real light incidence. Proceedings of the 14th International Conference on Computer Graphics Theory and Applications (GRAPP), Prague, Czech Republic.","DOI":"10.5220\/0007686803600366"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"168","DOI":"10.1016\/j.cageo.2011.09.012","article-title":"Multiview 3D reconstruction in geosciences","volume":"44","author":"Favalli","year":"2012","journal-title":"Comput. Geosci."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1016\/j.isprsjprs.2018.03.004","article-title":"Integration of aerial oblique imagery and terrestrial imagery for optimized 3D modeling in urban areas","volume":"139","author":"Wu","year":"2018","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1016\/j.oregeorev.2018.03.012","article-title":"Thermal infrared multispectral remote sensing of lithology and mineralogy based on spectral properties of materials","volume":"108","author":"Ninomiya","year":"2019","journal-title":"Ore Geol. Rev."},{"key":"ref_11","unstructured":"Rother, C., Kiefel, M., Zhang, L., Sch\u00f6lkopf, B., and Gehler, P.V. (2011). Recovering intrinsic images with a global sparsity prior on reflectance. Adv. Neural Inf. Process. Syst., 765\u2013773."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Fu, H., Zhou, T., and Sun, C. (2020). Object-Based Shadow Index via Illumination Intensity from High Resolution Satellite Images over Urban Areas. Sensors, 20.","DOI":"10.3390\/s20041077"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Bell, S., Upchurch, P., Snavely, N., and Bala, K. (2015, January 7\u201312). Material recognition in the wild with the materials in context database. Proceedings of the IEEE conference on computer vision and pattern recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298970"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"30073","DOI":"10.1364\/OE.25.030073","article-title":"Image processing pipeline for segmentation and material classification based on multispectral high dynamic range polarimetric images","volume":"25","author":"Valero","year":"2017","journal-title":"Opt. Express"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"259","DOI":"10.1016\/j.rse.2013.04.005","article-title":"High spatial resolution three-dimensional mapping of vegetation spectral dynamics using computer vision","volume":"136","author":"Dandois","year":"2013","journal-title":"Remote Sens. Environ."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Del Pozo, S., Lindenbergh, R., Rodr\u00edguez-Gonz\u00e1lvez, P., Blom, J.K., and Gonz\u00e1lez-Aguilera, D. (2015). Discrimination between sedimentary rocks from close-range visible and very-near-infrared images. PLoS ONE, 10.","DOI":"10.1371\/journal.pone.0132471"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1016\/j.compag.2018.05.002","article-title":"A multispectral machine vision system for invertebrate detection on green leaves","volume":"150","author":"Liu","year":"2018","journal-title":"Comput. Electron. Agric."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Feng, Q., Zhu, D., Yang, J., and Li, B. (2019). Multisource Hyperspectral and LiDAR Data Fusion for Urban Land-Use Mapping based on a Modified Two-Branch Convolutional Neural Network. ISPRS Int. J. Geo Inf., 8.","DOI":"10.3390\/ijgi8010028"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Audebert, N., Le Saux, B., and Lef\u00e8vrey, S. (2017, January 6\u20138). Fusion of heterogeneous data in convolutional networks for urban semantic labeling. Proceedings of the 2017 Joint Urban Remote Sensing Event (JURSE), Dubai, UAE.","DOI":"10.1109\/JURSE.2017.7924566"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1016\/j.isprsjprs.2017.05.002","article-title":"Simultaneous extraction of roads and buildings in remote sensing imagery with convolutional neural networks","volume":"130","author":"Alshehhi","year":"2017","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Wang, C., Ji, M., Wang, J., Wen, W., Li, T., and Sun, Y. (2019). An improved DBSCAN method for LiDAR data segmentation with automatic Eps estimation. Sensors, 19.","DOI":"10.3390\/s19010172"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1349","DOI":"10.1109\/TGRS.2015.2478379","article-title":"Unsupervised deep feature extraction for remote sensing image classification","volume":"54","author":"Romero","year":"2015","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Nalepa, J., Myller, M., Imai, Y., Honda, K., Takeda, T., and Antoniak, M. (2019). Unsupervised Segmentation of Hyperspectral Images Using 3-D Convolutional Autoencoders. IEEE Geosci. Remote Sens. Lett., 1\u20135.","DOI":"10.1109\/LGRS.2019.2960945"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Nevalainen, O., Honkavaara, E., Tuominen, S., Viljanen, N., Hakala, T., Yu, X., Hyypp\u00e4, J., Saari, H., P\u00f6l\u00f6nen, I., and Imai, N. (2017). Individual tree detection and classification with UAV-based photogrammetric point clouds and hyperspectral imaging. Remote Sens., 9.","DOI":"10.3390\/rs9030185"},{"key":"ref_25","first-page":"26","article-title":"Urban tree health assessment using airborne hyperspectral and LiDAR imagery","volume":"73","author":"Degerickx","year":"2018","journal-title":"Int. J. Appl. Earth Obs. Geoinf."},{"key":"ref_26","unstructured":"Wu, C. (2020, March 20). VisualSFM: A Visual Structure from Motion System. Available online: http:\/\/ccwu.me\/vsfm\/doc.html."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"204","DOI":"10.1111\/cgf.12009","article-title":"Simulation of Morphology Changes in Drying Leaves","volume":"32","author":"Jeong","year":"2013","journal-title":"Comput. Graph. Forum"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"366","DOI":"10.1016\/j.rse.2006.01.003","article-title":"Analysis of NDVI and scaled difference vegetation index retrievals of vegetation fraction","volume":"101","author":"Jiang","year":"2006","journal-title":"Remote Sens. Environ."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Jurado, J., Ortega, L., Cubillas, J., and Feito, F. (2020). Multispectral Mapping on 3D Models and Multi-Temporal Monitoring for Individual Characterization of Olive Trees. Remote Sens., 12.","DOI":"10.3390\/rs12071106"},{"key":"ref_30","first-page":"586","article-title":"Method for registration of 3-D shapes","volume":"Volume 1611","author":"Besl","year":"1992","journal-title":"Sensor Fusion IV: Control Paradigms and Data Structures"},{"key":"ref_31","first-page":"1","article-title":"Real-time kd-tree construction on graphics hardware","volume":"27","author":"Zhou","year":"2008","journal-title":"ACM Trans. Graph. (TOG)"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1007\/BF02616245","article-title":"Efficient algorithms for divisive hierarchical clustering with the diameter criterion","volume":"8","author":"Hansen","year":"1991","journal-title":"J. Classif."},{"key":"ref_33","unstructured":"H\u00e4rdle, W.K., and Simar, L. (2007). Applied Multivariate Statistical Analysis, Springer."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"881","DOI":"10.1109\/TPAMI.2002.1017616","article-title":"An efficient k-means clustering algorithm: Analysis and implementation","volume":"24","author":"Kanungo","year":"2002","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1016\/j.isprsjprs.2012.01.006","article-title":"3D terrestrial lidar data classification of complex natural scenes using a multi-scale dimensionality criterion: Applications in geomorphology","volume":"68","author":"Brodu","year":"2012","journal-title":"ISPRS J. Photogramm. Remote Sens."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/8\/2244\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,6,26]],"date-time":"2024-06-26T22:00:24Z","timestamp":1719439224000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/8\/2244"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,4,15]]},"references-count":35,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2020,4]]}},"alternative-id":["s20082244"],"URL":"https:\/\/doi.org\/10.3390\/s20082244","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,4,15]]}}}