Improving High-Throughput Phenotyping Using Fusion of Close-Range Hyperspectral Camera and Low-Cost Depth Sensor

doi:10.3390/s18082711

. 2018 Aug 17;18(8):2711.

doi: 10.3390/s18082711.

Improving High-Throughput Phenotyping Using Fusion of Close-Range Hyperspectral Camera and Low-Cost Depth Sensor

Peikui Huang¹, Xiwen Luo², Jian Jin³, Liangju Wang⁴, Libo Zhang⁵, Jie Liu⁶, Zhigang Zhang⁷

Affiliations

¹ Key Laboratory of Key Technology on Agricultural Machine and Equipment, Ministry of Education, South China Agricultural University, Guangzhou 510642, China. peikuihuang@stu.scau.edu.cn.
² Key Laboratory of Key Technology on Agricultural Machine and Equipment, Ministry of Education, South China Agricultural University, Guangzhou 510642, China. xwluo@scau.edu.cn.
³ Department of Agricultural and Biological Engineering, Purdue University, 225 S. University St., West Lafayette, IN 47907, USA. jinjian@purdue.edu.
⁴ Department of Agricultural and Biological Engineering, Purdue University, 225 S. University St., West Lafayette, IN 47907, USA. wang3335@purdue.edu.
⁵ Department of Agricultural and Biological Engineering, Purdue University, 225 S. University St., West Lafayette, IN 47907, USA. zhan2693@purdue.edu.
⁶ College of Engineering, Huazhong Agricultural University, Wuhan 430070, China. liujie@mail.hzau.edu.cn.
⁷ Key Laboratory of Key Technology on Agricultural Machine and Equipment, Ministry of Education, South China Agricultural University, Guangzhou 510642, China. zzg208@scau.edu.cn.

PMID: 30126148
PMCID: PMC6111299
DOI: 10.3390/s18082711

Improving High-Throughput Phenotyping Using Fusion of Close-Range Hyperspectral Camera and Low-Cost Depth Sensor

Peikui Huang et al. Sensors (Basel). 2018.

. 2018 Aug 17;18(8):2711.

doi: 10.3390/s18082711.

Authors

Peikui Huang¹, Xiwen Luo², Jian Jin³, Liangju Wang⁴, Libo Zhang⁵, Jie Liu⁶, Zhigang Zhang⁷

Affiliations

¹ Key Laboratory of Key Technology on Agricultural Machine and Equipment, Ministry of Education, South China Agricultural University, Guangzhou 510642, China. peikuihuang@stu.scau.edu.cn.
² Key Laboratory of Key Technology on Agricultural Machine and Equipment, Ministry of Education, South China Agricultural University, Guangzhou 510642, China. xwluo@scau.edu.cn.
³ Department of Agricultural and Biological Engineering, Purdue University, 225 S. University St., West Lafayette, IN 47907, USA. jinjian@purdue.edu.
⁴ Department of Agricultural and Biological Engineering, Purdue University, 225 S. University St., West Lafayette, IN 47907, USA. wang3335@purdue.edu.
⁵ Department of Agricultural and Biological Engineering, Purdue University, 225 S. University St., West Lafayette, IN 47907, USA. zhan2693@purdue.edu.
⁶ College of Engineering, Huazhong Agricultural University, Wuhan 430070, China. liujie@mail.hzau.edu.cn.
⁷ Key Laboratory of Key Technology on Agricultural Machine and Equipment, Ministry of Education, South China Agricultural University, Guangzhou 510642, China. zzg208@scau.edu.cn.

PMID: 30126148
PMCID: PMC6111299
DOI: 10.3390/s18082711

Abstract

Hyperspectral sensors, especially the close-range hyperspectral camera, have been widely introduced to detect biological processes of plants in the high-throughput phenotyping platform, to support the identification of biotic and abiotic stress reactions at an early stage. However, the complex geometry of plants and their interaction with the illumination, severely affects the spectral information obtained. Furthermore, plant structure, leaf area, and leaf inclination distribution are critical indexes which have been widely used in multiple plant models. Therefore, the process of combination between hyperspectral images and 3D point clouds is a promising approach to solve these problems and improve the high-throughput phenotyping technique. We proposed a novel approach fusing a low-cost depth sensor and a close-range hyperspectral camera, which extended hyperspectral camera ability with 3D information as a potential tool for high-throughput phenotyping. An exemplary new calibration and analysis method was shown in soybean leaf experiments. The results showed that a 0.99 pixel resolution for the hyperspectral camera and a 3.3 millimeter accuracy for the depth sensor, could be achieved in a controlled environment using the method proposed in this paper. We also discussed the new capabilities gained using this new method, to quantify and model the effects of plant geometry and sensor configuration. The possibility of 3D reflectance models can be used to minimize the geometry-related effects in hyperspectral images, and to significantly improve high-throughput phenotyping. Overall results of this research, indicated that the proposed method provided more accurate spatial and spectral plant information, which helped to enhance the precision of biological processes in high-throughput phenotyping.

Keywords: close-range hyperspectral camera; fusion; high-throughput phenotyping; low-cost depth sensor; plant 3D model.

PubMed Disclaimer

Conflict of interest statement

The authors declare no conflict of interest.

Figures

**Figure 1**
(a) Middleton Spectral Vision MSV 101 Hyperspectral Camera. (b) Kinect V2 depth sensors.

**Figure 2**
(a) Reference gauge with a regular chess pattern with 1 cm edge length, red circles are the selected reference points numbered from 1 to 8; (b) Technical drawing with size specifications.

**Figure 3**
The structure of imaging station.

**Figure 4**
The imaging principle of the pushbroom camera model. The hyperspectral camera with focal length f and camera center C. The image coordinate is o-uv, while the system 3D coordinate is O-XYZ. The object point in X axis is projected to the image coordinate v, while the object point in Y axis is projected to the image coordinate u.

**Figure 5**
The fusion principle of pushbroom camera and depth sensor model. Starting from T, the line camera with focal length f and principal point pv is moved along V. The object point x is projected to the image point x’, with the coordinate (u; v).The coordinates of the base platform, hyperspectral camera, and Kinect V2 were O-X₁Y₁Z₁, H-X₂Y₂Z₂, and K-X₃Y₃Z₃, respectively, which were aligned with each other.

**Figure 6**
(a) The 3D coordinate of the selected reference points from Kinect V2 3D point clouds; (b) The hyperspectral image coordinate of the selected reference points achieved automatically using the corner detected method (red marks).

**Figure 7**
The sample soybean leaf attached to the base platform covered by a sheet of black cloth.

**Figure 8**
The segmentation 3-D point cloud of soybean leaf from Kinect V2.

**Figure 9**
Using the fusion model proposed in this paper to derivate Kinect V2 depth information to hyperspectral image coordinate information.

**Figure 10**
Soybean leaf hyperspectral 3D model, based on the fusion model proposed in this paper.

See this image and copyright information in PMC

Cited by

A hyperspectral plant health monitoring system for space crop production.
Qin J, Monje O, Nugent MR, Finn JR, O'Rourke AE, Wilson KD, Fritsche RF, Baek I, Chan DE, Kim MS. Qin J, et al. Front Plant Sci. 2023 Jul 4;14:1133505. doi: 10.3389/fpls.2023.1133505. eCollection 2023. Front Plant Sci. 2023. PMID: 37469773 Free PMC article.
Rotational hyperspectral scanner and related image reconstruction algorithm.
Luo L, Li S, Yao X, He S. Luo L, et al. Sci Rep. 2021 Feb 8;11(1):3296. doi: 10.1038/s41598-021-82819-8. Sci Rep. 2021. PMID: 33558585 Free PMC article.
End-to-End Fusion of Hyperspectral and Chlorophyll Fluorescence Imaging to Identify Rice Stresses.
Zhang C, Zhou L, Xiao Q, Bai X, Wu B, Wu N, Zhao Y, Wang J, Feng L. Zhang C, et al. Plant Phenomics. 2022 Aug 2;2022:9851096. doi: 10.34133/2022/9851096. eCollection 2022. Plant Phenomics. 2022. PMID: 36059603 Free PMC article.
The Development of Hyperspectral Distribution Maps to Predict the Content and Distribution of Nitrogen and Water in Wheat (Triticum aestivum).
Bruning B, Liu H, Brien C, Berger B, Lewis M, Garnett T. Bruning B, et al. Front Plant Sci. 2019 Oct 30;10:1380. doi: 10.3389/fpls.2019.01380. eCollection 2019. Front Plant Sci. 2019. PMID: 31737009 Free PMC article.
Estimation of Leaf Nitrogen Content in Wheat Based on Fusion of Spectral Features and Deep Features from Near Infrared Hyperspectral Imagery.
Yang B, Ma J, Yao X, Cao W, Zhu Y. Yang B, et al. Sensors (Basel). 2021 Jan 17;21(2):613. doi: 10.3390/s21020613. Sensors (Basel). 2021. PMID: 33477350 Free PMC article.

See all "Cited by" articles

References

1. Bai G., Ge Y., Hussain W., Baenziger P.S., Graef G. A multi-sensor system for high throughput field phenotyping in soybean and wheat breeding. Comput. Electron. Agric. 2016;128:181–192. doi: 10.1016/j.compag.2016.08.021. - DOI
1. Li L., Zhang Q., Huang D. A review of imaging techniques for plant phenotyping. Sensors. 2014;14:20078–20111. doi: 10.3390/s141120078. - DOI - PMC - PubMed
1. Thenkabail P.S., Lyon J.G., Huete A. Hyperspectral Remote Sensing of Vegetation. CRC Press; Boca Raton, FL, USA: 2012. Advances in Hyperspectral Remote Sensing of Vegetation and Agricultural Croplands; pp. 3–32.
1. Gonzalezdugo V., Hernandez P., Solis I., Zarcotejada P. Using high-resolution hyperspectral and thermal airborne imagery to assess physiological condition in the context of wheat phenotyping. Remote Sens. 2015;7:13586–13605. doi: 10.3390/rs71013586. - DOI
1. Thomas S., Kuska M.T., Bohnenkamp D., Brugger A., Alisaac E., Wahabzada M. Benefits of hyperspectral imaging for plant disease detection and plant protection: A technical perspective. J. Plant Dis. Prot. 2018;125:5–20. doi: 10.1007/s41348-017-0124-6. - DOI

LinkOut - more resources

Full Text Sources
Other Literature Sources
- scite Smart Citations

[1] Bai G., Ge Y., Hussain W., Baenziger P.S., Graef G. A multi-sensor system for high throughput field phenotyping in soybean and wheat breeding. Comput. Electron. Agric. 2016;128:181–192. doi: 10.1016/j.compag.2016.08.021. - DOI

[2] Bai G., Ge Y., Hussain W., Baenziger P.S., Graef G. A multi-sensor system for high throughput field phenotyping in soybean and wheat breeding. Comput. Electron. Agric. 2016;128:181–192. doi: 10.1016/j.compag.2016.08.021. - DOI

[3] Li L., Zhang Q., Huang D. A review of imaging techniques for plant phenotyping. Sensors. 2014;14:20078–20111. doi: 10.3390/s141120078. - DOI - PMC - PubMed

[4] Li L., Zhang Q., Huang D. A review of imaging techniques for plant phenotyping. Sensors. 2014;14:20078–20111. doi: 10.3390/s141120078. - DOI - PMC - PubMed

[5] Thenkabail P.S., Lyon J.G., Huete A. Hyperspectral Remote Sensing of Vegetation. CRC Press; Boca Raton, FL, USA: 2012. Advances in Hyperspectral Remote Sensing of Vegetation and Agricultural Croplands; pp. 3–32.

[6] Thenkabail P.S., Lyon J.G., Huete A. Hyperspectral Remote Sensing of Vegetation. CRC Press; Boca Raton, FL, USA: 2012. Advances in Hyperspectral Remote Sensing of Vegetation and Agricultural Croplands; pp. 3–32.

[7] Gonzalezdugo V., Hernandez P., Solis I., Zarcotejada P. Using high-resolution hyperspectral and thermal airborne imagery to assess physiological condition in the context of wheat phenotyping. Remote Sens. 2015;7:13586–13605. doi: 10.3390/rs71013586. - DOI

[8] Gonzalezdugo V., Hernandez P., Solis I., Zarcotejada P. Using high-resolution hyperspectral and thermal airborne imagery to assess physiological condition in the context of wheat phenotyping. Remote Sens. 2015;7:13586–13605. doi: 10.3390/rs71013586. - DOI

[9] Thomas S., Kuska M.T., Bohnenkamp D., Brugger A., Alisaac E., Wahabzada M. Benefits of hyperspectral imaging for plant disease detection and plant protection: A technical perspective. J. Plant Dis. Prot. 2018;125:5–20. doi: 10.1007/s41348-017-0124-6. - DOI

[10] Thomas S., Kuska M.T., Bohnenkamp D., Brugger A., Alisaac E., Wahabzada M. Benefits of hyperspectral imaging for plant disease detection and plant protection: A technical perspective. J. Plant Dis. Prot. 2018;125:5–20. doi: 10.1007/s41348-017-0124-6. - DOI

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Improving High-Throughput Phenotyping Using Fusion of Close-Range Hyperspectral Camera and Low-Cost Depth Sensor

Affiliations

Improving High-Throughput Phenotyping Using Fusion of Close-Range Hyperspectral Camera and Low-Cost Depth Sensor

Authors

Affiliations

Abstract

Conflict of interest statement

Figures

Similar articles

Cited by

References

LinkOut - more resources

Full Text Sources

Other Literature Sources