Keywords

1 Introduction

In recent years, remote sensing with unmanned aerial vehicles (UAV) for precision agriculture [1, 2] has become a field of research in rapid progress, for different agricultural applications [3], using several types of data (visible, multi or hyper spectral) [4], and in several crop types notably in the viticulture [5].

Precision viticulture is an area of research that includes many applications, such as estimating growth [6], estimating evaporate-transpiration and harvest coefficients [7], vigor evaluation [8], water stress localization [9] or diseases detection [10,11,12,13,14,15,16,17,18,19].

Vine diseases detection (VDD) is a key issue for reducing the use of phytosanitary products and increasing the grapes production. So far, there is some researches on the different imaging systems for the VDD. Certain studies use images taken at the vine leaf level [10,11,12,13,14] which can be mounted in mobile robots. Other research is carried out on aerial images taken by drones at plot scale targeting the vine canopies [15,16,17,18,19]. The VDD at the canopy level requires the vine isolation from the background. In some research works, the isolation of vine is carried out by using vegetation index (NDVI, ExG, ExGR ...), and machine learning methods. However, these methods are not always effective, especially when the vine inter-rows are covered with green grass, which can be confused with the green color of the vine and leads to misclassification of the vine and the soil. To solve this problem, the authors in [20,21,22] have used 3-dimension (3D) information to separate the soil and the vine by depth information, using the digital surface model (DSM). The results show the importance of the 3D information. However, the combination of the deep learning approach and 3D information is still less explored.

This paper presents a method based on deep learning network segmentation, combined with 3D depth information for VDD in UAV images of vineyards partially or totally covered with green or yellow grass. This method reduces confusion between different classes (healthy vine, diseased vine, green grass, yellow grass,...) and only keep the detections on the vine vegetation.

This paper is organized as follow, materials and methods are described in Sect. 2, experiences and results are presented in Sect. 3, discussion and interpretation in Sect. 4, and the conclusion in Sect. 5.

2 Materials and Methods

This section details material, and method used for data acquisition, the proposed system, the design of the vine and non-vine mask (depth map - DM), construction of the rasters, deep learning segmentation method, and finally, correction of the segmentation output.

2.1 Materials and Acquisition

The UAV which acquires image data is a Quad-copter type drone. It has a flight autonomy of 20 min, and it embeds a GPS module and two image sensors of MAPIR Survey2 model. The first sensor operates in visible spectrum (RGB), and the second one in near infrared spectrum, which records 3-band images (Near Infrared (NIR), Red and Normalized Difference Vegetation Index (NDVI)). Both sensors have a high resolution of 16 mega pixels (4608 \(\times \) 3456).

The data acquisition method is realized by flying over the vineyard parcel at an altitude between 20 and 25 m and with an average speed of 10 km/h. The drone takes images of each area of the vine plot with a resolution of 1 cm\(^{2}\)/pixel and an overlap of 70% between each image taken. The acquired images are recorded with their GPS coordinates.

2.2 Method Overview

The system proposed in this study (Fig. 1) consists of three phases. The first one creates visible and infrared mosaic images for continuous view of the vineyard, and the DM (depth map). The second one segments the visible and infrared images by SegNet architecture, and merges the information of VDD. Finally, the last process corrects the result of the segmentation by using the DM.

Fig. 1.
figure 1

Overview of the proposed method for VDD.

Full size image

2.3 Depth Map

The design of the DM is carried out in two main stages, which are performed on images acquired by the drone in the two spectrum (visible and infrared). By using Agisoft Metashape software (version 1.5.5), the first step is to generate the DSM model, and the visible and infrared rasters (mosaic images), with the following steps; sparse point cloud (Tie points), dense point cloud (Dense cloud), digital surface model (DSM) and orthomosaic (Raster).

The second step is to extract the DM from the DSM model. It uses the same procedure in [21], which consists in three steps:

  • DSM filtering: the DSM is filtered by a low pass filter of size 20 \(\times \) 20, the filter has been chosen for smoothing the image and to keep only the digital terrain model (DTM).

  • Subtraction of DTM from DSM: this step eliminates variations in the terrain and keep only the height of the vineyards.

  • Contrast enhancement and thresholding: the result obtained by the subtraction has a weak dynamic of contrast. For this reason, a method for increasing the contrast based on the histogram was applied to improve the difference of the level between vine and soil. Then, an automatic thresholding (using Otsu’s Method) is applied to obtain a binary DM.

2.4 Segmentation

In our previous study [16], a VDD method were proposed using a SegNet architecture. This study gave a good results on two parcels containing a non-grassy soil (the soil of these plots was brown). However, false diseases detections were observed in the segmentation results. The aim of this study is to introduce the depth information to separate the vine vegetation and the soil (whatever the type of soil). Therefore, it filters out the soil and thus reduces the detection errors.

2.5 Correction and Fusion

The aim of the correction phase is to reduce the errors of the SegNet, using the fusion. Indeed, the segmentation result often presents confusion between the green grassy soil, and the vine vegetation, or, confusion between the discolored grassy soil and the diseased vine. The correction phase proposed in this study takes as input the result of the SegNet segmentation by fusion, and the binary DM (vine and non-vine). Each pixel of the two images is compared and corrected (if necessary) by the rules described in Table 1.

Table 1. Operation of the correction.
Full size table

3 Experimentation and Results

This section presents experimental procedures, quantitative and qualitative results. The experiments were carried out with Python 2.7 language, using the Tensorflow 1.8.0 libraries for the SegNet architecture, and GDAL 3.0.3 for reading and writing the rasters (whole view of the plot) and their GPS information. The operating system used is Linux Ubuntu 16.04 LTS (64 bits). The programs were executed on a machine with characteristics: Intel Xeon 3.60 GHz \(\times \) 8 processor, RAM of 32 GB, and a NVidia GTX 1080 Ti graphics card with an internal RAM of 11 GB.

3.1 Depth Map

To compute depth information, or the relief, we used depth from motion approach. The acquisition step is followed by the processing step, which consists in points matching between overlapped images. Matching points are represented by a sparse 3D point cloud, followed by a second processing to obtain a dense 3D point cloud. The DM is obtained by processing the DSM, which is created from the dense point cloud. The Fig. 2 represents an example of DM map, visible and infrared image.

Fig. 2.
figure 2

Example of DM result.

Full size image

3.2 Correction of SegNet Segmentation

SegNet segmentation is performed on a raster images with size of 12000 \(\times \) 20000 pixels. A non overlapping sliding window of 360 \(\times \) 480 pixels, is applied on the entire raster to segment each area of the parcel (visible and infrared spectra). It takes 45 min on average for each of them. Once the two rasters (visible and infrared) are segmented, they are merged using the segmentation fusion. Then, the DM is applied to the segmented image to isolate the background class. However, one can use background subtraction before the segmentation phase. But, we found that the SegNet is more precise when using soil classes. Table 2 shows the quantitative results of the SegNet segmentation by fusion, and its correction by the DM. Figures 3 and 4 are examples of qualitative results on healthy area with green grassy soil (Fig. 3), and another example of diseased area on discolored grassy soil (Fig. 4).

Table 2. Quantitative result on two temporal tests by measuring Recall (Rec.), Precision (Pre.), F1-Score/Dice (F1/D.) and Accuracy (Acc.) on the performance of SegNet by Fusion and corrected SegNet (values presented in percent).
Full size table
Fig. 3.
figure 3

Qualitative result of the healthy vine area using the depth map.

Full size image
Fig. 4.
figure 4

Qualitative result of the diseased vine area using the depth map.

Full size image

4 Discussion

This research work was set out with the aim of developing efficient methods for vine disease detection. Table 2 shows the quantitative results obtained for the SegNet segmentation experiments by fusion, and the corrected segmentation by DM. The results obtained are presented in terms of recall, precision, F1-score/Dice and accuracy, expressed in percentages. As shown in the accuracy column, the corrected method gives a better rate than the uncorrected method. This improvement is due to the correction of the soil areas and the vine vegetation. Also, the reduction of the over-detections of the disease areas, which can be observed on the individual results of each class.

Figures 3 and 4 represent respectively, the qualitative results, of the DM application on an area in good health and diseased. As can be seen in Fig. 3.c, the SegNet result by fusion gave several segmentation errors, in particular the vines detection and symptoms on the soil. These errors are mainly due to the presence of green grass mixed with a light brown color of the soil, which look like the color of vine disease. Figure 3.d shows a improvement of the segmentation result after correcting this result by the 3D information. Indeed, the correction brings a better distinction of the vine lines and reduces false detection of symptoms and vine vegetation on the grassy soil.

The second SegNet result on an area contaminated by Mildew disease (see Fig. 4.c) gave an over-detection of the symptomatic areas, which overflowed on the soil. This problem does not allow to evaluate the real diseased. Also, in some cases, it can cause confusion between the vine-rows that are contaminated. After the correction (Fig. 4.d), the result shows better interpretation and distinction of the vine-rows, and the detection of symptoms is observed only on the vine, and not on the soil.

5 Conclusion

This research work was set out with the aim of developing efficient methods for vine disease detection. We have developed a new method base on the deep learning segmentation approach and 3D depth map (DM). The method consists of three steps. The first one is mosaicking the visible and infrared pictures to obtain whole view of the vineyard, and their DM. The second step segments and merges visible and infrared rasters by using the SegNet architecture. Finally, the third step consists of correction of the SegNet result using the DM. This study showed that the proposed method reduces false detections of the vine vegetation, the vine symptoms, the soil, and therefore gives better precision and estimation on the disease map.