Application of Image Processing in Fruit and Vegetable Analysis: A Review

2.1 Issues and Challenges

We survey the methods used in the recognition of fruits and vegetables and in the identification of fruit diseases and note the issues and challenges that each method tries to address. These issues and challenges may serve as the basis for evaluating the different methods. The input images may contain fruits or vegetables of more than one variety in an arbitrary position and in any number. Many kinds of fruits and vegetables exhibit significant variations in shape, texture and color depending on their ripeness. For example, oranges can be green, yellow, or patchy and brown. Because using just one image feature to separate fruits and vegetables into classes might not be sufficient, it is necessary to extract and combine those features that are useful for recognizing the produce. The produce might be inside a plastic bag, which can add hue shifts and specular reflections. Different classifiers may produce different results, so the selection of the type of classifier to use must also be addressed. In the literature, available classifiers work on two classes only, but in the produce classification problem we consider more than two classes, so it is a major issue to use a binary classifier in a multiclass scenario. Subtracting the background may be necessary to reduce scene complexities, such as variations in illumination, sensor capturing artifacts, background clutter, shading and shadows. Consequently, the result of the system heavily depends on the efficiency of the image segmentation method. The performance of the fruit disease recognition system also depends upon the defect segmentation, and thus precise defect segmentation is required. It might be interesting to consider the number of training examples because more training examples require more time to train the system. The system must perform better in situations where the system is trained with less training examples.

2.2 Fruit and Vegetable Recognition and Classification

A number of studies have been conducted on image categorization. Veggie-Vision [5] was an initial attempt to develop a produce recognition system for use in supermarkets. The system could analyze color, texture and density, and thus was able to obtain more information. Density was calculated by dividing weight with the area of the fruit. The reported accuracy was approximately 95% when color and texture features were combined, but the top four responses were used to achieve such result. Rocha et al. [59] developed an approach that could combine many features and classifiers. The authors approached the multi-class classification problem as a set of binary classification problem in such a way that one can assemble together diverse features and classifier approaches custom-tailored to parts of the problem. They achieved a classification accuracy of up to 99% for some fruits, but they fused three features, namely, border-interior classification (BIC), color coherence vector (CCV), and unser features, and identified top two responses to achieve them. Their method obtained poor results for some types of fruit and vegetable, such as Fuji Apple. Arivazhagan et al. [2] combined color and texture features to classify the produce. They used minimum-distance classifier and achieved 86% accuracy over their dataset with 15 different types of produce. Faria et al. [22] presented a framework for classifier fusion for the automatic recognition of produce in supermarkets. They combined low-cost classifiers trained for specific classes of interest to enhance the recognition rate. Chowdhury et al. [11] recognized 10 different vegetables using color histogram and statistical texture features. They obtained a classification accuracy of up to 96.55% using neural network as a classifier. Danti et al. [15] classified 10 types of leafy vegetables using the BPNN classifier with a success rate of 96.40%. To form the feature vector, they first cropped and resized the image and then extracted the mean and range of hue and saturation channel of HSV image. Suresha et al. [63] obtained 95% classification accuracy over a dataset with eight types of different vegetables using texture measures in RGB color space. They used watershed segmentation to extract the region of interest as a pre-processing and decision tree classifier for training and classification purposes.

Dubey [16] and Dubey and Jalal [17, 20] proposed a framework for recognizing and classifying images of 15 different types produce. The approach involves segmenting an image to extract the region of interest, and then calculating the features from that segmented region, which is further used in training and classification by a multi-class support vector machine. Moreover, they proposed an improved sum and difference histogram (ISADH) texture feature for this kind of problem. The ISADH outperformed the other image color and texture features. Arefi et al. [1] developed a segmentation algorithm to guide a robot arm in picking ripe tomatoes using image-processing technique. They prepared a machine vision system to acquire images from tomatoes. The algorithm works in two phases: (1) the background is subtracted from the RGB color space and then the ripe tomato is extracted using a combination of RGB, HSI, and YIQ color spaces; and (2) the ripe tomato is localized using morphological features of the image. They achieved an accuracy of up to 96.36% for 110 tomato images.

Fruit detection greatly affects the robot’s harvesting efficiency because it is an unstructured environment with changing lighting conditions. Bulanon et al. [10] enhanced the portion occupied by fruit in images using a red chromaticity coefficient and adopted a circle detection method for classifying individual fruits. To improve fruit visibility, they acquired multiple views from different viewing angles for a portion of a tree canopy. Fruit visibility improved from 50% to about 90% by acquiring multiple views. Date palm fruits are popular in the Middle East and have a growing international presence. Sorting of dates can be a tedious job and a key process in the date palm industry. Haidar et al. [27] presented a method for classifying dates automatically based on pattern recognition and computer vision. Using an extracted image of an appropriately crafted mixture of 15 different visual features, they tried multiple classification methods. The performance of the methods ranged from 89% to 99%. Jimenez et al. [29] developed a method that can identify spherical fruits in the natural environment in which difficult situations are present: occlusions, shadows, bright areas, and overlapping fruits. Range and attenuation data are sensed by a laser range-finder sensor, and the 3-D position of the fruit with radius and reflectance are obtained after the recognition steps.

Vegetable quality is often measured in terms of color, shape, mass, firmness, size and absence of bruises, which can also serve as a basis for classifying fruits. Lino et al. [39] classified lemons and tomatoes on the basis of their size and color. Liu et al. [40] developed as system for recognizing peach in a natural setting. The system first obtains and image of a red peach region and then a matching expansion is used to recognize the entire region. The potential center point of the fitting circle is calculated by the intersection of the perpendicular bisector of the line on the contour. Finally, the center point and radius of the fitting peach circle are obtained by calculating the statistical parameters of the potential center points. Patrasa et al. [51] examined variations in antioxidant profiles among fruits and vegetables using pattern recognition tools. Classification is done on the basis of global antioxidant activity, levels of antioxidant groups (ascorbic acid, total anthocyanins, total phenolics), and quality parameters (moisture, instrumental color). Using hierarchical cluster analysis and principal component analysis (PCA), their system can discover the interrelationships among the parameters and thus identify different fruits and vegetables. Patel et al. [50] developed a fruit detection method using an improved algorithm that can calculate multiple features. The algorithm can assign different weights for different features such as color, intensity, edge and the orientation of the input image. The approximate locations of the fruit within an image are represented by the weights of the different features. The algorithm can detect up to 90% of different fruit images taken from different positions on a tree.

Thermal imaging is an approach for converting the pattern of invisible radiation of an object into visible images to facilitate the extraction and analysis of features. If temperature differences can be used to assist in the analysis, diagnosis, or evaluation of a product or process, then infrared thermal imaging technology can be successfully applied. The potential use of thermal imaging in the food and agriculture industry includes detecting diseases and pathogens in plants, predicting water stress in crops, predicting fruit yield, planning irrigation scheduling, detecting bruises in fruits and vegetables, evaluating the maturing of fruits, distributing temperature during cooking, and detecting foreign bodies in food material. Vadivambal and Jayas [67] reviewed the application of thermal imaging in the food and agriculture industry and highlighted the potential of thermal imaging techniques in various agricultural processes. The main advantage of using infrared thermal imaging is the non-contact, non-destructive, and non-invasive nature of the technique in finding the distribution of temperature in a short period of time. Seng and Mirisaee [61] combined different methods that can analyze color, size, and shape to increase the accuracy of recognition. Using the nearest neighbor classifier for the classification, they achieved an accuracy of up to 90%.

Researchers have begun to explore how mobile devices can be conveniently used for recording nutritional intake. Rahman et al. [57] introduced a concept for integrating the camera of mobile phones to capture the images of food consumed. These images can be processed automatically to identify the food items present in the image. They generated texture features from food images and demonstrated that this feature leads to greater accuracy for a mobile phone-based dietary assessment system.

2.3 Fruit Disease Recognition and Classification

Automatic detection of fruit diseases is necessary to detect immediately the symptoms as soon as they appear on the growing fruits. Fruit diseases can develop after harvesting, thereby causing major losses in yield and quality. To determine what disease control measures to take to avoid similar losses during the next harvesting period, it is crucial to recognize the symptoms of fruit diseases. Some diseases can also affect other parts of the tree, infecting twigs, leaves and branches. Some common diseases of apple fruits are apple scab, apple rot and apple blotch [28]. Apple scabs are gray or brown corky spots. Apple rot infections produce slightly sunken, circular brown or black spots that may be covered by a red halo. Apple blotch is a fungal disease and appears on the surface of the fruit as dark, irregular or lobed edges.

Because consumers demand high-quality food products, a fast, accurate and objective method for determining food quality is needed. A promising alternative is the use of computer vision as a cost-effective, non-destructive and automated technique. This inspection approach based on image analysis and processing has found a variety of different applications in the food and agriculture industry. Brosnan and Sun [6, 7] reviewed the progress that had been made in the development of computer vision, and emphasized the important aspects of the image-processing technique coupled with recent developments throughout the agricultural and food industry. Bennedsen and Peterson [3] developed a machine vision system for sorting apples according to surface defects, including bruises. The system could detect surface defects using a combination of a routine based on artificial neural networks and principal components, and three different threshold segmentation routines. When evaluated using eight apple varieties, the routines were able to detect 77%–91% of the individual defects and were able to measure 78%–92.7% of the total defective area. Bennedsen et al. [4] used rotating apples in front of a camera to capture multiple images and removed the dark areas on the apple surface efficiently. Pydipati et al. [55] identified four types of citrus diseases in reference to uninfected fruits using color co-occurrence methods and generalized squared distance in HSV color space, and achieved more than 95% accuracy. Kim et al. [31] introduced an approach for classifying grapefruit peel diseases. Their dataset consisted of the six known diseases of grapefruit peel and the uninfected one. The ROI of the fruit is generated by cropping over which intensity texture features are generated and classified using discriminative analysis. They gained 96% accuracy.

Bulanon et al. [8] explored the variation in temperature at different times of the day in citrus canopy as a potential approach for detecting oranges. Using a thermal infrared camera, they monitored the tree canopy in 24-h cycles and measured surface temperature, ambient temperature and relative humidity. The temperature profile of both the canopy and fruit demonstrated large temperature gradients from afternoon (16:00) until midnight. Using image-processing techniques, they segmented very efficiently the thermal images of the fruits during the period with the largest temperature difference. In another study, Bulanon et al. [9] fused a thermal image with a visible image of an orange canopy to improve fruit detection. A thermal infrared camera captured the thermal image while a digital color camera acquired the visible image. They applied two image-fusion approaches, namely, fuzzy logic and the Laplacian pyramid transform (LPT). The fuzzy logic approach performed better than the LPT, and both fusion approaches improved detection as compared with using thermal images alone.

Qin et al. [56] developed a hyperspectral imaging system for acquiring reflectance images of citrus samples in the spectral region ranging from 450 to 930 nm. They performed spectral information divergence (SID) classification of hyperspectral images of grapefruits to recognize canker from normal fruits and other citrus surface conditions by quantifying the spectral similarities using a predetermined canker reference spectrum. They reported an overall classification accuracy of 96.2% using an optimized SID threshold value of 0.008. Crowe and Delwiche [12] proposed the use of three cameras that can sense the reflectance in the visible region and narrow bands in the near-infrared region for simultaneous color evaluation and detection of fruit defect. The visible region was used for color grading, whereas a narrow band centered at 780 nm was for identifying concavity with structured illumination while a second band centered at 750 nm was for detecting dark spots under complex illumination. In another study, Crowe and Delwiche [13] combined two near-infrared images of a fruit with a real-time pipeline image-processing system. The information on structured illumination facilitated in distinguishing defects from concavities. They estimated the total projected area of defects on each fruit and accordingly classified the defects based on the defect pixel total.

Near-infrared hyperspectral imaging (NIR-HSI) is an emerging technique that combines imaging techniques with the classical NIR spectroscopy to obtain spatial and spectral information simultaneously from a field or a sample. The technique is fast, non-polluting, non-destructive, and relatively inexpensive per capita cost of analysis. Dale et al. [14] reviewed the use of NIR-HSI in agriculture and in quality control of agricultural food products. There is growing interest in the application of HSI in quality and safety assessments of agricultural food products. Lorente et al. [42] surveyed the literature for the use of hyperspectral imaging in the inspection of fruit and vegetables. They explained the different approaches to acquire the images and their use in the inspection of the internal and external features of produce.

Fernando et al. [23] used an unsupervised method based on a multivariate image analysis strategy that uses PCA to generate a reference eigenspace from a matrix obtained by unfolding spatial and color data from defect-free peel samples. Moreover, they introduced a multi-resolution concept to speed up the process. They tested about 120 samples of mandarins and oranges from four different cultivars, namely, Marisol, Fortune, Clemenules, and Valencia. They reported a success rate of 91.5% in detecting individual defects, and 94.2% in detecting both damaged and sound samples. Dubey and Jalal [18, 19] developed another method for detecting and classifying fruit diseases using image-processing techniques. The method could detect the region of defect using an image segmentation technique based on k-means clustering, and then extract the features from that segmented region for use by a multi-class support vector machine for training and classification purposes.

Gabriel and Aguilera [24] proposed a pattern recognition method for automatic detection of stem and calyx ends and damaged blueberries. The method could extract color and geometrical features. In selecting the best features, they tested five algorithms and found that the best classifiers were support vector machine and linear discriminant analysis. Using these classifiers, they were able to determine the orientation of blueberries in 96.8% of the cases. The average performance for mechanically damaged, shriveled, and fungally decayed blueberries were reported as 86%, 93.3%, and 97%, respectively. Apple fecal contamination is an important food safety issue. Kim et al. [32] were able to detect fecal-contaminated apples using a hyperspectral reflectance imaging technique in conjunction with PCA. To detect apples with fecal contamination, they identified three visible and two near-infrared wavelengths that could potentially be implemented in multispectral imaging systems. In another study, Kim et al. [33] reported that multispectral fluorescence approaches can also be incorporated to detect fecal contamination on apple surfaces. Pujari et al. [52, 53] identified diseased fruits to allow for grading normal fruits using BPNN classifier with color- and texture-based features in the RGB and YCbCr color spaces. They obtained nearly 88% success rate. Pujari et al. [54] also used the ANN/Knowledge base classifier and by combining the color and texture features in the RGB color space to classify powdery mildew on six types of fruits. Using only color-based features and decision tree, Kanakaraddi et al. [30] computed the severity level of pathogenic diseases in chili fruits.

Kleynen et al. [34] developed a multispectral vision system in the visible/NIR range with four wavelength bands. They classified fruit defects into four categories: slight, more serious, leading to rejection, and recent bruises. They used a correlation pattern matching algorithm to detect stem-ends and calyxes. For defect segmentation, they applied a pixel classification approach and non-parametric models of defective and non-defective fruits based on Bayes theorem. They achieved good classification rates for apples with serious defects and recent bruises. Leemans et al. [35] proposed an approach based on color information for detecting defects in “Golden Delicious” apples. The approach involves the following steps. First, a model is generated on the basis of variability in the normal color of the variety. Afterwards, each pixel of an image is compared with that of the model to segment the defects. Any pixel that matches with that of the model is considered as a healthy tissue. Leemans et al. [36] proposed a segmentation process based on Bayesian classification that uses information enclosed in a color image of a bi-color apple. The process could separate in segments most defects, such as bruises, bitter pit, fungi attack, scar tissue, frost damages, scab and insect attack.

An automated bruise detection system can help the fruit industry in reducing potential economic losses while providing food products of higher quality to consumers. Lu [43] explored the potential of near-infrared hyperspectral imaging in the spectral region between 900 and 1700 nm for detecting bruises on apples. Using a near-infrared hyperspectral imaging system, they were able to detect both new and old bruises on apples.

Li et al. [38] developd a lighting transform method based on a low-pass Butterworth filter with a cutoff frequency (i.e. the filter response will be maximally flat for D0 <7). They applied this method for transforming non-uniform intensity values on spherical oranges into uniform intensity values over the whole fruit surface. The frequency response of the Butterworth filter was maximally flat up to the cutoff frequency, after which it started decreasing. They found that a ratio method and an R and G component combination coupled with a big area and elongated region removal algorithm could be used to discriminate effectively stem-ends from defects. Mehl et al. [44] presented multispectral techniques using hyperspectral image analysis for detecting defects on three apple cultivars: Red Delicious, Golden Delicious, and Gala. The techniques involved (1) designing a multispectral imaging system from hyperspectral image analysis to characterize spectral features of apples for the specific selection of filters and (2) multispectral imaging for fast detection of apple contaminations. Using these techniques, they found good separation between the normal and contaminated Golden Delicious and Gala varieties, but rather limited separation for Red Delicious variety. Mehl et al. [45] extended their earlier study by developing a hyperspectral imaging technique for the detection of defects and contaminations on the surface of apples. Li et al. [37] developed an experimental hardware system based on computer imaging technology for sorting defects on the surface of apples. The hardware system can inspect simultaneously four sides of each apple on the sorting line. They also developed methods for image background removal, defect segmentation and identification of stem-end and calyx areas. They argued that the adoption of the experimental hardware system is practical and feasible. In current citrus manufacturing industries, color and caliper are key features for the automatic classification of fruits using computer vision approaches. However, human inspection is still the primary means for detecting defects on the surface of citrus. Lopez et al. [41] presented a computer vision system capable of detecting defects and also classifying the type of flaws in citrus. The system uses Sobel gradient to images to segment the faulty zones. Afterwards, color and texture features are extracted considering different color spaces. They employed several techniques for classification and obtained promising results.

Ouyang et al. [47] developed a synthesis segmentation algorithm for use in real-time online segmentation of images of strawberry diseases in greenhouses. The algorithm can eliminate the effects of uneven illumination through the “top-hat” transform, and can remove noise interferences by median filtering. The algorithm computes for the complete area of the fruit image by means of gray morphology, logical operation, OTSU, and mean shift segmentation. Subsequently, the algorithm normalizes the extracted eigenvalues, and then uses eigenvectors of the samples to train the support vector machine and BP neural network. Their results indicate that support vector machines have higher recognition accuracy than the BP neural network. Panli [48] developed a method for segmentation of stems using mathematical morphology. The method applies an attention selection model based on the phase of the Fourier transform to extract the fruit saliency map. In this way, defects on the fruit surface are detected. Fnally, the method uses a support vector machine for the classification using the color and texture of the defective part. They obtained a good classification accuracy.

Schatzki et al. [60] used X-ray imaging for detecting defects in apples, in which apples are characterized as infected or not based on their appearance in X-ray images. Human observers inspect sets of X-ray images for a given cultivar/orientation and record the recognition rates. When they viewed still X-ray images on a computer screen, they found acceptable recognition: 50% of the defective apples were recognized, with 5% of the supposedly good apples were classified as defective. Several thresholding and classification-based approaches are used for pixel-wise segmentation of “Jonagold” apples surface defects. According to Unay and Gosselin [65], segmentation accuracy improves when pixels are represented as a neighborhood. The authors add that multi-layer perceptrons are more promising than the other techniques in terms of computational expense and segmentation accuracy. Their approach generates more precise results on healthy fruits.

Zhang and Wu [68] proposed a fruit classification method based on a multi-class kernel support vector machine (kSVM). The method works as follows: a split-and-merge algorithm is used to remove the background of each image. A feature space is then composed from the extracted color histogram, texture and shape features of each fruit image. PCA is used to reduce the dimensions of the feature space. The authors constructed three kinds of multi-class SVMs, namely, directed acyclic graph SVM, max-wins-voting SVM, and winner-takes-all SVM. They also chose three kinds of kernels: Gaussian radial basis kernel, homogeneous polynomial kernel, and linear kernel. They reported that the max-wins-voting SVM with Gaussian radial Basis kernel recorded the best classification accuracy with 88.2%. Tian et al. [64] developed a multiple classifier system (MCS) based on the support vector machine for pattern recognition of wheat leaf diseases. Three different features, namely, color, texture and shape, are used as training sets. These features are classified by the low-level of MCS classifier to different mid-level categories, which are partly described by the symptom of crop diseases. From these mid-categories produced from low-level classifiers, mid-level features are then extracted. Finally, the high-level SVMs, which correct the errors made by the different feature SVMs, are trained to improve the performance of recognition. Their approach obtained a good success rate of recognition compared with the other classifiers for wheat leaf diseases.

3.1 Image Segmentation and Defect Segmentation

Image segmentation is a convenient and effective method for detecting foreground objects in images with stationary background. Background subtraction is a commonly used class of techniques for segmenting objects of interest in a scene. This task has been widely studied in the literature. Background subtraction techniques can be seen as a two-object image segmentation method, which often need to cope with variations in illumination and sensor capturing artifacts such as blur. Specular reflections, background clutter, shading and shadows in the images are the major factors that must be addressed. Therefore, it may be necessary to perform image segmentation by focusing only on the object’s description to reduce scene complexity. Rocha et al. [59] used a background subtraction method based on K-means clustering technique. Among several image segmentation techniques, only the K-means-based image segmentation method provides a trade-off between efficient segmentation and per unit cost. Some examples of image segmentation techniques are shown in Figure 2.

Figure 2.

Some Image Segmentation Result on the Fruit and Vegetable Images [20].

Precise defect segmentation is required in fruit disease classification. If the method used is imprecise, the features of the non-infected region will dominate over the features of the infected region. Whereas using K-means clustering techniques allow for the segmentation of defects of infected fruit images with three or four clusters, applying fruit background subtraction allows for segmentation with only two clusters. In defect segmentation, using only a single channel and two clusters are not sufficient, so more than two clusters and more than one channel of the color images should be considered for precise disease segmentation. In this experiment, images are partitioned into three or four clusters in which one cluster contains the majority of the diseased parts. The final decision of the number of clusters is done by empirical observation: once the c number of clusters is deemed sufficient for a particular problem, human intervention will no longer be required for further processing, that is, the process becomes fully automated. In our case, two and four clusters will suffice for classifying fruits and fruit diseases, respectively. Figure 3 shows some of the image segmentation results using the K-mean clustering technique.

Figure 3.

Some Defect Segmentation Results on the Diseased Fruit [19].

3.2 Feature Extraction

Some state-of-the-art color and texture features are extracted and used to validate the accuracy and efficiency of the system. The features used in the classification of fruits and vegetables and their diseases are global color histogram (GCH), color coherence vector (CCV), border/interior classification (BIC), local binary pattern (LBP), completed local binary patterns (CLBP), Unser’s feature (UNSER), and improved sum and difference histogram (ISDH).

1. Global Color Histogram (GCH)The GCH is the simplest approach for encoding the information present in an image [25]. A GCH is a set of ordered values, for each distinct color, representing the probability of a pixel being of that color. Uniform normalization and quantization are used to avoid scaling bias and to reduce the number of distinct colors [25].

2. Color Coherence Vector (CCV)Pass et al. [49] presented an approach for comparing images based on CCVs. Color coherence is defined as the degree to which image pixels of that color are members of a large region with homogeneous color. These regions are referred to as coherent regions. Coherent pixels are the parts of the contiguous region, whereas incoherent pixels are not. To compute for CCVs, the method blurs and discretizes the image’s color space to eliminate small variations between neighboring pixels. Afterwards, it finds the connected components in the image to classify the pixels of a given color bucket whether it is either coherent or incoherent. After classifying the image pixels, CCV computes two color histograms: one for coherent pixels and another for incoherent pixels. The two histograms are stored as a single histogram.

3. Border/Interior Classification (BIC)To compute for BIC, the method classifies image pixels as border or interior. A pixel is classified as interior if its four neighbors (top, bottom, left, and right) have the same quantized color. Otherwise, it is classified as border. After the image pixels are classified, two color histograms are computed: one for border pixels and another for interior pixels [62].

4. Local Binary Pattern (LBP)Given a pixel in the input image, LBP is computed by comparing it with its neighbors as follows [46]:

   (1)LBPN,R = ∑n = 0n−1s(vn − vc)2n, s(x) = {1, x ≥ 00, x < 0 (1)

   where v_c is the value of the central pixel, v_n is the value of its neighbors, R is the radius of the neighborhood and N is the total number of neighbors. Suppose the coordinate of v_c is (0, 0), then the coordinates of v_n are [R cos(2πn/N), R sin(2πn/N)]. The values of neighbors that are not present in the image grids may be estimated by interpolation. Let the size of image is I*J. After the LBP code of each pixel is computed, a histogram is created to represent the texture image:

   (2)H(k) = ∑i = 1I∑j = 1Jf(LBPN, R(i, j), k), k∈[0, K],f(x, y) = {1, x = y0, otherwise (2)

   where K is the maximal LBP code value. In this experiment, the value of “N” and “R” are set to “8” and “1”, respectively, to compute for the LBP feature.

5. Completed Local Binary Pattern (CLBP)LBP considers only signs of local differences (i.e., difference of each pixel with its neighbors), whereas CLBP considers both signs (S) and magnitude (M) of local differences as well as original center gray level (C) value [26]. The CLBP feature is the combination of three features, namely, CLBP_S, CLBP_M, and CLBP_C. CLBP_S is the same as the original LBP and used to code the sign information of local differences. CLBP_M is used to code the magnitude information of local differences:

   (3)CLBPN,R = ∑n = 0n−1t(mn, c)2n, t(x, c) = {1, x ≥ c0, x < c (3)

   where c is a threshold and set to the mean value of the input image in this experiment. CLBP_C is used to code the information of original center gray level value:

   (4)CLBPN,R = t(gc, cI), t(x, c) = {1, x ≥ c0, x < c (4)

   where threshold c_I is set to the average gray level of the input image. In this experiment, the value of “N” and “R” are set to “8” and “1”, respectively, to compute for the CLBP feature.

6. Unser’s Feature (UNSER)To extract the Unser feature, the method first finds the sum and difference of intensity values over a displacement of (d1, d2) of an image, and then it calculates two histograms (sum and difference histogram) and stores both histograms as a single histogram [66].

7. Improved Sum and Difference Histogram (ISADH)Dubey and Jalal [20] developed an efficient ISADH texture feature to encode the neighboring information of a pixel in the image on the basis of sum and difference histogram. The sum and difference are calculated with neighboring pixels in the x direction and then these outputs are simulated in the y-direction by calculating both the sum and difference in the y-direction. By considering x and y directions separately, the algorithm is able to encode the relation of any pixel with its neighboring pixels in both the x and y directions very efficiently.

3.3 Training and Classification

Rocha et al. [59] recently presented a unified approach that can combine many features and classifiers. The authors approached the multi-class classification problem as a set of binary classification problems in such a way one can assemble together diverse features and classifier approaches custom-tailored to parts of the problem. They defined class binarization as a mapping of a multi-class problem into two-class problems (divide-and-conquer) and referred to binary classifier as a base learner. For N-class problem N×(N–1)/2 binary classifiers will be needed where N is the number of different classes. According to the authors, the ij^th binary classifier uses the patterns of class i as positive and the patterns of class j as negative. They calculate the minimum distance of the generated vector (binary outcomes) to the binary pattern (ID) representing each class to find the final outcome. Test case will belong to that class for which the distance between the ID of that class and binary outcomes will be minimum.

Their approach can be understood by a simple three-class problem. Let three classes be x, y, and z. Three binary classifiers consisting of two classes each (i.e., x×y, x×z, and y×z) will be used as base learners, and each binary classifier will be trained with training images. Each class will receive a unique ID as shown in Table 1. To populate the table is straightforward. First, we perform the binary comparison x×y and tag the class x with the outcome +1, the class y with –1 and set the remaining entries in that column to 0. Thereafter, we repeat the procedure for the classifier x×z, and tag the class x with +1, the class z with –1, and the remaining entries in that column with 0. We repeat this procedure for the binary classifier y×z, and tag the class y with +1, the class z with –1, and set the remaining entries with 0 in that column, where the entry 0 means a “Don’t care” value. Finally, each row represents a unique ID of that class (e.g., y = [–1, +1, 0]). Each binary classifier results in a binary response for any input example. For example, if the outcomes for the binary classifier x×y, x×z, and y×z are +1, –1, and +1, respectively, then the input example will belong to that class that have the minimum distance from the vector [+1, –1, +1]. The final answer will be given by the minimum distance of

min dist ({+1, −1, +1},  ({+1, +1, 0}, {−1, 0, +1}, {0, −1, −1})).

We used the MSVM as a set of binary SVMs for the training and classification in both problems.

3.4 Results

To demonstrate the performance of the system, we used a dataset of fruits and vegetables obtained from a supermarket, comprising 15 different categories for a total of 2615 images: plum (264), Agata potato (201), Asterix potato (181), cashew (210), onion (75), orange (103), Tahiti lime (104), kiwi (157), Fuji apple (212), Granny Smith apple (155), watermelon (192), honeydew melon (145), nectarine (247), Spanish pear (158), and diamond peach (211) (Figure 4). The dataset contained fruit images under different lighting conditions with varying numbers of elements in an image. It also contained the pose differences among the fruits and cropping and partial occlusion. The presence of these features was allowed to make the dataset more realistic. A dataset of diseased apple fruits were also considered, which included four different categories forming a total of 391 images: apple blotch (104), apple rot (107), apple scab (100), and normal apples (80) (Figure 5). The variations in the type and color of the apples allowed the dataset to be more realistic. In the experiment, different numbers of images per class were used for training purposes. The average error was computed by calculating the sum of the average error of each class divided by the total number of classes. Figure 6 shows the average error for the classification of fruits and vegetables in relation to different features in both the RGB and HSV color spaces, whereas Figure 7 represents the average accuracy obtained for the classification of apple fruit diseases. The x-axis represents the number of images per class for the training, and y-axis represents the average error/average accuracy. The results illustrate that GCH has the highest average error (Figure 6) and the lowest average accuracy (Figure 7) for both the RGB and HSV color images. This is so because GCH has only the color information and it does not consider the relation among the neighboring pixels.

Figure 4.

Data Set used in the Fruit and Vegetable Classification Problem [20].

Figure 5.

Some Images of the Apples Infected with Scab, Rot, Blotch, and Normal Apples of the Dataset [19].

Figure 6.

Average Fruit and Vegetable Classification Error using the GCH, CCV, BIC, UNSER, and ISADH Features Considering MSVM as a Base Learner in the RGB and HSV Color Spaces [20].

Figure 7.

Apple Fruit Disease Average Classification Accuracy using GCH, CCV, LBP, and CLBP Features Considering MSVM as a Base Learner in the RGB and HSV Color Spaces [19].

The average error for CCV is less than that for the GCH (Figure 6). Moreover, the average accuracy of CCV is higher than that of GCH (Figure 7) because the former feature exploits the concept of coherent and incoherent regions. The BIC feature has a lower average classification error than CCV because the former feature considers the values of four neighboring pixels. The UNSER feature has lower average classification error than GCH, CCV, and BIC, and has higher classification than GCH and CCV (Figure 6). The reported error for fruit and vegetable classification in the RGB color space is 9.43%, 7.58%, 5.84%, 5.65%, and 4.56% for GCH, CCV, BIC, UNSER, and ISADH, respectively, whereas MSVM is trained with 60 images per fruit or vegetable. LBP performs better than GCH and CCV because LBP incorporates the neighboring information in a rotation-invariant manner (Figure 7). CLBP performs very well in both color spaces (Figure 7). It is also observed across the plots that both features perform better in the HSV color space than in the RGB color space. For the ISADH feature with 60 training examples per class, the reported classification error is 4.56% in RGB and 1.48% in HSV for fruit and vegetable classification. The ISADH texture feature outperforms the other features because the former uses the neighboring information of the x-direction with the neighboring information of the y-direction (Figure 6). In terms of disease recognition, the CLBP feature is a better choice.

We also made a comparison of the performance of SVM and KNN classifiers using ISADH texture features in both the RGB and HSV color spaces. In Figure 8, the comparison is illustrated for the fruit and vegetable classification problem. In Figure 9, the results are compared for the fruit disease recognition problem. The value K is considered as one. Across the plots, the performance of the SVM classifier is better than that of the well-known KNN classifier in both the RGB and HSV color spaces. On the basis of these results we gathered from several research papers, we argue that the ISADH texture feature is better suited with MSVM in both the RGB and HSV color spaces for solving fruit and fruit disease classification problems.

Figure 8.

Comparison of SVM and KNN Classifier using the ISADH Texture Feature in Both the RGB and HSV Color Spaces for Fruit and Vegetable Classification Problem [20].

Figure 9.

Comparison of SVM and KNN Classifier using the ISADH Texture Feature in Both the RGB and HSV Color Spaces for Fruit Disease Recognition Problem [21].

3.5 Comparison Among Existing Methods

We also compared the performance of different approaches for addressing fruit and vegetable classification (Table 2) and fruit disease recognition (Table 3) problems. We made the comparisons on the basis of the number of categories in the database, pre-processing steps involved, features extracted, color space used, classifiers used, and accuracy achieved.