Hass avocado ripeness classification by mobile devices using digital image processing and ANN methods

2.1 Device camera a digital image capture using RGB color model

In the initial stage of the artificial vision system, the mobile device camera captures a digital image using the red, green, and blue (RGB) color model. Then, the system removes any unwanted colors from the image and only retains the possible color ranges for the Hass avocado. Then, the inverted threshold segmentation algorithm separates objects from the background of the image. Next, this algorithm analyzes the image contours and characteristics and compares each observed object against the characteristics that are previously defined for Hass avocados. If there are any matches, the system calculates the RGB color average for each assessed area. The final stage of the artificial vision system comprises developing and training an ANN based on the characteristics and the average RGB color of multiple Hass avocados. After training, the ANN can match and predict the state of ripeness of the Hass avocado.

This Hass avocado analysis algorithm includes four general steps. Figure 1 illustrates the general process of identifying and classifying the Hass avocados based on a digital image.

Figure 1:

General process flowchart of classifying the Hass avocados.

2.1.1 Step 1

A smartphone digital camera captures a digital image in the RGB format. Then, the system resizes this image to the proposed resolution of 320 × 240 with an aim of reducing algorithmic complexities in the subsequent steps. The high-resolution tests performed while executing the final algorithm exhibited similar results; however, these tests increased the execution time significantly.

Next, the system preprocesses the image by filtering the largest possible number of colors or unwanted objects by replacing each pixel that does not belong to the possible Hass avocado color palette with white. After the preprocessing has been completed, only black, brown, and dark green shades remain in the image, as can be seen in Figure 2.

Figure 2:

(A) Original image and (B) image using a color filter.

2.2 Proposed artificial neural network (ANN)

The ANN implementation presented in this study is based on the multilayer perceptron model developed under the Open Computer Vision (OpenCV) library with three layers defined as shown in Figure 3. This neural network was used to classify the maturity of Hass avocado fruits. Four characteristics obtained from the DIP process raised above were used as input to the neural network, where each entry corresponds to R, G, B values, and the contrast obtained from the gray level co-occurrence matrix (GLCM). In addition, six neurons were used in the hidden layer, and there are three outputs that correspond to the maturity stages proposed for the Hass avocado (green, immature, and ripe).

Figure 3:

Artificial neural network.

Further, the learning or the training phase and the operation or the execution phase can be distinguished while using neural networks. In the first phase, the network is trained to perform a certain type of processing. Once an adequate level of training has been completed, the operation phase is initiated. Here, the network is used to perform the task for which it was trained. In our study, this task is to estimate and assign the desired classification.

2.3 Training phase

The objective of training an ANN is to ensure that a given application produces a set of desired or minimally consistent outputs for a set of inputs. Several training algorithms are available. This methodology uses the backpropagation algorithm, which supervises learning for the multilayer perceptron. This training phase has an ability to adapt and modify the network parameters so that the obtained output becomes as accurate as the output provided by the supervisor [26].

The backpropagation algorithm obtains the input pattern of the network, propagates these inputs to the output layer, calculates the error by comparing the obtained value against the expected value, propagates the error to the hidden layers (backward), and modifies the synaptic weights [26]. This process executes iteratively using all the patterns until the network converges to a desired error value.

Through training, ANNs obtain their own representation of the problem; therefore, they can obtain coherent solutions when presented with situations that they have not previously understood. Thus, ANNs considerably generalize previous cases to form new cases.

2.4 Data collection and analysis

The Hass avocado fruit crops were picked at the Villa Carolina site located in Vereda Volcanes, Santa Rosa de Cabal, Risaralda, Colombia, South America, at an altitude of 1617 m above sea level, latitude of 4°47.075ʹN and longitude of 75°38.016ʹW, with an average temperature of 19 °C. In terms of sampling, each crop comprised 65 Hass avocados harvested at their physiological ripeness from random trees and stored at an average temperature of 25 °C and relative humidity of 45%. During this storage time, the authors monitored the physicochemical and color changes of each fruit. For 15 days, DIP monitored the color changes of 20 Hass avocados, and the remaining 45 Hass avocados were subjected to laboratory analysis for assessing their physicochemical changes during the ripening process.

2.5 RGB value capture

The mobile application captured the color and texture characteristics for each Hass avocado, monitoring their color changes during the ripening process. Since the RGB values obtained using the algorithm depend on the lighting and distance between the lens and the fruit, the system captured multiple images of each fruit based on lighting changes and at distances ranging from 10 to 20 cm.

2.6 Physicochemical parameters

The dry matter quality parameter was determined according to the Mexican standard NMX-FF-016-SCFI-2006.

2.7 Statistics analysis

The results were expressed as mean ± standard deviation and were obtained with three repetitions for each one of the samples evaluated. The analysis of variance (ANOVA) was conducted using IBM SPSS Statistics Version 22 software, followed by the Duncan test. P values <0.05 were considered to indicate statistically significant differences between post-harvest days for dry matter and oil content.

The correlations between the G component of the RGB space and the dry matter content and weight loss were analyzed using Pearson’s correlation.

A linear regression was carried out for the estimation number of days using the algorithm developed. Accuracy and error were estimated.

3.1 Hass avocado recognition

While the tests conducted on different images denoted a variety of objects, we were able to discard any objects that did not resemble the characteristics of the Hass avocados. Furthermore, we were able to obtain different details and denote the image areas where the system observed fruit matches. In fact, the remaining black, brown, and dark green shades exhibited an effect on the Hass avocado identification process. Similarly, there were problems with the shadows reflected on the fruit. However, the use of the mobile device’s flash while capturing the image effectively resolved these problems. Figure 4 highlights the contours which presented similarities to Hass avocados.

Figure 4:

Hass avocado contour.

3.2 Data analysis

The Hass avocados are green during the physiological ripening process and change to dark brown as their horticultural ripening approaches. The information and characteristics obtained from the Hass avocados while studying and monitoring their ripeness during a 15-day period denoted that the green component of the RGB color space begins with higher values in comparison with the red and blue components. As it approaches its organoleptic or consumption ripeness, the value of the green component decreases until it becomes lower than the remaining two color-components. This behavior occurs in different lighting environments while capturing the image and provides valuable information for estimating the ripeness in new cases. Figure 5 shows the average RGB color space values for the 15-day avocado ripeness monitoring period.

Figure 5:

RGB values during 15 days of ripening.

The analyzed samples presented differences on the days on which the fruit reached its ripeness stage. Table 1 presents the behavior presented during the ripening process, exhibiting considerable similarity between the ripe and unripe states, which makes it difficult to distinguish them.

3.3 Color and chemical correlation

On an average, the samples reached the state of ripeness on day 11.36 ± 1.009, on which the fruit exhibited optimal characteristics for consumption. Similarly, in Figure 6, obtained from the performed physiochemical studies, according to the statistical analysis, significant differences were observed (p < 0.05) for post-harvest day 11 compared to the other days evaluated, where the highest content of dry matter and oil was presented. According to the dry matter content, an increase was observed from 23.20 ± 0.1912 to 27.72 ± 0.7140%, and in relation to the oil content, values were observed from 14.09 ± 0.1600 to 22.73 ± 0.4259%.

Figure 6:

A. Dry matter percentage and B. Oil content percentage.

Also, the G component of the RGB space has a correlation of −0.595, −0.946 and −0.521 with respect to the dry matter, weight loss and oil content, respectively; thus, it has been possible to verify the existing relation between the physicochemical changes and the color.

3.4 Classification results

Table 2 lists the results obtained while estimating the states of ripeness of 65 samples using the previously trained ANN. The neural network tested the datasets obtained from the images captured by the flash-enabled mobile device.

The final classification accuracy using the ANN is 88%, with the intermediate state exhibiting the highest error rate at 20%. The false classification of the unripe and ripe states due to the color similarities accounts for most of this error.

The estimation of the number of days defined by Equation (2) with a regression value R² of 0.819 accurately estimated 88% of the analyzed data with a 3-day error and 74% of the analyzed data with a 2-day error.

Equation 2