Machine Learning Approach in Orange Fruit Grading Using Close Range Photogrammetry

N.S. Osman
K.N. Tahar
A. Almhafdy Abdulbasit

Intensive fruit or vegetable sorting is a common mission in productive regions. In order to meet the market standards, it is produced according to quality levels that depend on maturity, weight, size, density, skin defects and others. The aim of this study is to automate the sort of oranges based on colour and size to specific grades. The experiment on the oranges colour grading using K-Means and the Isodata technique has been used in this study. The methodology consists of four phases which are data acquisition, data pre-processing, data segmentation and data analysis. Moreover, the results of the RGB value and measurement size of the oranges are based on the output that has been achieved. The RGB value is to determine the maturity of the oranges based on the RGB value. In addition, the measurement size of the oranges was computed using the 3DF Zephyr that generated the 3D images. The manual measurement and direct measurement using photogrammetry software will be able to compare the different sizes of the oranges. Based on the results obtained, the best type of oranges that represents Grade A by size and colour output is Grapefruit Australia. The reason is that the size measurement using a comparison between manual and the 3D builder tools shows the best outcome that, is a 0.1 cm difference, while the manual and 3D Zephyr difference is 4.28-cm. Consistent RGB percentage for both classifications show good ripe. The red colour percentage is 37.6%, the green colour is 37.5%, and the blue colour is 16.2%. Besides that, all the oranges are also supported with depth accuracy and a ground pixel size under the 3D images technique that is under the tolerance of one-pixel disparity. In conclusion, the grading of orange is determined using the size measurement from 3D images and colour classification using the unsupervised classification of K-Means or Isodata Classification.

Machine Learning Approach in Orange Fruit Grading Using Close Range Photogrammetry

Osman, N. S.,1 Tahar, K. N.2* and Abdulbasit A. Almhafdy3

1 Braintree Technologies Sdn Bhd, Block G1, UPM - MTDC Technology Centre, Universiti Putra Malaysia,

43400 Serdang, Selangor, Malaysia

2 Centre of Studies for Surveying and Geomatics, Faculty of Architecture, Planning and Surveying, Universiti

Teknologi MARA, 40450 Shah Alam, Selangor Darul Ehsan, Malaysia, E-mail: khairul0127@uitm.edu.my*

3 Department of Architecture, College of Architecture and Planning, Qassim University, Buraydah

Saudi Arabia

*Corresponding Author



Abstract

Intensive fruit or vegetable sorting is a common mission in productive regions. In order to meet the market standards, it is produced according to quality levels that depend on maturity, weight, size, density, skin defects and others. The aim of this study is to automate the sort of oranges based on colour and size to specific grades. The experiment on the oranges colour grading using K-Means and the Isodata technique has been used in this study. The methodology consists of four phases which are data acquisition, data pre-processing, data segmentation and data analysis. Moreover, the results of the RGB value and measurement size of the oranges are based on the output that has been achieved. The RGB value is to determine the maturity of the oranges based on the RGB value. In addition, the measurement size of the oranges was computed using the 3DF Zephyr that generated the 3D images. The manual measurement and direct measurement using photogrammetry software will be able to compare the different sizes of the oranges. Based on the results obtained, the best type of oranges that represents Grade A by size and colour output is Grapefruit Australia. The reason is that the size measurement using a comparison between manual and the 3D builder tools shows the best outcome that, is a 0.1 cm difference, while the manual and 3D Zephyr difference is 4.28-cm. Consistent RGB percentage for both classifications show good ripe. The red colour percentage is 37.6%, the green colour is 37.5%, and the blue colour is 16.2%. Besides that, all the oranges are also supported with depth accuracy and a ground pixel size under the 3D images technique that is under the tolerance of one-pixel disparity. In conclusion, the grading of orange is determined using the size measurement from 3D images and colour classification using the unsupervised classification of K-Means or Isodata Classification .

1. Introduction

Intensive fruit or vegetable sorting is a common mission in productive regions. In order to meet the market standards, it is produced according to quality levels that depend on maturity, weight, size, density, skin defects and others. In this study, the fruit selection and grading system are based on image processing techniques. Besides that, different types of extract features of fruit characteristics are available by capturing the fruit image. Therefore, the fruit will be graded into a good quality that uses other types of classifiers. Based on the sorting and grading system, calculating the size using a simple method of classification parameters, reconfiguration of the outputs and maintenance of production are basic regulations to be followed. One of the main reasons to further this study is to try to meet the requirements of the system, mainly in the accuracy of measurements to be graded. This work aimed to study many types of methods used for quality grading to develop an algorithm for detecting and sorting fruit from the image collected. A feature, such as the morphological feature, is used, and colour can be extracted. Furthermore, this study also focused on computer-assisted photogrammetric procedure, which is related to the colour of the fruits, their ripeness, and will be graded physically in order to classify them. Next, generating knowledge that increases production and processing efficiency determine the grade and quality of the fruit itself (Banot and Mahajan, 2016).


3D imaging is a high technology technique that is available to capture multiple images of any kind of external fruit quality to be measured. 3D imaging has a high demand, is time-consuming and has the techniques for better methods. Besides that, 2D modelling is also popular for managing the imaging systems to detect the external quality of fruits. However, 2D image analysis is not always a good technique for fruit image processing due to the uneven colour distribution of fruit and occlusion of systems from different viewing perspectives (Ghuman, 2016).

This is also to reduce human labour because it takes a longer period to complete all the processes that need to be completed. Since the sorting system is well recognised in the industry sector, the grading of oranges using close-range photogrammetry is one of the methods that can be applied to provide images of data by multi-analysis or photography that justify the grades efficiently without any mistakes (He et al., 2017).

2. Literature Review

Next, the image pre-processing phase consists of three steps, for example, image binarisation, morphological processing, and feature extraction. The photogrammetry also includes the steps that are performed in an automated way such as geo-referencing, image correction, camera calibration, orientation, orthophoto formation that are related with the required expert knowledge or to be determined manually. The first output consists of 2D representations of the 3D object in space, orthophotos of the image that can transform into a 3D representation of the object (Rahaman and Champion, 2019). Based on studies by Thomas et al., (2019), a dense point cloud is computed, whereby it compels to complete a dense pairing between oriented photographic images. The detection of gaps in 3D is to be completed either by using a volumetric representation with voxels or with triangulated surface meshing.

Accordingly, image resampling is implemented to get a textured 3D cloud or model. Furthermore, the method for measuring the stereo camera depth accuracy was validated with a stereo camera that was built from two SLRs (single-lens reflex) where the formula or calculation of comparison between the stereo camera depth resolution and human depth resolution were computed (Vázquez-arellano et al., 2016).

Moreover, the binarisation method translates the RGB image into grey and binary until the binary changes are made. Dialectal or morphological activity such as edge detection, contrast enhancement, noise removal, haze reduction image segmentation into regions, and skeletonisation will be examined before the next steps proceed. The segmentation concerned the development of thresholding and clustering, which combines and classifies the natural colour of the sorted and graded fruits. Meanwhile, image segmentation is used as an algorithm before extracting the features of the image. The operation of image segmentation run in different colour components. The image morphological technique is used to ‘grow’ or ‘thicken’ an object in the binary image (Zhou et al., 2019). Next, each point cloud will be transformed into red, green, blue (RGB) space to intensity, hue and saturation value (IHS). One of the aspirations in the analytical step is to prevent the obstacle that is caused by light intensity to detect the fruit surface presented a good algorithm for sorting and grading lemon fruits based on colour and size, related to the visual basic environment. Besides that, clustering is the process of grouping a set of objects in such a way that objects in the same group are more similar in some particular manner to each other than to those in other groups. A defect analysis of fruit is based on colour features using the K-mean clustering through the unsupervised procedures. This technique verified the original combination of pixels that were present in an image. This clustering method uses the minimum spectral distance formula to form clusters. The Isodata utility repeats the clustering of the image until either a maximum number of iterations has been performed, or the maximum percentage of unchanged pixels has been reached between two iterations (Zhang et al., 2019).

Grading is classifying vegetables and fruits into certain grades based on the scale, shape, colour and volume of the vegetables and fruits to a high market price. The visual grade is known prior to the analysis of the images about the maturity. Therefore, the image collection level was done within a day after the fruits were delivered to the laboratory so as not to damage the colour of the fruits. Besides that, the rate values of the Digital Number (DN) of the segmented images can be used to differentiate the ripe fruit from the unripe ones. Three general grades are regarded as an additional class consisting of five per cent tolerance for the foreign market. Furthermore, strawberry farmers, traders and distributors needed automated and energy saving in the classification of strawberry fruit shapes. The procedure of classifying the fruit shape and packing engage more than 60% of the working time of farmers.

Manual grading was costly, and the sorting function was affected due to a shortage of work in peak seasons; even the manual grading system has the best precision of quality checking of any vegetables and fruits. Besides that, based on the current technique, there will be a reference as a guide, if in future other people want to refer to the previous work that has been done (Ji et al., 2017).

3. Material and Methods

Basically, this study consists of four phases: data acquisition, data processing, data segmentation, and data analysis (Figure 1). In phase 1, data acquisition is a preparation of the instrument to be calibrated to the specific specification of the parameter, light system, angle and even the distance. In phase 2, the first step to be done is the image binarisation then the morphological processing that is related to the shape of the fruits. Next is the extraction processes that consist of colour technique that uses hue, saturation and value methods. During the process of phase 2, which is the data processing, there will be another process that will be conducted, i.e., the close-range photogrammetry for the 3D imaging system that will be run by structure from motion algorithms. This process using the 3DF Zephyr software will perform photo alignment, building dense cloud point, mesh creation and textured mesh creation. Furthermore, phase 3 is basically the phase for conversion and clustering steps that have been applied. The conversion algorithms were used is the RGB colour space method while the clustering is the K-Means and Isodata Classification. Phase 4 is the data analysis step that is the calculation of DNs and the classification of size and colour.

3.1 Data Acquisition

The data were obtained by capturing the images using the digital Nikon Coolpix AW110 camera. This camera also provides the navigation of a GPS receiver where sensors such as altimeter, barometer, and magnetometer have been included in its specifications. The sensor resolution of the camera is 16 megapixels and the total pixel contain 16790000 pixels. The depth is 1 inch, the height is 2.6 inches and its width is 4.3 inches. Other than that, the resolution can be highest at 1920 x 1080 and the minimum focal length is 5 mm which can be maximised until 25 mm in length.

Figure 1 Research methodology on orange fruit grading

Furthermore, the study area for this study can be any spacious area that can accommodate all the equipment such as the bucket, the fruits and the tripod stand. The 3D photography set up was chosen to capture the images.

Besides that, the light support is produced by natural sources, which is sunlight. The tripod height is 57 cm, while the angle viewing is 20 degrees and the distance between the camera and object is a range of 13 to 16 cm. Each sample of orange is taken a photo for about 30 to 50 images to produce the best images possible. In this way, the steps are repeated using the different types of oranges and the outcome is processed using the 3DF Zephyr, and the Erdas Imagine algorithms to record the results that are obtained. The main reason for the different distances between the camera and object is due to the size of the oranges. Besides that, the data are processed based on a suitable method before producing the final output, i.e., the 3D and 2D images that classify the grade of the oranges.

3.2 Data Processing

In this study, the close-range application has been used to determine the size of the fruit using a 3D imaging system which is the 3DF Zephyr software or algorithms. Thus, the 3D imaging system has a better viewing perspective due to the angle or dimension of the image captured and made. Through this method, the GPS was used to obtain the position of the oranges. After that, the image will be captured using the Nikon Coolpix AW110.

All the images that have been captured are added in the selection page. This free version of 3DF Zephyr only has a limited 50 images that can be applied. All types of oranges tested in this study have a range of between 30 to 50 images. The next step is the stereo settings, which are an advanced method. These settings have remained as defaults for all the oranges. At this stage, the 3DF Zephyr finds the camera position and orientation for each photo and builds a sparse point cloud model. Based on the estimated camera positions, the program calculates the information depth for each camera to be combined into a single dense point cloud which is three cameras that are the nearest (Figure 2).

Furthermore, the building mesh is to control its position manually or automatically that has already been referenced. In these mesh creation settings, an advanced technique was used that contained photo consistency-based optimisation. The maximum iterations are 40 and the maximum vertices are 10000000. These settings remained as defaults for all the oranges. Next, is the building texture that will produce a quality texture and generalise the parameter as well as the building of the mapping modes of the 3D point cloud (Figure 3). This processing was used as advanced settings. Furthermore, the maximum textured size is 8190 x 8190, while for maximum vertices number is 3000000. The image resolution is 100 per cent for the output that will be generated. This setting is the default for all the oranges that will be used.

Figure 2 Point Cloud; a) sparse dense, b) dense

Figure 3 The image of final output of textured mesh

Moreover, the accuracy of models of the known objects with simpler structure is investigated. The comparison between the manual measurement and direct measurement using photogrammetric was differentiated by the dimension of the object. Therefore, the accuracy can be calculated using the accuracy and ground pixel size method. But before the dimension was used to compute the measured, all the unnecessary point cloud were deleted and fill holes filtering were generated to prevent any miscalculated measurement of dimension.

3.2.1 Subset image and manual measurement

The first step to start processing is by subsetting the image using Erdas Imagine. The images of the oranges were subset, strictly following the shape of the oranges (Figure 4).

Figure 4 The subset of raw images of South Africa oranges into a desired size of images subset

This is to prevent any other detection of colour that will affect the RGB result at the final output. For the manual measurement, the oranges were measured using measurement tape to obtain the actual size of the oranges. This is to make a comparison between the 3D measurements using the 3DF Zephyr software and the 3D Build in Windows.

3.2.2 Image binarization

The binarisation process changes the original image to a grey colour. Therefore, using the subset image from the previous step, the K-Means classification under the unsupervised method was used to change the number of classes into two. With only two classes, the image will turn into black and white binary colour, which represents the grey colour as the outcome (Figure 5). This will ensure that the conversion of the image will help the following process to detect well in the morphological processing.

Figure 5 The results of conversion to gray colour using unsupervised methods

3.2.3 Morphological processing

Morphological image processing is a procedure that is related to the shape or morphology appearance in an image. Thus, by using the morphology technique, it is able to remove the imperfections by studying the structure of an image. The morphological processing uses the Erdas Imagine algorithm technique to obtain the perfect image without any error or distortion. The Noise reduction process helps to eliminate radiometric correction that applies an edge-preserving smoothing technique. This process was used after the conversion of the binarisation of the original colour to grey colour images. Haze reduction is a process that was applied after noise reduction application. Therefore, the atmospheric effects can cause the imagery to have a limited dynamic range, generally perceived as haziness or reduced contrast. This function helps to sharpen the image using a Tasselled Cap or Convolution method.

3.2.4 Extraction processing

Feature abstraction is the process of measuring or calculating the features from the image samples. Some of the fruits can be easily verified due to their colour and size. The feature extraction process is completed by using the Erdas Imagine software. This process is a conversion of the previous image that results from the morphological process that applied noise and haze reduction to change into RGB images. The intensity, hue, and saturation of the images show only the grey images. Therefore, the spectral method of IHS to RGB images was generated to obtain the RGB images, which contains a threshold for 255 or more classes with a specific value (Figure 6).

Figure 6 The results conversion from morphological process to IHS to RGB image

3.3 Data Segmentation

Data segmentation is the practice of identifying, categorising, labelling, and processing specific elements or sections of electronic data to provide accurate results to those who might be using them in the future. The automatic analysis and segmentation of terrestrial, aerial and 2D satellite images that are known as semantic classes are always referred to as image classification that has always been used in the area for research in photogrammetry, remote sensing, and computer vision.

3.3.1 K-Means classification

From the previous processing which was the conversion of IHS to RGB image, the next step was the K-Means Classification. Ten classes were generated because the minimum class number must be at least ten classes. Therefore, in this technique, ten classes were chosen. Thus, the result of the RGB value is obtained. The K-Means technique was used to compare the result of the colour based on the RGB value to determine the grading of the oranges. Each RGB value has its own value to be analysed. The value can specify the grade of the oranges by referring to each colour value that has the highest DN. Moreover, to verify the ripeness or maturity of the oranges, the red colour value needs to be higher than the blue and green values. From this automatic method, the technique will determine how many classes in the images are to be classes.

3.3.2 Isodata classification

Another classification was performed to compare the differences of results in the RGB value and the number of classes generated by Isodata classification technique. Isodata classification technique also used ten classes just as in the K-Means classification. While other details are set up as default values. Both classifications are under the unsupervised classification. The image that is used to generate this classification is also using the image conversion from the previous result of the IHS to RGB images. This technique will also generate a number of classes automatically.

3.4 Data Analysis

Data analysis is introduced as a process of cleaning, changing, and the pattern data to discover advantages of information for decision-making. The purpose of data analysis is to obtain good information from the data and to obtain decisions that are based on the data that are produced.

3.4.1 Calculation of digital number

The calculation of DNs is categorised as high, low, and average or a range of values that are based on collected images, where the decision-making is analysed using data that are obtained based on the orange fruit. This calculation can show the difference value of the RGB colour changes to identify the maturity, where each type of orange has a different RGB value. Furthermore, the DN will specify each red, green, and blue colour. From the DN, the value or percentage will justify the orange’s maturity colour based on the obtained low or high percentage. The percentage accuracy is calculated manually using the basic formula of sum and mean method.

3.4.2 Classification of size and colour

The classification method using size and colour was used to grade the fruits as good or bad. Thus, photogrammetry and the remote sensing method have been used to obtain the size and colour results. The unsupervised method of K-Means and the Isodata technique was conducted to justify the colour classification. The DN has been calculated using a specified calculation based on the RGB value performed. Therefore, the results will be produced as 2D images with specified classes. Besides that, the results from the close-range photogrammetry using the 3DF Zephyr will be in 3D images, which determine the size diameter of the orange. The classification of size and colour is the main target for the grading system in thèse studies. The grading consists of three types of grade which is grade A, grade B and grade C. The combination of 2D and 3D images will identify the grade of the orange.

4. Result and Analysis

4.1 Result of Size Based on 3D Images

Based on the result that has been achieved, four types of oranges have been used to identify the difference in size, to grade the oranges into the suitable grade by computing the 3D measurement using the 3DF Zephyr. The four types of oranges are Navel, Valencia, Grapefruit South Africa, and Grapefruit Australia. This result had gone through a process that involved aligning photos, building dense clouds, building mesh and textured mesh (Figure 7).

Based on the result of the navel oranges, the enhancement of texture for these oranges is presented through the 3D image (Figure 7a). Some of the bottom parts of the images contain holes that have been filled using the filling holes filtering, also the texture can be seen as if there are small deep pores or scales. This is to secure whole images that have a better representation and to ensure that no errors occur during the direct measurement. In terms of the actual size, this orange is the smallest among the others. Based on the result of the Grapefruit Australia oranges, the enhancement of texture for these oranges is presented through the 3D image, in which the small dots and marked black point can be clearly seen (Figure 7b). However, for this orange, the distortion for the bottom part is quite big but this can be covered using the filling holes filtering. The reason might be that the captured images are not well covered at the bottom parts of the orange. Nevertheless, the applied filtering technique will help to make the selection of points for direct measurement easier. In reality or actual shape, this orange is the largest.

Based on the result of the Grapefruit South Africa, the enhancement of texture for these oranges is presented through the 3D image (Figure7c). The texture from the images also show small dots, but are not as visible as those of Grapefruit Australia. In terms of size, in reality, this orange is the second largest. Some of the bottom parts of the images contain some holes that have been filled using the filling holes filtering. Furthermore, the purpose to cover the holes is to avoid any errors during the direct measurement and to have a better 3D representation of the images. Based on the Valencia Orange result, the enhancement of texture for these oranges is presented through the 3D image (Figure 7d). The texture from the images also shows small scales types pores, some bruised marks and white little dots. In reality, in terms of size, this orange is a medium size. Some of the bottom parts of the images contain some holes that have been filled using the filling holes filtering. Furthermore, the purpose to cover the holes is to avoid any errors during the direct measurement and to have a better 3D representation of the images.

4.2 Result of Colour Based on RGB Value

Based on the results that are obtained, the colour classification has been determined using two-classification methods, which are the K-Means and Isodata Classification. Both classifications are specified into ten classes. The purpose of this technique is to compare the RGB value to verify the grades of the oranges. The red colour represents ripeness, the green represents over-ripeness, and the blue colour is the bad ripe.

4.2.1 Grapefruit Australia

This orange used the K-mean and Isodata Classification to achieve the red, green, and blue value that can determine the maturity ripeness of the oranges, which can eventually be graded depending on the results. There will be two results, based on K-Means and Isodata Classification. In fact, this type of orange has the second largest size. From the K-Means classification results, the red colour value has the highest percentage at 37.6%, while the green colour is 37.5%, and the blue colour is 16.2%. The red and green colours indicate the ripeness of an orange that is in good condition.

Figure 7 The 3D result 3DF Zephyr; a) Navel Orange, b) Grapefruit Australia, c) Grapefruit South Africa,

d) Valencia Orange

Based on the obtained Isodata Classification result, there is no difference between the classification of K-Means and the Isodata Classification. However, each red, green and blue value has shown differences, where the highest value is the red colour, which is 37.6%. The maturity of the oranges was defined based on the red colour value. Meanwhile, the green value is 37.5% which shows that the over-ripeness of the oranges is also contained in the obtained results, and the blue colour indicates a bad ripe condition of the oranges which is 16.2%. This amount is the second-lowest percentage compared to the other oranges. Both classifications do not have differences for Grapefruit Australia (Figure 8).

4.2.2 Grapefruit South Africa

There will be two results, which is based on K-Means and Isodata Classification. In reality, this orange is the largest in size. Based on the K-Means result, three out of ten classes were detected during this classification. The highest colour value is red colour, which is 19.9% and supported with 19.8% of green colour. The blue colour only contributes 13.8% of the classification. Based on the Isodata Classification result, there is a difference between the classifications of K-Means and Isodata Classification. The difference can be seen where five classes have been identified using the latter technique. Besides that, the percentage of red colour and green colours is higher than the K-Means classification for Grapefruit Australia. The red colour contributes 29.4%, while the green colour is 29.3% and the blue colour is 14.7%. Therefore, the ripeness of the orange are more enhanced using the Isodata Classification method (Figure 9).

4.2.3 Valencia orange

There will be two results, which is based on K-Means and Isodata Classification. In reality, this orange is medium in size. Based on the obtained K-Means result, there is a difference between both classifications, of K-Means and Isodata Classification. Based on K-Means classification, three classes were detected out of the ten classes that have been formatted. Besides that, the red colour value that is obtained from K-Means Classification is 26.5%, the green colour value is 26.4%, and the blue colour is 22.4%. Based on the obtained Isodata Classification result, there is a difference between the classification of K-Means and Isodata Classification. The Isodata technique for the Valencia oranges only detects four classes out of the ten classes, which is one extra class from the K-Means classification. This result shows that the red colour consumes by these techniques is higher, that is 36.5%, the green colour is 36.4%, and the blue colour is 19.3%. This showed that the Isodata Classification enhanced the red value more than the K-Means classification ( ).

Figure 8 The result of Classification for Grapefruit Australia; (a) K-Means, (b) Isodata

Figure 9 The result of Classification for Grapefruit South Africa; a) K-Means Classification, (b) Isodata

Figure 10 The result of Classification for Valencia Orange; (a) K-Means, (b) Isodata

Figure 11 The result of Classification for Navel Orange; (a) K-Means, (b) Isodata

4.2.4 Navel oranges

There will be two results, which is based on K- Means and Isodata Classification. This orange is the smallest size in reality. Based on the K-Means technique results, five classes are detected from the ten classes that have been set up in the details information table. The red colour percentage value is 37.8%, while the green colour value is 36.7% and the blue colour value is 16%. Based on the obtained Isodata Classification result, there is a difference between the classification of K-Means and Isodata Classification. The number of classes that have been obtained is six out of the ten classes that have been set up earlier. Based on the comparison from both classifications, the entire colour in the Isodata Classification is higher than the K-Means classification. The red colour is 47.8%, the green colour is 47.7% and the blue colour value percentage is 16%. This showed that the K-Means classification enhanced the red value more than the Isodata Classification for the ripeness of the orange (Figure 11).

4.2.5 The percentage comparison between the Red, Green and Blue value

The Grapefruit Australia is based on the comparison of the red colour value, i.e., for maturity which indicates ripeness or a good condition. This is because of the consistency of the RGB value that has been obtained for both classifications that show the balance of ripe and over-ripe since red colour in their original state is a good condition. The actual ripe colour or in the original picture is the colour orange, while over-ripeness is represented as a slightly darker orange that is more towards a reddish colour. The value for the blue colour is quite high, which is the same as a navel blue colour value. This is because the Grapefruit Australia has black marks on the oranges, which are detected as bad ripe or the darken marks of damaged orange conditions. The Navel and Valencia oranges were bought as bad ripe oranges to show that ripeness can be qualified as lower-grade for this study. Nevertheless, the highest value contributes for red colour is the Navel oranges using the Isodata Classification method that is 47.8%, and for the K-Means the red colour for the Navel orange has the highest percentage of ripeness, which is 37.8%. The Isodata Classification proved that the blue value represented the bad ripe conditions where the highest percentage were the Valencia orange for both classifications: K-Means is 22.4% and the Isodata Classification is 19.3%. Furthermore, the best second results are the Navel orange since it has the highest ripeness percentage that is 47.8% using the Isodata Classification. Besides that, this classification shows that the Isodata Classification method is much more accurate because it computes a greater number of classes that increases the accuracy of the RGB colour that represents the images of the oranges.

Table 1 The percentage comparison between the Red, Green and Blue value

Types of orange

Red %

Green %

Blue %

K-Means

Isodata

K-Means

Isodata

K-Means

Isodata

Grapefruit Australia

37.6%

37.6%

37.5%

37.5%

16.2%

16.2%

Grapefruit South Africa

19.9%

29.4%

19.8%

29.3%

13.8%

14.7%

Valencia

26.5%

36.5%

26.4%

36.4%

22.4%

19.3%

Navel

37.8%

47.8%

36.7%

47.7%

16%

16%

Furthermore, the Grapefruit South Africa shows the lowest RGB colour for both classifications. This might be due to the image enhancement processing or data acquisition that lacks the technique to verify whether the oranges are in a good condition when in reality Grapefruit South Africa is the best condition compared to the other oranges. The red colour percentage is 29.4% for Isodata while K-Means is 19.9% (Table 1).

4.2.6 Grading of the Oranges

This grading of orange is determined through the size measurement from the 3D images and colour classification using an unsupervised classification of K-Means or Isodata Classification. Based on the obtained results, the best oranges that represents a Grade A is by size and colour output, which is the Grapefruit Australia. The reason is because the size measurement- using the comparison between the manual and 3D build tools, shows that the best outcome is 0.1 cm in difference, while the difference for manual and 3D Zephyr is 4.28 cm. The smallest difference from one of the comparisons is how the size measurements are determined to be the best grade. This is also supported by colour classification whereby the RGB colour percentage for a Grapefruit is consistent for both classifications that have no difference between the two techniques that are used. The red colour percentage is 37.6%, the green colour is 37.5%, and the blue colour is 16.2%. Besides that, all the oranges are also supported with depth accuracy and ground pixel size based on the 3D images technique that is under the tolerance of one-pixel disparity.

The second-best grade is Grade B, which is the Grapefruit South Africa, where the size measurement errors, difference is still considered as small and consistent compared to the others. The manual and 3D build difference is 3.65cm, while the difference between the manual and 3D Zephyr is 3.34cm. Thus, the supports of colour classification indicate the lowest blue colour to be the lesser bad ripe oranges with the minimal red and green colour percentage. In addition, the colour classification is also under tolerance for depth accuracy and ground pixel size.

Lastly, Grade C is probably classified as a bad representation in terms of size, colour, or for both size and colour classification. The Valencia and Navel oranges are graded as Grade C. This is because Valencia has shown a huge difference in both the manual and 3D build and the 3D Zephyr. Its colour classification has the largest percentage of the blue colour, which indicates the bad ripe conditions; its size measurement also has the largest difference that is of a very bad representation. However, for colour classification, its red and green colour percentages are high. These findings can also indicate that the oranges are actually over-ripe since they are bought to prove bad ripe oranges in the classification method. The only positive results for both the Navel and Valencia are their depth accuracy and ground pixel size that is under tolerance.

4.2.7 Overall analysis

Based on the results that have been obtained, both methods the 3D Zephyr software- to obtain the size, and the Erdas Imagine which is used for colour, are not providing satisfying results. This is because the 3D Zephyr measurement cannot be generated due to the shortage of measurement tools. The only available measurement tool is the quick measurement or direct measurement tools. The other measurement tool is area; volume is available but it is not suitable to get the diameter for the oranges. Nevertheless, the results that have been achieved are based on the diameter measurement, but they are not accurate. Besides that, the accuracy depth and ground pixel size were some of the supported results that helped to define the accurateness of the processing. Therefore, from observation, the best combination of both measurement sizes, accuracy depth and ground pixel size are the Grapefruit Australia and the Grapefruit South Africa. The reason is that the size measurement using the 3DF Zephyr is the closest to the manual measurement and is still in the range of one-pixel accuracy.

Furthermore, the next results are based on the classification technique using the Erdas Imagine to achieve the RGB value for the ripeness of the oranges. The red colour for this study represents good ripeness, while the green and blue are representing over-ripeness of the oranges and as the distortion that has been computed together. This distortion might be due to the lack of performed enhancement and the technique for the images- as data acquisition, was not conducted in proper ways. The effects of brightness or shadow can cause the final results to be of inferior quality. However, based on the observation results that have been achieved, the best oranges representing the RGB value is the Grapefruit Australia, where even its blue and green value is high, but with a balanced amount of red, green and blue showing. Thus, it can be the best results that have been obtained. The Grapefruit South Africa has also contributed a balanced red, green and blue value. Thus, it can be one of the good condition oranges.

Lastly, based on both results- for size and colour classification, the Grapefruit Australia is the best and can be categorised as the first-class grade that is Grade A. While the Grapefruit South Africa will be categorised as a second-class grade, which is Grade B, and finally for the third-class grade, which is Grade C, is for the Valencia and Navel oranges. These oranges have obtained the lowest results in value for both methods, in photogrammetry and remote sensing.

5. Conclusions

This study has used the 3DF Zephyr software that produces 3D images and the diameter size of the measurement of each orange that is collected. The result shows some negative and positive outcomes, however, the Grapefruit Australia has the best results to represent the size measurement. The different stereo viewing angles are computed by the size of the oranges since each has a different stereo view aspect when the images are captured. Nevertheless, this method needs to be improved since the obtained results consist of good and bad output.

The RGB value is the main data to determine which oranges have the best results. Therefore, the Grapefruit Australia is the best since both of its classifications have the same value of RGB and it is also expressed in percentage form. The red colour value represents a good ripe, the green colour represents an over-ripe, and the blue colour is bad ripe conditions. This method proves that colour can detect ripeness or maturity.

Eventually, the oranges can be graded using the colour classification method. Thus, from the results that are obtained, both size measurement and colour classification have been used to determine the grades of the oranges, where the Grapefruit Australia has shown the best results from its size and colour. The size measurement is compared with several techniques which use manual measurement, 3D Zephyr, and 3D build tools to obtain the best outputs. Accordingly, colour classification using the Erdas Imagine software were compared by K-Means and Isodata Classification. The purpose of this study is to sort the oranges by referring to both techniques for better selection in the grading system and to help to minimise the mistakes of grading the fruits. Furthermore, based on the results in this study, it can be concluded that the Grapefruit Australia is Grade A, the Grapefruit South Africa is Grade B, and Valencia or Navel orange is Grade C. This study can be improvised or improved using other techniques or fruits.

Acknowledgements

Faculty of Architecture, Planning, and Surveying, Universiti Teknologi MARA (UiTM), Research Management Centre (RMC) and Ministry of Higher Education (MOHE) are greatly acknowledged for providing the Fundamental Research Grant Scheme (Title: Scale Invariant Feature Transform (SIFT) and Speeded Up Robust Feature (SURF) Modelling In Fulfilling Multispectral Object Reconstruction, Grant No. FRGS/1/2021/WAB07/UITM/02/2) and GPK fund (Grant No. 600-RMC/GPK 5/3 (223/2020)) to enable this research to be carried out. The authors would also like to thank the people who were directly or indirectly involved in this research.

References

Banot, M. S. and Mahajan, P. M., 2016, A Fruit Detecting and Grading System Based on Image Processing-Review. International Journal of Innovative Research in Electrical, Electronics, Instrumentation and Control Engineering , Vol. 4(1), 47-52.

Ghuman, S. S., 2016, Clustering Techniques- A Review. International Journal of Computer Science and Mobile Computing, Vol. 5(5), 524–530.

He, J. Q., Harrison, R. J. and Li, B., 2017, A novel 3D Imaging System for Strawberry Phenotyping. Plant Methods, Vol. 13(1), https://doi.org/https://doi.org/10.1186/s13007-017-0243-x.

Ji, S., Ren, Y., Ji, Z., Liu, X. and Hong, G., 2017, An Improved Method for Registration of Point Cloud. Optik, Vol. 140, 451-458.

Rahaman, H. and Champion, E., 2019, To 3D or Not 3D : Choosing a Photogrammetry Workflow for Cultural Heritage Groups. Heritage 2019, Vol. 2(3), 1835-1851.

Thomas, A. S., Hassan, M. F., Ibrahim, M., Nasrull, M., Rahman, A., Sapuan, S. Z. and Ahmad, F., 2019, A Study On Close-Range Photogrammetry In Image Based Modelling And Rendering (IMBR) Approaches and Post-Processing Analysis. Journal of Engineering Science and Technology, Vol. 14(4), 1912–1923.

Vázquez-arellano, M., Griepentrog, H. W. and Paraforos, D. S., 2016, 3-D Imaging Systems for Agricultural Applications-A Review. Sensors, Vol. 16(5), 1-24, https://doi.org/10.3390/s16050618.

Zhang, Y., Teng, P., Shimizu, Y., Hosoi, F. and Omasa, K., 2016, Estimating 3D Leaf and Stem Shape of Nursery Paprika Plants by a Novel Multi-Camera Photography System. Sensors, Vol. 1 (6), 1-18, https://doi.org/10.3390/s16060874 .

Zhou, Y., Daakir, M., Rupnik, E. and Deseilligny, M. P., 2019, Improvement of Photogrammetric Accuracy by Modeling and Correcting the Thermal Effect on Camera Calibration. ISPRS Journal of Photogrammetry and Remote Sensing, Vol.148, 142-155.