Classification of external defects on soybeans based on color and fluorescence image features
2024 Volume 5 Issue 2 Pages 10-21
Details
2024 Volume 5 Issue 2 Pages 10-21
Accurate soybean sorting is an important but time- and labor-intensive process in soybean production. Therefore, inexpensive and accurate sorting machines are needed. Many of the currently reported models for discriminating external defects in soybeans have used deep learning, but the high cost of deep learning is an issue. Therefore, this study aimed to discriminate external defects in soybeans using fluorescent images in addition to color images, using a model that is less expensive than deep learning. Color images and fluorescence images of soybeans at an excitation wavelength of 365 nm were taken, and visually labeled into six categories: normal, wrinkled, peeled, pests, denatured, and insect-damaged grains. Image features were extracted for both color and fluorescence images, and classification models were constructed using Support Vector Machine (SVM) with three input patterns: (a) color image features alone, (b) fluorescence image features alone, and (c) color and fluorescence image features input. The test accuracy was 75.06%, 58.28%, and 76.91%, respectively. In addition, two and four classifications were devised in order to anticipate the needs of the field. The accuracy reached 82.06% and 94.99% for four category and two category simultaneous input of color and fluorescent images, respectively. Especially in the two categories, the discrimination accuracy exceeded 90%, indicating that a highly accurate discrimination model could be created. Furthermore, as a result of visualizing the features important for discrimination using the ablation study, it was found that fluorescent images were effective in addition to color images for discrimination in the six categories. The importance of shape information, namely, perimeter and circularity, was high in the discrimination model for all categories, indicating that shape information is the most important information for discriminating external defects in soybeans. From those results, it can be concluded that the combination of conventional color images and fluorescence images is effective for classifying soybean external defects.
Soybeans are a highly nutritious agricultural crop, rich in vegetable protein and fat. In particular, in terms of protein, soybeans have high quality protein with a good balance of essential amino acids. In addition, about 30% of the dry weight of soybeans is protein, which is far more than that of other crops1). Therefore, soybeans are used in a wide range of fields, from food to feed. In recent years, soybean production and consumption have been on the rise in recent years against the backdrop of a growing world population, especially in non-meat-eating regions2). On the other hand, global soybean inventories have been on a downward trend for the past five years, and the supply of soybeans continues to be insufficient to meet demand3). Global food shortages are expected to continue in the future as the world’s population increases, and there is a need to further improve soybean productivity and reduce losses in the production process.
In Japan, in particular, there are more varieties of processed soybean foods than in other countries, and many processed soybean foods have long been popular as common home cooking, including nimame (boiled soybeans), natto (fermented soybeans), shiro-miso (white miso), and bean confectionery, all of which emphasize appearance quality. Since almost all domestic soybeans are used for food products in Japan, the soybeans need to be sorted according to various processed soybean products. For these reasons, the sorting process is more important in Japan than in other countries, and realizing accurate sorting process will lead to reducing losses in the entire production process. After soybeans are harvested, they are dried for several weeks, threshed, and sorted before shipping.
The threshing process has been mechanized by farmers due to the relatively low cost of electric threshers.
In the sorting process, there are currently two types of mainstream sorting machines: mechanical sorting machines and color sorting machines. Mechanical sorting machines sort soybeans based on the grain size and shape, making it possible to select only rounded-shaped soybean grains. On the other hand, color sorting machines sort by the color of the grain coat, which detect normal grains, pests grains, and denatured grains. The color sorters are useful for sorting out external defects in soybeans. However, they are large and expensive, which are still difficult to be intriduced in small-and medium-scale farmers. Therefore, in the current situation, small- and medium-scale farmers bring their soybeans to large- scale farmers who have sorting machines, which often leads to the contamination of various soybean varieties. To solve these problems, the inexpensive and highly accurate sorting machine is needed.
Previous studies on the classification of external defects in soybeans have included the construction of deep learning models using color images as input and deep learning models using transmission images, both of which have been reported to be able to classify external defects with an accuracy of 90% or higher4)5). However, both of these methods have the disadvantages of conputation costs and data size due to the use of deep learning. In addition to color images, multispectral images from visible to near- infrared have been used for the early detection of strawberry spoilage6). More recently, it has been shown that changes in fluorescence response due to ultraviolet excitation can acquire information on fluorescent substances that cannot be captured visually and can detect minute scratches and defects on the surface of agricultural crops. In a previous study, Gomes et al. (2019) reported monitoring the in vivo absorption and transport of ZnO and ZnSO4 in soybean leaves using X-ray fluorescence and X- rays7). Other imaging methods besides color imaging include short-wavelength infrared imaging and hyperspectral imaging, but both are not suitable for implementation in inexpensive equipment due to the high cost of cameras and the huge amount of data. In previous studies using fluorescence images, Huang et al. (2022) reported that it is possible to detect rot and blemishes in strawberry6), and Konagaya et al. (2020) reported that it is possible to discriminate quality deterioration in tomatoes8). Thus, not only information from RGB color images, but also fluorescence images are premissing as the input for predicting soybean quality; by simultaneously acquiring and analyzing information from RGB and fluorescence images, a lower cost and more accurate model may be realized.
Therefore, the objective of this study was to classify the external defects on soybeans using fluorescence images with color images. Not only the usual three-channel RGB color images using a white light source, fluorescence images excited by a UV LED with a wavelength of 365 nm were taken in this study. A total of 49 image features were obtained from the obtained color and fluorescence images. We employed a typical machine learning model, Support Vector Machine (SVM), to compare the accuracy of the model with the three patterns of input information: input of only image features for color images, input of only image features for fluorescence images, and simultaneous input of image features for color and fluorescence images.
(1) Soybean samples
A Japanese soybean cultivar (‘Toyokomachi’) harvested in October 2022 at the Field Science Education and Research Center (Muramatsu Station), Faculty of Agriculture, Niigata University, was used as the experimental material. After harvesting, foreign matters such as pods, branches, and dust were removed, and the grains were dried at room temperature to a moisture content of about 15%. The weight of the grains was about 800 g. A total of 3593 soybean grains were provided in this study. As a pretreatment for imaging, the soybeans was manually sorted to classify the external defects. The external defects of soybeans were divided into six categories: normal, wrinkled, peeled, pests, denatured, and insect-damaged grains. Fig.1 shows examples of typical soybean images for each category.
(2) Selection of excitation wavelength based on excitation emission matrix
As a preliminary experiment for this study, an excitation emission matrix (EEM) of soybean epidermis was measured to find the optimum excitation wavelength for classification of soybean defect types. It is called a fluorescence fingerprint and can characterize substance-specific information. A previous study has used the EEM to estimate crude protein and lipid content in soybeans9). In the report, the main peaks of soybean powder appeared in the excitation wavelength range of 200-500 nm and the emission wavelength range of 220-600 nm. Following the previous study, a spectrofluorometer FP-8350 (JASCO, Japan) was used in this study. Three representative soybean grains were selected in each category, which resulted in a total of 6×3=18 soybeans. The measurement wavelengths were set to 200-735 nm for excitation and 220-600 nm for emission to cover the main fluorescence peaks reported in previous studies9). The band width was set to 5 nm for both excitation and emission, and the detection sensitivity was set to low to avoid saturation of the fluorescence intensity. To correct for changes in the sensitivity of the instrument during the measurement, the measured values were corrected using the integrated intensity of Raman scattering at emission wavelengths of 371-428 nm with 350 nm excitation for each measurement date10).
(3) Image acquisition
The imaging system constructed for this study is shown in Fig.2. Fig.2(a) and Fig.2(b) show the schematic diagrams of the imaging system from the front and the side view, respectively. A white LED (LDL2-80X16SW2, CCS, Japan) was used to capture color images, and a UV LED (LDL-71X12UV2-365- N, CCS, Japan) was used to capture fluorescence images. When capturing color images, a polarizing filter was placed in front of both the white LED and camera lens to remove specular reflection from the soybean surface. In addition, an UV-cut filter with a cutoff wavelength of 390 nm was attached to the camera lens to prevent UV reflected light from being detected in the fluorescence images. A single-lens reflex color camera (EOS kiss x7, Canon) was used, and a macro lens (DG MACRO, SIGMA) with a focal length of 70 mm was attached on the camera to achieve high image resolution.
The color and fluorescence images of soybean grains were taken with the imaging setup shown in Fig.2. A total of 24 soybeans (6 vertical and 4 horizontal) were uniformly placed in an image. The f-number and ISO sensitivity were set to 5.6 and 800, respectively. The shutter speed for color and fluorescence images was was 1/30 and 1/20 sec, respectively. The original image size was 5184 × 3456 pixels. Before the imaging process, the soybean images were visually labeled into six categories: normal, wrinkled, peeled, pests, denatured, and insect-damaged.
(4) Image analysis
MATLAB (MATLAB R2022a, MathWorks, USA) and a laptop computer (HM750PAW, LAVIE, NEC, Japan) were used for the analysis in this study. After capturing 24 soybean images of 6 × 4 (vertical × horizontal) using the image capture device shown in Fig.2, Otsu’s binarization11) was applied to the color images. And then, background noise other than soybeans was removed, and a mask was created by filling holes and merging regions on the grains. The mask created for the color image was also applied to the fluorescence image, and bounding boxes were created for each grain, which were used to crop the image of each grain. The cropped soybean images were labeled into six categories: normal, wrinkled, peeled, pests, denatured, and insect-damaged grains, and the same grains were named so that the color and fluorescence images corresponded. The final number of images obtained for each category is shown in Table 1.
In this study, these named images are henceforth referred to as pre-processed images. Image features were then acquired for each grain in each category from the preprocessed images. There were three types of image features acquired: color information, shape information, and Haralick texture information. The total number of image features acquired was 49 for both color and fluorescence images. A list of the features acquired from the image analysis is shown in Table 2. Texture features are image textural information that quantitatively measures the features that visually capture the form of pattern or the appearance of an object. Texture features have been widely used in the medical field, especially in MRI images and observation images using electron microscopes, because of their good compatibility with biological images. Recently, however, it has been found to be effective in the food field as well, and has been used for identification of citrus disease, digital images of bread, and electron microscope images of cream cheese12)13)14).
(5) Selection of the best model
In this study, common supervised learning classifiers include SVM, decision trees, and ensemble learners were investigated to select the optimal classification model for classification. As a method for selecting the optimal classifier, previous studies have reported the selection of the optimal classification model using the “Classification Learner” application in MATLAB15). Therefore, in order to select the optimal model for this study as well, the “Classification Learner,” an application in MATLAB (MathWorks, 2023a), was used to select the optimal classification model. The procedure for selecting a classification model using the “Classification Learner” is described below.
First, we need to prepare in advance a known input-output data set and labels for the data set to run the application. Therefore, we provide the training classifier with a dataset of all features (49 variables) for both color and fluorescence images of soybeans and the correct label for the classification in each soybean image. Next, we adopted a 5-fold cross- validation as the validation method to avoid over fitting. And then, the classification model was automatically trained on the given data with a decision tree, SVM, and ensembler classification model. The classification learner outputs a score for each model’s specified validation method after the model is automatically trained. In this study, the average accuracy of the 5-fold cross-validation was obtained.
(6) Selection of optimal offset
Offset is an important parameter for Haralick texture features. Gray-Level Co-Occurrence Matrix (GLCM) is first generated to calculate Haralick texture features. GLCM represents a histogram that captures the frequency of co-occurrence of two-pixel value at a certain offset: inter-pixel distance and orientation. In particular, the inter-pixel is distance between pixels, which has a significant effect on the value of the Haralick texture features. If the value of the distance between pixels is too small, the gray level differences will be barely noticeable. On the other hand, too large a value for the inter-pixel distance ignores information in regions close to the pixel of interest. If the offset is not selected appropriately, the GLCM will not be able to adequately represent the Haralick texture features of the area of interest in the target image. Therefore, the selection of appropriate offsets is necessary.
In this study, to select an appropriate offset, the following three steps were used: (i) the effect of inter- pixel distance on the contrast of Haralick texture features, (ii) acquisition of Haralick features under six offset conditions, and (iii) Comparison of the accuracy of the second-order linear SVM classification using the Haralick features as input under the six offset conditions.
(7) Selection of optimal number of gradation
A previous research has reported that the Haralick texture feature varies with the number of gradations in an image16). The number of image gradation represents the number of color information in an image, and generally, the images we see have 256 gradations. In this study, Haralick texture features for both color and fluorescence images were prepared with different number of gradations to select the most effective number of image gradations at inter-pixel distance of 15. Four patterns were used for the number of gradations: 32, 64, 128, and 256.
(1) Preliminary experimental results
The EEM results are shown in Fig.3. All EEMs shown in Fig.3 were correcred by the fluorescence intensity of water Raman scattering, in which the unit is Raman unit (R.U.). As shown in Fig.3, normal grains show strong peaks around excitation wavelength (Ex): 350-400 nm and emission wavelength (Em): 450-500 nm, while pests and denatured (black) grains showed little fluorescence in the same wavelength range. The fluorescence peak at Ex365 nm/Em450-500 nm is supposed to emit bluish white fluorescence in the fluorescence image. This peak is considered to be related to oil and Maillard compounds9). Therefore, it is possible that the oil and fat compounds in the soybean epidermis are responsible for the fluorescence. In the denatured grains (white), a strong peak is observed around Ex250-300 nm /Em300-350 nm. One of the chemical compounds of this peak has been suggested to be aromatic amino acids9).
The white portion of the denatured grains (white) is generally considered to be derived from mold, which is the cause of the white alteration. The fluorescence image and EEM results are also consistent with each other, since only normal grains and denatured grains (yellow) showed fluorescence at Ex365nm. Therefore, in this study, the excitation light at 365 nm, which is used in various fields and can be introduced inexpensively, was adopted for fluorescence imaging because it shows the most difference between normal grains, pests grains, denatured grains (black), denatured grains (white), and insect-damaged grains.
(2) Selection of optimal learning model
Table 3 shows the result of classification accuracy with different machine learning models obtained from the 5-fold cross validation. It was found that the quadratic linear SVM has the highest average accuracy than any other models. Therefore, the quadratic linear SVM was selected for classifying the external defects of soybeans for the following analysis. In this case, Box Constraint was set to 1 and Kernel Scale was set to auto. The auto for Kernel Scale was calculated using Heuristic method to calculate the appropriate scale factor.
(3) Selection of optimal offset
(i) Effect of inter-pixel distance on the contrast of Haralick texture features
To investigate the effect of inter-pixel distance on the “Contrast” value of Haralick texture features, representative images were first selected from each external defect category. Then, the “Contrast” value, which is a representative feature of the Haralick texture features, was calculated for each offset between 1 and 50 pixels for the selected images. The results of the “Contrast” values are shown in Fig.4. In Fig.4, the vertical axis represents the “Contrast”, the horizontal axis represents the inter-pixel distances, and the legend is the label of soybean external defect. As shown in Fig.4, in the color image, the variance of “Contrast” between each category is the largest around inter-pixel distance of 15. In addition, there is almost no difference in the “Contrast” value between each category at around 1 and 5 inter-pixel distances. This is thought to be because the distance between pixels is too small to represent differences in external defects. The difference of each category is also small at around 45 and 50 inter-pixel distance, which is because the features of external defects have been jumped over. On the other hand, in the fluorescence image, the difference between the categories is still seen at 15 pixels, but the difference is larger at 50 pixels. This may be because external defects are more emphasized in the fluorescent image than in the color image, and the range of external defects is larger in the fluorescent image. The results show that the difference in contrast values for each category is greatest when the distance between pixels is 15 for the color images and 50 for the fluorescence images.
(ii) Haralick feature acquisition
Based on the results of (i), six patterns of inter- pixel distances for offsets were selected: 1, 5, 10, 15, 20, and 50 pixels. The offsets made from the selected inter-pixel distances are shown in Table 4. The averaged and standardized values of the GLCMs in the four directions for each offset from No.1 to No.6 in Table 4 were used to calculate the Haralick texture features, and the 14 features were calculated from each of the color and fluorescence images.
(iii) Accuracy comparison of second-order linear SVM with Haralick texture features as input under six offset conditions
To select the most effective offset for classifying the external defect types of soybeans among the Haralick texture features using the six patterns of offsets obtained in (ii) for the selected quadratic linear SVM, Haralick texture features for both color and fluorescence images using each offset (28 variables) were prepared. The Image feature data consisting of 28 variables were prepared for each of the color and fluorescence images, and the supervised data for the six offset patterns were input to a second- order linear SVM to compare the accuracy. In addition, since the purpose of this study was to search for effective Haralick texture feature offsets for both color and fluorescence images, the Haralick texture feature data (28 variables) was prepared with unified offsets for color and fluorescence images.
The accuracy of the second-order linear SVM using Haralick texture feature data with each output offset of both color and fluorescence images as input is shown in Table 5. Table 5 shows that the average accuracy of the quadratic linear SVM is maximum at a inter-pixel distance of 15 pixels for the offsets.
Based on the results of (i), (ii) and (iii) above, the Haralick texture feature with a inter-pixel distance of 15 pixels at the offset was selected as the input data for the Haralick texture feature.
(4) Selection of optimal number of gradations
The accuracy of the quadratic linear SVM with Haralick texture feature data using each number of shades as input was shown in Table 6. As shown in Table 6, the average accuracy of the quadratic linear SVM is maximum at 256 shades for the offset inter- pixel distance of 15. Since the average accuracy also decreases with the decrease in the number of shades, it can be said that the Haralick texture features accurately represent the differences in external defects in the discrimination of external defects in soybeans with a larger number of shades. Based on the above results, the Haralick texture features obtained from the image with 256 gradations and a 15 inter-pixel distance were selected as the final input data.
(5) Classification result of soybean external defects
Three patterns of input data were created from the data obtained under the selected conditions: (a) color image features alone, (b) fluorescence image features alone, and (c) color and fluorescence image features. The results of discriminating the external defect type of soybeans by SVM using these (a), (b) and (c) as input data are shown in Fig.5. The figures are a confusion matrix where the vertical axis represents the actual labels and the horizontal axis represents the predicted labels. It is a prediction result for the test data. The discrimination accuracies in Fig.5 were (a) 75.06%, (b) 58.28%, and (c) 76.91%, respectively, and the accuracy was highest for (c) the simultaneous feature input model for color images and fluorescence images. The Recall and Precision for the “normal” label in model (c) were 78.4% and 80.0%, respectively.Here, “recall” is the “ratio of the number of predicted normal to the number of actually normal” and “precision” is the “ratio of the number of actually normal to the number of predicted normal”. In all models, there were many cases of misclassification between normal grains and wrinkled grains, between peeled grains and wrinkled grains, and between pests grains and denatured grains. Insect-damaged grains were the least accurate in all models, but this was because the number of samples for insect-damaged grains was much smaller than for the other categories. Three categories were used in this study to examine differences in the fluorescence properties of external defects: pests, denatured and insect damage, all of which are generally discarded. Therefore, they do not need to be classified in actual soybean sorting process. For this reason, we constructed an external defect discrimination model for soybeans in four categories, where pests, denatured, and insect-damaged grains were classified together as defect grains, and investigated how much the accuracy would increase. In the four category discrimination, the discrimination was performed with the same input as (c) color and fluorescence image features, which was the most accurate of the three input patterns in the six category discrimination. The discrimination results are shown in Fig.6.The model and parameters used in the discrimination are the same as those used in the six categories. The discrimination accuracy of Fig.6 was 82.06%. The discrimination accuracy was 5.15% higher than that of the six category model. The elimination of misclassification between pests and denatured grains, which were often misclassified, is thought to have contributed to this accuracy. Furthermore, When soybeans are sorted in the field, it is also necessary to classify soybeans according to their intended use. While the classification of normal grains, peeled grains, and wrinkled grains is necessary for products that make use of the shape of soybeans, such as black beans and cooked beans, classification is not necessary for applications in which soybeans are crushed, such as tofu, soybean paste, and soy sauce. Therefore, we constructed the classification model for soybeans with two categories of external defects, where the three categories of normal, peeled, and wrinkled grains are classified as “well formed” and the three categories of pests, denatured, and insect damaged grains are classified as “defect” to see how much the accuracy increases. In the two category discrimination, the discrimination was performed with the same inputs as in (c), the input model with the highest accuracy of the three input patterns for the six category discrimination. The discrimination results are shown in Fig.7. The accuracy of Fig.7 was 94.99%. It increased by 18.08% compared to the six category model. Furthermore, all previous studies on the use of deep learning for the discrimination of external defects in soybeans have had a discrimination accuracy of more than 90%4)5). This study found that by using fluorescence images in addition to color images and reducing the number of categories to two, discrimination accuracy comparable to deep learning discrimination models could be achieved.
(6) Misclassified images
In this study, misclassified soybean images were extracted. Fig.8 shows typical examples of misclassified images obtained by six category discrimination of the (c) input model for color and fluorescence images. In the image where a normal grain was misclassified as a wrinkled grain, the wrinkles on the contour of the soybean, which are characteristic of wrinkled grains, were not seen. Therefore,the reason for the misclassification could be that there were not enough valid features for wrinkled grains. Similarly, in the image where the peeled grains were misclassified as normal grains, the peeled skin, which is a characteristic of peeled grains, was seen in the center of the soybean, but the image was misclassified, suggesting that there were not enough valid features for the peeled grains. On the other hand, in the image where the denatured grains were misclassified as pests grains, the center of the soybean was blackened, especially in the fluorescence image, which may have led to the misclassification. Many of the appearance characteristics of denatured grains and pests grains are similar, and this is thought to be the reason for the increased misclassification between pests and denatured grains. In the image where the peeled grains were misclassified as wrinkled grains, both peeled and wrinkled grains were present the subject soybeans. The fact that many of the peeled grains had wrinkles at the same time may have caused the increased misclassification between peeled and wrinkled grains.
In addition, there were a few cases where normal grains were misclassified as pests grains, although the number was small. Images of normal grains misclassified as pests are shown in Fig.9. Fig.9 shows an image of soybeans that were manually sorted as normal grains. In the color image, the grains appear normal, while in the fluorescence image, black spots appear on the right side of the soybeans.It is bleived that these black spots were judged to be diseased and misclassified as pests grains. In several cases, the fluorescence image was misclassified as pest grains because such invisible information was read from the fluorescence image. EEM measurement results show that fluorescence due to oil and fat compounds is observed at Ex 365 nm/Em 475 nm. Therefore, the soybeans in the images where normal grains were misclassified as pests grains may have been inactivated by some effect on the oil and fat compounds in the soybean epidermis. In addition, previous research has shown that the internal components of soybeans, such as carbohydrates, change when affected by Purple Blotch, a common cause of pests and denatured grains. The research has also shown that the protein and lipid content of soybeans correlates with the severity of purple blotch disease. These results suggest that fluorescence may be able to detect early disease that cannot be visually confirmed.17).
(7) Consideration of feature importances
Ablation studies have been conducted with the objective of identifying the features that are crucial for machine learning, such as SVM18). In the ablation study, some of the features used in the training model are removed and re-trained, and the difference between the training model with all features and the training model with some of the features removed is calculated19). Ablation studies were therefore conducted with the objective of identifying the most significant features of the following three learning models: (a) color image features alone, (b) fluorescence image features alone, and (c) color and fluorescence image features discriminated into six categories. In this study, 48 variable inputs were prepared by removing one feature from the data of 49 different features to calculate the importance of the feature. Then, 49 patterns of data were created, each data was divided into training data and test data as in the previous learning models, and the accuracy of the model learned using the training data with respect to the test data was calculated. The difference between the accuracy of the six category discrimination model and the accuracy of the training model using the validation data was then visualized as the importance level. Fig.10 shows a graph visualizing the top five feature importance levels for each of the 6 input models and the six category discrimination model. As shown in Fig.10, (a) in the input model for color image feature values alone, the top five include the three shape information of perimeter, circularity, and eccentricity, indicating that shape information is considered important for discrimination. In addition, the highest feature importance, perimeter, decreased the discrimination accuracy by more than 3% of the discrimination accuracy of all feature data, indicating the strong contribution of shape information to the discrimination. On the other hand, in the (b) fluorescent image single input model, H(fl) is more important than the other features. In addition to the color information, the Haralick texture feature is among the top 5 features in the (b) fluorescent image single input model, indicating that it contributes to the discrimination. In (c), the model with the highest discrimination accuracy, only the shape information and the Haralick texture feature are in the top 5, indicating that these two pieces of information are important. (b) The fluorescent image input model alone was overwhelmingly less accurate than the other input models, so it was expected that the features obtained from the color image would occupy the top 5 positions. However, (c) In the feature importance of the simultaneous input model for color and fluorescence images, the Haralick texture features obtained from fluorescence images ranked 4th and 5th. This indicates that the discrimination accuracy of the fluorescence image alone was not high, but adding the information of the fluorescence image to the information of the color image improved the discrimination accuracy, and the fluorescence image contributed to improving the discrimination accuracy of the training model. These results indicate that while shape information is useful for discriminating the six categories, the addition of Haralick texture features in the fluorescence images further improves the accuracy.
Feature importance was also calculated for four category discrimination and two category discrimination. Fig.11 shows a graph visualizing the top five feature importance levels in the four and two category discriminant models. As shown in Fig.11, the feature importance in the four and two category discrimination model is similar in that the shape information and Haralick texture features are highly important. In addition, the top five feature importance values in the four and two category discrimination models are almost exclusively information from color images, indicating that information from color images is effective. These results indicate that in the six category discrimination model, fluorescence images are effective in addition to color images. In addition, the importance of perimeter and circularity of shape information is high in the discriminant model for all categories discrimination model, indicating that shape information is the most important information for discriminating external defects in soybeans. The Haralick texture features were the most important among the top 5 features other than shape information, and among the 14 features, the Entropy and Correlation features were the most frequently found. Entropy is a value that quantifies the distribution of pixel pair combinations in the GLCM, and it increases as more combinations are observed. Correlation is a measure of the degree of linear relationship between pixel pairs in the GLCM20). Both are indicators of the uniformity of pixel pairs in the GLCM, and image uniformity was important in distinguishing external defects in soybeans. In addition, a trend was obtained in which the information in the color image contributed significantly when the number of categories was reduced. This indicates that fluorescence images are effective in discriminating external defects of soybeans in detail, while the effectiveness of fluorescence images decreases and information from color images becomes more important when classifying external defects in large categories.
In this study, we aimed to construct a model with accuracy comparable to deep learning models by using machine learning, which is a lighter model with more input information. First, characteristic peaks at excitation wavelengths of 350 - 400 nm and fluorescence wavelengths of 450 - 500 nm were observed by excitation fluorescence matrix (EEM). Based on the results, we constructed a fluorescence imaging system with an excitation wavelength of 365 nm and acquired color and fluorescence images. Six categories were classified by dividing the input data into three patterns: (a) color image features alone, (b) fluorescence image features alone, and (c) color and fluorescence image features input model. The discrimination accuracy was 75.06%, 58.28%, and 76.91%, respectively. The input model with the highest accuracy was (c). Considering the needs of the field, we also classified the data into two and four categories. The accuracy was 82.06% for the four category model and 94.99% for the two category model, indicating that the discrimination accuracy increased for both the four category and two category models. In particular, for the two category model, the discrimination accuracy exceeded 90%, indicating that a highly accurate discrimination model could be created. In addition, some normal grains showed a significant change in appearance between the color and fluorescence images, suggesting a new possibility that fluorescence can detect early-stage diseases that cannot be confirmed visually.
When we verified the same conditions with another variety, the accuracy against the test data was 79.17%, which is almost the same level of results, and we believe that the method can be applied to other varieties as well. However, since there are many varietal differences in appearance quality in soybeans, such as seedling size, flattening rate, and seedling color, it is necessary to verify the accuracy when multiple varieties are mixed together as a future prospect.
