Development of High Accuracy Segmentation Model for Microstructure of Steel by Deep Learning

Abstract

We studied on automation of segmentation using deep learning, which has been remarkably developed in recent years.

For the microstructural image of ferrite-martensite dual phase steel, we tried to segment the ferrite phase, martensite phase, and ferrite grain boundary in different colors individually. We created two models, SegNet and U-Net that can perform segmentation with high accuracy and compared the accuracy with an existing method.

As a result, we demonstrated that models using deep leaning is more accurate than the existing method. In particular, U-Net model shows highly accuracy of segmentation for material microstructures.

1. Introduction

Development of new materials requires a large number of microstructural analyses. Most of these processes require much time because they are done manually or semi-automatically. Thus, it is important for high-efficiency material development to automate the present microstructural analysis process.

Above all, applying different colors to each phase of the microstructural image (hereafter, called “segmentation”) is time-consuming. Electron back scattered diffraction pattern (EBSD) and Trainable Weka Segmentation (TWS) built into ImageJ^1,2) have been used as segmentation methods for microstructures. EBSD is capable of segmenting based on crystal orientation.^3,4) However, this method takes much time for imaging and the accuracy of segmenting a region with high dislocation density is insufficient.

In contrast, TWS can segment more easily, and the accuracy of segmentation is higher than for EBSD.⁵⁾ However, filters to be applied in TWS are predesigned, so that it does not guaranteed to perform best segmentation for a given image.^6,7)

There are two main reasons why segmentation of microstructures is difficult. First, some knowledge of the material is required because it is necessary to comprehensively analyze not only the brightness value of the image, but also morphological information of the crystal grain. This suggests that automation of segmentation with a simple algorithm is difficult. For instance, in TWS, about 20 image-filters are implemented, although it is not guaranteed to extract sufficient microstructural features.

Another is that a robust algorithm for external noise, such as image capturing conditions, has not been fully developed. An optical microscope or a scanning electron microscope is generally used for microstructural observation, but the brightness of a micrograph may depend on the conditions of the observation. Therefore, some pre-processing is required for the segmentation model using TWS. Such noises can lead to loss of the generalizing capability of the segmentation model. Here, we worked on utilization of deep learning (DL) to solve this problem.

DL has been well-developed and applied in various fields. For instance, DL is the most prominent object recognition technology,^8,9) enabling recognition of human faces,^10,11) characters,^12,13) and speeches.¹⁴⁾ In other words, DL has succeeded in automating tasks that are difficult to quantify. This feature of DL is also expected to be applied to material engineering, and there have already been some reports of DL in the field of numerical analysis in material engineering.^5,15,16)

The DL algorithm seems attractive but is not universal. It has been reported that DL can generally create a model with high accuracy by using a large amount of training data.^17,18) However, it is difficult to prepare a large amount of training data in material development. Although a method to avoid this problem has been reported,¹⁶⁾ it is not easy, so we wanted to create an easy segmentation model using a DL algorithm that requires less data.

Thus, we worked on two projects: first was the creation of a segmentation model that can identify not only the brightness value, but also the morphological information of a crystal grain by DL. Another was to verify how much training data can be reduced by an algorithm that can train with a small amount of data.

2. Deep Learning Algorithms

FCN is known as a DL algorithm¹⁹⁾ that can perform regional segmentation, and it can classify to which class an image belongs in units of pixels. Furthermore, FCN is used in various fields because it can solve the problems of existing methods.^20,21) FCN has been improved since its first report, and its internal structures have been optimized according to the application field.^22,23,24)

Using FCN such as SegNet²²⁾ and U-Net,²³⁾ we created segmentation models that are likely to be useful in microstructural analysis.

2.1. SegNet

SegNet is a network structure reported at the IEEE/CVF International Conference on Computer Vision and Pattern Recognition (CVPR) in 2015, and it has an encoder-decoder structure (Fig. 1). The encoder unit abstracts the image information and reduces the image size by performing a process called convolution⁸⁾ and pooling,^8,25) and the decoder unit adjusts the pixel position by restoring the reduced image size. Thus, SegNet enables one to learn the morphological information because information of the adjacent pixels is used when abstracting information in the encoder unit. Moreover, SegNet has also been reported to a model with reduced computational complexity, and it is attractive in that the model does not depend on a computational resource.²²⁾

Fig. 1.

Network structure of SegNet.²²⁾

2.2. U-Net

U-Net is also a network structure reported at CVPR in 2015, whose basic idea is the same as SegNet. The difference between U-Net and SegNet is whether the encoder and decoder units are connected or not (Fig. 2). Thus, microstructures can be recognized with high resolution because U-Net enables one to use information prior to abstraction when segmenting an image. In addition, there are many applications utilizing this feature in the medical field.^23,26,27)

Fig. 2.

Network structure of U-net.²³⁾

3. Methods

As shown in Figs. 3 and 4, an optical microscope image of a typical microstructure in dual-phase (ferrite and martensite) steel (hereinafter, called the “original image”), and an image in which each phase was separately painted, were prepared (hereinafter, called the “teacher image”). Details of the two images are shown in Table 1. The original image was an RGB image and each RGB value ranged from 0 to 255. On the other hand, the teacher image was expressed by three types of brightness values, which represented ferrite, martensite, and grain boundary.

Fig. 3.

An original image. (Online version in color.)

Fig. 4.

Teacher image.

Table 1. Properties of images.

	Original Image	Teacher Image
Number of Photos	50	50
Size (px)	1536×1536	1536×1536
Range of Brightness	RGB 0–255	3 types

In this study, 40 microstructural images were used for the training model (hereinafter, called the “training data”), and 10 microstructural images were used for evaluation of model accuracy (test data).

3.1. Segmentation by Threshold

We focused on the segmentation method using a certain threshold to evaluate the validity of SegNet and U-Net. Brightness value of each phase was obtained from the training data, and the segmentation for microstructure was carried out using the obtained brightness value. In particular, we tried the segmentation method, applying different colors to each phase by selecting a certain value based on the average and scattering of the brightness values of each phase.

3.2. Segmentation by TWS

While image filters are self-studied in DL-based image processing such as SegNet, predesigned filters are used in TWS. This means that training to find best filters is not required, thus image processing by TWS is relatively speedy compared with DL-based image processing. However adapted image filters are not sometimes best ones for a given image, so that it is not enough to detect fine particles or narrow grain boundaries as shown in Fig. 5. In contrast, as described later, DL-based image processing, in particular U-Net has an advantage in segmentation accuracy.

Fig. 5.

(a) Ferrite-martensite microstructure, (b) labeled image by TWS (red: martensite, green: ferrite grain boundary, yellow: ferrite interior).

3.3. Segmentation by Deep Learning

We created two models, one each trained using SegNet and U-Net.

3.3.1. Quantitative Evaluation of Each Model

Matching area ratio of the segmentation image created by each model (output image) and teacher image were evaluated to check accuracy. It was possible that the segmentation model tended to judge the phase as ferrite because a major portion of the image used in this experiment was ferrite. Therefore, we evaluated the accuracy of segmenting each phase using the following Eq. (1):   

$$accuracy=\frac{{{\displaystyle \sum }}_i{{\displaystyle \sum }}_j\delta \left(i,j\right)}{i*j}$$(1)

  

$$where\delta \left(i,j\right)\{\begin{array}{c}1\mathrm{\hspace{0.17em} }if\mathrm{\hspace{0.17em} }{O}_{i,j}=T_{i,j}\\ 0\mathrm{\hspace{0.17em} }if\mathrm{\hspace{0.17em} }{O}_{i,j}\ne T_{i,j}\end{array}$$

  

$$\begin{array}{l}where{O}_{i,j}:\\ abrightnessvalueofcoordinate\left(i,j\right)\in outputimage\end{array}$$

  

$$\begin{array}{l}where{T}_{i,j}:\\ abrightnessvalueofcoordinate\left(i,j\right)\in teacherimage\end{array}$$

This evaluation was performed for 10 images because Eq. (1) represents the accuracy for one test image, and the average value obtained from 10 times evaluation was defined as the accuracy of the model.

3.3.2. Qualitative Evaluation of Each Model

An image mapping the correct and incorrect parts (accuracy map) was created by taking the difference in brightness value of each pixel between the output and teacher images. The accuracy map enabled visualization of the advantage and disadvantage of each model. In addition, the brightness values in the matched and unmatched regions between the output and teacher images were set as 0 and 255, respectively. Therefore, the matched and unmatched regions were black and white, respectively.

3.3.3. Number of Training Data Required to Train a Deep Learning Model

The required number of images to train U-Net model was estimated by training the model with different numbers of training data.

4. Results

4.1. Segmentation by Threshold

Average, scattering, and standard deviation of the brightness values in each phase are shown in Table 2. Brightness value histograms of each phase are shown in Fig. 6.

Table 2. Statistical data of each phase: brightness values.

	Ferrite	Grain Boundary	Martensite
Mean	161.81	131.32	98.41
Variance	529.58	552.22	307.51
Standard deviation	17.53	23.49	23.01

Fig. 6.

Brightness value histogram.

As shown in Table 2 and Fig. 6, it was difficult to separate the phases using the threshold because the distribution of the brightness value of each phase was overlapped.

Here, we focused on those brightness values that could divide the histograms as much as possible, and an output image was created with brightness values of 0 to 110 as the martensite phase, 111 to 140 as the grain boundary, and 141 to 255 as the ferrite phase (Fig. 7).

Fig. 7.

Output image by the threshold.

This result indicated that the accuracy of the segmentation model using the threshold was clearly insufficient.

4.2. Segmentation by Deep Learning

Output images created by SegNet and U-Net models are shown in Figs. 8 and 9, respectively.

Fig. 8.

Output image by SegNet.

Fig. 9.

Output image by U-net.

4.2.1. Quantitative Evaluation of Models

A quantitative evaluation of each model is summarized in Table 3. Accuracy of the DL models was higher than that of the threshold model. In particular, the threshold model could not separate the grain boundary and the other two phases precisely, whereas the DL models could achieve this.

Table 3. Accuracy of each model.

	All phases	Ferrite	Grain boundary	Martensite
Threshold (%)	76.33	78.81	45.70	81.66
SegNet (%)	91.42	87.82	60.17	98.44
U-Net (%)	95.10	91.03	89.80	98.31

In addition, the accuracy of U-Net was higher than that of SegNet.

4.2.2. Qualitative Evaluation of Each Model

The accuracy map created by each model is shown in Figs. 10(a)–10(c). The threshold model could not distinguish martensitic phase and grain boundary, whereas the DL model could. From the perspective of separating each phase, the accuracy of U-Net was highest.

Fig. 10.

(a) Accuracy map by threshold, (b) Accuracy map by SegNet, (c) Accuracy map by U-Net.

4.3. Number of Training Data Required to Train an FCN Model

Table 4 shows the change in accuracy when the U-Net models were created by reducing the number of training data to five images and an image. The result indicated that even the U-Net model with reduced training images was still more accurate than the SegNet model with 45 training images.

Table 4. Accuracy of each model.

	U-Net	U-Net	SegNet	Threshold
Number of images for training	5	45	45	45
Accuracy (mean)	93.69%	95.10%	91.4%	76.33%

5. Discussion

5.1. Validity of the Deep Learning Model in Segmentation

As shown in Fig. 6, the segmentation using the threshold could not separate each phase sufficiently because the brightness value range of each phase was overlapped. In particular, wrong judgments often occurred because a major portion of the histograms between the martensitic phase and the grain boundary, and the grain boundary and the ferrite phase, were overlapped. Furthermore, the segmentation using the threshold could not recognize the ferrite phase in the upper left position of the image because the brightness of the entire image was inhomogeneous due to the light conditions of the microscope used.

These results suggested that conditions of the microstructure observation affected the accuracy in the segmentation method using the conventional threshold.

On the other hand, as shown in Figs. 8 and 9, the segmentation using DP models was highly accurate, even in the comparison of phases having similar brightness values. This result was probably attributable to the feature of the FCN algorithm.²³⁾ In the FCN algorithm, information including the brightness value of the adjacent pixel segmented each pixel by performing convolution and pooling. Therefore, morphological information of each phase was recognized, thereby causing the improvement in the segmentation accuracy.^8,23,25) Features of the FCN algorithm were effective even when the brightness values of each phase were not similar. For instance, the brightness of the image was not always constant, and the brightness value of the microstructural image may have changed due to conditions of the microstructure observation.

Thus, the FCN algorithm probably also showed robustness against brightness scattering attributed to the above-mentioned problems because it took the relationship between adjacent pixels into consideration.

5.2. Validity of U-Net for Microstructural Image

High sensitivity for the detection of fine regions was required for the segmentation because the microstructure of steel is generally composed of fine grains and grain boundaries. On the other hand, information of the fine regions was eliminated by convolution and pooling for the FCN model.^8,22) Therefore, as shown in Figs. 8 and 9, SegNet could not sufficiently recognize the vicinity of the grain boundary and the boundary at each phase. In order to solve this problem, the U-Net model could use the information prior to convolution and pooling when judging the pixels.²³⁾ Therefore, as shown in Figs. 8 and 9, the fine regions could be recognized with high accuracy.

From the above, it was probably appropriate to use U-Net for the segmentation of the steel microstructure.

5.3. Number of Training Data Required to Train Deep Learning Models

As described in section 4.2.3, a segmentation model with sufficient accuracy could be created using U-Net even if there were a small number of training images. As shown in Table 4, results obtained in this study clarified that the accuracy of U-Net was higher than that of SegNet, even when the amount of training data was small. It was confirmed from the results that U-Net was designed to create a model with less training data.²³⁾

In general, there is a common problem in applying DL for material developments such that it is difficult to prepare a large number of training data in material development. U-Net does not require much training data for any microstructure when creating a segmentation model for the same type of microstructural image as in this experiment. From the above, U-Net is a network structure suitable for material development.

6. Conclusions

We constructed segmentation models of the microstructure of steel using multiple algorithms, and the following results were obtained:

(1) DL models can perform segmentation for the microstructure of steel with sufficient accuracy. Compared with the method using the threshold, segmentation of phase with similar a brightness value can be performed with high accuracy.

(2) DL models show a relatively high robustness against external noise, such as conditions of microstructural observation.

(3) The U-Net model has sufficient recognition accuracy for microstructure because the model has an internal structure with information on that microstructure.

(4) DL models can be trained using a small number of microstructural images when the images are quite similar. An algorithm of the DL models probably enables an automated microstructural analysis process with less effort.

References

• 1)   W. S.  Rasband: ImageJ, http://imagej.nih.gov/ij/, (accessed 2019-01-10).
• 2)   C. A.  Schneider,  W. S.  Rasband and  K. W.  Eliceiri: Nat. Methods, 9 (2012), 671.
• 3)   M. M.  Nowell and  S. I.  Wright: J. Microsc., 213 (2004), 296.
• 4)   S.  Zaefferer,  P.  Romano and  F.  Friedel: J. Microsc., 230 (2008), 499.
• 5)   Z.  Wang,  T.  Ogawa and  Y.  Adachi: ISIJ Int., 59 (2019), 1691. https://doi.org/10.2355/isijinternational.ISIJINT-2019-089
• 6)   M.  Taguchi,  S.  Hirokawa,  I.  Yasuda,  K.  Tokuda and  Y.  Adachi: Tetsu-to-Hagané, 103 (2017), 142 (in Japanese).
• 7)   I. A.  Carreras,  V.  Kaynig,  C.  Rueden,  K. W.  Eliceiri,  J.  Schindelin,  A.  Cardona and  H. S.  Seung: Bioinformatics, 33 (2017), 2424.
• 8)   A.  Krizhevsky,  I.  Sutskever and  G.  Hinton: Advances in Neural Information Processing Systems 25 (NIPS 2012), Curran Associates, Inc., New York, (2012), 1097.
• 9)   S.  Loffe and  C.  Szegedy: Proc. 32nd Int. Conf. on Machine Learning, Vol. 37, JMLR, Inc. and Microtome Publishing, Massachusetts, (2015), 448.
• 10)   Y.  Taigman,  M.  Yang,  M.  Ranzato and  L.  Wolf: Proc. 2014 IEEE Conf. on Computer Vision and Pattern Recognition, Vol. 1, IEEE, New York, (2014), 1701. https://doi.org/10.1109/CVPR.2014.220
• 11)   L.  Chittka and  A.  Dyer: Nature, 481 (2012), 154.
• 12)   Y.  LeCun,  Y.  Bengio and  G. E.  Hinton: Nature, 521 (2015), 436.
• 13)   Y.  LeCun,  L.  Bottou,  Y.  Bengio and  P.  Haffner: Proc. IEEE, 86 (1998), 2278.
• 14)   G. E.  Hinton,  L.  Deng,  D.  Yu,  G. E.  Dahl,  A.  Mohamed,  N.  Jaitly,  A.  Senior,  V.  Vanhoucke,  P.  Nguyen,  T.  Sainath and  B.  Kingsbury: IEEE Signal Process. Mag., 29 (2012), 82.
• 15)   Z.  Wang and  Y.  Adachi: Mater. Sci. Eng. A, 744 (2019), 661.
• 16)   S.  Azimi,  D.  Britz,  M.  Engstler,  M.  Fritz and  F.  Mücklich: Sci. Rep., 8 (2018), 2128.
• 17)   X.  Zhu,  C.  Vondrick,  C. C.  Fowlkes and  D.  Ramanan: Int. J. Comput. Vis., 119 (2016), 76.
• 18)   S.  Amari,  N.  Fujita and  S.  Shinomoto: Neural Comput., 4 (1992), 605.
• 19)   J.  Long,  E.  Shelhamer and  T.  Darrell: Proc. 2015 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), IEEE, New York, (2015), 3431.
• 20)   Y.  LeCun,  B.  Boser,  J.  Denker,  D.  Henderson,  R. E.  Howard,  W.  Hubbard and  L. D.  Jackel: Neural Comput., 1 (1989), 541.
• 21)   O.  Matan,  C. J.  Burges,  Y.  LeCun and  J. S.  Denker: Proc. Advances in Neural Information Processing Systems 4 (NIPS 1991), NIPS, New York, (1991), 488.
• 22)   V.  Badrinarayanan,  A.  Kendall and  R.  Cipolla: IEEE Trans. Pattern Anal. Mach. Intell., 39 (2017), 2481.
• 23)   O.  Ronneberger,  P.  Fischer and  T.  Brox: Proc. Int. Conf. on Medical Image Computing and Computer-Assisted Intervention (MICCAI 2015), Springer Science+Business Media, Berlin, (2015), 234.
• 24)   H.  Zhao,  J.  Shi,  X.  Qi,  X.  Wang and  J.  Jia: Proc. 2017 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), IEEE, New York, (2017), 2881.
• 25)   Y.  Boureau,  J.  Ponce and  Y.  LeCun: Proc. 27th Int. Conf. on Machine Learning, JMLR, Inc. and Microtome Publishing, Massachusetts, (2010), 111.
• 26)   K.  Murakami,  N.  Hashimoto,  S.  Kido,  Y.  Hirano,  S.  Mabu,  K.  Kondo and  J.  Ozawa: Proc. 32nd Annual Conf. of the Japanese Society for Artificial Intelligence, The Japanese Society for Artificial Intelligence, Tokyo, (2018), 1Z3-05 (in Japanese). https://doi.org/10.11517/pjsai.JSAI2018.0_1Z305
• 27)   K.  Ninomiya,  Y.  Furuyama,  J.  Ota and  H.  Suyari: Proc. 32nd Annual Conf. of the Japanese Society for Artificial Intelligence, The Japanese Society for Artificial Intelligence, Tokyo, (2018), 2J4-01 (in Japanese). https://doi.org/10.11517/pjsai.JSAI2018.0_2J401