Instrumentation, Control and System Engineering
Surface Defects Classification of Hot Rolled Strip Based on Improved Convolutional Neural Network
2021 Volume 61 Issue 5 Pages 1579-1583
Details
2021 Volume 61 Issue 5 Pages 1579-1583
Surface defect classification of hot-rolled strip based on machine vision is a challenge task caused by the diversity of defect morphology, high inter-class similarity, and the real-time requirements in actual production. In this work, VGG16-ADB, an improved VGG16 convolution neural network, is proposed to address the problem of defect identification of hot-rolled strip. The improved network takes VGG16 as the benchmark model, reduces the system consumption and memory occupation by reducing the depth and width of network structure, and adds the batch normalization layer to accelerate the convergence speed of the model. Based on a standard dataset NEU, the proposed method can achieve the classification accuracy of 99.63% and the recognition speed of 333 FPS, which fully meets the requirements of detection accuracy and speed in the actual production line. The experimental results also show the superiority of VGG16-ADB over existing classification models for surface defect classification of hot-rolled strip.
Surface quality defects of steel seriously affect the physical and chemical properties of steel products, and bring huge economic and commercial reputation losses to product manufacturers. For the defect detection of product surface in industrial production, the detector is required to be able to distinguish whether there are defects on the product surface or not, and it is practically designed as a two class classifier.1,2) Zhang et al. proposed a GRNN-PNN neural network method for the detection of iron defects, and got an overall pass rate of 96% by testing the surface quality of plane printed iron sheet.2) Generally, it is difficult to develop recognition algorithm for hot rolled strip because the high temperature and radiation intensity, as well as water vapor iron oxide scale and non-uniform illumination.3,4,5) Moreover, the demand for accurate identification of defect types is increasing rapidly for the need of tracing the changes of production environment, or status of equipments.
Therefore, correct classification of surface defects of hot rolled strip is very important but challenging job for strip production and quality control. Unlike the defects detection where the task is a binary classification problem, the defects usually involve multiple classes, and some models which can identify multiple defect types have been proposed.4,5,6,7,8,9) Traditional detection methods mainly include manual detection, eddy current detection magnetic flux leakage detection and vision-based steel surface defect detection, but these methods usually lead to a large number of missing and false detections, which cannot guarantee product quality and meet the real-time need of the field.10,11,12,13,14,15)
Fortunately, deep learning methods have recently been applied to a number of similar domains with success.16,17) Masci et al. developed a maximum pooling convolutional neural network (CNN) for surface defects detection of hot rolled strip, and achieved an accuracy of 98.57% with a recognition speed of 0.008 s.18) Li et al. firstly expanded the number of defect images by 10 times through data augmentation technique such as rotation, translation and noise addition, and proposed a 7-layer convolutional neural network for surface defect recognition where a speed of 0.001 s for each image detection, and the accuracy of 99.05% can be achieved.19) Although the deep learning-based models promoted the defects detection heavily, there is a room for improvement of detection speed and accuracy. In this work, an improved computational model named VGG16-ADB based on VGG16 convolutional neural network is proposed for surface defect images of hot rolled strip.
The data used in this work come from a surface defect dataset named NEU database which was constructed by the Northeast University, China. The database collects gray images of six typical surface defects of hot rolled strip, including rolled-in scale (Rs), scratch (Sc), pitted surface (Ps), inclusion (In), patches (Pa) and crazing (Cr). This dataset contains 300 images for each type of hot rolled strip. The original pixel resolution of each image is 200 × 200. In order to reduce the training time of network model parameters, the size of each image is adjusted to 128 × 128 before it is sent to the network. Figure 1 shows the sample images of six typical surface defects, each listed as one category.
Six typical defect sample images in NEU database.
Among many CNN models, VGG-Nets has good generalization performance, and its pre-training model is widely used for image recognition in different domains. Compared with previous networks such as Alex-Net, VGG16 uses smaller convolution kernels and deeper network structures, by which network training parameters can be reduced, and the capability of feature learning can be enhanced while maintaining the same field of receptivity as large convolution kernels. Although VGG16 network has a good learning ability, it needs a lot of storage space and training time because there are 140 M training parameters, which hinders the popularization of this model in real applications.
Therefore, this work focuses on the improvement of VGG16 model to meet the high requirements of recognition rate and real-time in surface defects detection of hot rolled strip. Firstly, instead of SGD optimization algorithm in the original VGG16 model, the Adam optimization algorithm which can automatically update the learning rate is selected to solve the possible oscillation problem in the process of parameter optimization and defined the model replacing the optimization algorithm as the VGG16-A.20,21) Secondly, because of 80% of the parameters in the VGG16 model are derived from the fully connected layer, the single full-connection layer with 512 neurons used as the final feature extraction step was adopted by the VGG16-AD model on the basis of VGG16-A, which greatly reduces the number of parameter of model. Thirdly, based on the VGG16-AD model, the batch normalization layer is added to further accelerate the convergence speed of the model.22) Finally, an improved VGG16-ADB network with 16 layers has been constructed and the architecture of improved convolutional neural network is shown in Fig. 2.
The architecture of our convolutional neural networks. Conv, FC, BN and MaxP refers to convolutional layer, fully connection layer, batch normalization layer and max pooling. (Online version in color.)
In this work, all of the experiments are implemented on a NVIDIA Quadro M4000 GPU hardware platform. The batch size is set to 20, the initial learning rate is 1e-6, and the decay of learning rate is 1e-7. The initial epoch number is set to 250 times on the model training, and this number is set to 40 after model is accelerated.
For six types of defects, 50% images (150 images for each of them) are randomly selected to train the model, and the remains are adopted for model test and evaluation. To overcome the influence of randomness of dataset partition, the experiment has been repeated five times, and the average performance is taken as the final evaluation for the proposed model. As VGG-Nets parameter initialization method did, the convolution layer weight of the proposed network model is initialized with the value of VGG16 pre-trained in ImageNet.
4.2. Prediction Performance of Improved Model 4.2.1. Classification Performance of Improved ModelIn order to apply the CNNs to the classification of specific steel surface defects and meet the real-time requirements of the steel production line, three improved VGG16 models, i.e., VGG16-A, VGG16-AD and VGG16-ADB, have proposed to optimize the parameters and structure of the prediction module in different levels, and the differences among these models are shown in Fig. 3.
Variants of the prediction module. (Online version in color.)
The prediction accuracy of each improved model is illustrated in Table 2, it can be seen that VGG16-ADB model achieves the best classification performance with the accuracy of 99.63%, and only 3 images were misclassified for 900 test images. In order to show the classification effect of each kind of defects more carefully, the confusion matrix is shown in Table 1. It can be seen that VGG16-ADB model can achieve 100% of classification accuracy for four defects, i.e., crazing, rolled-in scale, pitted surface and inclusion. Obviously, the proposed model is effective in the defect image classification.
| Cr | Rs | Ps | Pa | In | Sc | |
|---|---|---|---|---|---|---|
| Cr | 150 | 0 | 0 | 0 | 0 | 0 |
| Rs | 0 | 150 | 0 | 0 | 0 | 0 |
| Ps | 0 | 0 | 150 | 0 | 0 | 0 |
| Pa | 0 | 0 | 1 | 149 | 0 | 0 |
| In | 0 | 0 | 0 | 0 | 150 | 0 |
| Sc | 0 | 0 | 0 | 0 | 2 | 148 |
| VGG16 | VGG16-A | VGG16-AD | VGG16-ADB | |
|---|---|---|---|---|
| Number of parameters | 65079110 | 65079110 | 18912582 | 18913606 |
| Model size (M) | 248.26 | 248.26 | 72.15 | 72.15 |
| Training time (seconds) | 5000 | 360 | 302 | 270 |
| Average accuracy (%) | 99.43 | 99.36 | 99.30 | 99.63 |
In our experiment, only 2.63 seconds were spent when 900 images were tested with the proposed model, which means the average time to classify an image is 0.003 seconds and the detection speed is 333 FPS. In the actual production line, the maximum production speed is typically 30 m/s, and the shooting field of a single camera is 50–100 cm, which requires the detection equipment to have at least 30–60 FPS detection speed.18) Therefore, 333 FPS detection speed of the proposed VGG16-ADB model in this work can fully meet the real-time requirements in actual production.
4.3. Performance Comparison of Improved ModelsTo evaluate the performance of three improved models, convergence rate and parameters amount are used as measures of classification performance.
4.3.1. Comparison of Convergence RateConvergence rate is a measure of how fast the model completes training, which is an important index to evaluate deep learning method. To speed up the convergence and detection, this work introduces some improvement techniques into the VGG16 model, and the loss curve of each model is shown in Fig. 4.
Loss curve of training model. (Online version in color.)
Compared with VGG16, it can be seen from the Fig. 4 that when the optimization parameter of VGG16-A model is changed to Adam under the same structure and parameter quantity, the convergence speed is significantly faster. After further reducing the parameters and increasing the batch normalization layer, the convergence rate increases by 58 and 130 seconds respectively, and the fastest convergence time of VGG16-ADB model is 270 seconds. It is demonstrates that Adam algorithm is the most effective way to speed up the convergence rate, reducing the number of parameters and increasing the normalization layer can also effectively improve the convergence speed of the model, especially when they work together.
4.3.2. Model Parameters and SizeThe model size mainly depends on the parameters of the model, which is particularly important for the application of deep learning model in industrial production. The total number of parameters and model size statistics are already shown in Table 2.
Since the original VGG16 and the improved models have the same basic convolution layer for feature extraction, the different number of parameters among different models mainly comes from the prediction module. For instance, there are 65079110 parameters in the VGG16 model, and the number from the prediction module is 50339840, which is accounting for 77.35% of the total number. Therefore, reducing the number of full connection layers made up of a large number of neurons or cutting down neurons can effectively decrease the number of model parameters. Based on this idea, the total parameters is 18912582 in the VGG16-AD model, with 512 neurons in a single full connection layer, and the model size is 72.15 M which is only one third of the VGG16 model.
4.4. Comparisons with Other State-of-the-art MethodsTo further assess the effectiveness of the proposed VGG16-ADB model, several state-of-the-art methods are compared in the predictive accuracy and the detection time per image when the same NEU dataset is used. Two traditional machine learning-based approaches, i.e., Nearest neighbor clustering (NNC) and support vector machine (SVM) with different feature extraction techniques including adjacent evaluation completed local binary patterns (AECLBP) and generalized completed local binary patterns (GCLBP), are considered for comparison.13,23) Moreover, two deep-learning-based surface defect classification approaches including the end-to-end CNN model (CNNs) proposed by Li et al. and the Decaf model-based approach (Decaf + MLR) proposed by Ren et al. are also implemented.19,24)
It can be seen from Table 3 that although traditional machine learning methods can achieve good results with 97.93%, 98.93% and 99.11% accuracy for three models, the process of feature extraction and classification is slow. In contrast, the method based on deep learning with transfer learning strategy can get better classification accuracy and detection speed. Compared with the Decaf + MLR method only initialized by the decaf model, this work cuts down a lot of training parameters to avoid model under-learning, and uses Adam optimization algorithm and batch normalization to speed up the training speed. As a result, the proposed VGG16-ADB method achieves 0.3% higher accuracy than Decaf + MLR model and faster speed of 0.003 s. The results demonstrated the effectiveness of the proposed model in the classification of hot-rolled steel strip surface defects.
To verify the accuracy and speed improvement of our method, the model is also applied to detect defects on the surface of the magnetic tile where the dataset contains 1344 images with five types of defects, including blowhole, crack, break, fray and unevenness and one type of non-defect data,25) and each defect type is shown in Fig. 5. In our experiment, the dataset is divided into two parts, 70% of which are used for training, and the remaining 30% are used as test set. Due to the different image sizes and the unbalanced sample distribution, this work uniformly scales the image size to 224*224, and uses a data enhancement strategy to avoid over-fitting.
Examples of each defect type.
With the same experimental setup as above, the model we proposed was trained on the magnetic tile dataset. As a result, the detection accuracy we got was 99.4%, and the detection speed for each magnetic tile image was 90 FPS, while these values are 98.1% and 12 FPS in Wang’s work.25) This results show a good generalization ability of our proposed model.
The surface image quality of hot rolled strip is susceptible to environmental impact, which makes the defect recognition technology based on machine vision face many challenges. In this work, an improved convolutional network model named VGG16-ADB is proposed to solve the problem of insufficient feature extraction and slow defect recognition speed in traditional algorithms. The average classification accuracy of the model is 99.63%, which is higher than that of state-of-art methods. Moreover, the recognition speed of the network is 333 FPS, which fully meets the real-time requirements in actual production. The experimental results improve the detection of hot-rolled steel strip surface defects, and therefore can be helpful for steel industry.
This work was supported by the National Natural Science Foundation of China (Nos. 61472282, 61672035, and 61872004), Educational Commission of Anhui Province (No. KJ2019ZD05), Open Fund from Key Laboratory of Metallurgical Emission Reduction & Resources Recycling (KF2017-02), and Anhui Scientific Research Foundation for Returnees.
