2024 Volume 5 Issue 1 Pages 72-79
In snowy and cold regions, road surface conditions change rapidly due to snowfall and temperatures. Road surface conditions need to be grasped both in daytime and nighttime to maintain a safe and comfortable road space. However, such a method has not been proposed. Therefore, this paper proposes a method to estimate road surface conditions in both day and night by using confidence levels obtained from the daytime and nighttime classifiers. The proposed method can estimate road surface conditions throughout the daytime and nighttime. In the last part of this paper, the effectiveness of the proposed method is confirmed through experiments using actual images taken by an in-vehicle camera.
In snowy and cold regions such as Hokkaido, winter-specific traffic accidents occur1). One of the causes is the rapid change of road surface conditions due to snow accumulation and freezing, making it difficult to control vehicles. Anti-freezing agents are applied on highways in Hokkaido to prevent freezing. However, applying anti-freezing agents causes problems such as accelerating deterioration of road structures and vehicles2). In addition, applying anti- freezing agents on highways in Hokkaido accounts for about 30% of the total cost of snow and ice operation3). Thus, the decision of applying anti- freezing agents is made by human, decision, specifically by visual determination. Therefore, a more efficient application of anti-freezing agents is desirable, and it is necessary to develop a method that enables an accurate and detailed understanding of road surface conditions.
A smart anti-freeze spraying system has been proposed as a current method to estimate road surface conditions4). This system employs an automatic road surface classification system (CAIS) that classifies road surface conditions using acceleration sensors installed on the inner surface of tires. However, since the system uses sensors mounted on the inner surface of the tire, it can accurately estimate the road surface where the tire contacts the ground. However, it is only to estimate the road surface conditions in the tire trajectory area.
Various methods have also been proposed to estimate the road surface conditions using images. In reference5), a Support Vector Machine (SVM) is used to estimate the road surface conditions. In this method, features based on color and texture are calculated and put into SVM to estimate road surface conditions. However, the input data are hand-crafted features selected manually, and there is no guarantee that the most suitable features for classification are selected from the countless number of existing features. To solve this problem, a neural network with a convolutional layer (Convolutional Neural Network, CNN), which extracts features that increase the classifier’s accuracy, is considered effective. Colors and textures of road surface images are considered to be meaningful for classifying the road surface condition. However, the number of combinations of these elements is enormous. The convolutional layer automatically determines the combinations that are considered to be effective for classification. Therefore, a neural network with a convolutional layer is considered effective.
The authors proposed a method for estimating road surface conditions in winter based on in-vehicle camera images using Mobile Nets, a type of CNN, and demonstrated its effectiveness6). However, in reference6), the method was applied to daytime images and nighttime image classification was not discussed. To realize the practical use of the method, it is desirable to estimate road surface conditions both in daytime and nighttime. For this purpose, one solution is to build a nighttime-specific classifier using the method proposed in reference6). In this case, switching between the daytime and nighttime classifier is necessary to estimate road surface conditions. However, accuracy degradation is due to changes in image brightness depending on weather conditions and date and time. Switching between the daytime and nighttime classifier is challenging to maintain classification accuracy. In addition, if there is a sudden change in the image’s brightness, such as backlighting or reflections from the road surface, the method proposed in reference6) may not be able to ensure classification accuracy.
Therefore, this paper proposes a method to estimate road surface conditions in both daytime and nighttime. Specifically, a method that classifies road surfaces during daytime and nighttime by using the confidence levels obtained from each classifier is proposed. An Extreme Learning Machine (ELM)7) performs the daytime and nighttime classification, which uses the confidence levels obtained from the daytime and nighttime classifiers as input, making it possible to estimate the road surface conditions both during the daytime and nighttime.
This section explains the previous method for winter road surface condition classification used in reference6). In reference6), the road surface condition is classified into 6 labels using images obtained from an onboard camera. Fig.1 shows an overview of the previous method. In this method, As shown in Fig.2, the road surface part of the image is utilized to classify the road surface conditions as shown in Fig.3. Six types of road surface conditions are classified by Mobile Nets: Dry, Semi-wet, Wet, Slush, Ice-snow, and Fresh-snow. In the following subsection 2. (1), Mobile Nets8) is explained. In 2. (2), classification of road surface conditions based on the confidence level calculated by Mobile Nets is explained.
(1) Mobile Nets
The proposed method uses trained Mobile Nets, which are a type of CNN with a low computational complexity, as classifiers. In the convolutional layer, local features are extracted with a filter. Specifically, consider image data (F × F pixels) with N channels. In the general convolutional layer of CNNs, such as those in reference9) and referece10), M features are extracted by an operation using M filters, each of which has K × K × N parameter values CNN. In other words, the parameter amount of the general convolutional layer is as shown in Equation (1).
;%0A%09%09%09newWindow.document.open();%0A%09%09%09newWindow.document.write('<img src=%22./Graphics/5.1_72_013.jpg%22>');%0A%09%09%09newWindow.document.close();%0A%09%09)
Unlike most CNNs, Mobile Nets performs convolution computation separately for Depthwise convolution and Pointwise Convolution. In Mobile Nets, only spatial convolution is performed in Depthwise convolution, followed by channel convolution in Pointwise Convolution. Separating the channel and spatial convolutions reduces the amount of parameters and lowers the computational cost. Specifically, the number of parameters for convolution in Depthwise convolution is expressed by Equation (2).
;%0A%09%09%09newWindow.document.open();%0A%09%09%09newWindow.document.write('<img src=%22./Graphics/5.1_72_014.jpg%22>');%0A%09%09%09newWindow.document.close();%0A%09%09)
The number of parameters in Pointwise Convolution is expressed by Equation (3).
;%0A%09%09%09newWindow.document.open();%0A%09%09%09newWindow.document.write('<img src=%22./Graphics/5.1_72_015.jpg%22>');%0A%09%09%09newWindow.document.close();%0A%09%09)
Hence, the number of parameters in Mobile Nets is given by Equation (4).
;%0A%09%09%09newWindow.document.open();%0A%09%09%09newWindow.document.write('<img src=%22./Graphics/5.1_72_016.jpg%22>');%0A%09%09%09newWindow.document.close();%0A%09%09)
Therefore, compared to normal convolutional computation, Mobile Nets can reduce the number of parameters to the amount shown in Equation (5).
;%0A%09%09%09newWindow.document.open();%0A%09%09%09newWindow.document.write('<img src=%22./Graphics/5.1_72_017.jpg%22>');%0A%09%09%09newWindow.document.close();%0A%09%09)
(2) Estimation of the road conditions
In reference6), Mobile Nets calculates the following confidence levels ;%0A%09%09%09newWindow.document.open();%0A%09%09%09newWindow.document.write('<img src=%22./Graphics/5.1_72_M01.jpg%22>');%0A%09%09%09newWindow.document.close();%0A%09%09)
for each of the six road surface conditions
;%0A%09%09%09newWindow.document.open();%0A%09%09%09newWindow.document.write('<img src=%22./Graphics/5.1_72_018.jpg%22>');%0A%09%09%09newWindow.document.close();%0A%09%09)
In the previous method, the road surface condition is classified using the following equation for each obtained for each image.
;%0A%09%09%09newWindow.document.open();%0A%09%09%09newWindow.document.write('<img src=%22./Graphics/5.1_72_019.jpg%22>');%0A%09%09%09newWindow.document.close();%0A%09%09)
Here, L indicates the label of the road surface condition.
In this Section, the proposed method is explained. Fig.4 shows an overview of the proposed method. First, the proposed method obtains the confidence levels of each label of the input image from each of the daytime and nighttime classifiers via Mobile Nets. Then, using an ELM with these confidence levels as input, the proposed method estimates the road surface conditions.
In the following subsection 3.(1), the integration of the calculated confidence level is explained. In 3.(2), The classifier ELM is explained.
(1) Feature vectors obtained by the integration of confidence levels
This subsection describes a method for estimating road surface conditions day and night. The model of the classifier is Mobile Nets, described in the previous subsection. The proposed method uses a daytime image classifier and a nighttime image classifier, which are constructed by learning Mobile Nets with daytime and nighttime images. First, the daytime image classifier is used to estimate the input image, and the confidence level of the six predicted labels (Dry, Semi-wet, Wet, Fresh-snow, Ice-snow) is calculated and is expressed as follows.
;%0A%09%09%09newWindow.document.open();%0A%09%09%09newWindow.document.write('<img src=%22./Graphics/5.1_72_020.jpg%22>');%0A%09%09%09newWindow.document.close();%0A%09%09)
In the same way, the input image is estimated using a classifier trained by transferring Mobile nets using the nighttime image, and the confidence level for the six predictive labels is calculated and expressed as follows.
;%0A%09%09%09newWindow.document.open();%0A%09%09%09newWindow.document.write('<img src=%22./Graphics/5.1_72_021.jpg%22>');%0A%09%09%09newWindow.document.close();%0A%09%09)
The feature vector fimage is a feature vector consisting of a total of 12-dimensional confidence levels as shown in Equation (7).
;%0A%09%09%09newWindow.document.open();%0A%09%09%09newWindow.document.write('<img src=%22./Graphics/5.1_72_022.jpg%22>');%0A%09%09%09newWindow.document.close();%0A%09%09)
(2) ELM classifier for estimating the road surface condition
This section explains the calculation of confidence levels for each feature. The proposed method uses a classifier ELM to estimate the road narrowing conditions.
ELM is a type of single hidden layer feedforward neural networks (SLFNs), which consist of three layers of neural networks. It enables fast learning speed and universal approximation with a small amount of training data. The ELM is known as a classifier that can ensure classification accuracy even with relatively a small number of training data compared to other neural networks11)
First, as training data, we consider a set of feature vectors fimage and estimating labels ;%0A%09%09%09newWindow.document.open();%0A%09%09%09newWindow.document.write('<img src=%22./Graphics/5.1_72_M02.jpg%22>');%0A%09%09%09newWindow.document.close();%0A%09%09)
if the estimation label zi is Dry. The specific calculation of the ELM is as follows. By performing a feature transformation using the sigmoid function d(fimage) is calculated according to the following equation.
;%0A%09%09%09newWindow.document.open();%0A%09%09%09newWindow.document.write('<img src=%22./Graphics/5.1_72_023.jpg%22>');%0A%09%09%09newWindow.document.close();%0A%09%09)
where uo(k = 1,2, ..., K), and vk (k = 1,2, ..., K) are parameters of the sigmoid function G, and K is the number of nodes in the hidden layer. Next, the weight of the final layer, β, is calculated by the following equation.
;%0A%09%09%09newWindow.document.open();%0A%09%09%09newWindow.document.write('<img src=%22./Graphics/5.1_72_024.jpg%22>');%0A%09%09%09newWindow.document.close();%0A%09%09)
However, Z = [z1, z2, ..., zR] T, R is the number of the training data and D = [d(f1), d(f2), ..., d(fR)] T. Finally, in the test data, when the feature vector fimage is input to ELM, the output value is g = d(f)T β, and the class label is the class label corresponding to the node that outputs the largest value among g.
In this section, experiments are conducted to verify the effectiveness of the proposed method. In this experiment, the accuracy of the proposed method is confirmed by using images obtained from an in- vehicle camera mounted on a vehicle traveling on an expressway in Hokkaido, Japan. In subsection 4.(1), the data and evaluation criteria of the experiment is described.
In 4.(2), the comparative method used in the experiments is explained. In 4.(3), the results of the experiments are explained.
(1) Dataset and evaluation criteria
In the experiment, video images acquired from several road management vehicles equipped with onboard cameras that capture images of the vehicle’s front is utilized. The images were captured on a highway in Japan, with asphaltic pavements. The dataset used in the experiment is shown in Table 1.
The experiment is conducted using five-part cross- validation. The size of the cropped area (l × m) is 700 × 280 pixels, and the number of nodes in the hidden layer of the ELM, K is twice the dimensionality of the feature vectors. The training parameters of the Mobile
Nets are shown in Table 2.
The experiments are evaluated quantitatively using Equations (10) and (11).
;%0A%09%09%09newWindow.document.open();%0A%09%09%09newWindow.document.write('<img src=%22./Graphics/5.1_72_025.jpg%22>');%0A%09%09%09newWindow.document.close();%0A%09%09)
;%0A%09%09%09newWindow.document.open();%0A%09%09%09newWindow.document.write('<img src=%22./Graphics/5.1_72_026.jpg%22>');%0A%09%09%09newWindow.document.close();%0A%09%09)
The Recall and the Precision were derived from Equations (12) and (13), respectively.
;%0A%09%09%09newWindow.document.open();%0A%09%09%09newWindow.document.write('<img src=%22./Graphics/5.1_72_027.jpg%22>');%0A%09%09%09newWindow.document.close();%0A%09%09)
;%0A%09%09%09newWindow.document.open();%0A%09%09%09newWindow.document.write('<img src=%22./Graphics/5.1_72_028.jpg%22>');%0A%09%09%09newWindow.document.close();%0A%09%09)
(2) Comparative methods
This subsection describes the comparative method performed to confirm the experiment’s effectiveness. The experiment consists of four comparative methods. The following subsubsections 4.(2)a) ~4.(2)d) describe each of these comparative methods.
a) Comparative method 1
An overview of comparative Method 1 is shown in Fig.5(a). In comparative method 1, a mixed classifier trained with Mobile Nets is used to estimate road surface conditions using a mixed dataset of daytime and nighttime images. By comparing the proposed method with comparative method 1, the effectiveness of classifying the road surface condition of the proposed method throughout the daytime and nighttime is confirmed.
b) Comparative method 2
An overview of comparative method 2 is shown in Fig.5(b). In comparative method 1, the road surface condition is classified using a daytime classifier. By comparing the proposed method with Comparative Method 2, the effectiveness of the proposed method, specifically classifying daytime images, is confirmed.
c) Comparative method 3
An overview of comparative method 3 is shown in Fig.5(c). Comparative method 2 classifies the road surface condition using a nighttime classifier. By comparing the proposed method with comparative method 3, the effectiveness of the proposed method, specifically classifying nighttime images is confirmed.
d) Comparative method 4
Fig.5(d) shows an overview of comparative method 4. In the Comparative method 4 daytime and nighttime images are pre-sort. Comparative method 4 classifies daytime images using a daytime classifier and nighttime images using a nighttime classifier. By comparing the proposed method with the comparative method 4, the proposed method is compared with the classification under the ideal situation where the data is pre-sorted.
(3) Experimental results
This subsection confirms the effectiveness of the proposed method for estimating road surface conditions both day and night. The experimental dataset is composed of only daytime images, only nighttime images, and a mixture of daytime and nighttime images.
The experimental results are shown in Table 3, Table 4, and Table 5. Table 3 shows that the comparative method 1 achieves a correct classification rate of 0.680, while the proposed method achieves a rate of 0.812. In other words, the proposed method is 19.4% more accurate than the comparative method 1. Next, comparing the proposed method with the comparative method 2 on Table 4, the proposed method achieves the same level of accuracy even when focusing only on daytime classification. Comparing proposed method with comparative method 3 on Table 5, the proposed method achieves the same accuracy rate as the comparative method, even for nighttime images. Finally, comparing the proposed method in Table 3 with the comparative method 4, the proposed method achieves the same level of accuracy in F-measure as the comparative method 4, while the proposed method improves the accuracy by about 1% in accuracy. In another perspective, under the assumption that the data is pre-sort and the calculation cost is ignored, the comparative method 4 is less computationally expensive and easier to implement than the proposed method. However, the comparative method 4 is unrealizable since this is only under the assumption that the preprocessing ideally sorts the dataset. The proposed method realizes a higher accuracy than comparative method 4, which is a method that performs the classification under these ideal circumstances.
Table 6 shows the Recall and Precision of the proposed method. The precision of the label “Dry” is 0.881, achieving a classification result of approximately 90%. Therefore, the proposed method is less likely to incorrectly classify a non-dry surface as “dry”.
Fig.6 shows an example of improved classification accuracy by the proposed method. Fig.6(a) is an example of the proposed method for improving classification accuracy. In Fig.6(a), the correct label is Ice-snow. The proposed method correctly classifies the label as Ice-snow, while the comparative method 4 incorrectly classifies the label as Slush. In Fig.6(a), the road surface is dark in the image due to the sun’s backlight. Therefore, it is considered that the proposed method correctly classified the road surface in the nighttime image when the road surface was dark because it also considered the classifier learned by the transition.
Fig.6(b) shows an image in which the correct label is Semi-wet. The proposed method correctly classified the image as Semi-wet, while the comparative method 4 incorrectly classified it as Slush. The image shown in Fig.6(b) is a nighttime image in which street lights and vehicle headlights brightly illuminate the road surface near the analysis window. This indicates that the image resembled a bright daytime image. Therefore, the proposed method correctly estimates the nighttime image because the classifier trained on the daytime image is used in the nighttime image.
The proposed method can improve the classification accuracy in such situations where the brightness of the road surface suddenly changes.
In this experiment, all images were of asphaltic pavements. Since the pavement type may affect the analysis result, re-learning the model should be performed according to the pavement types.
This paper proposes a method for estimating road surface conditions throughout day and night. The proposed method enables the estimation of road surface conditions in ELM based on the confidence level calculated from Mobile Nets, thus enabling the estimation of road surface conditions in both day and night.
In reference12), a system to classify road surface conditions by the same label as Fig.3 is proposed. The system is currently used in practice. In operation, anti-freezing agents are applied when the road surface condition is not classified as “Dry”. However, the system involves human intervention to classify the road surface conditions. Therefore, the proposed method is expected to classify the road surface condition in this system automatically.
Also, studies on video analysis for winter road management, such as traffic volume survey method13) and estimation method of road narrowing condition by piled snow13) is proposed and have shown an accuracy of 90% or better as required performance. Therefore, in terms of practice use, the future work will be to realize a method to achieve the required 90% or better performance.
In terms of improvement of classification accuracy, by adding information such as time, weather, and temperature as input to the model, the accuracy of classification is expected to improve. Also, the dataset in this paper does not include evening images that fall between daytime and nighttime. Therefore, to further improve the effectiveness of the proposed method, it is necessary to expand and verify such a dataset in the future.
This work was partly supported by JSPS KAKENHI Grant Number JP22H01607, Japan.
