Blur Map Generation Based on Local Natural Image Statistics for Partial Blur Segmentation

Partial blur segmentation is one of the most interesting topics in computer vision, and it has practical value. The generation of blur maps is a crucial part of partial blur segmentation because partial blur segmentation involves producing a blur map and applying a segmentation algorithm to the blur map. In this study, we address two important issues in order to improve the discrimination of blur maps: (1) estimating a robust local blur feature to consider variations in the intensity amplitude and (2) a scheme for generating blur maps. We propose the ANGHS (AmplitudeNormalized Gradient Histogram Span) as a local blur feature. ANGHS represents the heavy-tailedness of a gradient distribution, where it is calculated from an image gradient normalized using the intensity amplitude. ANGHS is robust to variations in the intensity amplitude, and it can handle local regions in a more appropriate manner than previously proposed local blur features. Blur maps are affected by local blur features but also by the contents and sizes of local regions, and the assignment of blur feature values to pixels. Thus, multiple-sized grids and the EAI (Edge-Aware Interpolation) are employed in each task to improve the discrimination of blur maps. The discrimination of the generated blur maps is evaluated visually and statistically using numerous partial blur images. Comparisons with the results obtained by state-of-the-art methods demonstrate the high discrimination of the blur maps generated using the proposed method. key words: ANGHS, blur map generation, natural image statistics, partial blur segmentation


Introduction
The images taken by photographers often include partial blur. These images are called partial blur images, and much effort is required to segment these images [1]- [6]. Partial blur images are divided into two sub-classes according to the cause of blurring. One class comprises focal blur images, which are caused by the focal settings of the camera and distances to objects in a scene. Another class comprises motion blur images, which are caused by the exposure settings of the camera and the relative movements between the camera and objects. Examples of partial blur images are shown in Fig. 1. Many image segmentation techniques have been proposed for focal blur images [9]- [12] and motion blur images [13], [14]. Partial blur segmentation has received much attention in recent years because it is expected to be capable of segmenting both types of blur images with a single unified algorithm. Partial blur segmentation can be employed in more applications than focal and motion blur segmentation. For example, blur magnification [15] and image deblurring [13] are typical applications of focal and motion blur segmentation, respectively. Partial blur segmentation can be used for pre-processing in both applications. Thus, improving partial blur segmentation is an interesting topic, and it also has practical value. One of the standard approaches for partial blur segmentation involves generating a blur map and applying a segmentation algorithm to the blur map. Therefore, generating blur maps is a crucial part of partial blur segmentation. A blur map represents the blur degrees of an overall image. J [x] and I[x] indicate a sharp image and a partially blurred image, respectively. x ∈ X indicates the discrete coordinates in an image. If a set of blur kernels is defined by K = {k x [y]}, I[x] is given by Eq. (1).
y is an element in a kernel support S x = supp(k x [y]). Equation (1) shows that the blur kernel is different for each x. Therefore, the set of kernels is difficult to estimate using algorithms for spatially invariant blurs [16]. A typical approach to address this problem is estimating the blur features in local regions. In terms of this approach, we consider the following two important issues in order to improve discrimination of blur maps.
1. Estimating a robust local blur feature according to variations in the intensity amplitude. 2. A scheme for generating blur maps.
The local regions in a blurred image can be classified into four types as shown in Fig. 2. We denote each type of local region as S-E (Sharp region with Edge pixels), S-NE (Sharp region with No Edge pixels), B-E (Blur region with  Edge pixels), and B-NE (Blur region with No Edge pixels). The pixels related to the edges and textures in local regions contain useful information for estimating the blur degrees. Some local blur features can be employed to estimate the blur degrees using the local image gradient [1], [6] because the edge and texture information are characterized by the image gradient. However, the local image gradient is affected by the local intensity amplitude as well as by blur. For example, B-E often has a higher image gradient than S-NE. Thus, a local blur feature calculated from B-E would have a more similar value to S-E than that of S-NE in this case. Therefore, it is difficult to correctly classify these regions into sharp or blur regions using local blur features by directly utilizing the image gradient.
The basic idea of the proposed local blur feature involves obtaining feature values to correctly classify the four types of local regions as sharp or blur regions. The GHS (Gradient Histogram Span) proposed by Liu et al. [1] is a local blur feature based on natural image statistics. GHS has desirable characteristics as a blur feature because it effectively captures the blur responses of edge and texture pixels. However, GHS has difficulty handling local regions correctly because it is affected by the local intensity amplitude. Thus, in this study, we propose the ANGHS (Amplitude-Normalized Gradient Histogram Span), which is calculated from the normalized local gradient. The local gradient is normalized by the local intensity amplitude, which is estimated using an empirical method in this process. Normalization makes GHS robust to variations in the local intensity amplitude, and ANGHS can handle the local regions in a more appropriate manner than GHS.
Blur maps are affected by the local blur features but also by the generation scheme employed. Traditional blur feature map generation using window sliding is simple, but it often produces noisy blur maps. Thus, our proposed method employs the EAI (Edge-Aware Interpolation) [17] to generate blur maps. EAI propagates sparsely sampled local blur features based on the local color similarity, and the blur maps obtained have smooth and edge-preserved appearances. The sampling of local blur features has an important role in this process. We propose multiple-sized grids to sample the local regions. This technique can improve the stability of the blur maps. Learning the models and parameters in advance is not required in the proposed method, which makes the proposed method suitable for many applications.
We also compare the discrimination of blur maps based on statistical methods using numerous images. We investigated some characteristics of the proposed method using previously proposed performance evaluation criteria [18]. Comparisons of the results obtained using the dataset proposed by Shi et al. [6] demonstrated the high discrimination of the blur maps generated by the proposed method. The remainder of this paper is organized as follows. In Sect. 2, we introduce the state-of-the-art methods for blur segmentation. In Sect. 3, we describe the characteristics of local natural image statistics and explain ANGHS. In Sect. 4, we describe the proposed blur map generation scheme. In Sect. 5, we compare the results obtained by the proposed method and other state-of-the-art methods. Finally, we give our conclusions and suggestions for future research in Sect. 6.

Related Work
In the following, we introduce the state-of-the-art blur segmentation algorithms, which are characterized by the estimation of blur maps. Blur estimation methods can be divided into two types. One type of the methods estimates the kernel parameter directly. Chakrabarti et al. [3] formulated Eq. (1) as a problem for Fourier analysis, and they determined the kernel size by maximum a posterior estimation. Zhuo et al. [9] found that the kernel size can be calculated based on the ratio of the original image gradient relative to that of the re-blurred image and a Gaussian parameter to reblur the original image. Their method is correct at the edge pixels, and they used EAI to propagate a sparse blur map to an entire image. The propagation of sparse blur maps is also used in the proposed method. In fact, their blur maps and those obtained by the proposed method have similar characteristics. However, their method requires edge detection as a pre-processing, and the blur maps depend on the detection results. The other type of blur estimation methods involves convolving an image with the kernel as a smoothing operation and quantifying the effects of smoothing. Su et al. [4] found that the first few most significant eigen-images obtained by singular value decomposition have higher weights in a blur region, and they estimated a blur feature using these weights. Liu et al. [1] and Shi et al. [6] applied a machine learning algorithm to natural image statistics. First, a naive Bayes classifier learns the natural image statistics from the sharp and blur image patches, respectively. Next, these methods use the likelihood ratio of the learned classifier as a blur feature. These three methods employ local blur features, similar to our proposed method. However, the effects of the intensity amplitude on a blur feature are not considered in these previous methods. In addition, these methods use window sliding to generate blur maps. Window sliding moves a local window as a raster scan and directly assigns each local blur feature in the window to each center pixel. Window sliding is simple, but it often produces noisy blur maps. Moreover, the methods of Liu et al. and Shi et al. require numerous images for their learning classifiers.

Gradient Histogram Span
In this section, we describe the proposed local blur feature. First, we give a brief introduction to the GHS proposed by Liu et al. [1]. It is well known that the image gradients of natural images have a heavy-tailed distribution [19] because the distribution has a sharp peak at 0 and extensive tails. This is because most of the gradients in smooth areas are almost 0, whereas a few edge pixels have a large gradient. However, the distribution of the local regions is quite different depending on their contents. In Fig. 3, ED represents the empirical gradient distribution for each local region depicted in Fig. 2. We note that the vertical axes of the graphs in Fig. 3 represent the logarithms of the probabilities. The image gradient is calculated by convolving eight directional derivative filters. Both S-E and B-E include edge pixels, but the heavy-tailedness of each local region is quite different as illustrated in Fig. 3 (a) and (c), which is caused by attenuation of the image gradient in the blur region. Liu et al. denoted the heavy-tailedness of gradient distribution as GHS and proposed a method for quantifying the heavy-tailedness by fitting a GMM with two components. The probability density function for a GMM with two components is represented as follows: where ΔI[x] indicates the image gradient in a local region, μ 1 , μ 2 , and σ 1 , σ 2 are the averages and standard deviations of each Gaussian component, respectively, and w 1 and w 2 are the relative weights of each Gaussian component. σ 1 < σ 2 means the first and the second Gaussian components are fitted to small and large gradients, respectively. Liu et al. used max(σ 2 1 , σ 2 2 ) as GHS because the heavytailedness is affected greatly by edge pixels. GHS has the advantage that the calculated value is not affected by smooth areas because the method effectively captures the features of edge and texture pixels. We note that Liu et al. finally scaled GHS based on the maximum Michelson contrast in a local region. However, this post-process is not useful for removing the effect of the amplitude from GHS because the amplitude influences GHS as the variance of the sampled image gradient. Moreover, the Michelson contrast depends on the minimum intensity level, and scaling based on the contrast often has undesirable effects on GHS. Therefore, in the proposed method, we use the unscaled GHS to derive ANGHS.
Liu et al. [1] used the EM algorithm to fit the GMM with two components, but our proposed method employs kmeans clustering. First, eight gradient images are obtained by convolving eight directional derivative filters with an image. All of the gradient images are combined in each local region. Next, the combined local gradient is clustered based on |ΔI[x]| by naive-thresholding, which uses the average of |ΔI[x]| as a threshold. Subsequently, k-means clustering is used to adjust the initial clusters. Finally, μ 1 , μ 2 , and σ 1 , σ 2 are calculated from ΔI[x] in each cluster.
k-means clustering is simpler and converges faster than the EM algorithm. Moreover, GHS calculated by k-means clustering tends to obtain a larger difference between S-E and B-E than the EM algorithm, as illustrated by GC2 and GC2-EM in Fig. 3 (a) and (c). Each gradient affects both Gaussian components when using the EM algorithm. Thus, the effects of the gradient are relatively distributed to both Gaussian components. By contrast, each gradient affects either of the Gaussian components when using k-means clustering. k-means clustering captures the feature of edge pixels more directly than the EM algorithm and improves the discrimination of GHS.

Amplitude-Normalized GHS
GHS has the desirable feature that it is not affected by smooth areas, but the blur feature is affected by the intensity amplitude. For example, we consider simple a case where a kernel k[x], S = supp(k[x]) is convolved with one-dimensional edge signals to explain this effect. A onedimensional edge signal can be represented as follows: where H[x] is a discrete Heaviside step function, and A and B indicate the intensity amplitude and offset of the edge, respectively. When J[x] is convolved with k[x], the derivative of the convolved function with respect to x is represented as follows: where δ[x] is the unit impulse function, and * is convolution. Equation (6) shows that the gradients of the blurred edge signals are proportional to A. Therefore, GHS is affected by A because GHS is the variance of the image gradient, which is calculated from the extracted edge pixels. S-NE has a lower GHS value than B-E, as illustrated in Fig. 3 (b) and (c), because B-E has a higher local intensity amplitude than S-NE. Therefore, the proposed method uses ANGHS, which is calculated from ΔI[x]/A. We note that normalization is applied before GMM fitting in ANGHS unlike the scaled GHS proposed by Liu et al. Moreover, the amplitude used as the normalization factor is independent of the specific intensity level. Figure 4 (a) shows simple simulation images. The first column in Fig. 4 (a) illustrates the simulation images, which have four levels of light rectangles and a dark background. The other columns in Fig. 4 (a) illustrate the blurred simulation images obtained by Gaussian filters. We note that each intensity amplitude A is set to each intensity of the rectangles in this simulation for simplicity. Figure 4 (a) shows blurred simulation images for some representative values of s, where s varies in steps of 0.01 in this simulation. Each line in Fig. 4 (b) and (c) shows the behavior of the blur features according to A. The behavior of GHS depends on A, as illustrated in Fig. 4 (b). By contrast, the behavior of ANGHS is unchanged, as illustrated in Fig. 4 (c). We note that Fig. 4 (b) and (c) use different vertical scales because their relative behaviors according to A are important for showing the robustness to the intensity amplitude. According to this result, ANGHS is a more robust blur feature with respect to the local amplitude than GHS.
Determining the local intensity amplitude A is a crucial part of the proposed blur feature, but obtaining an appropriate estimate is also a difficult task. We estimate the local intensity amplitude based on the confidence interval of the local intensitiesÂ = a var (I[y]), y ∈ S x , where var (·) indicates the variance. When a = 1.96,Â is approximately the 95% confidence interval under the Gaussian assumption. This estimation is simple, but the estimated values ofÂ is sometimes extremely large or small. Extremely large and small estimates ofÂ often cause problems during the normalization of ANGHS. For example, if a local region has almost no texture, the estimated value ofÂ is extremely small, and ΔI[x]/Â has a high value. As a result, the value for the local region is similar to that for a sharp region. The opposite applies to a local region with an extremely largê A. Hence, we limit the range ofÂ as [Â min ,Â max ] to avoid estimation problems.Â max is defined simply as the practical maximum of the image gradient because local regions rarely have extremely large values forÂ.Â max = 255 if the intensity values for an image are represented by 8-bit numbers. On the other hand,Â min has significant effects on the generated blur maps because many B-NE and some S-NE have extremely small values forÂ. The effect ofÂ min on the generated blur maps is considered in Sect. 5.
In the rest of this section, we compare GHS and ANGHS based on a practical image.Â min = 30 is used in the comparison. Figure 5 shows the feature distributions for GHS and ANGHS. Both distributions are calculated from Fig. 1 (a) using window sliding with local region measuring 21 × 21. The red and blue lines in Fig. 5 show the feature distributions in the sharp and blur regions, respectively. The sharp and blur regions are defined as the ground truth, as defined by Shi et al. [6]. We note that Fig. 5 only shows the range of feature values [0, 0.5] for clarity because most of  the feature values are concentrated in the low range. The distributions of most of the sharp and blur regions calculated using GHS are overlapping, as illustrated in Fig. 5 (a). By contrast, there is relatively much less overlapping of the distributions calculated using ANGHS, as illustrated in Fig. 5 (b). This result demonstrates the higher discrimination of ANGHS.

Process Overview
In this section, we describe the proposed blur map generation scheme. An overview of the proposed blur map generation method is illustrated in Fig. 6. First, the proposed method uses multiple-sized grids to assign local regions, and local blur features are then sampled from the local regions. Next, sparse blur maps are generated by assigning each local blur feature to the center pixels of each local region, which are represented by pseudo-colors in Fig. 6. The 0 th to 95 th percentiles of the feature values are mapped from blue to red colors for clarity because the feature distribution of ANGHS is highly skewed to the left. Individual sparse blur maps are generated for each grid size. All of the individual sparse blur maps are then combined into a single combined sparse blur map. We note that the blur features of smaller regions are assigned to one of the unassigned neighboring pixels of the original position when the center pixels of smaller regions overlap with those of larger regions. The unassigned pixels are searched for in a clockwise direction from the top-left. The combined sparse blur map is difficult to use for segmentation because it has feature values only on the center pixels of each local region. Hence, EAI is employed to generate a full blur map from a combined sparse blur map. The high feature values in the full blur map are represented as dark colors in Fig. 6 in order to show the blur degrees for the image.

Sparse Blur Map Using Multiple-Sized Grids
The sampling of local blur features has an important role in the proposed blur map generation process. Sampling is often performed using edge detection algorithms [9], but the blur map obtained tends to be unstable because the sampled local regions depend on the edge detection method. The grid-based local regions uniformly sample local blur features from the overall image to improve the stability of the blur map.
Local blur features are affected by the contents and sizes of the local regions. The local regions assigned by small grids densely sample the local blur features, but their blur features are unstable, as shown in Fig. 6 because the blur features rely on a small number of pixels. By contrast, the blur features calculated from the local regions assigned by large grids are more stable, but they are sampled more sparsely than small grids. Thus, sampling with multiplesized grids balances this trade-off.
The combination of different sizes is based on userdefined values. We use grids measuring 11 × 11, 21 × 21, and 41 × 41 pixels as the experimental setting. Each length of the local regions is approximately equal to 1/64, 1/32, and 1/16 widths of video graphics array images, which are mostly used in the dataset prepared by Shi et al. [6]. The local intensity amplitude is estimated using larger regions based on one pixel to avoid problems with normalization at the ends of the local regions.

Full Blur Map Generation
The proposed method employs the EAI proposed by Levin et al. [17] to generate full blur maps. EAI can be used to propagate sparse features to the entire image based on local color similarities.
We represent a color image as where α is constant, and we use α = 0.0001 as the experimental value. D is an N × N diagonal matrix. When D i j (1 ≤ j ≤ N) indicates each element of D, it is represented as follows: where r i is the size of each local region. This setting emphasizes the cost of larger regions in Eq. (7). L is also an N × N matrix, and it is referred to as the matting Laplacian matrix. L i j is represented as follows: where δ i j is Kronecker delta, ω k indicates a local window, |ω k | is the number of pixels in ω k , μ k and σ k are the 3 × 1 averaged color vector and 3 × 3 covariance matrix, respectively, which are calculated from the pixel values in ω k , E 3 is 3 × 3 identity matrix. L i j has a value calculated by Eq. (9) only when (i, j) ∈ ω k , and L i j is 0 in other cases. Equation (7) is a quadratic equation for b. Therefore, Eq. (7) has a minimum value when ∂C(b)/∂b = 0 holds. After partially differentiating Eq. (7) with respect to b, the following equation is derived.
A full blur map is obtained as a solution of Eq. (10), and the estimated blur map is illustrated in Fig. 6.

Performance Evaluation
In the following, we present performance evaluations for the blur maps. The performance evaluations were conducted based on previously proposed evaluation criteria [18]. We are the maximum value and averaged value, respectively, which are related to a threshold p. |Info(p)| is the absolute Informedness [20], which is calculated from a segmented result by naive-thresholding. |Info| is invariant to the exchange labeled values of the sharp and blur regions, which is a desirable characteristic when evaluating the discrimination of blur maps. We also evaluated the performance when blur maps were segmented using Otsu's method [21] in order to illustrate the proposed method's performance in automatic applications.
We used the dataset prepared by Shi et al. [6] in the performance evaluation. This dataset contains 704 focal blur images and 296 motion blur images. Each image has a manually defined ground truth and four blur maps generated by state-of-the-art methods [1], [3], [4], [6]. In addition to these blur maps, we also evaluated the blur maps generated by Zhuo et al. [9]. The blur maps generated by [9] have similar characteristics to those obtained by the proposed method. We note that both the proposed method and that described by [9] use the EAI proposed by Levin et al. [17] to generate a full blur map. The proposed method adds a modification to the EAI as described in Eq. (8) in order to handle multiplesized grids.
First, the effects ofÂ min on the generated blur maps are shown in Fig. 7. The blur maps in the top and bottom rows were estimated from Fig. 1 (a) and (b), respectively. The ANGHS values obtained for the motion blur image were unstable whenÂ min = [1, 30] because of some strong edges along the blur direction and small intensity amplitudes in the blur regions. Large values forÂ min can avoid the problem with normalization in these areas, but setting excessively high values forÂ min will also cause problems. The body areas of the focal blur image had small ANGHS values when A min = 80, and the discrimination of the blur map was degraded. It is difficult to set appropriate values forÂ min automatically because the most appropriate values depend on each image. Thus, we investigated the suitable settings experimentally.
The dependence of the performance on different technical components of the proposed method is summarized in Table 1, where the results obtained for multiple-sized grids with ANGHS using k-means and the EM algorithm are denoted by GM and GM-EM, respectively. The re- Fig. 7 Effects of different values ofÂ min on the generated blur maps.   Table 1, GM was superior to the other methods using a broad range for A min . These results indicate the effectiveness of our proposed techniques.Â min = 30 andÂ min = 80 obtained the highest scores for the focal blur images and motion blur images, respectively. We used these settings forÂ min to make comparisons with the other algorithms. Examples of the blur maps and segmented results obtained by Otsu's method are shown in Fig. 8. The first col-umn in Fig. 8 shows the input images and ground truths. The other columns in Fig. 8 show each blur map and each segmented result obtained by Otsu's method. We note that pixels located near the image borders were not used in the evaluations because the blur features in these regions were often inaccurate. The unused areas of the ground truths and segmented results are colored in gray in Fig. 8. The blur maps generated by the proposed method and the method of Zhuo et al. appeared smoother than those produced by the other methods, which is attributable to differences in the approaches employed to generate the blur maps, as explained in Sect. 1. Moreover, the blur maps obtained by the proposed method exhibited better discrimination than those produced using the method of Zhuo et al.
Finally, the performance of each method is summarized in Fig. 9. Figure 9 (a) and (b) show the results in terms of the focal and motion blur segmentation, respectively. The blue, red, and green bars in Fig. 9 represent the summarized maximum values, the results obtained by Otsu's method, and the averaged values, respectively. Each error bar denotes the standard deviation. The values on the upper error bar denote the summarized values for all the images. The maximum segmentation performance and Otsu's method were superior to the other methods in all cases, as shown by the blue and red bars in Fig. 9, respectively. These results indicate the high discrimination of the blur maps generated using the proposed method. The blur maps obtained by the proposed method were highly sensitive to tuning the threshold because the differences between the maximum values and averaged values were large compared with the blur maps produced using the methods of Shi et al. and Liu et al. It is difficult to apply thresholding with a fixed parameter to the blur maps obtained by the proposed method, but automatic thresholding or other sophisticated segmentation algorithms are used more often than thresholding with a fixed parameter in automatic applications. Therefore, the proposed method will be superior to the other methods in automatic applications, as shown by the results obtained using Otsu's method.

Conclusions and Future Work
In this study, we addressed two important issues to improve the discrimination of blur maps. First, we considered the estimation of a local blur feature that is robust to variations in the intensity amplitude. The experimental comparison showed that the blur feature ANGHS is more robust than GHS to variations in the local amplitude. Thus, ANGHS can handle local regions in a more appropriate manner than GHS. Second, we proposed a scheme for generating blur maps in order to improve the discrimination of blur maps. Blur maps are affected by local blur features but also by the contents and sizes of the local regions, as well as the assignment of blur feature values to pixels. In our proposed method, we apply multiple-sized grids and EAI to each task. Comparisons of the results obtained using numerous partial blur images demonstrated the high discrimination of the blur maps generated with the proposed method. The components of the proposed method do not require learning any models and parameters in advance; therefore, the proposed method is suitable for many applications.
In future research, we aim to improve the performance of motion blur images because the current performance is insufficient. Liu et al. [1] showed that motion blur images have directional characteristics. They used these directional characteristics to determine the blur types after region segmentation or detection, but these characteristics can also be used for segmentation. For example, segmentation based on two-dimensional features using a blur feature and the blur direction may improve the performance of motion blur images. These improvements may enhance the generality and usability of partial blur segmentation algorithms. An appropriate value forÂ min depends on each image, but we use a fixed value forÂ min in our method. Thus, defining a specific criterion or the automatic selection of appropriate settings must be addressed.