This study improves the compression efficiency of Lee's colorization-based coding framework by introducing a novel colorization matrix construction and an adaptive color conversion. Colorization-based coding methods reconstruct color components in the decoder by colorization, which adds color to a base component (a grayscale image) using scant color information. The colorization process can be expressed as a linear combination of a few column vectors of a colorization matrix. Thus it is important for colorization-based coding to make a colorization matrix whose column vectors effectively approximate color components. To make a colorization matrix, Lee's colorization-based coding framework first obtains a base and color components by RGB-YCbCr color conversion, and then performs a segmentation method on the base component. Finally, the entries of a colorization matrix are created using the segmentation results. To improve compression efficiency on this framework, we construct a colorization matrix based on a correlation of base-color components. Furthermore, we embed an edge-preserving smoothing filtering process into the colorization matrix to reduce artifacts. To achieve more improvement, our method uses adaptive color conversion instead of RGB-YCbCr color conversion. Our proposed color conversion maximizes the sum of the local variance of a base component, which resulted in increment of the difference of intensities at region boundaries. Since segmentation methods partition images based on the difference, our adaptive color conversion leads to better segmentation results. Experiments showed that our method has higher compression efficiency compared with the conventional method.