Visual search was investigated using a color target presented among heterogeneous distractors with several colors within a certain color difference in the OSA uniform color scales. The search time did not depend on color difference between the target and the distractors, but on color categories of the target and distractors. In the conditions that a single color category occupied a large number of distractors search time was longer for the targets in the same category than for the targets in other categories, although the color differences were constant. We proposed a categorical color search model, in which categorical color perception had an important role to determine response time of visual search. Our results suggest that the higher-order color mechanism influences the heterogeneous multi-color visual search.