The previous research which investigated the effect of verbal encoding on the memory of visual information has yielded contradictory results: facilitation and inhibition of performance by verbal encoding. However, it has not been elucidated why and how these conflicting results were brought about. The main purpose of this study is to show that verbal encoding not only facilitates but also impairs performance by taking task demand and characteristics of visual representations into account. Another purpose is to investigate how verbal encoding would influence the memory of color information, which is irrelevant for the memory task. In Experiment 1, the results showed that verbal encoding could have two opposite effects in the same task. In Experiment 2, the results showed that the effect of verbal encoding on the memory of color would differ from that on the memory of the shapes of pictures.