We administered a semantic association judgment test to patients with aphasia, and investigated their semantic processing. The subjects were 35 aphasics and 10 healthy individuals (control group). Each participant was instructed to select one item that was the most related to the target item from five choices. We controlled the semantic relationship between the choices and the target item. For example, when the target was “dog,” one item related to both situation and category (SC: “cat”), one related only to situation (S: “house”), one related only to category (C: “elephant”), and the others did not relate to either situation or category (N1 and N2: “bream” and “eraser”). The results showed that the responses by the patients with aphasia were more varied than those by the controls. The number of N1 and N2 responses was significantly larger in the patients with aphasia than those by the controls. The proportion of related items in the subgroup of mild aphasia (good auditory comprehension group) was significantly higher than that in the subgroup of severe aphasia (poor comprehension group). These results suggest that there are functional associations between non-verbal semantic processing, and severity and auditory comprehension in patients with aphasia.