Most scholars say that "freedom of speech" includes "freedom not to speak". But what about the Freedom not to speak is infringed? How can we decide when freedom not to speak is restricted? The aim of this paper is to demonstrate that question, focusing on cases and theory in the United States. By analyzing many cases, I distinguished the "freedom not to speak" based on "freedom of thought" from "freedom not to speak" based on "freedom of speech", and maintain that the former is infringed if government coerces someone to speak on a political or ideological matter, the latter is infringed only if the listener understand that the expression which is not related to political or ideological matter coerced by government is being attributed to the speaker. That is to say, the freedom not to speak based on "freedom of speech" means the right of an individual to not have expressions contrary to his thinking attributed to him by government. Underlying that understanding, I urge, the harm caused by the forced expression is not to the speaker, but to the listener's interest. This understanding may have many difficulties, but it also has much practical significance worthy of consideration.