Previous metaphor studies have paid much attention to nominal metaphors and predicative metaphors and little attention has been given to adjective metaphors. The most adjective metaphor studies have only examined how the acceptability of adjective metaphors can be explained by the pairing of adjective modifier’s and head noun’s modalities. Sakamoto & Utsumi (2009) showed that adjective metaphors, especially those modified by color adjectives, tend to evoke negative meanings. Sumihisa et al (2011) examined whether evoking negative meanings is the unique feature of adjective metaphors through comparison among nominal metaphors and predicative metaphors for the Japanese language and revealed that meanings of metaphors are basically affected by meanings of vehicles, but when vehicles themselves had neutral meanings, negative meanings were evoked more frequently for adjective metaphors among the other types of metaphors. The purpose of this study, therefore, explores the reason why adjective metaphors evoke negative meanings more frequently than the other types of metaphors. For this purpose, we examined what kind of meanings associated with topics or vehicles affect the comprehension of metaphors. Our psychological experiments revealed that meanings associated from vehicles affect the comprehension of metaphors. And when metaphorical expressions have vehicles with positive or negative meanings, metaphorical expressions show the same meanings as the vehicles. On the other hand, when metaphorical expressions have vehicles with neutral meaning, only adjective metaphors evoke negative meanings. Our results suggest that the comprehension of adjective metaphors is selectively affected by the negative meanings associated with adjectives as vehicles.