Phonetic symbolism is a phenomenon that speech sound evokes images based on sensory experiences and thus often discussed with the similarity for cross-modal correspondence. Hirata and Kita (2012) showed cross-modal congruence between brightness/darkness and Chinese speech sound with and without aspiration in Chinese speakers by using Garner's task. In the present study, we examined whether Japanese speakers, which do not have any Chinese knowledge, show cross-modal correspondence to the Chinese speech sound or not. We conducted the same experiment as previous research to Japanese speakers with no Chinese experience. As a result, Chinese speech sounds with aspiration, which resemble voiceless consonants, were not matched with brightness, whereas those without aspiration, which resemble voiced consonants, were not matched with darkness. This result is different to its pattern in Chinese speaker and consequently suggests that phonetic symbolism is affected by the knowledge of the phonemes of its language.