This paper demonstrates a new quantitative approach to identify what is behind universally sensed sound-symbolism and sound-symbolism detected only by speakers of a particular language. For this purpose, we presented 70 locomotion videos to English and Japanese speakers and asked them to create a word that would sound-symbolically match each action. Then the participants rated each action on 6 semantic dimensions. Multivariate analysis detected what level of sound-unit (e.g., phonetic features, phoneme, mora) are linked to each semantic dimension. Results revealed that certain sound-meaning links (e.g., voicing and heaviness) are more consistent than others within and across languages. Furthermore, language-specific sound-symbolism was found in the particular sound-units (e.g., the sequence of a voiced initial consonant and a middle-low vowel and heaviness in English). This implies that the language-specific sound-symbolism is motivated by the phonological system each language possesses, whereas universal sound-symbolism appears at the more abstract level of sound component.