This paper demonstrates a new quantitative approach to identify what is behind universally sensed sound symbolism and sound symbolism detected only by speakers of a particular language. We presented 70 locomotion videos to Japanese and English speakers and asked them to create a word that would sound-symbolically match each action, then to rate the action on five semantic dimensions. Multivariate analyses revealed that certain sound-meaning links (e.g., voicing and speed) were more consistent than others within and across languages. Language-specific sound symbolism was also found for some sound-meaning links (e.g., the affricate manner of articulation was associated with light motions in Japanese, but with heavy motions in English). This implies that cross-linguistically shared and language-specific parts of sound symbolism are intricately intertwined within each language. This research underscores the importance of a bottom-up approach which can exploratorily investigate the complex sound-symbolic systems as a whole.