Music notation and English word reading have similar visual processing requirements. It remains unclear how the two skills influence each other. Here we investigated the modulation of music reading expertise on visual processing of English words through an ERP study. Participants matched English real, pseudo, and non-words preceded by musical segments or novel symbol strings in a sequential matching task. Musicians showed smaller N170 amplitude in response to English non-words preceded by musical segments than by novel symbol strings in the right hemisphere. This effect was not observed in real or pseudo-words, or in any of non-musicians’ responses. Similar to English non-words, musical segments do not have morphological rules or semantic information, giving rise to this modulation effect. This finding suggested a shared visual processing mechanism in the right hemisphere between music notation and English non-word reading, which may be related to serial symbol processing as suggested by previous studies.