Several studies have demonstrated that language encodes geographical information. That is, the relative longitude and latitude of city locations can be extracted from language. Whether people actually rely on these linguistic features is less clear. Recent studies have suggested that language statistics plays a role in geographical estimates, but these studies rely on map drawings, a fundamentally perceptual task. The current study investigated the extent to which people rely on map representations and statistical linguistic frequencies by using a linguistic task. Participants saw U.S. city pairs in their iconic positions (a more northern city is presented above a more southern city, or a more western city is presented to the left of a more eastern city), and in their reverse-iconic positions (a more southern city is presented above a more northern city, or a more eastern city is presented to the left of a more western city). For iconic city pairs both in the east – west (Seattle – Boston) and north – south (Memphis – Miami) configurations, RTs were determined by the iconicity. No effect was obtained for statistical linguistic frequencies. However, when city pairs were presented in a reverse-iconic configuration, for both horizontal (Boston – Seattle) and vertical (Miami – Memphis) orientations, both perceptual and linguistic factors explained RTs. These findings support the idea that cognition relies on a shallow heuristic, a linguistic system, and a fine-grained and more precise perceptual simulation system.