Are numerals divorced from a sense of the actual quantities they represent? We show that accessing a sense of how much a numerical symbol actually represents is a surprisingly difficult and non-trivial process. Irrespective of numerical size or distance, direct comparison of the relative quantities represented by symbolic and non-symbolic quantities leads to performance markedly worse than when comparing two non-symbolic quantities. Experiment 2 shows that this effect cannot be attributed to differences in perceptual processing streams. Experiment 3 shows that there is no additional cost of mixing two formats that are both symbolic; that is, the decrement in mixing formats is specific to mixing symbolic and non-symbolic representations. Our data are consistent with the view that numerical symbols operate primarily as an associative system in which relations between symbols come to overshadow those between symbols and their quantity referents.