How are we able to reason about abstract concepts that lie resolutely beyond the reach of perception? One strategy is to ground understanding in space. Numbers, for instance, are known to interact with egocentric space during rapid numerical judgments. A range of experimental results have demonstrated that, among literate Western people, this “mental number-line” goes from left to right, with smaller numbers associated with left space, and larger numbers with right space. But what is the nature of this “space”? Previous work has conflated multiple possible egocentric frames of reference—head-based, eye-based, action-based—leaving it unclear which space is interacting with number. In the present paper, two studies investigated whether a single centrally-located button, stationary in hand- and eye-based coordinates, can nevertheless exhibit different spatial properties in virtue of task-specific activity. In a go/no-go paradigm, participants judged the magnitude (Exp. 1) and parity (Exp. 2) of single-digit numbers. Crucially, they responded only with the index or middle finger of a single hand. While judging magnitude (Exp. 1), participants were faster to respond to smaller numbers with the more leftward finger, and larger numbers with the more rightward finger, regardless of the hand being used. This effect disappeared when judging parity (Exp. 2), replaced by finger-specific associations on the left hand only. In sum, in a task-sensitive way, participants associated numbers with egocentric space—but a behavioral space defined relative to embodied interaction rather than head- or eye-based reference frames. We discuss implications for number representation and the nature of “space” in embodied activity.