Visual perspective taking (VPT) – people’s ability to represent the physical world from another person’s viewpoint – plays a fundamental role in social cognition. However, little is known about whether and when VPT can be triggered spontaneously without any explicit verbal prompting. In six studies, we measured spontaneous VPT as the tendency to read an ambiguous number from another agent’s imagined perspective (“6”) rather than from one’s own default visual perspective (“9”). We found that the likelihood of spontaneous VPT varied systematically with the target agent’s behavior. The strongest trigger for spontaneous VPT was the agent’s goal-directed reaching, followed by object-directed gaze, and lastly the agent’s mere presence in the scene. Furthermore, observing an agent’s reaching or gaze toward an object triggered viewers’ spontaneous VPT even for objects with which the agent was currently not engaged.