Action recognition is important for social interactions. Because little is known about the visual tuning properties of processes involved in action recognition, we examined the visual tuning properties of action recognition by means of a behavioral adaptation paradigm. Participants were adapted to images showing a person hitting or waving and subsequently categorized test images showing an ambiguous action as either hitting or waving. We found the perception of the test images to be significantly biased away from the adapted action (action adaptation aftereffect (AAA)). Subsequent experiments ruled out that the AAA was not merely driven by the adaptation of local visual contrast or the emotional content of the action. However adaptation to action words (e.g. “hitting” or “waving”) did not induce an AAA. Finally we found evidence for the AAA being modulated by the social context in which an action is embedded, suggesting high level influences on action recognition.