Social evaluations depend on our ability to interpret other people’s behavior. In adults, these evaluations are influenced by our perception of the competence and motivation of the agent: helping when it is difficult to help is praiseworthy; not helping when it is easy to help is reprehensible. Here we look at young children’s capacity to make competence attributions and its relation to their social evaluations. We find that as early as 18-months, infants can use the time and effort associated with achieving a goal-directed action to distinguish agents, and that infants prefer more competent agents. When asked to choose between two agents who act as moral bystanders and refuse to engage in a helpful action, we find a sustained preference for the more competent agent until the age of three, when the preference is reversed. We argue that the ability to calculate the cost and benefits of goal-directed action originates in early childhood and plays a fundamental role in moral reasoning.