From something as simple as judging the time to more complicated behaviours like answering trivia questions, our cognitive systems always provide us with a representation of confidence: the probability of being correct. The development of confidence has been a long-standing issue in cognitive and developmental science. However, most studies assess children’s confidence through either extensively trained numerical or verbal scales (“I am sure”), or by asking children to gamble on their answer. These measures stand to confuse metacognition with the development of language and inhibitory control. Here, we validate a novel model and task that measures individual and developmental differences in confidence relatively (“Are you more confident in X or Y”). Subsequently, we apply this task to demonstrate that metacognitive abilities of children aged 5–8 show significant development in the domain of intuitive number representations. These results are discussed in a broader context of theory and measurement of metacognition.