Many aspects of our experiences do not have to be explicitly remembered, but can be inferred based on our knowledge of the regularities in our environment. Such knowledge can operate at multiple levels of abstractions. For example, this could lead to recall for the height of a particular person to be influenced not only by general knowledge about heights of people, but also by specific knowledge about the height of men and women. We assess the relative contribution of this type of prior knowledge on reconstructive memory. In a series of behavioral studies we first assessed peoples a priori expectations of the heights of men and women. We show that peoples a priori expectations are in line with the true distribution of heights in the population. We then tested memory performance in a continuous recall task in which subjects had to reconstruct from memory the height of people shown earlier in a sequence. The stimuli were either naturalistic images of males and females or gender-ambiguous silhouettes. Our results suggest not only that prior knowledge can improve average recall, but also that knowledge can come from multiple levels of abstraction such as gender and the overall height of people.