Congenital prosopagnosia refers to a lifelong impairment in face identification that is present from birth. In contrast to acquired prosopagnosia, where the deficit is due to brain damage, people with congenital prosopagnosia never evolve a functional face recognition system in the first place but develop compensatory processing strategies to overcome their deficit. In order to assess both deficit and compensatory processing in congenital prosopagnosia, we conducted a series of experiments with a large group of 15 prosopagnosic participants. The tests administered covered different aspects of face and object recognition: Identification under unlimited and restricted viewing times, the influence of rotation in depth, recall of target stimuli after one year, and recognition of famous faces. Based on the test results, we propose a categorization of congenital prosopagnosia along the lines of an apperceptive, associative or amnestic deficit and discuss this categorization scheme with respect to potential neuroanatomical differences underlying the deficit.