Words divide the world into labeled categories. Languages vary in the categories they label, sometimes to the point of making cross-cutting divisions of the same space. Previous work suggests two opposing hypotheses about how communication contributes to category emergence: 1) these spaces lack an objective shared similarity structure, and communication dynamically creates one of a number of optimally shareable category structures; 2) the category structures resulting from communication are not necessarily optimal, but diverge from a shared similarity space in language-specific ways. We had participants categorize images drawn from a continuous space in two conditions: a) non-communicative, by similarity, b) communicative, dynamically creating categories when playing a partnered communication game. The memory demands of communication lead to reliance on salient images and early conventions, resulting in non-optimal category structures compared to non-communicative participants. This supports the hypothesis that communication leads to categories that diverge non-optimally from a shared similarity space.