The visual system, like the brain more broadly, relies heavily on categorical representations. It is easier to spot a visual difference that crosses a category boundary, e.g., between blue and green, or between vertical and oblique. Here we show that topological relations between objects are similarly categorical. When asked to detect changes between object arrangements, participants were better at detecting those changes that crossed hypothesized category boundaries, such as 'overlapping', or 'touching', compared to equally-sized changes that did not. These effects were magnified at increased memory load, presumably because this categorization forms a more efficient code. This finding, consistent with previous computational modeling work, suggests that categorical relations are critical for remembering and comparing complex images.