We explore the interaction between local-global information processing in visual perception, leveraging a visual phenomenon known as crowding, whereby the perception of a target is impaired by the presence of nearby flankers. Most of the established models explain the crowding effect in terms of local interactions. However, recent experimental results indicate that a classical crowding effect, the deterioration in the discrimination of a vernier stimulus embedded in a square, is alleviated by the presence of additional flankers (uncrowding). Here, we propose that crowding and uncrowding arise from cortical inferences about hierarchically organized groups, and formalize this concept using a hierarchical Bayesian model. We show that the model reproduces crowding and uncrowding for flanked vernier discrimination. More generally, the model provides a normative explanation of how visual information might simultaneously flow bottom-up, top-down, and laterally, to allow the visual system to process local and global features in the visual scene interactively.