Implicit bias has recently gained much attention in scholarly attempts to understand and explain different forms of social injustice by identifying causally relevant mental states in individual’ minds. Here we question the explanatory power of implicit bias in a particular type of injustice, testimonial injustice, and more generally in what we call speech injustice. Testimonial injustice occurs when the audience deflates a speaker’s credibility due to the speaker’s perceived social identity (Fricker, 2007). We identify two drawbacks of a widely accepted explanation attributing testimonial injustice to prejudices (e.g. implicit bias) in the mind of the hearer, and argue that further understanding of this phenomenon can be gained from a structural explanation that appeals to discursive conventions and interlocutors’ positions in the communicative exchange.