Policy decisions on political, economic, legal, and health issues are often made by groups that rarely exceed 40 members and are typically much smaller. Given that wisdom is usually attributed to large crowds, should committees be larger? Using computer simulations and mathematical analyses we show that group accuracy, averaged across the range of difficulty that would be encountered in real-world tasks, is often maximized for moderate-sized groups. The result holds whenever the accuracy in easy tasks is above chance more than the accuracy in difficult tasks is below chance and the easy tasks are encountered more often. Note that for the result to hold, it is not necessary to assume selective sampling of group members according to their individual accuracy. A smaller committee that would produce more accurate decisions in our model can simply be selected randomly out of a larger group of experts.