The visual system takes advantage of redundancies in the scene by extracting summary statistics from groups of similar items. Similary, in social situations, we routinely make snap judgments of crowds of people. Reading "crowd emotion" is critical for guiding us away from danger (e.g., mass panic or violent mobs) and towards help from friendly groups. Scrutinizing each individual's expression would be too slow and inefficient. How the brain accomplishes this feat, however, remains unaddressed. Here we report a set of behavioral and fMRI studies in which participants made avoidance or approach decisions by choosing between two facial crowds presented in the left and right visual fields (LVF/RVF). Behaviorally, participants were most accurate for crowds containing task-relevant cues-avoiding angry crowds/approaching happy crowds. This effect was amplified by sex-linked facial cues (angry male/happy female crowds), and highly lateralized with greater recognition of task-congruent stimuli presented in LVF. In a related fMRI study, the processing of facial crowds evoked right-lateralized activations in the dorsal visual stream, whereas similar processing of single faces preferentially activated the ventral stream bilaterally. Our results shed new light on our understand of ensemble face coding, revealing qualitatively different mechanisms involved in reading crowd vs. individual emotion.