According to the competition account of lexical selection in word production, conceptually driven word retrieval involves the activation of a set of candidate words in left temporal cortex, and competitive selection of the intended word from this set, regulated by frontal cortical mechanisms. However, the relative contribution of these brain regions to competitive lexical selection has remained unclear. In the present study, five patients with left prefrontal-cortex lesions (overlapping in ventral and dorsal lateral cortex), eight patients with left lateral temporal-cortex lesions (overlapping in middle temporal gyrus), and 13 matched controls performed a picture-word interference task. Distractor words were the picture name itself (congruent condition), semantically related or unrelated to the picture. Semantic interference (related vs unrelated), tapping into competitive lexical selection, was examined. An overall semantic interference effect was observed for the control and left-temporal groups separately. The left-frontal patients did not show a robust semantic interference effect as a group. The left-temporal patients had increased semantic interference in the error rates relative to controls. Error distribution analyses indicated that these patients had more hesitant responses for the related than for the unrelated condition. We argue that left middle temporal lesions affect the lexical activation component, making lexical selection more susceptible to errors. By contrast, the top-down regulation over competitive lexical selection in picture-word interference does not seem to be dependent on the left lateral prefrontal cortex.