Large brains are metabolically expensive but support skills (or cognitive abilities, knowledge, information, etc.) that allow overcoming ecological and social challenges, with social challenges being thought to strongly promote large-brain evolution by causing evolutionary arms races in cognition yielding exaggerated brain sizes. We formulate a mathematical model that yields quantitative predictions of brain and body mass throughout ontogeny when individuals evolve facing ecological but no social challenges. We find that ecological challenges alone can generate adult brain and body mass of ancient human scale, showing that evolutionary arms races in cognition are not necessary for extreme brain sizes. We show that large brains are favored by intermediately challenging ecological environments where skills are moderately effective and metabolically expensive for the brain to maintain. We further show that observed correlations of cognitive abilities and brain mass can result from saturation with skill maintenance of the brain metabolic rate allocated to skills.