Slowing is a common feature of ageing, yet a direct relationship between neural slowing and brain atrophy is yet to be established in healthy humans. We combine magnetoencephalographic (MEG) measures of neural processing speed with magnetic resonance imaging (MRI) measures of white- and gray-matter in a large (N=617, 18-88yrs), population-derived cohort (www.cam-can.org) to investigate the relationship between age-related structural differences and visual and auditory evoked response delay. Using a novel technique, we show that visual-evoked responses exhibit a constant delay, whereas auditory-evoked responses exhibit delay that accumulates over time. Visual delay is mediated by white-matter microstructure in the optic radiation, presumably reflecting increased transmission time, whereas auditory delay is mediated by gray-matter differences in auditory cortex, presumably reflecting less efficient local processing. Our results demonstrate that age has dissociable effects on neural processing speed, and that these effects relate to different types of brain atrophy.