Developmental improvements in working memory are important in the acquisition of new skills, like reading and maths. Current accounts of the brain systems supporting working memory rarely take development into account. However, understanding the development of these skills, and in turn where this development can go awry, will require more sophsiticated neuropsychological accounts that fully consider the role of development. The current study investigated how structural brain correlates of components of the working memory system change over developmental time. Verbal and visuospatial short-term and working memory were assessed in 153 children between 6 and 16 years and latent components of the working memory system were derived using principal component analysis. Further, fractional anisotropy and cortical thickness maps were derived from T1-weighted and diffusion-weighted MRI and processed using eigenanatomy decomposition, an advanced dimensionality reduction method for neuroimaging data. We were then able to explore how the structural brain correlates of working memory gradually shifted across childhood. Regression modelling indicated greater involvement of the corpus callosum and posterior temporal white matter in younger children for performance associated with the executive part of the working memory system, while thickness of the occipitotemporal cortex was more predictive in older children. These findings are consistent with an account in which increasing specialisation leads to shifts in the contribution of neural substrates over developmental time, from early reliance on a distributed system supported by long-range connections to later reliance on specialised local circuitry. Furthemore, our findings emphasise the importance of taking development into account when considering the neural systems that support complex cognitive skills, like working memory.