Genomes evolve through a medley of mutation, drift, and selection, all of which act heterogeneously across genes and lineages. The pacemaker models of genomic evolution describe the resulting patterns of evolutionary rate variation: genes that are governed by the same pacemaker exhibit the same pattern of rate heterogeneity across lineages. However, the relative importance of drift and selection in determining the structure of these pacemakers is unknown. Here, we propose a novel phylogenetic approach to explain the formation of pacemakers. We apply this method to a genomic dataset from holometabolous insects, an ancient and diverse group of organisms. We show that when drift is the dominant evolutionary process, each pacemaker tends to govern a large number of fast-evolving genes. In contrast, strong negative selection leads to many distinct pacemakers, each of which governs a few slow-evolving genes. Our results provide new insights into the interplay between drift and selection in driving genomic evolution.