Understanding the origin of species is as Darwin called it "that mystery of mysteries". Yet, how the processes of evolution give rise to non-interbreeding species is still not well understood. In an empirical search for a genetic basis, transcription factor DNA binding has been identified as an important factor in the development of reproductive isolation. Computational and theoretical models based on the biophysics of transcription factor DNA binding have provided a mechanistic basis of such incompatibilities between allopatrically evolving populations. However, gene transcription by such binding events occurs embedded within gene regulatory networks, so the importance of pair-wise interactions compared to higher-order interactions in speciation remains an open question. Theoretical arguments suggest that higher-order incompatibilities should arise more easily. Here, we show using simulations based on a simple biophysical genotype phenotype map of spatial patterning in development, that biophysics provides a stronger constraint, leading to pair-wise incompatibilities arising more quickly and being more numerous than higher-order incompatibilities. Further, we find for small, drift dominated, populations that the growth of incompatibilities is largely determined by sequence entropy constraints alone; small populations give rise to incompatibilities more rapidly as the common ancestor is more likely to be slightly maladapted. This is also seen in models based solely on transcription factor DNA binding, showing that such simple models have considerable explanative power. We suggest the balance between sequence entropy and fitness may play a universal role in the growth of incompatibilities in complex gene regulatory systems.