Species barriers, expressed as hybrid inviability and sterility, are often due to epistatic interactions between divergent loci from two lineages. Theoretical models indicate that the strength, direction, and complexity of these genetic interactions can strongly affect the expression of interspecific reproductive isolation and the rates at which new species evolve. Nonetheless, empirical analyses have not quantified the frequency with which loci are involved in interactions affecting hybrid fitness, and whether these loci predominantly interact synergistically or antagonistically, or preferentially involve loci that have strong individual effects on hybrid fitness. We systematically examined the prevalence of interactions between pairs of short chromosomal regions from one species (Solanum habrochaites) co-introgressed into a heterospecific genetic background (Solanum lycopersicum). We used a diallel crossing design to generate most pairwise combinations of 15 chromosomal segments from S. habrochaites in the background of S. lycopersicum (i.e., 95 double introgression lines). We compared the strength of hybrid incompatibility (either pollen sterility or seed sterility) expressed in each double introgression line to the expected additive effect of its two component single introgressions. We found that: epistasis was common among co-introgressed regions; epistastic effects were overwhelmingly antagonistic (i.e., double hybrids were less unfit than expected from additive single introgression effects); and, epistasis was substantially more prevalent in pollen fertility compared to seed fertility phenotypes. Together, these results indicate that high-order interactions frequently contribute to postzygotic sterility barriers in these species. This pervasive epistasis leads to the decoupling of the patterns of accumulation of isolation loci and isolation phenotypes, and is expected to attenuate the rate of accumulation of hybrid infertility among lineages over time (i.e., giving diminishing returns as more reproductive isolation loci accumulate). This decoupling effect might also explain observed differences between pollen and seed fertility in their fit to theoretical predictions of the accumulation of isolation loci, including the snowball effect.