TY - JOUR T1 - Phylogenomic species tree estimation in the presence of incomplete lineage sorting and horizontal gene transfer JF - bioRxiv DO - 10.1101/023168 SP - 023168 AU - Ruth Davidson AU - Pranjal Vachaspati AU - Siavash Mirarab AU - Tandy Warnow Y1 - 2015/01/01 UR - http://biorxiv.org/content/early/2015/07/23/023168.abstract N2 - Background Species tree estimation is challenged by gene tree heterogeneity resulting from biological processes such as duplication and loss, hybridization, incomplete lineage sorting (ILS), and horizontal gene transfer (HGT).Mathematical theory about reconstructing species trees in the presence of HGT alone or ILS alone suggests that quartet-based species tree methods (known to be statistically consistent under ILS, or under bounded amounts of HGT) might be effective techniques for estimating species trees when both HGT and ILS are present.Results We evaluated several publicly available coalescent-based methods and concatenation under maximum likelihood on simulated datasets with moderate ILS and varying levels of HGT. Our study shows that two quartet-based species tree estimation methods (ASTRAL-2 and weighted Quartets MaxCut) are both highly accurate, even on datasets with high rates of HGT. In contrast, although NJst and concatenation using maximum likelihood are highly accurate under low HGT, they are less robust to high HGT rates.Conclusion Our study shows that quartet-based species-tree estimation methods can be highly accurate under the presence of both HGT and ILS. The study suggests the possibility that some quartet-based methods might be statistically consistent under phylogenomic models of gene tree heterogeneity with both HGT and ILS.AbbreviationsCA-MLconcatenated analysis using maximum likelihoodGTR+GammaGeneralized Time Reversible model of site evolution with Gamma distributed rates across sites HGT: horizontal gene transferHGThorizontal gene transferILSincomplete lineage sortingMSCmulti-species coalescentMLmaximum likelihoodMLBSmulti-locus bootstrappingMQSSTmaximum quartet support species treeMSCmulti-species coalescent ER -