RT Journal Article SR Electronic T1 The Equidistance Index of Population Structure JF bioRxiv FD Cold Spring Harbor Laboratory SP 033852 DO 10.1101/033852 A1 Yaron Granot A1 Omri Tal A1 Saharon Rosset A1 Karl Skorecki YR 2016 UL http://biorxiv.org/content/early/2016/03/22/033852.abstract AB Measures of population differentiation, such as Fst, are traditionally derived from a partition of heterozygosities within and between populations. However, the emergence of population clusters from multilocus analysis is a function of genetic structure (departures from panmixia) rather than of diversity. If the populations are close to panmixia, slight differences between the mean pairwise distance within and between populations (low FST) can manifest as strong separation between the populations, thus population clusters are often evident even when the vast majority of diversity is partitioned within populations rather than between them. Moreover, because Fst is also a function of internal diversity, it does not directly reflect the strength of separation between population clusters. For any given FST value, clusters can be tighter (more panmictic) or looser (more stratified), and in this respect higher Fst does not always imply stronger differentiation. Finally, FST as a measure of structure or population distance is a 'supervised’ measure, in the sense that target populations have to be predefined (samples labeled). In this study we propose a measure for the partition of structure, denoted Est, which is more consistent with results from clustering schemes. Crucially, our measure is based on a statistic of the data that is a good measure of internal structure, mimicking the information extracted by unsupervised clustering or dimensionality reduction schemes. To assess the utility of our metric, we ranked various human (HGDP) population pairs based on FST and EST and found substantial differences in ranking order. In some cases examined, most notably among isolated Amazonian tribes, Est ranking seems more consistent with demographic, phylogeographic and linguistic measures of classification compared to Fst. Thus, Est may at times outperform FST in identifying evolutionarily significant differentiation.