On the transition of genetic differentiation from isolation to panmixia: What we can learn from GST and D

Population genetic differentiation characterizes the repartition of alleles among populations. It is commonly thought that genetic differentiation measures, such as GST and D, should be near zero when allele frequencies are close to their expected value in panmictic populations, and close to one when they are close to their expected value in isolated populations. To analyse those properties, we first derive analytically a reference function f of known parameters that describes how important features of genetic differentiation (e.g. gene diversity, proportion of private alleles, frequency of the most common allele) are close to their expected panmictic and isolation value. We find that the behaviour of function f differs according to three distinct mutation regimes defined by the scaled mutation rate and the number of populations. Then, we compare GST and D to f, and demonstrate that their signal of differentiation strongly depends on the mutation regime. In particular, we show that D captures well the variations of genetic diversity when mutation is weak, otherwise it overestimates it when panmixia is not met. GST detects population differentiation when mutation is intermediate but has a low sensitivity to the variations of genetic diversity when mutation is weak. When mutation is strong the domain of sensitivity of both measures are altered. Finally, we also point out the importance of the number of populations on genetic differentiation measures, and provide recommendations for the use of GST and D.