Effect of epistasis and linkage on fixation probability in three-locus models: An ancestral recombination–selection graph approach

We study the probability of ultimate fixation of a single new mutant arising in an individual chosen at random at a locus linked to two other loci carrying previously arisen mutations. This is done using the Ancestral Recombination–Selection Graph (ARSG) in a finite population in the limit of a large population size, which is also known as the Ancestral Influence Graph (AIG). An analytical expansion of the fixation probability with respect to population-scaled recombination rates and selection intensities is obtained. The coefficients of the expansion are expressed in terms of the initial state of the population and the epistatic interactions among the selected loci. Under the assumption of weak selection at tightly linked loci, the sign of the leading term, which depends on the signs of epistasis and initial linkage disequilibrium, determines whether an increase in recombination rates increases the chance of ultimate fixation of the new mutant. If mutants are advantageous, this is the case when epistasis is positive or null and the initial linkage disequilibrium is negative, which is an expected state in a finite population under directional selection. Moreover, this is also the case for a neutral mutant modifier coding for higher recombination rates if the same conditions hold at the selected loci. Under the same conditions, deleterious mutants are disfavored for ultimate fixation and neutral modifiers for higher recombination rates still favored. The recombination rates between the modifier locus and the selected loci do not come into play in the leading terms of the approximation for the fixation probability, but they do in higher-order terms.