I'd like to test the possibility of positive selection at the tips of a tree for an organism with is a strong GC->AT mutation bias. I'm afraid, however, that this situation might represent a violation of the model assumptions so that I could run a risk of obtaining a false positive result. I was wondering if anyone knows how robust currently used models/methods are to this situation.

Hi,

Are all organisms in your phylogeny evolving under a GC->AT bias? If yes, then this should be alright, since your underlying nucleotide substitution matrix can accommodate this with its equilibrium frequencies. If the GC->AT bias emerges at some point in your tree, then this might be cause for concern, and you may need to set up a custom model to account for this.

Thanks noob for your reply!

Yes, I believe that all organisms - which are extremely closely related - are likely accumulating A+T nucleotides. The organisms differ at the tips by only one or a couple of SNPs from each other in their entire genomes. So those are the mutations that I would like to look at.

Sorry for being slow, but if there is an overall and pronounced accumulation of A+T nucleotides (in an organism already rich in A+T), is this not a violation of the equlibrium (stationary?) nucleotide frequency assumption? But maybe it does not matter?

Hi again,

The way the standard nucleotide models obtain an instantaneous substitution rate matrix is by multiplying each column of a symmetric exchangability matrix by the equilibrium frequency for that column. This means that if, at equilibrium, A was 4 times more frequent than C, then the C->A entry in the instantaneous substitution rate matrix would be 4 times higher than the A->C one. Whether this setup can adequately capture the details of your process depends on how far from equilibrium it is. You note that the organism is already AT enriched, which suggests that it might be at equilibrium, or at least might be well approximated locally (you mentioned low divergence) by a process at equilibrium.

Agreed with noob. In principle if the process is away from equilibrium at the root of the tree (but how would you know?) this could have an effect that would be fixable by adding branches that are not AT enriched, so as to allow inference of the root state of the AT-rich subtree, and inferring frequency parameters separately for the subtree (ML inference rather than taking the usual empirical shortcut).

At the moment we don't have code for doing this, and I expect it won't make a noticeable difference. Also it would require having homologous non-AT-rich sequences and a willingness to assume you know when the mutation bias kicked in. I'd ignore it in practice but yes, it is a potential caveat in principle.

Thanks Konrad and noob for your input.