Dear Sergei,

As a convergence/sanity check, I recently tried to repeat a run. I used the same input (213 sequences of 328 codons, 012345). The program ran for a few hours instead of a few days and then terminated. The screen output read:

[Preliminary Step 1] A total of 423 branches. Fitting a nucleotide model to approximate branch lengths...
[Preliminary Step 2] Nucleotide LogL = -8257.96
4 additional processes aborted (not shown)

I can't process the output file (it's empty). The error files are all blank. I'm not sure whether messages.log.mpinode0 might be useful--happy to provide it. The last line is "InitMPIOptimizer:Finished with the setup. Maximum of 429 transferrable parameters."

Any ideas? These same inputs worked before. I was only using a different seed.

Thanks again,
Sarah

Dear Sergei,

All the other message logs end at the same big block involving synRate, node, given tree. (e.g., ...givenTree.HK68_BI_17938_69.synRate;givenTree.Node419.synRate)

The cluster where I run this hasn't reported any outages.

I'm trying to repeat this error but have to wait in a queue for a while--I'll let you know if the next run gets past this step, but I might not know for a couple of days.

Thanks,
Sarah

p.s. I walked through the messages log for node0 for the successful and unsuccessful run. There's nothing different between them, except that the unsuccessful run's log terminates early. I've sent an email to the cluster admins to confirm that there were no 'sub'outages.

Dear Sergei,

Thank you for the offer. It looks like maybe there isn't a problem with the code (or it's a rare one!). I ran the program for three hours last night and it reached the GA analysis step before I had it scheduled to end. In the run I described above, it actually only ran for 41 min (walltime) and of course never reached the GA step.

I'll let you know if I run into this problem again, and I'll hopefully have a few more hypotheses to work with.

Thanks again,
Sarah