Welcome, Guest. Please Login
YaBB - Yet another Bulletin Board
 
  HomeHelpSearchLogin  
 
[ADDITION] Batch Mode For Standard Analyses (Read 9576 times)
Sergei
YaBB Administrator
*****
Offline


Datamonkeys are forever...

Posts: 1658
UCSD
Gender: male
[ADDITION] Batch Mode For Standard Analyses
Aug 21st, 2006 at 3:08pm
 
In the 0.9920060821 or later version (starting Aug 21st, 2006 build) there is now an option to script all the responses to prompts and such into a batch file. This is useful for running repetitive standard analyses, or batch jobs.

The logic is as follows.
  • Execute a standard analyses (or any other batch file which requires prompts), and record down the responses that you want to script (see the attached example for details)
  • Format the inputs as an associative array, in the order they are needed
  • Execute a batch file, passing it the list of inputs as an argument. This will remove the need for user interaction


As an example, I am attaching a script which will execute a SLAC selection analysis on a specified sequence file. Multiple files can be handled by wrapping such code inside a loop (one iteration per file name)

Code:
RequireVersion ("0.9920060821");

stdinRedirect = {}; /* an associative array, with string entries, used to replace the standard input */
					/* entries are always case sensitive and are given in the order they will be
					   expected (smallest key is first, etc)
					*/

filePath		 = "/Users/sergei/HyPhy/data/hiv.seq"; 		/* data file to be analysed, replace this with a path to the file that exists on your system */

stdinRedirect["00"] = "Universal"; 							/* genetic code */
stdinRedirect["01"] = "New Analysis"; 							/* analysis type */
stdinRedirect["02"] = filePath; 								/* data file path */
stdinRedirect["03"] = "Custom";								/* Model options */
stdinRedirect["04"] = "012345";								/* Model string */
stdinRedirect["05"] = "y";										/* Use data file tree */
stdinRedirect["06"] = filePath + ".fit";						/* save model fit to */
stdinRedirect["07"] = "Estimate dN/dS only";					/* dN/dS options */
stdinRedirect["08"] = "Single Ancestor Counting";				/* Which analysis to run (SLAC) */
stdinRedirect["09"] = "Full tree";								/* SLAC options */
stdinRedirect["10"] = "Averaged";								/* How to treat ambiguities */
stdinRedirect["11"] = "Approximate";							/* Which test statistic to use */
stdinRedirect["12"] = "0.1";									/* Significance level */
stdinRedirect["13"] = "Export to File";						/* Output options */
stdinRedirect["14"] = filePath+".csv";							/* Save tabulated output to */
stdinRedirect["15"] = "Skip";									/* Rate class estimate */

/* read the batch file to run */

batchFileToRun	= HYPHY_BASE_DIRECTORY + "TemplateBatchFiles" + DIRECTORY_SEPARATOR + "QuickSelectionDetection.bf";
ExecuteAFile	 (batchFileToRun, stdinRedirect);
 



Sergei
Back to top
 

Associate Professor
Division of Infectious Diseases
Division of Biomedical Informatics
School of Medicine
University of California San Diego
WWW WWW  
IP Logged
 
avilella
YaBB Newbies
*
Offline


I love YaBB 1G - SP1!

Posts: 35
Re: [ADDITION] Batch Mode For Standard Analyses
Reply #1 - Aug 22nd, 2006 at 8:40am
 
This is really cool stuff.

Now that this feature is in place, it would be really nice to have a Bio::Tools::Run::Phylo::HyPhy.pm

Do you have any stats about which are the most popular analyses in HyPhy?

I could start writing the bioperl wrappers for those.

Thanks again,

Albert.
Back to top
 
 
IP Logged
 
Sergei
YaBB Administrator
*****
Offline


Datamonkeys are forever...

Posts: 1658
UCSD
Gender: male
Re: [ADDITION] Batch Mode For Standard Analyses
Reply #2 - Aug 22nd, 2006 at 8:43am
 
Dear Albert,

[quote author=avilella  link=1156198140/0#1 date=1156261215]This is really cool stuff.

Now that this feature is in place, it would be really nice to have a Bio::Tools::Run::Phylo::HyPhy.pm

Do you have any stats about which are the most popular analyses in HyPhy?
[/quote]

That'd be awesome. Based on what people cite, I'd say tests of selection, model selection, relative rate and ratio tests and molecular clocks.

Cheers,
Sergei
Back to top
 

Associate Professor
Division of Infectious Diseases
Division of Biomedical Informatics
School of Medicine
University of California San Diego
WWW WWW  
IP Logged
 
avilella
YaBB Newbies
*
Offline


I love YaBB 1G - SP1!

Posts: 35
Re: [ADDITION] Batch Mode For Standard Analyses
Reply #3 - Aug 22nd, 2006 at 8:49am
 
Pushing a bit here Smiley

If you could paste a generic stub for each of those, that would be great. Something with sensible default parameters...
And the name as in "Mymodel.pm" that you would prefer.
Back to top
 
 
IP Logged
 
NP
YaBB Newbies
*
Offline



Posts: 11
Re: [ADDITION] Batch Mode For Standard Analyses
Reply #4 - Jun 12th, 2007 at 4:04pm
 
Hi Sergei and everyone,

I´m trying to use the script at the bottom for a phylogeny reconstruction using JTT+F  +Gamma. Unfortunately, it is not working, giving the error:

[b]Error:
Operation * is not defined for associative lists
Current BL Command:Build Formula:ExecuteAFile(batchFileToRun,stdinRedirect)

Check errors.log for details on execution errors.[/b]

How could I deal with it?

By the way, should I include the absolute path of my dataset in filePath, or relative to the HYPHY binary?
Should I launch the program using
HYPHY (script)
or instead
HYPHY < (script)
?

Thank you very much indeed!

Nicolás.-


[code]
RequireVersion ("0.9920060821");

stdinRedirect = {};

filePath   = "/homes-ufq/nicolas/HYPHY_Source/PSIBLAST_7ma_SSIII-A.thaliana_N-term_edit_namesGIonly.phy";

stdinRedirect["00"] = "8";
stdinRedirect["01"] = "5";
stdinRedirect["02"] = "1";
stdinRedirect["03"] = filePath;
stdinRedirect["04"] = "1";
stdinRedirect["05"] = "1";
stdinRedirect["06"] = "1";
stdinRedirect["07"] = "JONES+F";
stdinRedirect["08"] = "2";
stdinRedirect["09"] = "1";
stdinRedirect["10"] = "4";
stdinRedirect["11"] = "1";

batchFileToRun = HYPHY_BASE_DIRECTORY + "TemplateBatchFiles" + DIRECTORY_SEPARATOR + "SequentialAddition.bf";
ExecuteAFile  (batchFileToRun, stdinRedirect);
[/code]
Back to top
 
 
IP Logged
 
Sergei
YaBB Administrator
*****
Offline


Datamonkeys are forever...

Posts: 1658
UCSD
Gender: male
Re: [ADDITION] Batch Mode For Standard Analyses
Reply #5 - Jun 12th, 2007 at 4:52pm
 
Dear Nicolas,

Looks like you are running an old version of HyPhy (maybe). Also, you need to use explicit inputs (not numerical equivalents) when composing user-input redirection as in the example at the bottom of the message (this example assumes that you paste some rough tree into the alignment to estimate global model parameters from - this really speeds up the process). Also, since you are calling 'SequentialAddition' directly, you can skip the dialog options needed to select this analysis first.

HyPhy will understand absolute paths and also paths relative to the current batch file (not the binary).

As far as calling goes, try:

Code:
$HYPHY BASEPATH=/absolute/path/to/HyPhy/ your/batch/file.bf
 



Finally, HyPhy is quite slow when it comes to phylogenetic inference (because it is a secondary feature), but you can infer trees with complex models (at least for small data sets). Also, many analyses will run in parallel on an MPI cluster if you have access to one.

Cheers,
Sergei

Code:
 RequireVersion ("0.9920060821");

 stdinRedirect = {};

 filePath  = "/Users/sergei/HyPhy/data/hiv.aa";

 stdinRedirect["00"] = "Nucleotide/Protein";
 stdinRedirect["01"] = filePath;
 stdinRedirect["02"] = "Given Order";
 stdinRedirect["03"] = "No Constraint";
 stdinRedirect["04"] = "Complete NNI";
 stdinRedirect["05"] = "4";
 stdinRedirect["06"] = "JONES+F";
 stdinRedirect["08"] = "Rate Variation";
 stdinRedirect["09"] = "Gamma";
 stdinRedirect["10"] = "4";
 stdinRedirect["11"] = "First Three";
 stdinRedirect["12"] = "Get from a user tree";
 stdinRedirect["13"] = "y";

 batchFileToRun = HYPHY_BASE_DIRECTORY + "TemplateBatchFiles" + DIRECTORY_SEPARATOR + "SequentialAddition.bf";
 ExecuteAFile (batchFileToRun, stdinRedirect);
 

Back to top
 

Associate Professor
Division of Infectious Diseases
Division of Biomedical Informatics
School of Medicine
University of California San Diego
WWW WWW  
IP Logged
 
NP
YaBB Newbies
*
Offline



Posts: 11
Re: [ADDITION] Batch Mode For Standard Analyses
Reply #6 - Jul 16th, 2007 at 8:49pm
 
Dear Sergei, thanks for your answer.

Unfortunately, the input "4" for rate classes doesn´t seem to be working; HYPHY sends a How many rate classes? message immediately followed by an Error message (similar to the one I posted before).

How could I pass this value as an input from an associative array, then?

Thank you very much!
Back to top
 
 
IP Logged
 
Sergei
YaBB Administrator
*****
Offline


Datamonkeys are forever...

Posts: 1658
UCSD
Gender: male
Re: [ADDITION] Batch Mode For Standard Analyses
Reply #7 - Jul 19th, 2007 at 4:15pm
 
Dear NP,

Oops; the options list is case sensitive and there was also a typo in my original example. The code below should work...

Code:
RequireVersion ("0.9920060821");

stdinRedirect = {};

filePath = "/Users/sergei/HyPhy/data/hiv.aa";

stdinRedirect["00"] = "Nucleotide/Protein";
stdinRedirect["01"] = filePath;
stdinRedirect["02"] = "Given Order";
stdinRedirect["03"] = "No Constraint";
stdinRedirect["04"] = "Complete NNI";
stdinRedirect["05"] = "4";
stdinRedirect["06"] = "JONES+F";
stdinRedirect["08"] = "Rate variation";
stdinRedirect["09"] = "Gamma";
stdinRedirect["10"] = "4";
stdinRedirect["11"] = "First 3";
stdinRedirect["12"] = "Get from a user tree";
stdinRedirect["13"] = "y";

batchFileToRun = HYPHY_BASE_DIRECTORY + "TemplateBatchFiles" + DIRECTORY_SEPARATOR + "SequentialAddition.bf";
ExecuteAFile (batchFileToRun, stdinRedirect);
 



Cheers,
Sergei
Back to top
 

Associate Professor
Division of Infectious Diseases
Division of Biomedical Informatics
School of Medicine
University of California San Diego
WWW WWW  
IP Logged
 
NP
YaBB Newbies
*
Offline



Posts: 11
Re: [ADDITION] Batch Mode For Standard Analyses
Reply #8 - Aug 31st, 2007 at 9:22am
 
Thanks Sergei, but unfortunately I can´t run the script yet.

Now I´m getting the following error:

How many rate classes should we have?Error:
Ran out of input in buffer during a redirected standard input read.
Current BL Command:Choice List for Global parameters with choice list:SKIP_NONE. Store result in globalParameters

Any chance of solving this?

Regards,

Nico.-
Back to top
 
 
IP Logged
 
Sergei
YaBB Administrator
*****
Offline


Datamonkeys are forever...

Posts: 1658
UCSD
Gender: male
Re: [ADDITION] Batch Mode For Standard Analyses
Reply #9 - Aug 31st, 2007 at 10:44am
 
Dear Nico,

Could you paste in the exact script you are running?

Cheers,
Sergei
Back to top
 

Associate Professor
Division of Infectious Diseases
Division of Biomedical Informatics
School of Medicine
University of California San Diego
WWW WWW  
IP Logged
 
NP
YaBB Newbies
*
Offline



Posts: 11
Re: [ADDITION] Batch Mode For Standard Analyses
Reply #10 - Sep 17th, 2007 at 11:08am
 
Sorry for the delay, I am sending you the code I´m using.

[code]
RequireVersion ("0.9920060821");

stdinRedirect = {}; /* an associative array, with string entries, used to replace the standard input */
     /* entries are always case sensitive and are given in the order they will be
       expected (smallest key is first, etc)
     */

filePath = "/home/nicolas/HYPHY_Source/CLUSTAL_7ma_SSIII-A.thaliana_N-term_edit_namesGIonly.phy";   /* data file to be analysed, replace this with a path to the file that exists on your system */

stdinRedirect["00"] = "Nucleotide/Protein";        /* Data type: Nucleotide/Protein */
stdinRedirect["01"] = filePath;   /* datafile */
stdinRedirect["02"] = "Given Order";        /* Addition Order: given order */
stdinRedirect["03"] = "No Constraint";        /* Topology Constraint: No restrictions */
stdinRedirect["04"] = "No Swapping";        /* Branch Swapping: no */
stdinRedirect["06"] = "JONES+F";  /* Standard Model */
stdinRedirect["08"] = "Rate variation";       /* Model Options: Branch lengths from distribution, estimated parameters */
stdinRedirect["09"] = "Gamma";        /* Distribution: gamma with shape parameter alpha */
stdinRedirect["10"] = "4";        /* Rate classes */
stdinRedirect["11"] = "First 3";        /* Starting 3 taxa tree: first 3 */

/* read the batch file to run */

batchFileToRun = HYPHY_BASE_DIRECTORY + "TemplateBatchFiles" + DIRECTORY_SEPARATOR + "SequentialAddition.bf";
ExecuteAFile (batchFileToRun, stdinRedirect);

/* batchFileToRun = "/home/nicolas/HYPHY_Source/TemplateBatchFiles/SequentialAddition.bf";
ExecuteAFile  (batchFileToRun, stdinRedirect);
*/

[/code]

Hope this helps... Thank you in advance!
Back to top
 
 
IP Logged