We're having issues compiling HyPhy on both of our lab servers.  One is a 8-core Mac and the other a 16-core Ubuntu 9.10 system.  I am specifically writing about the Ubuntu system, but the issue is essentially the same in both.  We'd like to install the MPI version, but even the MP version does not install properly.  The MP version runs super slow (many times slower than my laptop) and slows down (non-HyPhy) active processes that were already running.

The MPI version simply fails to compile properly and the compilation then defaults to MP.  I would greatly appreciate any help in solving this issue!!

Thanks!
Dan.

Here's the terminal output from the last compilation attempt:

antat@whitney:Scripts $ bash buildFromSVN.sh MPI
cp: cannot create regular file `MPI/Library/.svn/all-wcprops': Permission denied
cp: cannot create regular file `MPI/Library/.svn/entries': Permission denied
cp: cannot create regular file `MPI/Library/.svn/text-base/build.sh.svn-base': Permission denied
cp: cannot create regular file `MPI/Library/.svn/text-base/README.svn-base': Permission denied
cp: cannot create regular file `MPI/Library/.svn/text-base/setup.py.svn-base': Permission denied
cp: cannot create regular file `MPI/Library/Examples/.svn/all-wcprops': Permission denied
cp: cannot create regular file `MPI/Library/Examples/.svn/entries': Permission denied
cp: cannot create regular file `MPI/Library/Examples/HBL/.svn/all-wcprops': Permission denied
cp: cannot create regular file `MPI/Library/Examples/HBL/.svn/entries': Permission denied
cp: cannot create regular file `MPI/Library/Examples/HBL/.svn/text-base/F81.bf.svn-base': Permission denied
cp: cannot create regular file `MPI/Library/Examples/HBL/.svn/text-base/HKY85.bf.svn-base': Permission denied
cp: cannot create regular file `MPI/Library/Examples/HBL/data/.svn/all-wcprops': Permission denied
cp: cannot create regular file `MPI/Library/Examples/HBL/data/.svn/entries': Permission denied
cp: cannot create regular file `MPI/Library/Examples/HBL/data/.svn/text-base/hiv.nuc.svn-base': Permission denied
cp: cannot create regular file `MPI/Library/Examples/Python/.svn/all-wcprops': Permission denied
cp: cannot create regular file `MPI/Library/Examples/Python/.svn/entries': Permission denied
cp: cannot create regular file `MPI/Library/Examples/Python/.svn/text-base/BasicHyPhy.py.svn-base': Permission denied
cp: cannot create regular file `MPI/Library/Examples/R/.svn/all-wcprops': Permission denied
cp: cannot create regular file `MPI/Library/Examples/R/.svn/entries': Permission denied
cp: cannot create regular file `MPI/Library/Examples/R/.svn/text-base/BasicHyPhy.R.svn-base': Permission denied
cp: cannot create regular file `MPI/Library/LibraryModules/.svn/all-wcprops': Permission denied
cp: cannot create regular file `MPI/Library/LibraryModules/.svn/entries': Permission denied
cp: cannot create regular file `MPI/Library/LibraryModules/Python/.svn/all-wcprops': Permission denied
cp: cannot create regular file `MPI/Library/LibraryModules/Python/.svn/entries': Permission denied
cp: cannot create regular file `MPI/Library/LibraryModules/Python/_HyPhy/.svn/all-wcprops': Permission denied
cp: cannot create regular file `MPI/Library/LibraryModules/Python/_HyPhy/.svn/entries': Permission denied
cp: cannot create regular file `MPI/Library/LibraryModules/Python/_HyPhy/.svn/text-base/__init__.py.svn-base': Permission denied
cp: cannot create regular file `MPI/Library/LibraryModules/Python/HyPhy/.svn/all-wcprops': Permission denied
cp: cannot create regular file `MPI/Library/LibraryModules/Python/HyPhy/.svn/entries': Permission denied
cp: cannot create regular file `MPI/Library/LibraryModules/Python/HyPhy/.svn/text-base/__init__.py.svn-base': Permission denied
cp: cannot create regular file `MPI/Library/LibraryModules/R/.svn/all-wcprops': Permission denied
cp: cannot create regular file `MPI/Library/LibraryModules/R/.svn/entries': Permission denied
cp: cannot create regular file `MPI/Library/LibraryModules/R/.svn/text-base/HyPhy.R.svn-base': Permission denied
cp: cannot create regular file `MPI/Library/Link/.svn/all-wcprops': Permission denied
cp: cannot create regular file `MPI/Library/Link/.svn/entries': Permission denied
cp: cannot create regular file `MPI/Library/Link/.svn/text-base/HyPhy.py.svn-base': Permission denied
cp: cannot create regular file `MPI/Library/Link/.svn/text-base/THyPhy.cpp.svn-base': Permission denied
cp: cannot create regular file `MPI/Library/Link/.svn/text-base/THyPhy.h.svn-base': Permission denied
cp: cannot create regular file `MPI/Library/Link/.svn/text-base/THyPhy_wrap.cxx.svn-base': Permission denied
cp: cannot create regular file `MPI/Library/SWIGWrappers/.svn/all-wcprops': Permission denied
cp: cannot create regular file `MPI/Library/SWIGWrappers/.svn/entries': Permission denied
cp: cannot create regular file `MPI/Library/SWIGWrappers/.svn/text-base/THyPhy_python.cpp.svn-base': Permission denied
cp: cannot create regular file `MPI/Library/SWIGWrappers/.svn/text-base/THyPhy_R.cpp.svn-base': Permission denied
Linux
Checking for curl
/usr/bin/ld: cannot find -lssl
collect2: ld returned 1 exit status
Curl seems to be absent (setting up compiler options skip CURL code)
+-----------------------------------------------------------+
|Building a multi-threaded HYPHYKernelMP with setconcurrency|
+-----------------------------------------------------------+
COMPILER=g++, gcc
COMPILER_FLAGS= -w -c -fsigned-char -O3 -fpermissive -I/Network/Servers/coloma.plb.ucdavis.edu/Users/antat/Downloads/hyphy/MPI/Source -I/Network/Servers/coloma.plb.ucdavis.edu/Users/antat/Downloads/hyphy/MPI/Source/SQLite -D INTPTR_TYPE=long -D __UNIX__  -D _SLKP_LFENGINE_REWRITE_  -D__HYPHY_NO_CURL__ -D __MP__ -D __MP2__ -fopenmp
Building HYNetInterface.cpp
Complete
Building Net.cpp
Complete
Building baseobj.cpp
Complete
Building batchlan.cpp
Complete
Building batchlan2.cpp
Complete
Building bayesgraph.cpp
Complete
Building bayesgraph2.cpp
Complete
Building bgm.cpp
Complete
Building bgm2.cpp
Complete
Building calcnode.cpp
Complete
Building calcnode2.cpp
Complete
Building category.cpp
Complete
Building error.cpp
Complete
Building fisher_exact.cpp
Complete
Building hyphyunixutils.cpp
Complete
Building likefunc.cpp
Complete
Building likefunc2.cpp
Complete
Building list.cpp
Complete
Building matrix.cpp
Complete
Building nexus.cpp
Complete
Building parser.cpp
Complete
Building parser2.cpp
Complete
Building polynoml.cpp
Complete
Building regex.cpp
Complete
Building scfg.cpp
Complete
Building sequence.cpp
Complete
Building site.cpp
Complete
Building strings.cpp
Complete
Building main-unix.cxx
Complete
Building SQLite file sqlite3.c
Complete
Linking HYPHYMP
g++ -w -fsigned-char -D__HYPHY_NO_CURL__ -o HYPHYMP obj_MP2/HYNetInterface.cpp.o obj_MP2/Net.cpp.o obj_MP2/baseobj.cpp.o obj_MP2/batchlan.cpp.o obj_MP2/batchlan2.cpp.o obj_MP2/bayesgraph.cpp.o obj_MP2/bayesgraph2.cpp.o obj_MP2/bgm.cpp.o obj_MP2/bgm2.cpp.o obj_MP2/calcnode.cpp.o obj_MP2/calcnode2.cpp.o obj_MP2/category.cpp.o obj_MP2/error.cpp.o obj_MP2/fisher_exact.cpp.o obj_MP2/hyphyunixutils.cpp.o obj_MP2/likefunc.cpp.o obj_MP2/likefunc2.cpp.o obj_MP2/list.cpp.o obj_MP2/main-unix.cxx.o obj_MP2/matrix.cpp.o obj_MP2/nexus.cpp.o obj_MP2/parser.cpp.o obj_MP2/parser2.cpp.o obj_MP2/polynoml.cpp.o obj_MP2/regex.cpp.o obj_MP2/scfg.cpp.o obj_MP2/sequence.cpp.o obj_MP2/site.cpp.o obj_MP2/sqlite3.c.o obj_MP2/strings.cpp.o -lm -lpthread -fopenmp -ldl
obj_MP2/sqlite3.c.o: In function `exprDup':
sqlite3.c:(.text+0x2c1ec): warning: memset used with constant zero length parameter; this could be due to transposed parameters
Finished
Linux
Checking for curl
/usr/bin/ld: cannot find -lssl
collect2: ld returned 1 exit status
Curl seems to be absent (setting up compiler options skip CURL code)

Dear Dan,

The build scripts for HyPhy are quite primitive; I apologize for your issues.

Try the following.

1). When you run buildFromSVN.sh from /path/to/hyphy/trunk/Scripts/ the script will create a directory /path/to/hyphy/HYPHY (where /path/to needs to be replaced with the appropriate bit for your system).

2). cd to /path/to/hyphy/HYPHY

3). Try running $bash build.sh MPI from this directory

4). If this fails, cp build.sh to build_mpi.sh, edit build_mpi.sh and modify the variables for compilers as follows

[code]
COMPILER="mpic++"; COMPILERC = "mpicc"
[/code]

5). Run $bash build_mpi.sh MPI

The behavior of the MP version is odd. We have an Ubuntu 9.xx system and I haven't seen anything like that. Try giving HYPHYMP a CPU=2 (or another number) to force it to run on fewer than the total number of processors in the system

Sergei

Hi Sergei,

Thank you so much for the prompt response!  

We're still getting a compilation error; something to do with a missing lpmich.  Also, even though we have Curl on that machine, the compilation script cannot find Curl.  However, I don't know how much of a problem the Curl thing is.

As far as the MP version, I have two simple questions:
1) How do I call HYPHYMP to specify that it run on a certain number of cores?
2) Is the MP or the MPI better?  That is, what would run faster on a multicore machine?

Thanks again for your help!
Here's the new compilation output:

antat@whitney:HYPHY $ sh build_mpi.sh MPI
Linux
[: 98: Linux: unexpected operator
[: 98: Linux: unexpected operator
Checking for curl
/usr/bin/ld: cannot find -lssl
collect2: ld returned 1 exit status
Curl seems to be absent (setting up compiler options skip CURL code)
[: 213: Linux: unexpected operator
[: 213: Linux: unexpected operator
+-----------------------------------------------------------+
|Building a single-threaded HYPHYKernelMPI for MPI          |
+-----------------------------------------------------------+
COMPILER=mpic++, mpicc
COMPILER_FLAGS= -w -c -fsigned-char -O3 -fpermissive -I/Network/Servers/coloma.plb.ucdavis.edu/Users/antat/Downloads/hyphy/HYPHY/Source -I/Network/Servers/coloma.plb.ucdavis.edu/Users/antat/Downloads/hyphy/HYPHY/Source/SQLite -D INTPTR_TYPE=long -D __UNIX__  -D _SLKP_LFENGINE_REWRITE_  -D__HYPHY_NO_CURL__ -D __HYPHYMPI__ -D _SLKP_LFENGINE_REWRITE_
File HYNetInterface.cpp is up to date
File Net.cpp is up to date
File baseobj.cpp is up to date
File batchlan.cpp is up to date
File batchlan2.cpp is up to date
File bayesgraph.cpp is up to date
File bayesgraph2.cpp is up to date
File bgm.cpp is up to date
File bgm2.cpp is up to date
File calcnode.cpp is up to date
File calcnode2.cpp is up to date
File category.cpp is up to date
File error.cpp is up to date
File fisher_exact.cpp is up to date
File hyphyunixutils.cpp is up to date
File likefunc.cpp is up to date
File likefunc2.cpp is up to date
File list.cpp is up to date
File matrix.cpp is up to date
File nexus.cpp is up to date
File parser.cpp is up to date
File parser2.cpp is up to date
File polynoml.cpp is up to date
File regex.cpp is up to date
File scfg.cpp is up to date
File sequence.cpp is up to date
File site.cpp is up to date
File strings.cpp is up to date
File main-unix.cxx is up to date
SQLite File sqlite3.c is up to date
Linking HYPHYMPI
mpic++ -w -fsigned-char -D__HYPHY_NO_CURL__ -o HYPHYMPI obj_MPI/HYNetInterface.cpp.o obj_MPI/Net.cpp.o obj_MPI/baseobj.cpp.o obj_MPI/batchlan.cpp.o obj_MPI/batchlan2.cpp.o obj_MPI/bayesgraph.cpp.o obj_MPI/bayesgraph2.cpp.o obj_MPI/bgm.cpp.o obj_MPI/bgm2.cpp.o obj_MPI/calcnode.cpp.o obj_MPI/calcnode2.cpp.o obj_MPI/category.cpp.o obj_MPI/error.cpp.o obj_MPI/fisher_exact.cpp.o obj_MPI/hyphyunixutils.cpp.o obj_MPI/likefunc.cpp.o obj_MPI/likefunc2.cpp.o obj_MPI/list.cpp.o obj_MPI/main-unix.cxx.o obj_MPI/matrix.cpp.o obj_MPI/nexus.cpp.o obj_MPI/parser.cpp.o obj_MPI/parser2.cpp.o obj_MPI/polynoml.cpp.o obj_MPI/regex.cpp.o obj_MPI/scfg.cpp.o obj_MPI/sequence.cpp.o obj_MPI/site.cpp.o obj_MPI/sqlite3.c.o obj_MPI/strings.cpp.o -lpthread -lm -lmpich -ldl
/usr/bin/ld: cannot find -lmpich
collect2: ld returned 1 exit status
Finished

Hi Dan,

Please edit the build script further (around line 198) to replace


Code (]
LINKER_FLAGS=$CURL_LINKER_LIBS" -lpthread -lm -lmpich -ldl "
[/code):

with [code] LINKER_FLAGS=$CURL_LINKER_LIBS" -lpthread -lm -ldl "

To respond to your questions:

1). Try $/path/to/HYPHYMP CPU=N BASEPATH=/path/to

and replace N with the desired number of cores. Note that not all analyses will be able to utilize all the cores

2). This depends on the analysis -- only some of them can take advantage of MPI at all for example. Can you give me an idea of what you are trying to do with HyPhy?

Sergei

Hi Sergei,

Thanks again for the immediate response!

The compilation seems to have worked.  I'll test it out both the MP and MPI versions asap.  I've read that for the MPI is best to use a wrapper script.

I'm actually drafting another post about what method to use.  I'll be looking for positive selection on a transcriptome-wide level.  I'm planning to use you GenomeFitters batch files.  By the way, kudos and thanks for building such an awesome and flexible analysis package!

So, we have several thousand codon alignments or orthologous genes from 4 species, where 1 species has 2 varieties for a total of 5 terminal branches.  There is a relatively far outgroup (i.e. the closest draft genome) that we've added to an equivalent set of codon alignments, so in those alignments we have 6 terminals.

I'm trying to decide whether to run PARRIS (without partioning), because of the very small size and low sequence divergence in the data, or SLAC and FEL with a high nominal alpha level (say 0.25 or 0.3).  Maybe all of the above is best?

Given time limitations on this study (i.e. we need to get it done within the next month) I was not planning to use REL.  Currently we have ~14,000 alignments, although I may filter these down with more stringent 1-to-1 orthology assignments and dropping all alignments with less than 5 or 10 SNPs.

Thanks,
Dan.

Hi Sergei,

I tested them and both MP and MPI work.

Thanks again,
Dan.