HyPhy message board - MPI compilation

tlefebure

YaBB Newbies

Offline

Posts: 36
Cornell University

MPI compilation
Aug 25^th, 2006 at 9:54am

Hello,
I'm now trying to compile with the MPI flag, and obtain this error:
Code:

[tristan@babylon HYPHY_Source] ./build.sh MPI
Linux
Checking for curl
Curl seems to be present
COMPILER=g++, gcc
COMPILER_FLAGS= -w -c -fsigned-char -O3 -fpermissive -I/home/tristan/bin/HYPHY_Source/Source -I/home/tristan/bin/HYPHY_Source/Source/SQLite
+-----------------------------------------------------------+
|Building a single-threaded HYPHYKernelMPI for MPI	    |
+-----------------------------------------------------------+
Building baseobj.cpp
In file included from category.h:33,
		     from likefunc.h:33,
		     from baseobj.cpp:44:
batchlan.h:333:18: error: mpi.h: No such file or directory
baseobj.cpp: In function &#8216;bool GlobalShutdown()&#8217;:
baseobj.cpp:224: error: &#8216;MPI_COMM_WORLD&#8217; was not declared in this scope
baseobj.cpp:224: error: &#8216;MPI_Comm_rank&#8217; was not declared in this scope
baseobj.cpp:225: error: &#8216;MPI_Comm_size&#8217; was not declared in this scope
baseobj.cpp:301: error: &#8216;MPI_Finalize&#8217; was not declared in this scope

I have lam install (I use it to run mrbayes with MPI, and it works fine). Librairies and binaries seem present:

Code:

[tristan@babylon HYPHY_Source] locate mpicc
/etc/alternatives/mpicc
/etc/alternatives/mpicc.1.gz
/usr/bin/mpicc.lam
/usr/bin/mpicc
/usr/share/man/man1/mpicc.lam.1.gz
/usr/share/man/man1/mpicc.1.gz
[tristan@babylon HYPHY_Source] locate mpic++
/etc/alternatives/mpic++.1.gz
/etc/alternatives/mpic++
/usr/bin/mpic++.lam
/usr/bin/mpic++
/usr/share/man/man1/mpic++.lam.1.gz
/usr/share/man/man1/mpic++.1.gz
[tristan@babylon HYPHY_Source] locate mpi.h
/home/tristan/tmp/mrbayes-3.1.2/macmpi.h
/home/tristan/tmp/wise2.2.0/src/dyc/dbmpi.h
/usr/include/lam/mpi.h

Is someone knows what's the problem?
(I did not found the solution for MP, and thus try to solve the problem using MPI, yes that's not nice...)
Thanks, Tristan

Back to top

WWW

IP Logged

Sergei YaBB Administrator Offline Datamonkeys are forever... Posts: 1658 UCSD Gender:	Re: MPI compilation Reply #1 - Aug 25^th, 2006 at 10:15am Dear Tristan, Try editing the build.sh file to use 'mpic++' in place of 'g++' as the compiler and see what happens. Also, you may need to add a -I include path for wherever the mpi.h file is. Cheers, Sergei
Back to top	Associate Professor Division of Infectious Diseases Division of Biomedical Informatics School of Medicine University of California San Diego WWW IP Logged

tlefebure

YaBB Newbies

Offline

Posts: 36
Cornell University

Re: MPI compilation
Reply #2 - Aug 25^th, 2006 at 10:59am

thanks Sergei,
I changed "g++" by "mpic++" (lign 6 of build.sh), and I removed "-lmpich" from the lign 148, and the compilation worked. Great.

than what I did:

Code:

echo "localhost cpu=4" > lamhosts
lamboot lamhosts
mpirun -np 4 ./HYPHYMPI BatchFiles/speedtest.bf

and it worked, the four cpus were used. Nevertheless the time needed is not that good: 31s, which is the same time used by a single cpus (which is already too slow)...

Back to top

WWW

IP Logged

Sergei

YaBB Administrator

Offline

Datamonkeys are forever...

Posts: 1658
UCSD
Gender: male

Re: MPI compilation
Reply #3 - Aug 25^th, 2006 at 11:03am

Dear Tristan,

speedtest.bf is not written to take advantage of MPI - only MP (it's actually very difficult to efficiently split up a single likelihood evaluation in a distributed environment). A number of standard analyses are MPI enables though, and you should see speed-ups there.

Try to run MPITest.bf to see that the MPI build is working properly.

I am still puzzled as to why gcc would produce such poor code for the Xeon - it is even slower than an old Athlon XP running at 1.33 GHz. Does your gcc version have a 'fast' flag? Perhaps try setting that in the MP (or SP) build and see if CPU specific tweaking will help.

Cheers,
Sergei

Back to top

Associate Professor
Division of Infectious Diseases
Division of Biomedical Informatics
School of Medicine
University of California San Diego

WWW

IP Logged

tlefebure

YaBB Newbies

Offline

Posts: 36
Cornell University

Re: MPI compilation
Reply #4 - Aug 25^th, 2006 at 12:49pm

Here is the results with MPITest.bf:

Code:

[tristan@babylon HYPHY_Source] mpirun -np 4 ./HYPHYMPI BatchFiles/MPITest.bf
Running a HYPHY-MPI test

Detected 4 computational nodes
Polling slave nodes...
Polling node 2...
OK
Polling node 3...
OK
Polling node 4...
OK

Measuring simple job send/receieve throughput...
Node     2 sent/received 8072.2 batch jobs per second
Node     3 sent/received 9841 batch jobs per second
Node     4 sent/received 10086.4 batch jobs per second

Measuring relative computational performance...
Master node reference index:    1304509
Slave node   1 index:    1367370.     104.82% relative to the master
Slave node   2 index:    1368910.     104.94% relative to the master
Slave node   3 index:    1352810.     103.70% relative to the master

Then I tried to compile with SP with the -fast and -m64 flags, but:
Code:

[tristan@babylon HYPHY_Source] ./build_mod2.sh SP
Linux
Checking for curl
Curl seems to be present
COMPILER=g++, gcc
COMPILER_FLAGS= -w -c -fast -m64 -fsigned-char -O3 -fpermissive -I/home/tristan/tmp/HYPHY_Source/Source -I/home/tristan/tmp/HYPHY_Source/Source/SQLite
+--------------------------------------+
|Building a single threaded HYPHYKernel|
+--------------------------------------+
Building baseobj.cpp
cc1plus: error: unrecognized command line option "-fast"
Error during compilation

Something interesting is that with MP, with the flag CPU=1, one cpu will be used at 100%, while with CPU>1, a single cpu is still used but this time at 120% (which speed up the computation: 31 -> 22s).

By the way, regarding MPI, why the 4 cpus are used at 100% with speedtest since only one is necessary?

I'am afraid I'm lost... thanks Sergei and sorry for all these questions. My aim is to convert a pipeline that test recombination and positive selection at the genome scale to hyphy. Today the pipeline use many different programs (phyml, phypack, paml,...). It seems that with hyphy, most of the pipeline might be summarized in a single batch file and would furthermore be deployed easily on a cluster.
Tristan

Back to top

WWW

IP Logged

Sergei

YaBB Administrator

Offline

Datamonkeys are forever...

Posts: 1658
UCSD
Gender: male

Re: MPI compilation
Reply #5 - Aug 25^th, 2006 at 1:40pm

Dear Tristan,

I still don't understand why gcc would generate such poorly optimized Xeon code, but things like that happen. I'd recommend playing with various optimization flags, especially Xeon (or Pentium) specific, and see if anything makes a difference. All of our Linux boxes run on AMD processors, so I can't easily test the Xeon problem. I think Intel provides its Linux c++ compilers (highly tuned) for free - you may want to check that out.

Re: genomic pipeline. It should indeed be possible to make a single batch file, possibly calling other batch files, to run your data through. We have selection and recombination analyses built in (standard analyses), and I am currently working on updating selection screens to allow for testing in the presence of recombination.

You may also want to take a look at Multimedia File Viewing and Clickable Links are available for Registered Members only!! You need to

Login - a set of HyPhy scripts that Konrad Scheffler and colleagues developed - for a method to test for selection in the presence of recombination.

I'll be happy to answer any questions you may have along the way.

Cheers,
Sergei

Back to top

Associate Professor
Division of Infectious Diseases
Division of Biomedical Informatics
School of Medicine
University of California San Diego

WWW

IP Logged

tlefebure

YaBB Newbies

Offline

Posts: 36
Cornell University

Re: MPI compilation
Reply #6 - Aug 25^th, 2006 at 2:28pm

Ouh, I found at least one answer few second before I give up.

On Xeon 64bit (= Xeon EM64T = nocona cpu), you should compile with the following flags:
Code:

COMPILER_FLAGS=" -w -c -O2 -march=nocona -pipe -fsigned-char -fpermissive -I`pwd`/Source -I`pwd`/Source/SQLite ";

now it takes 11s to perform speedtest.bf with SP.

(that does not fixed the MP problem)
Tristan

Back to top

WWW

IP Logged

Sergei YaBB Administrator Offline Datamonkeys are forever... Posts: 1658 UCSD Gender:	Re: MPI compilation Reply #7 - Aug 25^th, 2006 at 2:33pm Dear Tristan, Thanks so much for the tip! I'll put that into the build.sh file for the next update. Does the MP build (with the new flags) take the same amount of time as the single build? Cheers, Sergei
Back to top	Associate Professor Division of Infectious Diseases Division of Biomedical Informatics School of Medicine University of California San Diego WWW IP Logged

tlefebure

YaBB Newbies

Offline

Posts: 36
Cornell University

Re: MPI compilation
Reply #8 - Aug 25^th, 2006 at 2:56pm

Test with speedtest.bf
SP -> 12s
MP CPU=1 -> 12s
MP CPU=2 -> 10s
MP CPU=4 -> 11s

Test with speedtest_2.bf:
SP -> 84s
MP CPU=1 -> 85s
MP CPU=2 -> 85s
MP CPU=4 -> 88s

So, MP still does not work. To the opposite of speedtest.bf, the results with speedtest_2. are similar to the one obtained before modifying the flags (Multimedia File Viewing and Clickable Links are available for Registered Members only!! You need to

Login). (?)

If you want to make some test on a EM64T Xeon, I can open you a session here. (I will be out of reach for one week)
Tristan

Back to top

WWW

IP Logged

Sergei YaBB Administrator Offline Datamonkeys are forever... Posts: 1658 UCSD Gender:	Re: MPI compilation Reply #9 - Aug 25^th, 2006 at 3:53pm Dear Tristan, Quote: If you want to make some test on a EM64T Xeon, I can open you a session here. (I will be out of reach for one week) I'd like to try the Xeon - this odd behavior is a bit troubling. Cheers, Sergei
Back to top	Associate Professor Division of Infectious Diseases Division of Biomedical Informatics School of Medicine University of California San Diego WWW IP Logged

	Welcome, Guest. Please Login