Mpirun mpirexec PBS scripts are different

A simple MPI Program Execution

Systems & access: learning to use: Linux computing resource

Example of an MPI program mpitest. c

/* program hello *//* Adapted from mpihello.f by drs */#include <mpi.h>#include <stdio.h>int main(int argc, char **argv){int *buf, i, rank, nints, len;char hostname[256];MPI_Init(&argc,&argv);MPI_Comm_rank(MPI_COMM_WORLD, &rank);gethostname(hostname,255);printf("Hello world!  I am process number: %d on host %s\n", rank, hostname);MPI_Finalize();return 0;}

Example of MPI program execution using mpiexec

The preferred means of starting MPI programs on the Linux cluster is mpiexec. the main advantage of using mpiexec over mpirun is that there is no need for sourcing any setup file during the program execution. however, you still must source the setup file before compiling your program. it is a lot easier to write the PBS script file using mpiexec than mpirun. the example MPI program above cocould be executed with the following PBS script that uses mpiexec.

Sample mpitest. PBS using mpiexec

#!/bin/csh#*** The "#PBS" lines must come before any non-blank non-comment lines ***#PBS -l walltime=30:00,nodes=4:myri:ppn=2set mycommand = "./mpitest"  # note "./" is used to not rely on $PATHset myargs    = ""           # arguments for $mycommandset infiles   = ""           # list of input filesset outfiles  = ""           # list of output filesif ($?PBS_NODEFILE && $?PBS_JOBID) then#count the number of processors assigned by PBSset NP = `wc -l < $PBS_NODEFILE`echo "Running on $NP processors: "`cat $PBS_NODEFILE`elseecho "This script must be submitted to PBS with 'qsub -l nodes=X'"exit 1endif# set up the job and input files, exiting if anything went wrongcd "$PBS_O_WORKDIR" || exit 1if ($?infiles) thenforeach file ($infiles)cp -v $file /scratch/ || exit 1endendifcd /scratch || exit 1# run the command, saving the exit valuempiexec $PBS_O_WORKDIR/$mycommand $myargsset ret = $?# get the output filescd "$PBS_O_WORKDIR"if ($?outfiles) thenforeach file ($outfiles)cp -vf /scratch/`basename $file` $fileendendifecho "Done   " `date`exit $ret

 

Sample run

To run a job, the first step is to source the setup file:

  hpc-master: source /usr/usc/mpich/1.2.5..12/gm-cdk/setup.csh

The second step is to compile the program:

  hpc-master: mpicc -o mpitest mpitest.c

The third step is to submit the. PBS file to the queue. When the job submission is successful, the qsub command returns the job's process Number:

  hpc-master: qsub mpitest.pbs12240.hpc-pbs.usc.edu

The output files for the above run are created in the same directory where the qsub command was executed. the output files are created using the same filename as your PBS script filename with an extension that includes des "e" for "error" or "O" for "output" followed by the process number. in our example above the file mpitest. PBS. e12240 contains the error messages and the file mpitest. PBS. o12240 contains the output of the program. the output of the program can be viewed by listing the output files:

  hpc-master: ls -l mpitest.pbs.*-rw-------    1 nirmal   rds             0 Jul 16 14:59 mpitest.pbs.e12240-rw-------    1 nirmal   rds          1442 Jul 16 14:59 mpitest.pbs.o12240

  hpc-master: more mpitest.pbs.*::::::::::::::mpitest.pbs.e12240::::::::::::::::::::::::::::mpitest.pbs.o12240::::::::::::::----------------------------------------Begin PBS Prologue Wed Dec  8 16:49:24 PST 2004Job ID:         12240.hpc-pbs.usc.eduUsername:       nirmalGroup:          rdsNodes:          hpc0632 hpc0633 hpc0634 hpc0636PVFS: mounting at /scratchEnd PBS Prologue Wed Dec  8 16:49:27 PST 2004----------------------------------------Warning: no access to tty (Bad file descriptor).Thus no job control in this shell.Starting 12240.hpc-pbs.usc.edu Wed Dec 8 16:49:27 PST 2004Initiated on hpc0636Running on 8 processors: hpc0636 hpc0636 hpc0634 hpc0634 hpc0633 hpc0633 hpc0632 hpc0632Hello world!  I am process number: 4 on host hpc0632Hello world!  I am process number: 3 on host hpc0634Hello world!  I am process number: 0 on host hpc0635Hello world!  I am process number: 2 on host hpc0634Hello world!  I am process number: 6 on host hpc0631Hello world!  I am process number: 1 on host hpc0635Hello world!  I am process number: 5 on host hpc0632Hello world!  I am process number: 7 on host hpc0631Done    Wed Dec 8 16:49:28 PST 2004--------------------------------------------------Begin PBS Epilogue Wed Dec  8 16:49:33 PST 2004Job ID:         12240.hpc-pbs.usc.eduUsername:       nirmalGroup:          rdsJob Name:       mpitest.pbsSession:        4020Limits:         neednodes=4:myri:ppn=2,nodes=4:myri:ppn=2,walltime=00:02:00Resources:      cput=00:00:00,mem=12648kb,vmem=25752kb,walltime=00:00:04Queue:          quickAccount:Nodes:          hpc0632 hpc0633 hpc0634 hpc0636Killing leftovers...End PBS Epilogue Wed Dec  8 16:49:35 PST 2004--------------------------------------------------