8.4.3. TMpiRun

In this case, many processes are started on different nodes. MPI uses the distributed memory paradigm: each process have is own address space. All processes run the same macro and define their own objects. If you create a big object in the evaluation/master code section, all processes allocate it (this is why, generally, the main dataserver object is created in the onMaster part to prevent from creating as many dataserver as there are slaves).

  • the constructor calls MPI_Init for the initial process synchronisation. This step is automatical, as long as one is running through the on-the-fly C++ compilator thanks to the root command or in python. In the peculiar case of standalone compilation please refer to the provided exemple and the discussion on how to handle this in Macro “relauncherCodeFlowrateMpiStandalone.C”.

  • startSlave either exits immediately for the master process (id=0) or starts evaluation loop for other ones.

  • depending if we are on the master process or not, onMaster is true or false.

  • stopSlave puts fake items for evaluation and then exits. Evaluation processes get it, stop their loop, exit from startslave, and usually jump the master bloc instructions. Unlike threads, the master process is not waiting for evaluation processes.

  • the destructor calls MPI_Finalize for the final process synchronisation.

TMpiRun constructor has one argument, a pointer to a TEval object.

// Creating the threaded runner
TMpiRun mrun(&code);

To run a macro in a MPI context, you have to use the mpirun command. Here is a simple way to run our example:

mpirun -n 8 root -l -q -b RosenbrockMacro.C

Here, we launch root on 8 cores (-n 8); -q option (quit) is needed to exit the ROOT interpreter at the script end; -b option (batch) is needed when running on many nodes, preventing opening display. The mpirun command has other options not mentioned here.

In general, one runs a MPI job on a cluster with a batch scheduler. The previous command is put in a shell script with batch scheduler parameters. The ROOT macro does not use viewer, but saves results in a file. They will be analysed in a post interactive session using all the ROOT facilities.

If one wants to run in a compiled way, this cannot be done just by adding a “+” to the command line. Effectively, if all processes try to compile using the same output file, conflicts occur. One way to do is to run a first ROOT session without mpirun to compile your macro. Then, if you run a second mpi root session with the single “+”, processes will use the pre-compiled macro. You can compile your macro with the command:

gROOT->LoadMacro("Rosenbrock.C++");

LoadMacro compiles it but does not execute it. Another possibility to run a code in a compile way is to consider the standalone compilation which consists in considering Uranie as a set of libraries, as already discussed in Standard compilation.

Warning

The TMpiRun implementation requires also at least 2 cores (one being the master and the other one the core on which assessors are run). If only one core is provided, the loop will run infinitely.