default process allocation process changed for master branch since 12/20/2021

[wzamazon]

Thank you for taking the time to submit an issue!

Background information

What version of Open MPI are you using? (e.g., v3.0.5, v4.0.2, git branch name and hash, etc.)

master branch

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

git clone; then compiled from source

If you are building/installing from a git clone, please copy-n-paste the output from git submodule status.

 4d07260d9f79bb7f328b1fc9107b45e683cf2c4e 3rd-party/openpmix (v1.1.3-3319-g4d07260d)
 9ac0b7ecee2c97c357bf6751fdaab7a10e62df14 3rd-party/prrte (psrvr-v2.0.0rc1-4133-g9ac0b7ecee)

Please describe the system on which you are running

• Operating system/version: Alinux2
• Computer hardware: intel
• Network type: EFA

---

Details of the problem

Starting from around 12/20/2021, the default process allocation policy seems to have changed for the master branch.

To give an example, say I want to run 8 MPI processes on 2 nodes, and I use the following command:

    mpirun -n 8 --map-by ppr:4:node --machinefile ../../2instances my_application

Previous, the processes with MPI rank 0 to 3 will be allocated on the 1st node, the processes with MPI rank 4 to 7 will be allocated on the 2nd node.

Now, the allocation has changed to a round-robin style. e.g. rank 0, 2, 4, 6 are on 1st node, and rank 1, 3, 5, 7 are on the 2nd node.

To reproduce, I wrote the following an testing program:

#include <mpi.h>
#include <unistd.h>
#include <stdio.h>

int main(int argc, char **argv)
{
	int rank, size;
	char hostname[256];

	MPI_Init(&argc, &argv);
	MPI_Comm_size(MPI_COMM_WORLD, &size);
	MPI_Comm_rank(MPI_COMM_WORLD, &rank);

	gethostname(hostname, 256);

	printf("rank: %d hostname: %s\n", rank, hostname);

	MPI_Finalize();
	return 0;
}

and got the following output:

rank: 0 hostname: c5n-st-c5n18xlarge-1
rank: 1 hostname: c5n-st-c5n18xlarge-2
rank: 2 hostname: c5n-st-c5n18xlarge-1
rank: 3 hostname: c5n-st-c5n18xlarge-2
rank: 4 hostname: c5n-st-c5n18xlarge-1
rank: 5 hostname: c5n-st-c5n18xlarge-2
rank: 6 hostname: c5n-st-c5n18xlarge-1
rank: 7 hostname: c5n-st-c5n18xlarge-2

My questions are:

1. Is this change intentional? My concern is that such a change will have profound impact on applications.
2. What is the easiest way to make mpirun to go back to old behavior?

[rhc54]

Members of the OMPI community asked that the rank-by policy default to the map-by policy. In this case, you are mapping by "node", and so the default ranking policy is also by "node". You can override this by setting the ranking policy to whatever you like - e.g., --rank-by slot, which would return you to the old behavior.

FWIW: it is entirely possible that older versions of OMPI defaulted to the "new" master behavior - i.e., that the master behavior prior to the PRRTE update was actually incorrect.

[wzamazon]

Thank you! --rank-by slot returns to original behavior.

[wzamazon]

@rhc54 @jsquyres

Would there be a notes about this change in the NEWS of open mpi 5.0?

Also, when a user use -N x option with open mpi 5.0, they will asked to use --map-by ppr:x:node. However, the two are not exactly same, because of the rank-by method is node now. Should we consider to change the warning message to add --rank-by slot to make them consistent with -N?

[bwbarrett]

We (OMPI) actually owns making those choices. I'm not sure we actually want to emit the warning with -N ?, because that will screw up applications that redirect mpirun output and -N ? has been part of our CLI for a long time. One of the advantages of the work Ralph's done with PRRTE's interface is that the OMPI community can decide what we want the behavior to be.

[wzamazon]

We (OMPI) actually owns making those choices.

According to the earlier comment from Ralph, I thought OMPI has made the decision to make rank by method the same as map by method, which is the motivation of the change?

[rhc54]

"OMPI" is a multi-headed beast - difficult to get everyone to agree on a mapping default. I think Brian is correct that MPI would prefer "dense" packing (i.e., with "rank-by slot" being the default here), while others would prefer something else.

I'll deal with this in the upcoming change to PRRTE, and with the "-N" deprecation warning.

[rhc54]

Fortunately, turning off the "-N" deprecation is trivial - just change a "true" to "false" on a function call 😄

[rhc54]

Okay, a brief status report on this issue. First, let me explain the origin of the change. There has been general agreement that when we map by an object (e.g., --map-by node), user expectation is that the ranks will follow the mapping. In other words, users expect that we first place rank=0 on the first node, then we place rank=1 on the next node, etc. The revised ranking policy followed that plan.

However, there is an open question regarding what to do with the ppr directive. We split the mapping and ranking into two discrete steps, so let's first consider the mapping (placement) step. One could argue that ppr:2:node should be executed cyclically - i.e., you put the first proc on the first node, the next proc on the next node, etc. This is commonly referred to as the "load balancing" method. In that scenario, users might expect a ranking similar result to the above description - rank=0 would be on the first node, rank=1 on the next - since you are effectively mapping by node until enough procs are on each node.

Another point-of-view is that this mapping should be executed "densely" - i.e., you put 2 procs on the first node, then 2 procs on the next node, etc. This is commonly known as "front loading". In that scenario, users might expect the result to be rank=0 and rank=1 on the first node, rank=2 and rank=3 on the next node, etc.

Note that there is a significant difference in the result should the user also specify the total number of procs to be executed, and that number is less than the ppr times the number of nodes. In the first approach, you get equal load on all the nodes with any remainder being evenly distributed on the first nodes of the allocation. In the second approach, you fully load the first nodes of the allocation until all procs are placed - potentially leaving completely empty nodes at the back of the allocation.

Given that difference, PRRTE only implements the "load balancing" method for the ppr directive (there are other combinations of command line options to get "front loading") when placing the procs. Accordingly, it was thought that we should default to ranking by node since the backend mapper is effectively mapping by node.

We have gone back/forth on the ranking policy in this case forever - it all depends on who you talk to that day. There is no clear answer. The changes I am making will allow the OMPI community to decide for themselves how this should be interpreted so you don't get impacted (like what happened here) by somebody else's requirement.

What I have done for the OMPI personality is to leave the ppr mapping policy as "load balance", but default ranking policy to the "dense" behavior for this option. I have also added an MCA parameter by which OMPI can control the output of deprecation warnings, and set its default to "do not warn". This should mirror (as closely as possible) the behavior of prior releases. You folks can adjust from there.

[wzamazon]

@rhc54 Thank you for the detailed explanation!

What I have done for the OMPI personality is to leave the ppr mapping policy as "load balance", but default ranking policy to the "dense" behavior for this option.

Has this already been done? Can you point me to the commit that does it?

[rhc54]

I can point you to the PR - the PRRTE community needs to discuss it prior to commit: openpmix/prrte#1176

[wzamazon]

I can see the referenced PR was merged openpmix/prrte#1176, and ompi master behavior has changed by. Thank you!