Is OpenMPI supporting RDMA?

[ghostplant]

Thank you for taking the time to submit an issue!

What version of Open MPI are you using? (e.g., v1.10.3, v2.1.0, git branch name and hash, etc.)

openmpi-bin=1.10 (installed from Ubuntu 16.04 universe repository)

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

We have some metals equipped with MELLANOX OFED ConnectX-5 Ethernet Cards supporting RDMA data transfer. When using OpenMPI to launch applications, is it automatically take advantage of RDMA channel, or how to configure mpiexec to enable it?

Please describe the system on which you are running

• Operating system/version: Ubuntu 16.04 x86-64
• Computer hardware: C4140
• Network type: Mellanox MT28800 ConnectX-5

---

[jsquyres]

FWIW: Open MPI v1.10 is ancient. You should probably update to a newer version of Open MPI -- the current latest is v3.1.2.

But yes, even as far back as v1.10 -- assuming that both Open MPI and the IB communications stack was installed correctly -- Open MPI uses RDMA and NIC offloading to take advantage of HPC-class NICs.

[ghostplant]

@jsquyres
Thank you! I just saw a not-fast send/recv performance, and I don't know how to verify the running OpenMPI really detect ConnectX-5 RDMA successfully without using other slower communication channel. Is there a way to confirm the RDMA offloading?

[jsquyres]

I'll direct you to Mellanox to help you with that...

@jladd-mlnx @artpol84 Can one of you followup with @ghostplant?

[ghostplant]

@jedbrown @jladd-mlnx @artpol84 Great! Thank all of you, and I'll arrange some detailed logs, and arguments for the application execution short later!

[ghostplant]

There are 3 same hardware metals with their hostname node-130, node-131, node-132 respectively.
And script below is how I run the mpi test program under Ubuntu 16.04 (I had already updated openmpi-bin to v3.1.2):

node-130# mpirun --allow-run-as-root --bind-to none --mca oob_tcp_if_include enp216s0 \
          --mca btl_tcp_if_include enp216s0 -x NCCL_SOCKET_IFNAME=enp216s0 --map-by slot \
          -np 3 --host node-130,node-131,node-132 /opt/mpi-test-example.bin

vsetenv PMIX_SERVER_TMPDIR failed
vsetenv PMIX_SERVER_TMPDIR failed
vsetenv PMIX_SERVER_TMPDIR failed
vsetenv PMIX_SERVER_TMPDIR failed
vsetenv PMIX_SERVER_TMPDIR failed
vsetenv PMIX_SERVER_TMPDIR failed
vsetenv PMIX_SERVER_TMPDIR failed                                                                                                                   [71/1851]
--------------------------------------------------------------------------
No OpenFabrics connection schemes reported that they were able to be
used on a specific port.  As such, the openib BTL (OpenFabrics
support) will be disabled for this port.

  Local host:           node-130
  Local device:         mlx5_0
  Local port:           1
  CPCs attempted:       udcm
--------------------------------------------------------------------------
vsetenv PMIX_SERVER_TMPDIR failed
vsetenv PMIX_SERVER_TMPDIR failed
vsetenv PMIX_SERVER_TMPDIR failed
vsetenv PMIX_SERVER_TMPDIR failed
vsetenv PMIX_SERVER_TMPDIR failed
vsetenv PMIX_SERVER_TMPDIR failed
[node-130:00035] 2 more processes have sent help message help-mpi-btl-openib-cpc-base.txt / no cpcs for port
[node-130:00035] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

(Following outputs are from MPI program after MPI_Init() is invoked, and everything after warning above is fine and success)
...
...

You could see the MPI program got some warnings at the beginning of the running program when calling MPI_Init(), but the program can still run normally including MPI_Send/MPI_Recv/.. So I am not sure whether HPC RDMA Offloading is enabled or not and what the warnings mean.

More INFO:

1. the interface enp216s0 is the only network interface that support MLNX OFED RDMA, and all 3 nodes are of the same hardware;
2. The MLNX_OFED driver is installed as well;

node-130/node-131/node-132# lspci | grep mell -i
d8:00.0 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5]

node-130/node-131/node-132# ip route
default via 10.244.48.1 dev eno3 onlink
10.244.48.0/24 dev eno3  proto kernel  scope link  src 10.244.48.130
192.168.2.0/24 dev enp216s0  proto kernel  scope link  src 192.168.2.130

node-130/node-131/node-132# dpkg -l | grep mlnx-ofed-kernel-dkms
ii  mlnx-ofed-kernel-dkms                           4.2-OFED.4.2.1.0.0.1.gf36c870              all          DKMS support for mlnx-ofed kernel modules

[ghostplant]

If I run all the same test using openmpi-bin v1.10 (from Ubuntu 16.04 apt repo), I got a different warning at the startup of MPI program:

--------------------------------------------------------------------------
WARNING: No preset parameters were found for the device that Open MPI
detected:

  Local host:            node-131
  Device name:           mlx5_0
  Device vendor ID:      0x02c9
  Device vendor part ID: 4119

Default device parameters will be used, which may result in lower
performance.  You can edit any of the files specified by the
btl_openib_device_param_files MCA parameter to set values for your
device.

NOTE: You can turn off this warning by setting the MCA parameter
      btl_openib_warn_no_device_params_found to 0.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
No OpenFabrics connection schemes reported that they were able to be
used on a specific port.  As such, the openib BTL (OpenFabrics
support) will be disabled for this port.

  Local host:           node-131
  Local device:         mlx5_0
  Local port:           1
  CPCs attempted:       udcm
--------------------------------------------------------------------------

[abeltre1]

@ghostplant did you try installing a later version of openmpi like @jsquyres suggested. That may have a lot of benefits when running MPI over InfiniBand (IB).

[ghostplant]

@abeltre1 Yeal, I tried both v3.1.2 and v1.10.2, but both got the following warning when calling MPI_Init(0,0):

--------------------------------------------------------------------------
No OpenFabrics connection schemes reported that they were able to be
used on a specific port.  As such, the openib BTL (OpenFabrics
support) will be disabled for this port.

  Local host:           node-130
  Local device:         mlx5_0
  Local port:           1
  CPCs attempted:       rdmacm, udcm
--------------------------------------------------------------------------

[ghostplant]

Hi, I just wanted to know what kind of --mca should I use for Mellanox Technologies MT28800 Family [ConnectX-5] whose linux network interface name is enp216s0 ?

I tried --mca oob_tcp_if_include enp216s0 --mca btl_tcp_if_include enp216s0 but seems the performance is 20% slower than I wrote a RDMA C/S directly using librdmacm.