Native Infiniband / RoCE / Omni-Path Support (RDMA)

RDMA support for Infiniband, RoCE (RDMA over Converged Ethernet), and Omni-Path in BeeGFS is based on the Open Fabrics Enterprise Distribution ibverbs API (


To enable RDMA, the BeeGFS client kernel modules have to be compiled with Infiniband support. Client Infiniband support is enabled by setting the corresponding buildArgs option in the client autobuild file (/etc/beegfs/beegfs-client-autobuild.conf). This file also contains more details on the values that you need to set to enable Infiniband support.


Up to BeeGFS v7.0:
The BeeGFS OpenTk communication library ( for userspace services comes pre-built with and without native Infiniband support. You can use the following command to enable the shared library version with native Infiniband support:
$ beegfs-setup-rdma

(Note that the command above is automatically executed after the beegfs-opentk-lib package installation.)

BeeGFS v7.1 and newer:
Please install the libbeegfs-ib package. BeeGFS will then enable RDMA support automatically, if hardware and drivers are installed.

Verifying Infiniband Connectivity

At runtime, you can check whether your IB devices have been discovered by using the BeeGFS online configuration tool. The tool mode LISTNODES will show a list of all registered services and their configured network interfaces in order of preference. The word "RDMA" will be appended to interfaces that are enabled for the native Infiniband protocol. Use the following command to list the services and their available network interfaces:
$ beegfs-ctl --listnodes --nodetype=storage --details
$ beegfs-ctl --listnodes --nodetype=meta --details
$ beegfs-ctl --listnodes --nodetype=client --details

To check whether the clients are connecting to the servers via RDMA or whether they are falling back to TCP because of configuration problems, use the following command to list established connections on a client:
$ beegfs-net

(Note that the command above reads information from /proc/fs/beegfs/<clientID>/X_nodes.)

In addition to the commands above, the log files also provide information on established connections and connection failures (if you are using at least logLevel=3). See /var/log/beegfs-X.log on clients and servers.

General Infiniband Tuning Settings

Note: A typical source of trouble is to have the ibacm service (/etc/init.d/ibacm) running on the machines. This service causes RDMA connections attempts to stall and should be disabled in all nodes.

Note: RHEL kernel 3.10.0-327 introduced a new security heuristic that resulted in an incompatibility with how BeeGFS services used to handle RDMA connections. This problem was addressed in BeeGFS 2015.03-r16. Please, make sure that you use a compatible release.

Intel/QLogic TrueScale Infiniband Tuning

Intel Omni-Path Architecture (OPA) Tuning

Mellanox Infiniband Tuning

Additional Notes

In an RDMA-capable cluster, some BeeGFS communication (especially communication with the management service, which is not performance-critical) uses TCP/IP and UDP/IP transfer. On some systems, the default "connected" IP-over-IB mode of IniniBand and Omni-Path does not seem to work well and results in spurious problems. In this case, you should try to switch the IPoIB mode to "datagram" on all hosts:
$ echo datagram > /sys/class/net/ibX/mode

Back to User Guide - Tuning and advanced Configuration
Valid XHTML :: Valid CSS: :: Powered by WikkaWiki