Install Commands Walk-Through (Quickstart Guide)

This section provides example commands for the manual installation procedure. It is assumed that you did read the previous sections of the chapter Manual Installation to have a basic understanding of the relevant BeeGFS services and installation steps. This Walk-Through refers to BeeGFS v7.1. For release v7.0 click here.

Table of Contents (Page)


Example Setup

In this example, we use the following hardware and software configuration:

Note on dedicated hosts for each service: In this example, we are using dedicated hosts for all BeeGFS services. This is just to show the different installation steps for each service. BeeGFS allows running any combination of services (including client and storage/metadata service) on the same machine.
Especially the management and admon daemons are not performance-critical and thus are typically not running on dedicated machines.

Step 1: Package Download and Installation

First, the BeeGFS package repository file for your distribution needs to be downloaded. Repository files for all distributions are available.

For our RHEL 7 example, we download this repository file and store it in the directory "/etc/yum.repos.d/" on all nodes:
$ ssh root@node0X wget -O /etc/yum.repos.d/beegfs_rhel7.repo

Now we can install the packages from the repository. For RHEL 7, the package manager is yum.

$ ssh root@node01 yum install beegfs-mgmtd                               # management service
$ ssh root@node02 yum install beegfs-meta libbeegfs-ib                   # metadata service; libbeegfs-ib is only required for RDMA
$ ssh root@node03 yum install beegfs-storage libbeegfs-ib                # storage service; libbeegfs-ib is only required for RDMA
$ ssh root@node04 yum install beegfs-client beegfs-helperd beegfs-utils  # client and command-line utils
$ ssh root@node05 yum install beegfs-admon                               # admon service (optional)

Note on helperd and utils: The additional beegfs-helperd contains the userspace helper daemon for the client, which provides logging and DNS lookup functionality to the client module. The beegfs-utils package contains e.g. the beegfs-ctl command line tool which can show statistics and perfom administrative tasks.

BeeGFS v7.1 and up: To enable support for remote direct memory access (RDMA) based on the OFED ibverbs API, please install the additional libbeegfs-ib package.
BeeGFS will automatically enable RDMA on startup, if the corresponding hardware and drivers are present.

Now that all services are installed, the next step is to configure the automatic client module build.

Step 2: Client Kernel Module Autobuild

Note: This step is only relevant if you have RDMA-capable network hardware (InfiniBand, Omni-Path, RoCE). Otherwise, skip this step.

To enable support for remote direct memory access (RDMA) based on the OFED ibverbs API, the corresponding BeeGFS client kernel module build parameters need to be added. (Use your favorite text file editor, we will use vim below.)

$ ssh node04
$ vim /etc/beegfs/beegfs-client-autobuild.conf

Find the option "buildArgs" and set it to enabled ibverbs support:

If you installed separate OFED kernel modules, add the OFED_INCLUDE_PATH:
buildArgs=-j8 BEEGFS_OPENTK_IBVERBS=1 OFED_INCLUDE_PATH=/usr/src/openib/include

Note on OFED_INCLUDE_PATH: Defining this is only required if you installed separate kernel driver modules. If you are not sure, check the returned path information of this command:
$ modinfo ib_core
Separate OFED modules are usually located in the /lib/modules/<kernel_version> directory.

Afterwards, force a rebuild of the client kernel module:

$ /etc/init.d/beegfs-client rebuild

Step 3: Basic Configuration

Before we can run the services, we need to make a few basic settings.

Management Service

The management service needs to know where it can store its data. It will only store some node information like connectivity data, so it will not require much storage space and its data access is not performance critical. Thus, this service is typically not running on a dedicated machine.

$ ssh root@node01
$ /opt/beegfs/sbin/beegfs-setup-mgmtd -p /data/beegfs/beegfs_mgmtd

Metadata Service

The metadata service needs to know where it can store its data and where the management service is running. Typically, you will have multiple metadata services running on different machines.

Optionally, you can also define a custom numeric metadata service ID (range 1..65535). As this service is running on a server with name "node02" in our example, we will also pick number "2" as metadata service ID here.

$ ssh root@node02
$ /opt/beegfs/sbin/beegfs-setup-meta -p /data/beegfs/beegfs_meta -s 2 -m node01

Note on extended attributes: By default, the metadata service will store metadata as extended attributes (EAs) on the underlying file system for performance reasons. Those EAs have to be supported by your underlying file system. Please see the chapter Metadata Server Tuning on how to enable EAs for ext4 file systems.

Storage Service

The storage service needs to know where it can store its data and how to reach the management server. Typically, you will have multiple storage services running on different machines and/or multiple storage targets (e.g. multiple RAID volumes) per storage service.

Optionally, you can also define a custom numeric storage service ID and numeric storage target ID (both in range 1..65535). As this service is running on a server with name "node03" in our example, we will pick number "3" as ID for this storage service and we will use "301" as storage target ID to show that this is the first target ("01") of storage service "3".

$ ssh root@node03
$ /opt/beegfs/sbin/beegfs-setup-storage -p /mnt/myraid1/beegfs_storage -s 3 -i 301 -m node01

To add a second storage target on this same machine:
$ /opt/beegfs/sbin/beegfs-setup-storage -p /mnt/myraid2/beegfs_storage -s 3 -i 302


The client needs to know where the management service is running.

$ ssh root@node04
$ /opt/beegfs/sbin/beegfs-setup-client -m node01

The client mount directory is defined in a separate configuration file. This file will be used by the beegfs-client service startup script. By default, BeeGFS will be mounted to /mnt/beegfs. Thus, you need to perform this step only if you want to mount the file system to a different location.

$ ssh root@node04
$ vim /etc/beegfs/beegfs-mounts.conf

The first entry defines the mount directory. The second entry refers to the corresponding config file for this mountpoint.

Admon Service

The optional admon service needs to know how to reach the management server. (Use your favorite text file editor, we will use vim in this example.)

$ ssh root@node05
$ vim /etc/beegfs/beegfs-admon.conf

Find the option "sysMgmtdHost" and set it to "sysMgmtdHost=node01".

In addition, please make sure that the clocks of your servers are synchronized, by using a service like NTP. This is specially important for the Admon service to work.

Step 4: Service Startup

BeeGFS services can be started in arbitrary order by using the corresponding init.d or systemctl service scripts. All services create log files (/var/log/beegfs-...).

$ ssh root@node01 systemctl start beegfs-mgmtd
$ ssh root@node02 systemctl start beegfs-meta
$ ssh root@node03 systemctl start beegfs-storage
$ ssh root@node04 systemctl start beegfs-helperd
$ ssh root@node04 systemctl start beegfs-client
$ ssh root@node05 systemctl start beegfs-admon

To run the Admon GUI (the ##.jar## file can also be copied to other machines):
$ java -jar /opt/beegfs/beegfs-admon-gui/beegfs-admon-gui.jar

Note on mount cancelling: BeeGFS clients have a mount sanity check and cancel a mount operation if servers are unreachable. If you want to mount even with unreachable servers, set sysMountSanityCheckMS=0 in the file /etc/beegfs/beegfs-client.conf.

Note on SELinux: Some distributions (e.g. RHEL 6) enable SELinux by default. If you are seeing an "Access denied" error when you access the BeeGFS mount, read this.

Congratulations, your parallel file system is now up and running!

Step 5: Check Connectivity

Note: This step is especially relevant if you have RDMA-capable network hardware (InfiniBand, Omni-Path, RoCE) to ensure that RDMA is used as transport protocol. However, you can also ensure that BeeGFS is using the intended routes to the servers by using the commands below.

Check the detected network interfaces and transport protocols from a client node with the following commands:

$ ssh node04

$ beegfs-ctl --listnodes --nodetype=meta --nicdetails
$ beegfs-ctl --listnodes --nodetype=storage --nicdetails
$ beegfs-ctl --listnodes --nodetype=client --nicdetails
$ beegfs-net                # Displays connections the client is actually using
$ beegfs-check-servers      # Displays possible connectivity of the services
$ beegfs-df                 # Displays free space and inodes of storage and metadata targets

Please check your log files (/var/log/beegfs-X.log), if you do not see expected RDMA connections and verify that the libbeegfs-ib is installed.

Make sure that the interfaces are listed in your order of preference (i.e. the primary interface should be listed first in the output of beegfs-ctl --listnodes) and that you only see interfaces that you want BeeGFS to use.

Note that BeeGFS clients establish connections only when needed and also drop idle connections, so you might e.g. not see metadata server connections in beegfs-net until you perfomed a metadata operation on the client mount like e.g. ls.

Proceed to next Step...

Next step: Read Additional Notes

Back to table of contents
Valid XHTML :: Valid CSS: :: Powered by WikkaWiki