BeeGFS Wiki : FAQ Categories PageIndex Login Search:

Frequently Asked Questions (FAQ)

General Questions Who should use BeeGFS? Is BeeGFS suitable for production use? What are the minimum system requirements to run BeeGFS? Which network types are supported by BeeGFS? Can I test-drive BeeGFS for free? Do I need a SAN or any kind of shared storage to run BeeGFS? Does BeeGFS provide high-availability? Can I do everything that can be done with the Admon GUI from the command-line? Is BeeGFS open-source? What does the abbreviation FhGFS mean? How does BeeGFS distribute metadata? How to backup metadata? How can I add a new storage node or metadata node? How can I add a new storage target? How can I move a storage target to a different server? How can I move a metadata target to a different server? How can I move the management service to a different server? Can I export BeeGFS via Samba/CIFS to connect Windows hosts? Can I export BeeGFS via NFS? Does BeeGFS have a limitation for the maximum number of groups per user? Does BeeGFS support hardlinks? Does BeeGFS support MPI-IO?

Installation & Startup Where can I find information about why an Admon-based installation of BeeGFS fails? We are not using one of the distributions you are providing packages for. What can we do to try BeeGFS? Do I need to be ''root'' to run BeeGFS? Is it possible to run storage servers and clients on the same machine? Is it possible to run multiple instances of a BeeGFS service on the same machine? Is it possible to have both BeeGFS and FhGFS services running on the same machines? Do I need a special Linux kernel for BeeGFS? Does BeeGFS need time synchronization between different machines? How to upgrade a BeeGFS service to a newer minor verion? How to upgrade from one BeeGFS major release series to another? Can I upgrade directly from release 2014.01 to release 6.x? Why do I get an 'Access denied' error on the client even with correct permissions? How can I configure BeeGFS to use ACLs? How much disk-space is required for metadata? BeeGFS 6.18 and RHEL 7.5 (CentOS, SL) error message: Not a directory The BeeGFS client of v6.18 or v7.0 does not build with Mellanox OFED 4.3 drivers. What can I do? The BeeGFS client of v7.1.2 does not build with Mellanox OFED 4.5 drivers. What can I do? How do I configure and use the client DKMS package?

Configuration & Tuning Where can I find system tuning recommendations? How can I remove a node from an existing file system? Is it possible to free up a storage target before removal? Can I configure connection-based Authentication? I did some testing and want to start over again. Is there an easy way to delete all residues from the old file system? My hosts have more than one network interface. Is there a way of configuring BeeGFS to use only certain interfaces or certain networks? Is it possible to force a certain node ID or target ID for BeeGFS servers? What needs to be done when a server hostname has changed? My client refuses to mount because of an 'unknown storage target' What happens when I delete a file that is still being used by a process? Is it possible to disable the use of RDMA by BeeGFS services for a specific network interface? Can I prevent new files being created on certain storage or metadata targets? What to do when my beegfs-storage reports too many open files? How to find out which file a specific storage chunk is related to? (V6.x) How is BeeGFS affected by Spectra and Meltdown? How to change the secret in an already secured cluster

---

[General Questions]

Who should use BeeGFS?

Everyone with an interest in fast, flexible, and easy to manage storage. This is typically the case for HPC clusters, but BeeGFS was designed to also work well on smaller deployments like a group of workstations or even heterogeneous systems with different hardware architectures.

Is BeeGFS suitable for production use?

Yes, absolutely! BeeGFS is not a proof-of-concept or research implementation. BeeGFS was developed for production use right from the start and is nowadays used in production at thousands of sites all around the globe. ThinkParQ, a Fraunhofer spin-off company, was specifically founded by the key people behind BeeGFS to provide professional support and services for BeeGFS in cooperation with international turn-key solution partners.

What are the minimum system requirements to run BeeGFS?

Currently, native BeeGFS client and server components are available for Linux on x86, x86_64 and PowerPC/Cell architectures. In general, all BeeGFS components can run on a single machine with only a single CPU core, a single hard disk and less than 1GB of RAM. But this is probably not what you want to do in practice, so here are some recommendations for your hardware configuration:

• Storage Server:
• A storage server should have at least 4GB of RAM to have enough memory available for caching and client connection buffers. 1GB of RAM per attached disk is generally a good choice, depending on the actual type of disk and number of file system clients.
• If you are using traditional SATA hard drives (i.e. not SSDs or fast SAS drives), you will typically want to use a RAID with at least 8 drives per server.
• Even though storage server tasks are generally not very CPU-intensive, the Linux buffer caches can produce high CPU load with fast RAID arrays, so it is often good to have a fast CPU here to enable high throughput, especially on high speed networks.
• Keep an eye on the balance between number of disks (in the whole system and per server), network interconnect bandwidth, and number of file system clients. (E.g. depending on your use case, it might not make sense to use more than 12 hard drives per server if the servers are equipped with only a 10Gb interconnect.)

• Metadata Server:
• If you are working primarily with large files then there is not much work to do for the metadata server and you might just want to run it on the same machines that are running as storage servers and on the same hard disk partitions.
• On clusters with very different kinds of workloads, you might still want to run the metadata server daemon on the storage server machines (to have a high number of metadata servers without spending the extra money for dedicated machines), but you should use dedicated disks inside your storage servers for metadata.
  Since metadata servers are doing a lot of small disk random access (i.e. they are reading and writing a lot of small files) there might be a significant performance impact for storage and metadata access if both are located on the same disks. We also often see in pratice that RAID controllers have (performance) problems when managing different RAID volume types for metadata and storage servers, so you might want to have metadata on a separate RAID controller.
  You might also want to use a dedicated network interface for metadata, since streaming network transfers can have an impact on metadata access latency at the network level.
• CPU-usage on metadata servers can be high if you have a high number of clients dealing with many (small) files. So make sure to use a fast CPU and fast drives (usually SSDs) to guarantee low access latency to metadata.
• The amount of space needed for metadata in comparison to the total storage capacity depends on the actual use case (i.e. the total number of files to be stored). For a scratch file system, the space needed for metadata is typically about 0.3% to 0.5% of the total storage capacity, but sometimes this is also way too much. See here for more details.

• Client:
• Since the client components of BeeGFS are designed to be very light-weight, there are no special RAM or CPU requirements. Regarding the supported kernels, see here.

• Management Daemon:
• The management daemon uses only minimal CPU cycles and memory. The required disk space is around 1 MB and the memory consumption is minimal. The network throughput created by the management daemon is very low, too. Access to the management daemon is not relevant for file system performance. Just run this daemon on any of the storage/metadata servers or on your cluster master. Nonetheless, if the management daemon is configured to use RDMA for communication (default: TCP), the memory consumption could become perceivable on large clusters, because every single connection would require some MB of memory.

• Admon Daemon:
• The admon daemon is frequently storing and querying a lot of values from its internal database, so it will run some storage transactions and use a moderate amount of RAM and CPU cycles. It is often fine to run it on any of the other storage/metadata server machines, but your cluster master might be a better place for this daemon to make sure that there is no impact on file system performance.

• Mon Daemon:
• The mon daemon collects and stores node statistics in regular intervals. How much storage space you need, depends on the size of your installation, how long the data is kept and the collection interval. By default, the database won't need much storage space (not more than a few GB) unless you have a really huge cluster. In most cases it will also be sufficient to run the daemon and the InfluxDB server on a host with a few GB of RAM and a small CPU (maybe on the same node the management runs).

Which network types are supported by BeeGFS?

BeeGFS supports all TCP/IP based networks and the native InfiniBand protocol (based on OFED ibverbs), which also includes Omni-Path and RDMA over Converged Ethernet (RoCE). Servers and clients can handle requests from/to different networks at the same time (e.g. your servers can be equipped with Infiniband and Ethernet interconnects and some clients connect via native InfiniBand while other clients connect via TCP/Ethernet). Clients with multiple connection paths (like InfiniBand and Ethernet or multiple Ethernet ports) can also do network failover if the primary connection path fails.

Can I test-drive BeeGFS for free?

Yes, BeeGFS can be downloaded and used free of charge without any limitations.

Do I need a SAN or any kind of shared storage to run BeeGFS?

No, BeeGFS is not a storage area network (SAN) file system and it does not rely on shared access to storage. Typical hardware configurations for BeeGFS consist of multiple servers with internal (or external non-shared) RAID storage.

Does BeeGFS provide high-availability?

Starting with the 2012.10 release series, BeeGFS provides optional metadata and file contents redundancy (replication). In the 2015.03 release series this concept was extended with additional high availability features for transparent failover. Please see here for more information (or here if you still use a release prior to 2015.03).

Besides that, you are also free to implement cold failover with active/active server pairs based on shared storage and external tools like e.g. Heartbeat or Pacemaker. This will allow access to your existing data in case of a machine failure, although BeeGFS is using write caching on the servers, so cold failovers might not be fully transparent for running applications.
To setup active/active pairs with failover based on external tools, you will need to run a new instance of the failed daemon on another machine, so that two instances of the same daemon are running on one machine (using different network protocol ports) when one machine is down. To make the failover BeeGFS daemon instance appear like the original daemon, you will need to also move IP addresses to the failover machine and make sure the failover daemon instance uses the same NodeID on the failover host. (See section "Configuration & Tuning" on this page for information on how to manually define a NodeID instead of using the hostname as NodeID.)
Note that the customer wiki contains more details on this topic.

Can I do everything that can be done with the Admon GUI from the command-line?

Yes, installation, configuration and status queries can also be done via the command line:

• Installation basically comes down to installing the RPM or DEB the packages from the yum/apt repositories and setting a few config file values.
• For starting and stopping the services (including the client mount), there are init scripts: /etc/init.d/beegfs-X.
• Run-time configuration changes (like striping settings) and status queries can be made with the BeeGFS Online Configuration Tool (beegfs-ctl on a client).

Is BeeGFS open-source?

Yes, see here: Source Download

What does the abbreviation FhGFS mean?

FhGFS is the old name of BeeGFS, which is now named after the nice and busy animals that work together as a swarm and do their important job mostly unnoticed in the background.

FhG: The German name of the Fraunhofer Society is Fraunhofer Gesellschaft, its official abbreviation is FhG.
FS: FS is short for File System.

Note: Since the meaning of the letter G in FhGFS is not always obvious, some people started calling FhGFS the Fraunhofer Global File System or Fraunhofer GFS or sometimes even just Fraunhofer.

How does BeeGFS distribute metadata?

BeeGFS distributes metadata on a per-directory basis. This means each directory in your filesystem can be managed by a different metadata server. When a new directory is created, the system automatically picks one of the available metadata servers to be responsible for the files inside this directory. New subdirectories of this directory can be assigned to other metadata servers, so that load is balanced across all available metadata servers.

How to backup metadata?

As extended attributes are typically not copied by default by many backup tools, here are three different ways to backup the metadata, which is stored in extended attributes. However, keep in mind that these are just examples. Other tools like rsync also have options to preserve extended attributes and hardlinks and thus could be used to backup BeeGFS metadata.

Note that starting with the 2012.10 release series, BeeGFS metadata also uses hardlinks on the underlying file system. It is important that these hardlinks are preserved when metadata is backed up and restored.

• Method 1: Using GNU tar without support for extended attributes
• Older GNU tar versions did not have support for extended attributes, so two metadata backup steps are required with tar and getfattr.
• Backup of the data:

  $ cd /path/to/beegfs/meta_parent_dir
  $ tar czvf /path/to/archive/meta.tar.gz metadir/
  $ getfattr -R -d -P metadir/ > /path/to/archive/ea.bak

• Restoring the backup:

  $ cd /path/to/beegfs/meta_parent_dir
  $ tar xvf /path/archive/meta.tar.gz 
  $ setfattr --restore=/path/to/archive/ea.bak

• Method 2: Using GNU tar with support for extended attributes
• Recent upstream GNU tar versions support extended attributes, RHEL6 tar versions are patched to support the backup of extended attributes.
• On RHEL, at least version tar-1.23-7.el6 (RHEL6.3) should be used. Earlier RHEL tar versions have a serious bug, which would make it impossible to restore backup files.
• Backup of the data:

  $ cd /path/to/beegfs/meta_parent_dir
  $ tar czvf /path/to/archive/meta.tar.gz metadir/ --xattrs

• Restoring the backup:

  $ cd /path/to/beegfs/meta_parent_dir
  $ tar xvf /path/to/archive/meta.tar.gz --xattrs

• Method 3: Using star (http://freecode.com/projects/star)
• star is an alternative to GNU tar, which supports backups of extended attributes.
• Backup of the data:

  $ cd /path/to/beegfs/meta_parent_dir
  $ star czv artype=exustar -xattr -f /path/to/archive/meta.tar.gz metadir/ 

• Restoring the backup:

  $ cd /path/to/beegfs/meta_parent_dir
  $ star xzv -xattr -f /path/to/archive/meta.tar.gz metadir/

If you are running the backup e.g. on a weekly basis while the BeeGFS file system is being accessed by users, you should run beegfs-fsck afterwards to resolve any inconsistencies. If you are running the backup to move metadata to a different machine or to different disks, it is recommended to stop the BeeGFS services before starting the backup to avoid inconsistencies from metadata files being modified while the backup is running.

Additional Notes:

• To reduce the overall runtime, you might want to use your backup tool combined with other tools like "xargs" (see parameter "--max-procs") or "GNU parallel" to run multiple processes in parallel, each on a subset of the directory structure.
• To be sure that EAs have been backuped, it is recommended to test that the commands above work in your environment. This test could be done with a subdir (e.g. metadir/inodes/1D/4F/) to check if a backup and restore of this directory works. Restored data should be tested with 'getfattr -d metadir/inodes/1D/4F/some_file' to check if those files still have BeeGFS extended attributes.
• The hardlink count of a metadata file can be checked with the stat tool (e.g. stat metadir/dentries/.../some_file), which will show a links count greater than 1 if a file has hardlinks.

How can I add a new storage node or metadata node?

Adding a new node to BeeGFS is very simple and does not require a downtime. You just have to take the following steps. Please, read the whole procedure before taking any action.

1. Firstly, make sure that the following properties are set to "true" in file /etc/beegfs/beegfs-mgmtd.conf, in order for the management service to allow new nodes and targets to be added to the system. You will find a detailed documentation of those properties on the bottom of the file.
sysAllowNewServers = true
sysAllowNewTargets = true


2. If you change the properties above, please restart the management service afterwards.

   $ service beegfs-mgmtd restart



3. After that, install BeeGFS packages on the new node and configure the services, as described in the installation procedure.


4. Now, start the service, as seen in the example below.

   $ service beegfs-storage start



5. Now, use the command beegfs-check-servers to list the server nodes in your system and check if your new node is listed. If not, check the log files of the added service and the management service to see if any error message informs you why the node could not be added.

   $ less /var/log/beegfs-mgmtd.log
   $ less /var/log/beegfs-meta.log
   $ less /var/log/beegfs-storage.log



6. Finally, set the management properties back to false, to prevent accidental registrations of new nodes and targets in your system. Please, restart the management service afterwards.
sysAllowNewServers = false
sysAllowNewTargets = false

How can I add a new storage target?

Adding a new storage target to BeeGFS is very simple and does not require a downtime. You just have to take the following steps. Please, read the whole procedure before taking any action.

1. Firstly, make sure that the following property is set to "true" in file /etc/beegfs/beegfs-mgmtd.conf, in order for the management service to allow new targets to be added to the system. You will find a detailed documentation of this property on the bottom of the file.
sysAllowNewTargets = true


2. If you change the property above, please restart the management service afterwards.

   $ service beegfs-mgmtd restart



3. After that, add the storage target on the storage server, as described in the installation procedure.


4. Now, restart the storage service, as seen in the example below.

   $ service beegfs-storage restart



5. Now, use the command beegfs-df to list all nodes and targets in your system and check if your new target is listed. If not, check the log files of the added service and the management service to see if any error message informs you why the node could not be added.

   $ less /var/log/beegfs-mgmtd.log
   $ less /var/log/beegfs-storage.log



6. Finally, set the management property back to false, to prevent accidental registrations of new targets in your system. Please, restart the management service afterwards.
sysAllowNewTargets = false

How can I move a storage target to a different server?

Take the following steps to move a storage target to a different machine. Please, read the whole procedure before taking any action.

1. Firstly, we suggest downtime to move a storage target to a new server!


2. Then, edit the file /etc/beegfs/beegfs-storage.conf on the current machine and remove the path of the moving storage target from the comma-separated list of storage targets' directories, defined by option storeStorageDirectory. If the service is configured to run in multi-mode, be careful to use the configuration file and directory of the right instance.


3. Stop the storage service on the current machine and unmount the storage target device.

   $ service beegfs-storage stop
   $ umount /data/mydisk



4. Start the storage service afterwards if it has some remaining storage targets. If you don't need the storage service running on the machine, please uninstall it.

   $ service beegfs-storage start



5. Check if the storage service is already installed on the new machine. If not, please install it. Do not start the service at this point.
• If the storage service is already installed on the new server machine and before the move you had multiple instances of the service running on different machines, and now you want to have 2 instances of the service running on the same machine, each using a different node ID, please configure the service to run in multi-mode.


• In case you don't mind having the storage target associated to the node ID used by the new server, you don't need to configure the storage service to be multi-mode. Later on, in a future step of this procedure, you will be able to simply add the storage targets to the existing instance of the storage service. Only configure the service to be multi-mode if you really want to keep the moved storage target associated to the previous node ID.


6. Check if the storage target device can be moved or connected to the new machine. If so, mount it on a directory of the new machine and make sure it is configured to be mounted at boot time.


7. Otherwise, if the storage target device cannot be moved or connected to the new machine, copy the data from a backup (or from the device, remounted somewhere else) to a storage device of new machine.


8. Then, edit the file /etc/beegfs/beegfs-storage.conf on the new machine and add the path of the moved storage target to the comma-separated list of storage targets' directories, defined by option storeStorageDirectory. If the service is configured to run in multi-mode, be careful to use the configuration file and directory of the right instance.


9. Make sure that the service directory contains the right nodeID file.
• If you are moving a storage target to a machine that already has a storage service running and this service is not in multi-mode, remove all files whose names match the patterns *node*ID and *Node*ID located at the storage target directory being moved. This will cause the storage service to recreate those files with the node ID used by the existing storage service daemon.


• In any other case, make sure that the file nodeID exists on the service directory and that it contains the ID that the storage service daemon should use to identify itself with the management service. If the file does not exist, just create it with the same content of the originalNodeID file.


10. Start or restart the storage service.

    $ service beegfs-storage restart



11. Test if the target is working properly.
• Check if log files contain any error message.

  $ less /var/log/beegfs-mgmtd.log
  $ less /var/log/beegfs-meta.log
  $ less /var/log/beegfs-storage.log
  $ less /var/log/beegfs-client.log



• List all storage targets of your system and check if the moved one appears online.

  $ beegfs-ctl --listtargets --longnodes --state



• If you moved a storage target to an existing storage service running on another server, you will see the previous target listed as unreachable. That happens because the management service still remembers it. If you want to make the management service to forget it, please use the command below.

  $ beegfs-ctl --removenode <NodeID>

How can I move a metadata target to a different server?

Take the following steps to move a metadata target to a different machine. Please, read the whole procedure before taking any action. This description focuses on the transfer of the storage device from one server to another, but it is possible to use a backup. If you are going to move the metadata target by means of a backup, please read the backup metadata instructions.

1. Firstly, we suggest downtime to move a metadata target to a new server!


2. Stop the metadata service on the current machine and unmount the metadata target device. If the service is configured to run in multi-mode, be careful to stop the right instance.

   $ service beegfs-meta stop
   $ umount /data/ssd



3. If you don't need the metadata service running on the current machine, please uninstall it.


4. Check if the metadata target device can be moved or connected to the new machine. If so, mount it on a directory of the new machine and make sure it is configured to be mounted at boot time.


5. Otherwise, if the metadata target device cannot be moved or connected to the new machine, copy the data from a backup (or from the device, remounted somewhere else) to a storage device of the new machine.


6. Check if the metadata service is already installed on the new machine. If not, please install it. If the metadata service is already installed on the new server machine, please configure the service to run in multi-mode. Do not start the service at this point.


7. Then, edit the file /etc/beegfs/beegfs-meta.conf on the new machine and add the path of the moved metadata target to option storeStorageDirectory. If the service is configured to run in multi-mode, be careful to use the configuration file and directory of the right instance.


8. Make sure that the file nodeID exists on the service directory and that it contains the ID that the metadata service daemon should use to identify itself with the management service. If the file does not exist, just create it with the same content of the originalNodeID file.


9. Start or restart the metadata service.

   $ service beegfs-meta restart



10. Test if the target is working properly.
• Check if log files contain any error message.

  $ less /var/log/beegfs-mgmtd.log
  $ less /var/log/beegfs-meta.log
  $ less /var/log/beegfs-storage.log
  $ less /var/log/beegfs-client.log



• List all metadata nodes of your system and check if the moved one appears online.

  $ beegfs-ctl --listnodes --nodetype=meta --reachable

How can I move the management service to a different server?

Take the following steps to move the management service to a different machine. Please, read the whole procedure before taking any action.

1. Firstly, we suggest downtime to move a management target to a new server!


2. Stop the management service on the current machine. If the service is configured to run in multi-mode, be careful to stop the right instance.

   $ service beegfs-mgmtd stop
   $ umount /data/ssd



3. If you don't need the management service running on the current machine, please uninstall it.


4. Check if the storage device where the management directory resides can be moved or connected to the new machine. If so, unmount it, move it, mount it on a directory of the new machine, and make sure it is configured to be mounted at boot time. The management directory is defined by option storeMgmtdDirectory in file /etc/beegfs/beegfs-mgmtd.conf.


5. Otherwise, if the device cannot be moved or connected to the new machine, copy the management directory from a backup (or from the device, remounted somewhere else) to a partition of new machine.


6. Check if the management service is already installed on the new machine. If not, please install it. Do not start the service at this point. If the management service is already installed on the new server machine, please configure the service to run in multi-mode.


7. Then, edit the file /etc/beegfs/beegfs-mgmtd.conf on the new machine and set option storeMgmtdDirectory to the path to the management service directory. If the service is configured to run in multi-mode, be careful to use the configuration file and directory of the right instance.


8. Make sure that the file nodeID exists on the service directory and that it contains the ID that the management service daemon should use to identify itself. If the file does not exist, just create it with the same content of the originalNodeID file.


9. If the new machine has a different hostname, edit the configuration file of each other BeeGFS service running in your system, which are associated to that management service, and write the new hostname as the value of option sysMgmtdHost.


10. Make sure that the new machine is reachable from the other BeeGFS servers, specially if you are using the client config options connInterfacesFile or connNetFilterFile to restrict which network interfaces can be used by BeeGFS services. Its IP address may be changed freely, because BeeGFS keeps no record of such addresses.


11. Start or restart the management service.

    $ service beegfs-mgmtd restart



12. Test if the target is working properly.
• Check if log files contain any error message.

  $ less /var/log/beegfs-mgmtd.log
  $ less /var/log/beegfs-meta.log
  $ less /var/log/beegfs-storage.log
  $ less /var/log/beegfs-client.log



• List all management nodes of your system and check if the moved one appears online.

  $ beegfs-ctl --listnodes --nodetype=mgmt --reachable

Can I export BeeGFS via Samba/CIFS to connect Windows hosts?

Yes, you can. Please, take the following steps.

1. Configure a BeeGFS client host to be the Samba server. Make sure you are using at least:
• BeeGFS 2015.03-r12, which contains optimizations for Samba, and
• Samba 4.2, which contains optimizations for BeeGFS.


2. Add a share definition to file /etc/samba/smb.conf for the BeeGFS mountpoint, as shown in the example below. For more information on these options, please have a look at the Samba documentation.

   [beegfs]
   comment = BeeGFS file system
   public = yes
   path = /mnt/beegfs
   browsable = yes
   writable = yes
   read only = no



3. Restart the samba server service. Now, you should be able access the BeeGFS file system from a Samba client.

In case you are configuring a Clustered Samba cluster, take the following additional steps.

1. Add the following definitions to file /etc/samba/smb.conf.

   [global]
   ...
   clustering = yes
   netbios name = BeeGFS *
   clustering = yes
   ...
   idmap config * : backend = autorid
   idmap config * : range = 1000000-1999999
   ...
   vfs objects = fileid
   fileid:algorithm = fsid
   
   [BeeGFS]
   ...
   strict locking = yes
   kernel oplocks = yes
   blocking locks = yes
   mmap = no



2. When configuring the cluster manager CTDB, add the following settings to file /etc/default/ctdb.

   CTDB_RECOVERY_LOCK=/mnt/beegfs/.ctdb/reclock
   CTDB_NODES=/etc/ctdb/nodes
   CTDB_PUBLIC_ADDRESSES=/etc/ctdb/public_addresses
   CTDB_MANAGES_SAMBA=yes
   CTDB_SERVICE_SMB=smbd
   CTDB_SERVICE_NMB=nmbd



3. For Samba Versions > 4.8 the recovery lock option CTDP_RECOVERY_LOCK option is moved to /etc/samba/smb.conf, where the following entry must be added:

   [cluster]
   recovery lock = /mnt/beegfs/.ctdb/reclock



4. Turn on global file and append locks in BeeGFS by changing the following options in the BeeGFS client configuration file /etc/beegfs/beegfs-client.conf.

   tuneUseGlobalAppendLocks = true
   tuneUseGlobalFileLocks = true



5. Configure the files /etc/ctdb/nodes and /etc/ctdb/public_adresses according to the environment.

Hints

1. Consider exporting BeeGFS via NFS, which usually has a better performance than Samba and is supported by recent Windows versions.


2. If you are using Windows Samba clients, please consider adding the option case sensitive = true to the share definition above, in order to make the BeeGFS export case sensitive. This setting has a positive performance impact on the client, as case-insensitive lookups perform fewer system calls on the exported file system.


3. Another way of improving performance of Windows clients, but keeping the BeeGFS export case-insensitive for ASCII file names, is to store the BeeGFS metadata in a case-insensitive XFS partition, formatted using the command mkfs.xfs with the option -n version=ci, as seen in the example below. This allows now to set the Samba option case sensitive = true, as suggested above, so that Samba is led to perform less syscalls on the exported file system. However, this means that all filenames are case-insenstive, even for Linux clients.

   $ mkfs.xfs -d su=128k,sw=8 -l version=2,su=128k -isize=512 -n version=ci /dev/sdx -f

Can I export BeeGFS via NFS?

Yes, you can. Starting with the 2014.01 release series, BeeGFS supports Linux kernel NFS server exports. For previous release series, it is recommended to use unfs3 to export BeeGFS via NFS 3.

In order to serve files via NFS, configure a BeeGFS client host to be a NFS server and re-export the BeeGFS client mountpoint via NFS. On the NFS server system, set the following beegfs-client.conf option to ensure that file attributes are refreshed also on the additional code paths tha NFS server:


Example NFSv4 export for kernel NFS server in file /etc/exports:


Example NFSv4 mount:


Before starting the kernel NFS server and mounting the NFS client, disable NFSv4 file leases to ensure that NFS does not keep files open after they have actually been closed by the application:


To have NFSv4 file leases also persistently disabled after a server reboot, add the following line to /etc/sysctl.conf:


Make sure to start the NFS server after the BeeGFS client. If you are using systemd,
add this file to /etc/systemd/system/multi-user.target.wants/beegfs-client.service:

and run


Important Notes

• Kernel NFSv3 has some limitations that prevent BeeGFS from supporting it. For example, NFS clients exchange cookies with the servers for identifying files when they are no longer on the server cache. The maximum size of such cookies in NFS 3 is too small for encoding BeeGFS entry lookup information. In NFS 4, the maximum cookie size was raised, enabling BeeGFS to use NFS cookies. Although kernel NFS 3 exports might sometimes work in your system, eventually, they are very likely to fail as soon as cookies are used (i.e. when the NFS server starts dropping entries from its cache and the cookie information is needed for lookups). So, make sure that you use kernel NFS version 4.

• It is a good idea to restrict the NFS protocol versions used in your system, as explained in the section A4 of the NFS documentation. The best way to do that is to add the definition RPCMOUNTDOPTS="--no-nfs-version 3" to the file /etc/sysconfig/nfs, which is read by the script /etc/rc.d/init.d/nfs.

• Beware that disabling NFS 3 in the server configuration and even restarting the NFS server machine may not be enough for enforcing NFS 4 in your network, because NFS stores allowed connections and sessions on disk. So, when the NFS server is restarted, the allowed connections are loaded from disk and thus, old NFS clients could still be allowed to reconnect via NFS 3. Only after all NFS clients are restarted, NFS 4 will really be used by all parts.

Does BeeGFS have a limitation for the maximum number of groups per user?

Some network file systems have a limitation regarding the number of groups that a user can be part of for the access permission check to work correctly. BeeGFS does not have such a limitation, so a user can be in as many groups as the operating system supports.

Does BeeGFS support hardlinks?

In BeeGFS v7 and earlier versions, hardlinks are only supported within a directory and not over different directories.

Does BeeGFS support MPI-IO?

Yes, BeeGFS supports MPI-IO. No special steps are needed to use MPI-IO on BeeGFS, it will simply work out of the box.

---

[Installation and Startup]

Where can I find information about why an Admon-based installation of BeeGFS fails?

If you tried to install BeeGFS by using the Administration and Monitoring GUI, there are two log facilities that can provide useful information:

• The installation log file on the server: You can read it by choosing "Installation" -> "Installation Log File" in the GUI
• The log file of the Java GUI: This log file is stored in the directory ".beegfs inside your home directory (beegfs-admon-gui.log).

We are not using one of the distributions you are providing packages for. What can we do to try BeeGFS?

If you want binaries and cannot install the available BeeGFS rpm or deb packages directly, you can try to download the rpm packages for Suse or Red Hat and unpack them with rpm2cpio and cpio (rpm2cpio packname | cpio -i).

Or you can just build BeeGFS from source code: BeeGFS Source Download

Do I need to be ''root'' to run BeeGFS?

Yes and no. You do not need root access to start the servers, as these are all userspace processes (of course, you need to change the configuration files to store the file system data and log files at places, where you have write access).
The client is a kernel module. To load the module and mount the file system, you normally need to have root privileges. As an alternative, it is also possible to grant permissions to execute the corresponding commands for non-root users via /etc/sudoers.

Is it possible to run storage servers and clients on the same machine?

Yes, it is. You do not need dedicated hosts for any service in BeeGFS. For example, it is possible to have one host running a management daemon, a metadata server, a storage server and a client at the same time.

Is it possible to run multiple instances of a BeeGFS service on the same machine?

Yes, it is. Please see MultiMode for details.

If you are using BeeGFS version 7 or newer and your system uses systemd, service templates are used to control
multiple instances. Init scripts are not used on systemd installations.

Starting with the 2012.10 release series, the standard BeeGFS service init scripts (/etc/init.d/beegfs-XY) can manage multiple instances of the same daemon on a machine. To enable support for this, see the comment on MULTI_MODE in /etc/default/beegfs-XY.

For multi-mode, you will need to create a separate configuration file for the other daemon instance, using different network ports, a different storage directory, a different log file and so on. If the second daemon instance on a machine should become part of the same file system instance (i.e. it registers at the same management daemon as the first daemon instance on this machine), then you would also need to set a different NodeID manually for the second daemon instance. (See "Configuration & Tuning" section on this page for information on how to manually set the NodeID.)

For the client, multi-mode is also available, but the client mount config file (/etc/beegfs/beegfs-mounts.conf) also allows specification of multiple mount points without enabled multi-mode, by adding one or more additional lines of mountpoint and corresponding client config file. (Note: Make sure to specify different client ports in the different client config files.)

The BeeGFS-on-demand script (/opt/beegfs/sbin/beegfs-ondemand-v2) from the beegfs-utils package is another possible way to run a separate BeeGFS instance, especially on a per-job basis for clusters or in cloud environments.

Is it possible to have both BeeGFS and FhGFS services running on the same machines?

Yes, it is. This specially useful in upgrade scenarios, where the new system needs to coexist with the old system for a while.

Each version will run as a separate instance of the file system, and their respective services won't be able to communicate with each other. They will have different configuration files (located at /etc/fhgfs and /etc/beegfs), different startup scripts (e.g.: /etc/init.d/fhgfs-client and /etc/init.d/beegfs-client), different commands (e.g.: fhgfs-ctl and beegfs-ctl), and, of course, must be mounted by clients on different paths (by default: /mnt/fhgfs and /mnt/beegfs).

After installing the packages of the new version, make sure that the ports used by services of one instance are different from ports used by services of the other instance. These ports are defined on the top of the configuration file of each service. For example, in file /etc/beegfs/beegfs-client.conf, you would have to change the options connClientPortUDP and connHelperdPortTCP to prevent the beegfs-client and beegfs-helperd from trying to use the same ports as fhgfs-client and fhgfs-helperd.

Alternatively, instead of changing each port individually, you can use option connPortShift to shift all ports defined on the configuration file at the same time, according to the specified value.

Please note that the installation of the new version does not require the services of the old version to be stopped. Therefore, no downtime is required.

Do I need a special Linux kernel for BeeGFS?

BeeGFS client modules up to version 7.1 require at least kernel version 2.6.18, but apart from that, BeeGFS does not need a special kernel: The client kernel modules were designed patchless (so you don't need to apply any kernel patches and don't even need to reboot to install and run the BeeGFS client) and the server components of BeeGFS run as userspace daemons, so they are independent of the kernel version.

From BeeGFS version 7.2 onwards, a new kernel support policy is introduced. We only support the latest mainline LTS kernels as well as the latest ones that are shipped with the distributions we officially support (that are RHEL/CentOS, SLES, Debian and Ubuntu). This provides several advantages:

• The significantly reduced kernel list enables the development team to focus on the supported ones and make sure they actually work. Unlike before, each of them can be thoroughly tested.
• A lot of conditional code in the kernel module can be removed, which increases code quality and makes maintaining easier.
• We can ship the module as a binary, so the whole compile-on-startup process can go away. This also means that you don't have to install the development packages anymore.
• The package manager takes care of dependencies, automatically picks the right version and makes sure you never break the client by updating something to an unsupported version (e.g. the kernel).

In case you want to use a different kernel, we provide a toolset to compile it yourself. Self compiled kernels are not supported by us though.

Does BeeGFS need time synchronization between different machines?

Yes, the time of all BeeGFS client and server machines needs to be synchronized for various reasons, e.g. to provide consistent file modification timestamps and for consistent ID generation. Make sure all server clocks are set to the correct time and date (e.g. with date or ntpdate) before starting up BeeGFS services.
A service like ntp can then be used to keep the clocks of all machines in sync.

How to upgrade a BeeGFS service to a newer minor verion?

For the upgrade of a BeeGFS service within a major release to a newer minor version, for example from 6.2 to 6.3, only two steps are necessary. (1) Upgrade the BeeGFS service (the recommended way is to upgrade using the package manager of your linux distribution), (2) Restart the BeeGFS service. If you have buddy mirroring enabled, we recommend to use the following sequence, in order to avoid unnecessary failovers or resyncs (executing steps (2) - (4) quickly one after the other, but not simultaneously): (1) Upgrade the package on both nodes, (2) stop the daemon on the primary node, (3) restart the daemon on the secondary node, (4) start the daemon on the primary node. It is always good to have a look at the changelog file before upgrading. Within one major version of BeeGFS, different minor versions can be used together in a BeeGFS system. Although no downtime is necessary, it is recommended to restart the service when load is not very high.

How to upgrade from one BeeGFS major release series to another?

For an upgrade from version 2009.08 to version 2011.04, see here for instructions.
For an upgrade from version 2011.04 to version 2012.10, see here for instructions.
For an upgrade from version 2012.10 to version 2014.01, see here for instructions.
For an upgrade from version 2014.01 to version 2015.03, see here for instructions.
For an upgrade from version 2015.03 to version 6, see here for instructions.

Can I upgrade directly from release 2014.01 to release 6.x?

Skipping major releases is not extensively tested and therefore we recommend to upgrade first to the 2015.03 release (as described here) and in a second step to release v6 (as described here). But don't worry, the intermediate upgrade step is simple.
After upgrading beegfs-mgmtd, beegfs-meta, beegfs-storage (and the optional beegfs-admon) services to release 2015.03, update also the BeeGFS packages of one client node to the 2015.03 release (beegfs-client, beegfs-helperd, beegfs-utils). Then check on this client node that the following commands return the expected output without any error messages:


Now, you can stop the BeeGFS services and proceed with the upgrade to release 6.x. (The remaining client nodes can be upgraded directly from the 2014.01 release to the v6 release).

Why do I get an 'Access denied' error on the client even with correct permissions?

Please check if you have SELinux enabled on the client machine. If it is enabled, disabling it should solve your problem. SELinux can be disabled by setting SELINUX=disabled in the config file /etc/selinux/config. Afterwards, you might need to reboot your client for the new setting to become effective.

How can I configure BeeGFS to use ACLs?

In BeeGFS, Access Control Lists (ACLs) are stored as extended file attributes of metadata files. Therefore, you need to make sure that extended attributes are enabled in the underlying file systems of the metadata services. For example, in an ext4 file system, the options used to mount the storage devices must include user_xattr. Nevertheless, in recent Linux distributions, this option is set by default when file systems that support extended attributes are mounted.

In addition, the client nodes need to run the Linux kernel version 3.1 or above. The ACL enforcement facilities needed by BeeGFS are not supported by older kernel versions.

After that, make sure that you are using at least the BeeGFS release 2015.03-r13, which fully supports ACLs. Then, edit the metadata configuration file (/etc/beegfs/beegfs-meta.conf) and check if the following options are set to true.
By default, the option storeUseExtendedAttribs is set to true during installation. This particular option can only be changed before the first service startup, while the options storeClientXAttrs and storeClientACLs may also be enabled later.

Then, edit the client configuration file (/etc/beegfs/beegfs-client.conf) and check if the following options are set to true.
If you change the configuration files, remember to restart their respective services afterwards.

How much disk-space is required for metadata?

This depends on things like the average file size that you have or how many files you want to be able to create in total.

In general, we recommend to have about 0.3% to 0.5% of the total storage capacity for metadata. However, this number is based on gathered statistics of file systems at different cluster sites and thus might or might not fit for your case. Sometimes, it also is way too much, so you might want to start with a smaller metadata capacity and simply be prepared to add more metadata capacity if it should actually be needed later.

As a rule of thumb, 500GB of metadata capacity are sufficient for about 150 million files, if the underlying metadata storage is formatted with ext4 according to the recommendations in the metadata server tuning guide: Metadata Server Tuning

More specifically, for every file that a user creates, one metadata file is created on one of the metadata servers. For every directory that a user creates, two directories and two metadata files are created on one of the metadata servers.

For file metadata, if the underlying local metadata server file system (e.g. ext4) is formatted according to our recommendations with large inodes (e.g. "mkfs.ext4 -I 512", as described here: Metadata Server Tuning) then the BeeGFS metadata as extended attribute fits completely into the inode of the underlying local file system and does not use any additional disk space. If the underlying metadata server file system is not formatted with large inodes, then the underlying local file system will need to allocate a full block (usually 4KB) to store the BeeGFS metadata information in addition to using up one inode.

Access Control Lists (ACLs) and user extended attributes are also stored as extended attributes in the corresponding metadata files and thus add to the required disk space, depending on whether they still fit into the inode or whether a new block needs to be allocated.

For each directory, one inode and one directory contents block (usually 4KB) are used on the underlying local file system until there are so many subentries in the directory that another directory contents block needs to be allocated by the underlying file system. How many entries fit into one block depends on things like user file name length, but usually 10+ entries fit into one directory block.
So if a user creates e.g. many directories with only one file inside them, this will significantly increase the used number of inodes and disk space on the underlying local file system compared to the case where the same number of files is stored in fewer directories.

Note that while ext4 is generally recommended for metadata storage because of its performance advantages for BeeGFS metadata workloads compared to other local Linux file systems, xfs has the advantage of using a dynamic number of inodes, meaning new inodes can be created as long as there is free disk space. ext4 on the other is based on a static number maximum number of inodes that is defined when the file system is formatted via mkfs (e.g. "mkfs.ext4 -i <number>"). So it can happen with ext4 that you have disk space left but run out of available inodes or vice versa. Use "df -ih" to see information on available inodes for mounted file systems.

BeeGFS 6.18 and RHEL 7.5 (CentOS, SL) error message: Not a directory

If BeeGFS 6.18 is used on a linux kernel that starts with 3.10.0-862 , then you see this error when executing %ls%. In this case, please upgrade to the last version.

The BeeGFS client of v6.18 or v7.0 does not build with Mellanox OFED 4.3 drivers. What can I do?

There is a patch for this issue and by clicking here you can see how to get and apply that patch.

The BeeGFS client of v7.1.2 does not build with Mellanox OFED 4.5 drivers. What can I do?

There is a patch for this issue and by clicking here you can see how to get and apply that patch.

How do I configure and use the client DKMS package?

Please have a look at the client DKMS documentation for more details.

---

[Configuration and Tuning]

Where can I find system tuning recommendations?

There are a lot of tuning possibilities to increase performance - not only in BeeGFS, but also in the Linux kernel, the underlying local file systems, the hardware drivers etc. Please have a look at the Tuning Guide for tips and recommendations on system tuning.

How can I remove a node from an existing file system?

Use the BeeGFS Control Tool beegfs-ctl (contained in the beegfs-utils package) if you need to unregister a server from the file system:

• $ beegfs-ctl --removenode --nodetype=meta <nodeID> (for a metadata node)
• $ beegfs-ctl --removenode --nodetype=storage <nodeID> (for a storage node)

Note: Removing a node will not automatically migrate the files that were stored on this node, thus use this option carefully. If you want to move files from a storage server to the other storage servers before removing it, see here: "Is it possible to free up a storage target before removal?"

Is it possible to free up a storage target before removal?

The beegfs-ctl tool has a mode called "migrate", which allows you to move all files from a certain storage target to other storage targets. (This only works for storage targets, not metadata targets.)


Migration is directory-based and currently single-threaded, so a single migration instance may perform well below the capabilities of the hardware. It is possible to start multiple non-interfering instances of "beegfs-ctl --migrate" on the same client (or different clients) for different directory trees, e.g. one instance for /mnt/beegfs/subdir1 and another instance for /mnt/beegfs/subdir2.

In addition, it is important to make sure that the files being migrated are not modified during the process. Otherwise, changes to the files while migration is in progress can get lost, as the migration first creates a complete copy of the file and then removes the original file. One option is to mount the file system as readonly so that users can continue to read their data while migration is in progress.

Can I configure connection-based Authentication?

Yes, by using the option "connAuthFile", which can be set in the configuration files of the BeeGFS services. This option defines a pre-shared secret and requests are only accepted from connections that can provide the pre-shared secret. In that option you can set the path to a file that can be of any type.

Note that without connection-based Authentication, it is possible to mount the BeeGFS instance from any machine that can connect to your servers. Since BeeGFS relies on the client side user authentication, the root user on any machine is treated as a root user by BeeGFS, and by extension, their files belong to root on all machines that have the same BeeGFS mounted. This can enable privilege escalation. Therefore we advise you to enable connection-based Authendication in every case where the administrator does not have full control over the network and all client machines.

Example:
1.create a file which contains a shared secret

2. scp the file to all host in the cluster (mgmtd, meta, storage, client, admon, mon).
3. edit all config files of all all services you are currently using (incl. helperd admon/mon) on all host in the cluster and configure the "connAuthFile=/etc/beegfs/connauthfile" with the absolute path/filename to the file which contains a shared secret
4. restart the services


After all services have correctly restarted no client without the shared secret file can connect the cluster.

Without the shared secret file the mgmtd will log:

I did some testing and want to start over again. Is there an easy way to delete all residues from the old file system?

To revert your installation to a completely clean file system, you can follow these steps:

1. Stop all clients and servers (via the Admon GUI or via /etc/init.d/beegfs-X stop)
2. Delete the data directories of the metadata servers, storage servers and the management daemon (these are the directories named "store...Directory" in the corresponding /etc/beegfs/beegfs-X.conf config files)
3. Start all servers and clients again

My hosts have more than one network interface. Is there a way of configuring BeeGFS to use only certain interfaces or certain networks?

Yes, there are two different settings that can be used to achieve this:

• In the config file of each service, there is an option called connNetFilterFile. It can be used to activate IP address range based filtering. This setting will allow outgoing connection attempts of a BeeGFS service only to the set of specified IP addresses, which is useful if a client (or other service) should not try to connect to certain exported interface IPs of another node, e.g. because some of the interface IPs might not be reachable from the current network or are meant for internal use only.
• In the config file of each service, there is an option called connInterfacesFile. By specifying a file with a list of interfaces (one per line) here, you can control which interfaces a BeeGFS service registers at the management daemon and in which order they should be tried during connection attempts by other nodes. For instance, if your node has eth0 and eth1, but you only add eth1 to the interfaces file, then the other servers/clients won't know that there is another interface for that server and hence will not try to connect to other interface.

Is it possible to force a certain node ID or target ID for BeeGFS servers?

Short answer:
Yes. Use the corresponding setup script of each service to manually define IDs:
Long answer:
First of all you should know that BeeGFS uses 2 different kinds of IDs for a server node.

• A string-based node ID. By default, the hostname is used for this.
• A numeric ID. By default, this ID is sequentially generated by the management daemon in 2015.x and higher releases (and randomly generated in 2014.x releases).
• Numeric IDs are 16-bit values in range 1..65535.

The string-based node ID is most important to the user/administrator, because it is the ID you will see in log messages to conveniently identify the servers. But internally, BeeGFS uses the numeric IDs, mostly because they can be used more efficiently.

Each BeeGFS server daemon checks for special files inside their storage directory during startup. To force certain IDs instead of having them generated automatically, you would create these special files before first startup of a BeeGFS server daemon.
The names of these special files are:

• nodeID: string-based node ID
• nodeNumID: numeric node ID
• targetID: string-based target ID (only for storage server targets)
• targetNumID: numeric target ID (only for storage server targets)

Example:
Assuming you want to set the string-based node ID of your first storage server to "storage01" and the numeric ID to "1". The first storage server also provides the first storage target, so you would want to set the string-based target ID to "target01" and the numeric target ID to "1". (The storeStorageDirectory in /etc/beegfs/beegfs-storage.conf for this example is set to /mnt/myraid/beegfs_storage.)
To force these IDs, you would use the commands below before starting up the beegfs-storage daemon for the first time:
The ID settings can be confirmed by checking the server log file (/var/log/beegfs-storage.log) after starting up the daemon. Or by querying the management server:
Important notes:

• These IDs are an essential part of all lookups in BeeGFS. Do not change IDs after starting up a daemon for the first time (or BeeGFS would no longer be able to retrieve the data that was stored based on the old ID).
• An ID only needs to be unique in its class. So a metadata server daemon and a storage server daemon may have the same node ID, but two metadata servers (or two storage servers) may not use the same ID.
• Numeric IDs were introduced with the 2012.10 release series. Before that, only string-based IDs existed.
• String-based IDs and numeric IDs are strongly tied together. It is not possible to change the string-based ID, while still using the old numeric ID.

What needs to be done when a server hostname has changed?

Scenario: 'hostname' or '$HOSTNAME' report a different name than during the BeeGFS installation and BeeGFS servers refuse to start up. Logs tell the nodeID has changed and therefore a startup was refused.

Note that by default, node IDs are generated based on the hostname of a server. As IDs are not allowed to change, see here for information on how to manually set your ID back to the previous value: "Is it possible to force a certain node ID or target ID for BeeGFS servers?"

My client refuses to mount because of an 'unknown storage target'

Scenario: While testing BeeGFS, you removed the storage directory of a storage server, but kept the storage directory of the management server. Now the BeeGFS client refuses to mount and prints an error about an unknown storage target to the log file.

What happened to your file system: When you start a new beegfs-storage daemon with a given storage directory, the daemon initializes this directory by assigning an ID to this storage target path and registering this targetID at the management server.
When you delete this directory, the storage server creates a new directory on next startup with a new ID and also registers this ID at the management server. (Because the storage server cannot know what happend to the old directory and whether you might have just moved the data to another machine, so it needs a new ID here.)

When the client starts, it performs a sanity check by querying all registered targetIDs from the mangement server and checks whether all of them are accessible. If you removed a storage directory, this check fails and thus the client refuses to mount. (Note: This sanity check can be disabled, but it is definitely a good thing in this case and saves you from more trouble.)

Now you have two alternative options...

Solution A: Simply remove the storage directories of all BeeGFS services to start with a clean new file system:

1) Stop all the BeeGFS server daemons, i.e. beegfs-mgmtd, beegfs-meta, beegfs-storage: 2) Delete ("rm -rf") all their storage directories. The paths to the server storage directores can be looked up in the server config files:

• storeMgmtdDirectory in config file /etc/beegfs/beegfs-mgmtd.conf
• storeMetaDirectory in config file /etc/beegfs/beegfs-meta.conf
• storeStorageDirectory in config file /etc/beegfs/beegfs-storage.conf

3) Restart the daemons ("/etc/init.d/beegfs-... start" or use the GUI).

Now you have a fresh new file system without any of the previously registered targetIDs.

Solution B: Unregister the invalid targetID from the management server:
For this, you would first use the beegfs-ctl tool (the tool is part of the beegfs-utils package on a client) to list the registered target IDs: Then check the contents of the file "targetNumID" in your storage directory on the storage server to find out which targetID is the current one that you want to keep.
For all other targetIDs from the list, which are assigned to this storage server but are no longer valid, use this command to unregister them from the management daemon: Afterwards, your client will no longer complain about the missing storage targets.

Note: There are options in the server config files to disallow initialization of new storage directories and registration of new servers or targets, which are not set by default, but should be set for production environments. See storeAllowFirstRunInit and sysAllowNewServers.

What happens when I delete a file that is still being used by a process?

When a file is deleted while it is still open by a process, its inode is temporarily moved to the directory disposal on the metadata node. Similarly, its data chunks are moved to directories with the same name on the storage targets. Later, when the file is closed, its inode and chunks are finally erased. So, it is normal to see files in the disposal directories, as long as processes still hold them open.

However, if a failure occurs on the metadata server machine and those processes don't have the chance of closing the files, the disposal entries won't be erased. In order to remove such disposal entries, you can execute the command below.
If a disposal file cannot be deleted because it is still being used by some process, you will see a message like the one below.
Nothing to worry about. Just wait the process end and that disposal file will be deleted. If you want to identify such process, please run the command below on your client nodes:
In case the process is aborted, the Linux Kernel will automatically close all files the process was using and the BeeGFS client module will keep on trying to send the close operation to the metadata server. The client will do that also if the network was disconnected at the time of the close operation.

In case the client node crashes or is rebooted, the disposal files will be removed by the metadata server after about 30 minutes, when the client node is marked as dead.

Is it possible to disable the use of RDMA by BeeGFS services for a specific network interface?

Yes, it is. Use the option connTcpOnlyFilterFile to set the path to a text file that specifies IP subnets that must have RDMA disabled for outgoing communication. Add the option to the configuration file of a server daemon or client, as follows.
This option works similarly to option connNetFilterFile. You need to add one subnet per line in classless notation (e.g. 192.168.10.0/24).

Can I prevent new files being created on certain storage or metadata targets?

Yes, you can do this by making BeeGFS assume that no space is left on a certain storage or metadata target. Based on this, the target will get moved into the emergency pool where it will not be used for new files while there are other storage targets still available in the normal or low pool. To do this, you can create a file named "free_space.override" with the contained number representing the amount of free space, so typically zero.

If your storage target directory (storeStorageDirectory in beegfs-storage.conf) is e.g. /mnt/myraid/beegfs_storage, then you would use this:

If your metadata target directory (storeMetaDirectory in beegfs-meta.conf) is e.g. /beegfs/meta, then you would use this:
It will take a few minutes until the information is updated. You can check the pool assignment of a storage or metadata target by looking at corresponding column of "beegfs-df" tool output. When you want to restore everything back to normal later, you can simply delete the "free_space.override" file. All this can be done at runtime without restarting any of the services.

What to do when my beegfs-storage reports too many open files?

This usually happens when a user application leaks open files, e.g. it creates a lot of files and forgets to close() them due to a bug in the application. (Note that open files will automatically be closed by the kernel when an application ends, so this problem is usually temporary.)

There are per-process limits and system-wide limits (accounting for all processes on a machine together) to control how many files can be kept open at the same time on a host.

To avoid applications from opening too many files at once and make sure that such application problems do not affect servers, it makes sense to reduce the per-process limit for normal applications of normal users to a reasonably low value, e.g. 1024 via the "nofile" setting in "/etc/security/limits.conf".

If your applications actually need to open a lot of files at the same time and you need to raise the limit in the beegfs-storage service, here are the steps to do this:

1) You can check the current limit for the maximum number of open files through the "/proc" file system, e.g. for running beegfs-storage processes on a machine:


2) The beegfs-storage and the beegfs-meta processes can try to increase their own limits through the config option "tuneProcessFDLimit"", but this will be subject to the hard limits that were defined for the system. If the beegfs-storage service fails to increase its own limit, it will print a message line to its log file ("/var/log/beegfs-storage.log").
Set the following in "/etc/beegfs/beegfs-storage.conf" to let the beegfs-storage service try to increase its own limit to 10 million files:


3) You can increase the system-wide limits (the limits that account for all processes together) to 20 million at runtime by using the following commands:


4) Make the changes from the previous step persistent across reboots by adding the following lines to "/etc/sysctl.conf" (or a corresponding file in the subdir "/etc/sysctl.d"):


5) Add the following line to "/etc/security/limits.conf" (or a corresponding file in the subdir "/etc/security/limits.d") to increase the per-process limit to 10 million.
If this server is not only used for BeeGFS, but also for other applications, you might want to set this only for processes owned by root.


6) Now you need to close your current shell and reopen a new shell on the system make the new settings effective. You can then restart the beegfs-storage process from the new shell and look at its limits:

How to find out which file a specific storage chunk is related to? (V6.x)

In some occasions, you may need to discover which file path a specific storage chunk is associated to, e.g. if you see a chunk ID (entry ID) appearing in a log message.

This information can be retrieved by executing the commands below to scan all files in BeeGFS, dump their paths and chunk IDs to a text file, and then, search for the information associated with the chunk ID.

Another way of doing that is to use a third-party tool like beegfs-chunk-map to produce a database mapping chunk IDs to file paths that can be queried later.

How is BeeGFS affected by Spectra and Meltdown?

If the patches that were created to solve these security vulnerabilities are enabled, it has an effect that depends on the workload. We have more information at this page.

How to change the secret in an already secured cluster

In order to exchange the connauthsecret on an already configured/secured cluster please follow the manual below. Because this type of environment can be quite large we would suggest using some parallel copy and command execution like (pscp.)pssh

1.Create the new connauthfile


2. Create a file with a list of host for distribution:


3. Stop all running BeeGFS helperd and clients on all host in/etc/beegfs/myscphosts.txt


4. Copy the new file to all host in /etc/beegfs/myscphosts.txt


5. Restart the server services in this order:


6. Start all BeeGFS helperd and clients on all host in /etc/beegfs/myscphosts.txt:


7. Check access to BeeGFS:

---

List of all categories

• [Source]
• 2020-06-02 14:21:15

Valid XHTML :: Valid CSS: :: Powered by WikkaWiki