BeeGFS Buddy Mirroring™
Table of Contents (Page) |
|---|
General Notes on Mirroring
Starting with the 2012.10 release series, BeeGFS provides support for metadata and file contents mirroring. Mirroring capabilities are integrated into the normal BeeGFS services, so that no separate services or third-party tools are needed. Both types of mirroring (metadata mirroring and file contents mirroring) can be used independent of each other.
In the 2015.03 release series the built-in mirroring was extended with additional high availability features for file contents. (If you use a version prior to 2015.03, please see here for more information: Mirroring in 2012.10 and 2014.01 Releases)
Starting with BeeGFS v6.0, the metadata mirroring was also extended with high availability features. (If you use version 2015.03, please see here: Buddy Mirroring in 2015.03 Releases)
Storage and metadata mirroring with high availability is based on so-called buddy groups. In general, a buddy group is a pair of two targets that internally manage data replication between each other. The buddy group approach allows one half of all servers in a system to fail while all data is still accessible. It can also be used to put buddies in different failure domains or different fire domains, e.g. different racks or different server rooms.
Storage Buddy Mirroring: 4 Servers with 1 Target per Server |
|---|
 |
Storage server buddy mirroring can also be used with an odd number of storage servers. This works, because BeeGFS buddy groups are composed of individual storage targets, independent of their assignment to servers, as shown in the following example graphic with 3 servers and 2 storage targets per server. (In general, a storage buddy group could even be composed of two targets that are attached to the same server.)
Storage Buddy Mirroring: 3 Servers with 2 Targets per Server |
|---|
 |
Note that this is not possible with metadata servers, since there are no metadata targets in BeeGFS. An even number of metadata server is needed so that every metadata server can belong to a buddy group.
In normal operation, one of the storage targets (or metadata servers) in a buddy group is considered to be the primary, whereas the other is the secondary. Modifying operations will always be sent to the primary first, which takes care of the mirroring process. File contents and metadata are mirrored synchronously, i.e. the client operation completes after both copies of the data were transferred to the servers.
If the primary storage target or metadata server of a buddy group is unreachable, it will get marked as offline and a failover to the secondary will be issued. In this case, the former secondary will become the new primary. Such a failover is transparent and happens without any loss of data for running applications. The failover will happen after a short delay, to guarentee consistency of the system while the change information is propagated to all nodes. This short delay also avoids unnecessary resynchronizations if a service is simply restarted, e.g. in case of a rolling update. For more information on possible target states see here:
Please note that targets and servers that belong to a buddy group are also still available to store unmirrored data, so it is easily possible to have a filesystem which only mirrors a certain subset of the data.
A failover of a buddy group can only happen if the BeeGFS Management service is running. That means, that no failover will occur, if the node with the beegfs-mgmtd service crashes. Therefore, it is recommended to have the beegfs-mgmtd service running on a different machine. However, it is not necessary to have a dedicated server for the beegfs-mgmtd service, as explained here.
| Note: Keep in mind that mirroring is not a replacement for backups. If files are accidentally deleted or overwritten by a user or process, mirroring won't help you to bring the old file back. So you are still responsible to do regular backups of your important bits. |
Enabling and disabling Mirroring
By default, mirroring is disabled for a new file system instance. Both types of mirroring can be enabled with the beegfs-ctl command line tool. (The beegfs-ctl tool is contained in the beegfs-utils package and is usually run from a client node.)
Before metaadata or storage mirroring can be enabled, buddy groups need to be defined, as these are the basis for mirroring. Please see here for more information on how to define and manage buddy groups:
Storage mirroring can be enabled on a per-directory basis, so that some data in the file system can be mirrored while other data might not be mirrored. On the medatada side, it is also possible to activate or deactivate mirroring per directory, but certain logical restrictions apply. For example, for a directory to be mirrored effectively, the whole path to it must also be mirrored.
Mirroring settings of a directory will be applied to new file entries and will be derived by new subdirectories. For instance, if metadata mirroring is enabled for directory /mnt/beegfs/mydir1, then a new subdirectory /mnt/beegfs/mydir1/mydir2 will also automatically have metadata mirroring enabled.
After metadata mirroring is enabled for a file system using the beegfs-ctl --mirrormd command, the metadata of the root directory will be mirrored by default. Therefore, newly created directories under the root will also have metadata mirroring enabled. It is possible to exclude new folders from metadata mirroring by creating them using beegfs-ctl --createdir --nomirror. For more information about metadata mirroring, please see here:
To enable file contents mirroring for a certain directory, see the built-in help of the beegfs-ctl tool. (Remember to define buddy groups first.)
$ beegfs-ctl --setpattern --buddymirror --help
File contents mirroring can be disabled afterwards by using beegfs-ctl mode --setpattern without the --buddymirror option. However, files that were already created while mirroring was enabled will remain mirrored.
To check the metadata and file contents mirroring settings of a certain directory or file, use:
$ beegfs-ctl --getentryinfo /mnt/beegfs/mydir/myfile
To check target states of storage targets, use:
$ beegfs-ctl --listtargets --nodetype=storage --state
Restoring Metadata and Storage Target Data after Failures
If a storage target or metadata server is not reachable it will be marked as offline and won't get data updates. Usually, when the target or server re-registers it will automatically be synchronized from the remaining mirror in the buddy group (self-healing). However, in some cases it might be necessary that you manually start a synchronization process. For more information on how to do that and on how to monitor synchronization, please see here:
Related Topics
- Mirroring in 2012.10 and 2014.01 Releases
- Definition and Management of Mirror Buddy Groups for Storage Targets
- Target States (Reachability & Consistency)
- Resynchronization of Metadata Servers and Storage Targets
Back to User Guide - Table of Contents
