BeeGFS Buddy Mirroring in 2015.03 Releases
Table of Contents (Page) |
|---|
General Notes on Mirroring
Starting with the 2012.10 release series, BeeGFS provides support for metadata and file contents mirroring. Mirroring capabilities are integrated into the normal BeeGFS services, so that no separate services or third-party tools are needed. Both types of mirroring (metadata and file contents) can be used independent of each other.
In the 2015.03 release series this concept was extended with additional high availability features. If you use a version prior to 2015.03, please see here for more nformation:
Mirroring can be enabled on a per-directory basis, so that some data in the file system can be mirrored while other data might not be mirrored. Data mirroring on the storage targets is based on so-called buddy groups. In general, a buddy group always consists of two targets. Please see here for more information on how to define and manage buddy groups:
Storage Buddy Mirroring: 4 Servers with 1 Target per Server |
|---|
 |
Storage server buddy mirroring can also be used with an odd number of storage servers. This works, because BeeGFS buddy groups are composed of individual targets, independent of their assignment to servers, as shown in the following example graphic with 3 servers and 2 storage targets per server. (In general, a storage buddy group could even be composed of two targets that are attached to the same server.)
Storage Buddy Mirroring: 3 Servers with 2 Targets per Server |
|---|
 |
In normal operation, one of the targets in a buddy group is considered to be the primary target, whereas the other is the secondary target. Modifying operations will always be sent to the primary target first, which takes care of the mirroring process. File contents are mirrored synchronously, i.e. the client operation completes after both copies of the data were transferred to the servers.
If the primary storage target of a buddy group is unreachable, it will get marked as offline and a failover to the secondary target will be issued. In this case, the former secondary target will become the new primary target. Such a failover is transparent and happens without any loss of data for running applications. The failover will happen after a short delay, to guarentee consistency of the system while the change information is propagated to all nodes. This short delay also avoids unnecessary resynchronizations if a service is simply restarted, e.g. in case of a rolling update. For more information on possible target states see here:
Please note that targets that belong to a buddy group are also still available to store unmirrored data, so it is easily possible to have a filesystem, which only mirrors a certain subset of the data.
The current 2015.03 stable release series still uses the metadata mirroring mechanisms from previous release series, which are described in "Mirroring in 2012.10 and 2014.01 Releases". For the old-style metadata mirroring, garnularity is a complete directory, so that the entries of a single directory are always mirrored on the same mirror server, but metadata for different directories is mirrored on different metadata servers.
| Note: Keep in mind that mirroring is not a replacement for backups. If files are accidentally deleted or overwritten by a user or process, mirroring won't help you to bring the old file back. So you are still responsible to do regular backups of your important bits. |
Enabling and disabling Mirroring
By default, mirroring is disabled for a new file system instance. Both types of mirroring can be enabled with the beegfs-ctl command line tool. (The beegfs-ctl tool is contained in the beegfs-utils package and is usually run from a client node.)
Mirroring settings of a directory will be applied to new file entries and will be derived by new subdirectories. For instance, if metadata mirroring is enabled for directory /mnt/beegfs/mydir1, then a new subdirectory /mnt/beegfs/mydir1/mydir2 will also automatically have metadata mirroring enabled.
To enable metadata mirroring for a certain directory, see the built-in help of the beegfs-ctl tool:
$ beegfs-ctl --mirrormd --help
To enable file contents mirroring for a certain directory, see the built-in help of the beegfs-ctl tool. (Remember to define buddy groups first.)
$ beegfs-ctl --setpattern --buddymirror --help
File contents mirroring can be disabled afterwards by using beegfs-ctl mode --setpattern without the --buddymirror option. However, files that were already created while mirroring was enabled will remain mirrored.
To check the metadata and file contents mirroring settings of a certain directory or file, use:
$ beegfs-ctl --getentryinfo /mnt/beegfs/mydir/myfile
To check target states of storage targets, use:
$ beegfs-ctl --listtargets --nodetype=storage --state
Restoring Storage Target Data after Failures
If a target is not reachable it will be marked as offline and won't get data updates. Usually, when the target re-registers it will automatically be synchronized from the remaining mirror target in the buddy group (self-healing). However, in some cases it might be neccessary that you manually start a synchronization process. For more information on how to do that and on how to monitor synchronization, please see here:
Restoring Metadata after Failures
The currently available metadata mirroring approach, which was originally introduced in the BeeGFS 2012 release series, does not provide high-availability and automatic self-healing. Information on restoring metadata manually can be found here.
(The new metadata buddy mirroring, which is currently under development, will work similar to the buddy mirroring concept for storage servers and will also provide similar high-availability and self-healing.)
Related Topics
- Mirroring in 2012.10 and 2014.01 Releases
- Definition and Management of Mirror Buddy Groups for Storage Targets
- Target States
- Synchronization of Storage Targets
Back to User Guide - Table of Contents
