Resynchronization of Storage Targets and Metadata Servers



Note: Metadata buddy mirroring was introduced in BeeGFS v6.0. If you are using version 2015.03, which only supports storage buddy mirroring, please see here:
- Resynchronization of Storage Targets in 2015.03 Releases


Table of Contents (Page)

  1. Automatic Resynchronization
  2. Manual Resynchronization
  3. Display Resynchronization Information
  4. Replacement of a failed device
 


Automatic Resynchronization


In general, if a secondary target or server is considered to be out-of-sync, it is automatically set to the consistency state needs resync (see "Target and Node States" for more information on states) by the the management daemon.
The storage target resynchronization process for self-healing is coordinated by the primary target. The standard process tries to avoid unnecessary transfer of files. Therefore the primary target saves the time of the last successful communication with the secondary target. Only files which where modified after this timestamp will be resynchronized by default. To avoid losing cached data, a short safety threshold timespan will be added (defined by sysResyncSafetyThresholdMins in beegfs-storage.conf).
Since metadata are much smaller than storage contents, there is no timestamp-based mechanism in place, and instead the full mirrored metadata of the metadata server will be sent to its buddy during the resynchronization process.


Manual Resynchronization


In some cases it might be useful or even neccessary to manually trigger resynchronization of a storage target or metadata server. One case, for example, is a storage system on the secondary target that is damaged beyond repair. In this scenario all data of that target might be lost and a new target needs to be brought up with the old target ID. The automatic resync won't be sufficient then, because it would only consider files after the last successful communication of the targets. Another case for a manual resync override is when a fsck of the underlying local file system (e.g. xfs_repair) has removed old files.

The beegfs-ctl tool can be used to manually set a storage target or metadata server to the needs resync state. Please note that this does not trigger a resync immediately, but does only inform the management daemon about the new state. The resync process then will be started by the primary of that buddy group a few moments later.

As said before, the primary target saves the time of the last successful communication with the secondary target. Without additional parameters, this timestamp will be used to shorten resynchronization times as much as possible. But it is also possible to override this timestamp to resynchronize a longer timespan or to resynchronize everything in the case described previously.

Please see the help of beegfs-ctl for more information on available parameters:
 $ beegfs-ctl --startresync --help 


If a resynchronization is already running and you want to abort it and start anew, you can do so by passing the --restart parameter to beegfs-ctl. If you don't, the current process keeps running and your request will be ignored. This is particularly useful if the system started an automatic resynchronization after a secondary target became reachable again, but you know that the timestamp-based approach is not sufficient. For example, this might be the case if your complete underlying filesystem broke before the secondary target was started, i.e. the target is completely empty and needs a full synchronization. Note that restarting a running resync is only possible for storage targets because metadata servers never do a partial resynchronization.

The following command could be used to stop the automatic resynchronization and start a full resynchronization instead.

 $ beegfs-ctl --startresync --nodetype=storage --targetid=X --timestamp=0 --restart 


Display Resynchronization Information


The beegfs-ctl command line tool can be used to display information on an ongoing resynchronization process.

Please see the built-in help of beegfs-ctl for more information on available parameters:
 $ beegfs-ctl --resyncstats --help 



Replacement of a failed device


If a storage or metadata target device of a buddy mirror group fails, and it cannot be recovered by RAID mechanisms, then the mirrored data can be restored with the commands presented above. In order to do this, it is necessary to recreate the device with the same IDs that the original target had. In the following example BeeGFS v6 and CentOS 7 are used. The steps are the same for newer BeeGFS versions and other Linux Distributions but the commands might differ.

Step-by-step example replacing a storage target



The following IDs are needed: nodeID, nodeNumID, targetID and targetNumID


The nodeIDs and targetIDs can be verified by the following commands:
$ beegfs-ctl --listtargets --nodetype=storage --longnodes

The targetNumIDs can be verified by the contents of the file 'targetNumIDs' in the BeeGFS management target, where the targetIDs are in hexadecimal form:
$ beegfs-ctl --listtargets --nodetype=storage --longnodes
TargetID   NodeID
========   ======
     101   beegfs-storage storage01 [ID: 1]
     201   beegfs-storage storage02 [ID: 2]

$ echo "obase=16; 201" | bc  # To convert decimal values to hexadecimal values
C9
$ cat /beegfs/mgmtd/targetNumIDs
0-5A9D4595-1=65
0-5A9E9232-2=C9
$ echo $((0xC9))             # To convert hexadecimal values to decimal values
201



If target 201 failed, the new target needs to have the nodeID=storage02, nodeNumID=2, targetID=201 and targetNumID=0-5A9E9232-2 . In the following we present the steps for the case that an internal device needs to be replaced and systemd is used to start and stop the BeeGFS services. Depending on the environment, the steps can be different.
  1. If not already done, stop the storage daemon:
    $ systemctl stop beegfs-storage
  2. Verify that the mirror buddy of the failed device is primary
    $ beegfs-ctl --listmirrorgroups --nodetype=storage
  3. Disable the storage daemon
    $ systemctl disable beegfs-storage
  4. Edit /etc/fstab, by commenting or removing the entry of the failed device. (We recommend the usage of labels or UUIDs instead of paths like /dev/sdX)
  5. Power off the server
  6. Replace the failed device
  7. Power on the server
  8. Verify the new device
  9. Format the device according to your setup (Setup recommendations)
  10. Edit /etc/fstab, by uncommenting and adapting or adding the entry for the new device (See the Setup recommendations for mount options)
  11. Reboot to check that everything is mounted correctly after a reboot
  12. Run the BeeGFS storage setup script.
    $ /opt/beegfs/sbin/beegfs-setup-storage -p /data/beegfs/beegfs_storage -s 2 -i 201 -m <mgmtd_hostname> -S storage02 -C
    For more information about that command, please run:
    $ /opt/beegfs/sbin/beegfs-setup-storage -h
  13. Create the file /data/beegfs/beegfs_storage/targetID with the targetID that was identified before (in our case: 0-5A9E9232-2):
    $ echo 0-5A9E9232-2 > /data/beegfs/beegfs_storage/targetID
  14. Start the storage daemon
    $ systemctl start beegfs-storage
  15. Verify that everything is running
    $ systemctl status beegfs-storage
    $ beegfs-ctl --listtargets --nodetype=storage --longnodes --state
  16. Start a full resynchronization. Shortly after starting the storage daemon, the secondary target most probably doing the resynchronization to the last timestamp. By adding --restart to the --startresync command it doesn't matter, if a resync is running:
    $ beegfs-ctl --startresync --targetid=201 --nodetype=storage --timestamp=0 --restart
  17. During the resync, the state changes of the targets can be verified by running:
    $ watch -n 1 beegfs-ctl --listtargets --nodetype=storage --longnodes --state
  18. Using --resyncstats the resync can be checked in detail:
    $ beegfs-ctl --resyncstats --targetid=201 --nodetype=storage
  19. When the secondary target returned to be in the Consistency=Good state, the system is back.
  20. Now the storage daemon can be enabled again:
    $ systemctl enable beegfs-storage

Step-by-step example replacing a meta node


The following IDs are needed: nodeID, nodeNumID. They can be retrieved by the following command:
$ beegfs-ctl --listtargets --nodetype=meta --longnodes
TargetID   NodeID
========   ======
       3   beegfs-meta meta03 [ID: 3]
       4   beegfs-meta meta04 [ID: 4]


If meta node 4 failed, the new node needs to have the nodeID=meta04 and nodeNumID=4. In the following we present the steps for the case that an internal device needs to be replaced and systemd is used to start and stop the BeeGFS services. Depending on the environment, the steps can be different.
  1. If not already done, stop the meta daemon:
    $ systemctl stop beegfs-meta
  2. Verify that the mirror buddy of the failed device is primary
    $ beegfs-ctl --listmirrorgroups --nodetype=meta
  3. Disable the meta daemon
    $ systemctl disable beegfs-meta
  4. Edit /etc/fstab, by commenting or removing the entry of the failed device. (We recommend the usage of labels or UUIDs instead of paths like /dev/sdX)
  5. Power off the server
  6. Replace the failed device
  7. Power on the server
  8. Verify the new device
  9. Format the device according to your setup (Setup recommendations)
  10. Edit /etc/fstab, by uncommenting and adapting or adding the entry for the new device (See the Setup recommendations for mount options)
  11. Reboot to check that everything is mounted correctly after a reboot
  12. Run the BeeGFS meta setup script:
    $ /opt/beegfs/sbin/beegfs-setup-meta -p /path/to/device/mount -s 4 -m <mgmtd_hostname> -S meta04 -C
    For more information about that command, please run:
    $ /opt/beegfs/sbin/beegfs-setup-meta -h
  13. Start the meta daemon
    $ systemctl start beegfs-meta
  14. Verify that everything is running
    $ systemctl status beegfs-meta
    $ beegfs-ctl --listtargets --nodetype=meta --longnodes --state
  15. Start a full resynchronization. That should happen automatically shortly after starting the daemon. If not, it can be started manually:
    $ beegfs-ctl --startresync --nodeid=4 --nodetype=meta
  16. During the resync, the state changes of the targets can be verified by running:
    $ watch -n 1 beegfs-ctl --listtargets --nodetype=meta --longnodes --state
  17. Using --resyncstats the resync can be checked in detail:
    $ beegfs-ctl --resyncstats --nodeid=4 --nodetype=meta
  18. When the secondary node returned to be in the Consistency=Good state, the system is back.
  19. Now the meta daemon can be enabled again:
    $ systemctl enable beegfs-meta



Back to Buddy Mirroring Overview
Valid XHTML :: Valid CSS: :: Powered by WikkaWiki