= BeeGFS Changelog (v6.x Release Series) = == Changes in 6.19 (release date: 2018-08-28) == Enhancements: * client: Added support for Linux Kernels 4.15, 4.17, and 4.18 * general: Documentation of timeout options in config files improved * storage: Added support for ZFS inode quota for ZFS versions >=0.7.4 * meta: Improved logging on mirroring failure caused different return codes * meta: Abort mirror buddy resync early in case of errors * client: Added support for OFED 4.3 * client: Added an option to limit concurrent connection attempts in order to prevent network issues * fsck: Prevent fsck from being run while nodes are needing a resync Fixes: * client: Fixed issue that prevented ACLs from being enforced immediately after a change * servers: Fixed issue that could lead to servers shutting down when an empty UDP packet was received * meta: Fixed race that could lead to remote IO errors when two clients tried to create the same file simultaneously * meta: Fixed resyncer issue that could lead to aborted resync when directories were deleted during resync * client: Fixed compatibility with Kernels provided by RedHat 7.5 * client: Fixed an issue that prevented multiple simultaneous BeeGFS mounts on Linux Kernel 4.14 and newer == Changes in 6.18 (release date: 2018-03-12) == Enhancements: * fsck: Must be run as root now * fsck: Inform user when fsck changes node state for repair action * client: Added support for Linux kernel 4.14 * general: Improvements on a number of error log messages to be more intuitive * ctl: Added migration of extended attributes and ACLs * ctl: Removed units from CSV output for consistency (--getquota --withzero --csv) * BeeOND: Remove kernel module at exit if no other BeeGFS or BeeOND instance is mounted Fixes: * fsck: Fixed issue that could lead to fsck aborting prematurely with "unordered operation" message * fsck: Do not abort in some cases where root inode is duplicated * storage: Fixed several chunk attribute and mirroring issues * storage: Improved logging during fsck * client: Fixed iov_iter behaviour when pipe is full (fixes an issue with NFS) * client: Fixed problems during unmount using newer kernel 4.13 and 4.14 * general: Fixed potential issue when fsck or metadata server was stopped before it was fully initialized * ctl: Improved output when filtering quota info by ID range * ctl: Fixed swapped values in resync job stats == Changes in 6.17 (release date: 2017-10-25) == Enhancements: * ctl: Extended sanity checks in modes --addmirrorgroup and --createfile. Fixes: * client: Allow NULL for empty ACLs in set_acl (e.g. used by NFS). [Thanks to ETH Zurich for reporting.] * client: Implement ITER_PIPE for linux 4.9 and newer. [Thanks to ETH Zurich for reporting.] * client: Don't try communication with storage nodes known to be offline. == Changes in 6.16 (release date: 2017-09-15) == General Changes: * client: Add support for linux-4.13. * auto_package: Only build bundled thirdparty libs on user request. Fixes: * ctl: Return proper return code (i.e. 0) if "--addmirrorbuddygroup --automatic --force" runs without error. * ctl: Properly validate striping chunk size for mode "--createfile". * client: Fix occassional coherency issues between clients. == Changes in 6.15 (release date: 2017-08-29) == Fixes: * admon: Fix startup issues on first run (i.e. if config files don't exist). [Thanks to CERN for reporting.] * ctl: Fix help text for "--createfile". * client: Don't try to access inode information after writeback has finished in paged mode I/O. [Thanks to DESY for reporting.] == Changes in 6.14 (release date: 2017-08-10) == Enhancements: * admon: Add Chinese language support. [Thanks to JZTech for providing this.] * beeond: Add option to use buddy mirroring. [Thanks to Huawei for suggesting this.] Fixes: * meta: Fix potential deadlock on modification of extended attributes. [Thanks to RiseFX for reporting.] * meta: Properly clean mirrored file sessions when clients are removed from the system. [Thanks to HPCnow for reporting.] * meta/storage: Fix potential memory leak when setting resync state. * client: Add compatibility with OverlayFS. [Thanks to HPCnow for reporting.] * client: Properly handle startup if no network interfaces were detected. [Thanks to Fred Hutch for reporting.] * beeond: Use the same name for the status file in beeond stoplocal and beeond stop. [Thanks to KIT for reporting.] == Changes in 6.13 (release date: 2017-07-26) == General changes: * all: Added packages for Debian 9. * admon: Remove obsolete RAID10 pattern from GUI. [Thanks to Huawei for reporting.] * fsck: Start resync in metadata mirrored systems after fsck has completely finished, instead of resyncing for each repair action independently. Fixes: * client: Don't try to perform fsync on offline targets. [Thanks to ClusterVision and Alphacruncher for reporting.] * fsck: Fix cases in which fsck spuriously reports failed repair actions, although they succeeded. * meta: Fix possible race of mkdir and rmdir during meta resync that could abort resync. * meta: Fix possible race of rename and other operations during meta resync that could abort resync. == Changes in 6.12 (release date: 2017-06-30) == General Changes: * client: Add support for linux 2.6.18 - 4.11. * client: Add support for Mellanox OFED 4.1. * mgmtd: Avoid race condition when shutting down mgmtd that could lead to spurious resyncs of secondary metadata servers. [Thanks to Megware and University of Jena for reporting.] * mgmtd: Allow to start with an empty node and target state files on disk. [Thanks to University of Geneva for reporting.] * meta: Rename file msessions to mirroredSessions to better reflect its purpose. * client: Don't consider node consistency state information for unmirrored data. Fixes: * ctl: Don't use buddy group IDs as nodelist for "--createdir --nomirror". * meta: Don't try to resync a modification event for a failed create-exclusive. * client: Fall back to syslog logging if helperd was not properly initialized. [Thanks to HPCnow for reporting.] == Changes in 6.11 (release date: 2017-05-26) == General Changes: * client: Add option sysRenameEbusyAsXdev to return EXDEV instead of EBUSY if rename() is called on open files. (Tools like "mv" can handle EXDEV as return value.) Fixes: * mgmtd: Immediately save target states to disk after first startup to avoid initialization errors on consecutive starts, if daemon was stopped shortly after first start. [Thanks to University of Geneva for reporting.] * meta: Properly restore sessions to deleted, but still opened files on restart and resync (could lead to unremovable directories). [Thanks to Megware for reporting.] * ctl: Properly document option --unmounted in mode "setpattern". * ctl: Return proper error code if opening a directory fails in mode "migrate". * ctl: Don't print false error message "Some IDs were skipped because of invalid values" when using --range in mode "getquota". [Thanks to anonymous customer from automotive industry for reporting.] == Changes in 6.10 (release date: 2017-04-28) == Fixes: * client/storage: Fix possible client error on systems using TCP when a storage server is restarted while operations are in progress. [Thanks to anonymous US Elite Defense Contractor for reporting.] * storage: Fix cases in which file sessions were not properly removed after resync and on close file errors. == Changes in 6.9 (release date: 2017-04-19) == Enhancements: * meta: Faster initialization of usable storage targets on startup. * mgmtd: Optimize save and restore of server states on unclean server shutdown. Fixes: * client: Fix client error when removing the last mirror buddy group from the system with beegfs-ctl. * client: Don't retry communication if signals are pending before the first connection attempt to the servers. * utils: Don't print units in output of "beegfs-ctl --getquota --csv". Known Issues: * ctl: The reachability check "beegfs-ctl --listnodes --nodetype=client --reachable" will report "Numeric ID collision" errors for clients versions before 6.9, due to a change in the message format. == Changes in 6.8 (release date: 2017-03-31) == General Changes: * client: Added support for kernel 2.6.32-642.15.1.el6.x86_64 (RHEL6). * client: Added support for Mellanox OFED 4.0. Enhancements: * mgmtd: Optimize save and restore of server states upon mgmtd restart. * utils: Add flag "--errorcodes" to "beegfs-ctl --getnodes" and "beegfs-checkservers". If set, the scripts will return a non-zero exit code if a node is unreachable. [Thanks to Albert Einstein Insitute Potdsdam for suggesting.] Fixes: * all: Fix detection of systemd on newer systems. * meta: Fix cases in which a secondary metadata server could be set into "bad state" although it is actually good. This could happen if file locks were held during a resync or restart of the server. * meta: Fix possible false modification event on rename during metadata resync that will lead to false errors reported. * storage: Fix potential issue that leads to a locked up storage server resyncer (resync never finishes without server restart). [Thanks to ETH Zurich for reporting.] * storage: Don't shutdown with error message if server gets shut down while resync is running. * client: Fix possible race between link and write that could sometimes lead to a non-responsive client. * client: Prevent processes from going into uninterruptible sleep under certain circumstances, if signals are pending before the first connection attempt to the servers. [Thanks to University of Oslo for reporting.] * client/meta: Retry mkdir instead of aborting with "Remote I/O error" if a client has received state updates before all metadata servers have. * utils: Fix output of "beegfs-check-servers" (properly print "Unreachable" for unreachable nodes instead of "connect failed on all available routes"). * utils: Don't abort "beegfs-ctl --setquota" with bad_alloc if parameter syntax is wrong. * utils: Don't allow to set mirror buddy groups with non-existing nodes as members. == Changes in 6.7 (release date: 2017-03-09) == General Changes: * repo: Changed Debian repository signature from SHA1 to SHA512 for compatbility with apt v1.4. [Thanks to FU Berlin for reporting.] * mgmtd: Enforce sysTargetOfflineTimeoutSecs to be greater than 30 seconds to avoid human configuration errors. Fixes: * storage: Fix problems with new security heuristic in recent RDMA drivers that could lead to RDMA failures and loss of RDMA connectivity if errors occur during write on server with enabled quota. [Thanks to Megware and anonymous US Elite Defense Contractor for reporting.] * meta: Fix possible memory leak on file unlink on secondary server of a mirror buddy group. [Thanks to FU Berlin for reporting.] * client: Don't retry RDMA communication if read/write fails with EFAULT, but signal error immediately. == Changes in 6.6 (release date: 2017-03-02) == Enhancement: * storage: Increase efficiency of state exchange between the servers of a mirror buddy group. * client: Added support for linux-4.9 and Debian/Ubuntu kernel 4.8.0. General Changes: * meta: Don't sync privileged extended attribute namespaces (e.g. security) on resync. [Thanks to Alphacruncher for reporting.] Fixes: * opentk: Fix manual rebuild, if InfiniBand autodetection fails. [Thanks to HPE and Megware for reporting.] * general: Fix possible server shutdown if reading a RDMA event failed. * meta: Fix possible server shutdown if saved sessions file is corrupt on startup. * meta: Fix possible memory leak in buddy mirrored setups, if messages sent to the server return an error. == Changes in 6.5 (release date: 2017-02-17) == General Changes: * general: Adjusted default timeouts for offline-detection of storage and metadata servers to handle network delays better. * mgmtd: Use separate thread pool for quota checks to better handle situations of high load on systems with a large quota UID range. Fixes: * meta: Fix loading of sessions after server restart for files that were deleted while opened, if the metadata server is configured to not use extended attributes for its on-disk data. == Changes in 6.4 (release date: 2017-02-03) == Enhancements: * utils: Add option to beegfs-check-servers to provide a client config file for systems with multiple BeeGFS instances. [Thanks to ScalableInformatics.] General Changes: * meta: On shutdown, only log client session information in case of errors. Fixes: * storage: Fix rare cases in which automatic block device detection for retrieving quota information did not work correctly. * meta/client: Fix possible file reference leak (file was not deleted from memory) in experimental native cache mode. [Thanks to anonymous US Elite Defense Contractor for reporting.] * meta: Fix unlikely file reference leak after setting owner on a file with quota enabled. * meta: Fix possible stalled process on shutdown if a resync is in progress. * meta: Fix potential stalled process if a resync of files, which were modified during the resync, fails. * mgmt: Improve node state saving in case of unexpected mgmtd failure. * admon: Fix package download URL. * client: Fix potential wrong link count if creating a file fails. [Thanks to Megware and XMA for reporting.] * client: Fix unlikely case that could lead to stalled processes when renaming a file and calling stat on its parent directory. * client: Fix possible unintended user-memory-access in MkfileWithStripeHints ioctl. * client: Fix handling of empty (and thus incorrect) input data in CreateFile ioctl. * client: Do not allow to use S_IFDIR in CreateFile ioctl. == Changes in 6.3 (release date: 2016-12-28) == Enhancements: * fsck: Dispose unused files before running checks (if not running read-only). * build: Enable cross compilation for aarch64 in thirdparty tools Makefile. General Changes: * servers: Allow /0 in netfilter config file as "allow all". Fixes: * auto_package: Fix help text for version parameters. * client_devel: Fix typos in examples. * client: Properly unlock mmap I/O pages in case of communication errors. * all: Fix compilation with gcc-6.2. * client: Fix compilation with RHEL 5.x kernels. * client: Fix problems in (experimental) native cache mode where written pages were not correctly sent to servers. == Changes in 6.2 (release date: 2016-12-05) == Enhancements: * admon: Add sendmail support for email notifications. General Changes: * storage/ctl: Introduce settings for unlimited quota for individual users if default quota are set. * meta/storage: Perform a failover if primary server gets stuck in an erroneous state. Fixes: * client/meta: Fix issue in "native" cache mode (experimental) that could lead to files which were not fully removed from disk when deleted. * fsck: Fix possible shutdown on RHEL 5.x due to a bug in libstdc++ versions prior to 4.6. * client: Fix potential client failure if connection retries fail under certain circumstances. == Changes in 6.1 (release date: 2016-11-16) == Fixes: * client: Fix mixed read/readv, write/writev on kernels before linux-3.16. == Changes in 6.0 (release date: 2016-11-14) == This is a new major release. The primary focus for this new release was the introduction of high availability for metadata servers based on mirroring. New Features: * Metadata Buddy Mirroring: BeeGFS now allows the definition of so-called buddy mirror groups of metadata servers to provide high availability and self-healing in case of a metadata server failure. General Changes: * Changing versioning scheme from YYYY.MM to MAJOR.MINOR. BeeGFS 2015.03 was the 5th BeeGFS major release and thus the new major release is 6.0. Enhancements: * client: Performance optimizations for "native" cache mode (experimental). * client: Support for security xattrs. * storage: Quota support for ZFS as underlying storage file system. * fsck: Changed database backend to custom database and other optimizations to improve runtime. * client: Improved handling of applications that mix read/write and mmap I/O. === Compatibility Notes === * Supported distributions: RHEL 5/6/7 (and Fedora), SLES 11/12 (and OpenSuse), Debian 7/8 (and Ubuntu). * Supported Linux kernel versions (client): 2.6.18 - 4.8. * Servers and clients from different major release series are not compatible and cannot be used together in the same file system instance. * If you want to upgrade a 2015.03 release, see here for upgrade notes: http://www.beegfs.io/release/beegfs_6/UpgradeNotes.txt