= BeeGFS Changelog (2015.03 Release Series) = == Changes in 2015.03-r27 (release date: 2017-10-25) == General Changes: * client: Add support for SLES11 SP4 kernel linux-3.0.101-107.sles11sp4. * client: Add support for MLNX OFED 4.1. Fixes: * client: Allow NULL for empty ACLs in set_acl (e.g. used by NFS). [Thanks to ETH Zurich for reporting.] == Changes in 2015.03-r26 (release date: 2017-08-10) == General Changes: * client: Add support for linux-3.10.0-693.el7.x86_64 in RHEL 7.4. Fixes: * meta/storage: Fix potential memory leak when setting resync state. == Changes in 2015.03-r25 (release date: 2017-07-13) == Enhancements: * mgmtd: Improve node state saving in case of unexpected mgmtd failure. General Changes: * client: Add support for MLNX OFED 4 and Intel OFED+ 7.4.2.0.6. [Thanks to GWDG and HLRN / Zuse Insitute Berlin for reporting.] Fixes: * storage: Fix potential issue that could lead to a locked up storage server resyncer (resync never finishes without server restart). [Thanks to ETH Zurich for reporting.] * utils: Don't print units in output of "beegfs-ctl --getquota --csv". [Thanks to University of Mannheim.] * meta: Fix creation of new files in folders with default ACLs. [Thanks to ClusterVision for reporting.] == Changes in 2015.03-r24 (release date: 2017-03-14) == Fixes: * general: Fix detection for the presence of systemd on recent distributions. * mgmtd: Use separate thread pool for quota checks to better handle situations of high load on systems with a large quota UID range. * storage: Fix problems with new security heuristic in recent RDMA drivers that could lead to RDMA failures and loss of RDMA connectivity if errors occur during write on server with enabled quota. [Thanks to Megware and Corvid for reporting.] == Changes in 2015.03-r23 (release date: 2017-02-10) == Enhancements: * admon: Add sendmail support for email notifications. * utils: Add option to beegfs-check-servers to provide a client config file for systems with multiple BeeGFS instances. [Thanks to ScalableInformatics] General Changes: * servers: Allow /0 in netfilter config file as "allow all". * storage/ctl: Introduce settings for unlimited quota for individual users if default quota are set. Fixes: * fsck: Fix possible shutdown on RHEL 5.x due to a bug in libstdc++ versions prior to 4.6. * meta: Fix read of extended attributes on hard linked files after deletion of one link. [Thanks to Universita della Svizzera Italiana for reporting] * meta: Properly move extended attributes if move of file fails during server restart. * storage: Fix rare cases in which automatic block device detection for retrieving quota information did not work correctly. * client: Fix potential wrong link count if creating a file fails. [Thanks to Megware and XMA for reporting] * client: Properly unlock mmap I/O pages in case of communication errors. * client: Fix compilation with RHEL 5.x kernels. * client: Fix handling of empty (and thus incorrect) input data in CreateFile ioctl. * client: Do not allow to use S_IFDIR in CreateFile ioctl. * client: Fix possible unintended user-memory-access in MkfileWithStripeHints ioctl. == Changes in 2015.03-r22 (release date: 2016-11-16) == Fixes: * client: Fix mixed read/readv, write/writev on kernels < 3.16. == Changes in 2015.03-r21 (release date: 2016-11-08) == * General Changes: * client: Add support for additional mount options in beegfs-mounts.conf for multi-mode setups. * client: Add support for linux-3.10.0-514.el7. * Enhancements: * fsck: Add option to remove unreadable chunks with malformed names from storage targets. * client: Support aio in buffered mode. * Fix: * storage: Properly abort all threads on storage resync, if resync is aborted. * meta: Properly stop modification event flusher when fsck is cancelled. == Changes in 2015.03-r20 (release date: 2016-10-19) == * General Changes: * general: Override quota enforcement configuration setting of servers with quota enforcement value from mgmtd. * Enhancements: * fsck: Include sanity checks and warn if a storage server has malformed chunk file names or directory structures. [Thanks to University of Oslo for reporting.] * Fix: * meta: Correctly check and enforce quota limits on chgrp. [Thanks to University of Mannheim for reporting.] * meta: Fix possible update of ctime on file read. [Thanks to Max Planck Institute for Metabolism Research for reporting.] * fsck/meta: Fix possible metadata server stall in fsck online mode in modification log enable phase. == Changes in 2015.03-r19 (release date: 2016-09-27) == * Enhancements: * ctl/meta: Allow users to set stripe pattern for directories they own (also see configuration option sysAllowUserSetPattern). * storage: Improve server failover handling under heavy system load. * Fix: * meta: Fix occasional lockup during session restore on startup. [Thanks to University of Heidelberg for reporting.] == Changes in 2015.03-r18 (release date: 2016-08-16) == * General Changes: * client: Updated to be compatible with linux-2.6.18 up to linux-4.7. * fsck: Run fsck in online mode by default. * Enhancements: * BeeGFS API: Add support for buddymirror stripe pattern. * general: Add log topics to logging facility of servers. By using log topics, the log levels can be set on a more fine-grained level, i.e. a log level can be set for each log topic individually. * ctl: Improve output of getquota for underlying filesystems which don't support inode quota. The used inodes are no longer displayed as 0. * Fixes: * meta: Ignore empty ACL when creating files or directories. * admon: Fix listing of buddymirror groups and the targets in the stripe pattern GUI. * admon: Fix "remote log file" function for clients in certain cases. == Changes in 2015.03-r17 (release date: 2016-07-25) == * Enhancements * storage: Don't do read-ahead (ignore tuneFileReadAheadSize) for direct I/O. [Thanks to ScalableInformatics for reporting.] * Fixes: * fsck: Fix potential shutdown during tests when running fsck in offline mode. [Thanks to University of Mannheim for reporting.] * meta: Update ctime/mtime when setting/changing/removing extended attributes. * mgtmd: Fix potential shutdown if quota enforcement is disabled. [Thanks to JZTech for reporting.] * admon: Fix display of buddy mirror groups in Admon GUI. == Changes in 2015.03-r16 (release date: 2016-07-05) == * General Changes: * meta/storage: Added support for RDMA with latest OFED drivers (e.g. mlnx_ofed 3.3, linux-4.6, RedHat 7.2 with kernel 3.10.0-327.22.2.el7). [Thanks to Mellanox and University of Mannheim for reporting.] * Enhancements: * fsck: Added option to abort and restart a running file system check. == Changes in 2015.03-r15 (release date: 2016-06-30) == * General Changes: * fsck: Wait longer for management daemon to respond when downloading node lists. This prevents fsck from failing to start if the management daemon takes too long to respond in busy systems. * Fixes: * storage: Don't restrict the length of storeStoragePath to 1024 characters. [Thanks to HPC Advisory Council for reporting] * admon: Fix possible daemon shutdown if time of storage servers is not synchronized. [Thanks to JZTech for reporting.] * client: Fixed some rare cases, in which files created with O_EXCL could be shown as deleted in lsof. * client: Fixed potential client stalls when mixing read/write and mmap I/O under heavy load. * Enhancements: * client: Added option tuneCoherentBuffers to enable/disable strong coherency of mixed read/write and mmap I/O. == Changes in 2015.03-r14 (release date: 2016-06-09) == * General Changes: * client: Updated to be compatible with linux-2.6.18 up to linux-4.6. * Enhancements: * general: Add option to ignore system users for quota checks. * general: Add option to set default quota limits. * mgmtd: Optimize recalculation of exceeded quotas after setting new limits. * meta: Store open metadata sessions on disk when stopping and recover sessions on start. * ctl: New option to read list of users for quota from file. * Fixes: * admon: Fixed possible failure of daemon restart when restarted via GUI. * admon: Automatically give focus to selected frame in GUI. * mgmtd: Fix possible error if a node gets deleted while exceeded quota is being propagated to nodes. * ctl: Properly cleanup open files in storage bench mode if an error occured during execution. == Changes in 2015.03-r13 (release date: 2016-05-09) == * General Changes: * client: Make built-in config option for communication retry timeouts consistent with config file (relevant for setups without config file, e.g. BeeOND). * Enhancements: * ctl: Add csv as output format. [Thanks to Kevin Van Workum from Sabalcore Computing Inc. for contributing.] * beeond: Optimized pdsh-based parallelization during startup. * Fixes: * meta: Fix possible discard of extended attributes on file move. * admon: Fixed occasional usage of MB as unit, where MiB should be used. * fsck: Fixed possible false positives for duplicated entry IDs in online mode. [Thanks to MPI for Biology of Ageing for reporting.] == Changes in 2015.03-r12 (release date: 2016-04-18) == * Fixes * meta/storage: Fixed possible wrong modification time of files in cases where modification time was set manually directly after closing a file. * storage: Fixed possible memory leak in InternodeSyncer component. * meta: Fixed retrieval of extended attributes if application buffer is larger than the extended attribute value. * client: Fixed ID mapping for applications that use user namespaces. [Thanks to DESY for reporting and suggesting a fix.] * client: Fixed encoding of NFS handle for BeeGFS root inode in NFS exports. * client: Changed tuneEarlyCloseResponse optimization default to false, due to conflicts with some applications. == Changes in 2015.03-r11 (release date: 2016-03-31) == * General Changes: * client: Updated to be compatible with linux-2.6.18 up to linux-4.5. * Enhancements: * client: Added support for custom mount options in beegfs-mounts.conf. Space-separated columns in beegfs-mounts.conf now are: * mountpoint (e.g. /mnt/beegfs) * config file (e.g. /etc/beegfs/beegfs-client.conf) * kernel module name (e.g. beegfs) [optional] * comma-separated custom mount options (e.g. ro) [optional] * admon: Added support for multiple GUI instances connecting to different BeeGFS instances from the same host. * fsck: Changed database backend to custom database and other optimizations to improve runtime. * client: Improved handling of applications that mix read/write and mmap I/O. [Thanks to ClusterVision and Zettar for reporting.] * Fixes: * storage: Fixed default values in documentation of tuneFile{Read,Write}Size in config file. * fsck: Fixed deletion of files, which were partially stored on targets that no longer exist. * client: Fixed handling of write retries in case of storage buddy failover. * meta: Fixed ctime calculation for environments with time offset between metadata servers and storage servers. [Thanks to MPI for Metabolism Resarch for reporting.] == Changes in 2015.03-r10 (release date: 2016-01-26) == * General Changes: * client: Updated to be compatible with kernel of SLES11 SP4 (3.0.101-68) [Thanks to University of Cape Town for reporting.] * Enhancements: * storage: Added quota support for ZFS as underlying storage file system. * Fixes: * client/meta: Fixed problem with umask handling when ACLs are enabled. [Thanks to University of Mannheim for reporting.] * migration script: Fixed access permissions for /etc/beegfs. [Thanks to Texas A&M Universtity for reporting.] == Changes in 2015.03-r9 (release date: 2015-12-23) == * Enhancements: * client: Add support for security xattrs. [Thanks to STZRE/Daimler for reporting.] * Fixes: * meta: Check maximum xattr size before sending to client. [Thanks to STZRE/Daimler for reporting.] * ctl: Ignore old config option "tuneMaxClientMirrorSize". [Thanks to Cambridge MRC Laboratory of Molecular Biology for reporting.] * migration script: Also update path of option connTcpOnlyFilterFile. [Thanks to Texas A&M Universtity for reporting.] == Changes in 2015.03-r8 (release date: 2015-12-08) == * General Changes: * client: Updated to be compatible with linux-2.6.18 up to linux-4.3. * client: Added support for RHEL 7.2 kernel. [Thanks to University of Mannheim for reporting.] * Improvements: * fsck/meta: Performance optimizations and improved delay handling. * Enhancements: * client: Added new cache mode "native", using the kernel's page cache (experimental). * Fixes: * admon GUI: Fixed issue, which could lead to flickering tree menu and sporadic delays. * ctl: Fixed wrong selection of files in mode "--find" and "--migrate". == Changes in 2015.03-r7 (release date: 2015-10-14) == * Fixes: * admon GUI: Fixed built-in version code. == Changes in 2015.03-r6 (release date: 2015-10-12) == * General Changes: * client: Updated to be compatible with linux-2.6.18 up to linux-4.1. [Thanks to Scalable Informatics for reporting.] Fixes: * admon GUI: Fixed compatibility with recent MacOS versions. [Thanks to M:Tier Ltd. for reporting.] * migration script: Fixed newline issue with multiple entries in beegfs-mounts.conf. [Thanks to University of Oslo for reporting.] * migration script: Fixed permissions of new directories. * mgmtd: Fixed incorrect selection of emergency pool for metadata servers in some cases. [Thanks to Aarhus Bioinformatics Research Centre for reporting.] * all: Fixed some log messages mentioning "fhgfs" instead of "beegfs". [Thanks to GrauData for reporting.] == Changes in 2015.03-r5 (release date: 2015-09-18) == * Improvements: * all: Support for ZFS in systemd scripts. * Fixes: * mgmtd: Fixed problem where quota was not correctly calculated for users that appeared multiple times in system databases. [Thanks to University of Oslo for reporting.] == Changes in 2015.03-r4 (release date: 2015-09-11) == * Fixes: * fsck: Fixed startup issues when using ACL option in client config. == Changes in 2015.03-r3 (release date: 2015-09-08) == * Fixes: * client: Fixed typo in paths of multi-mode startup scripts. == Changes in 2015.03-r2 (release date: 2015-08-14) == * Fixes: * client: Removed calls to special vfs read/write methods, which could lead to cache usage problems with some applications. == Changes in 2015.03-r1 (release date: 2015-08-11) == This is a new major release. The primary focus for this new release was the introduction of high availability for storage servers based on mirroring and on the new BeeOND ("BeeGFS on-demand") package to setup temporary file system instances e.g. on a per-job basis on compute node disks (to remove I/O load from the global storage system). If you are upgrading from a 2014.01 release, make sure to read the compatibility section below. New Features: * Buddy Mirroring: BeeGFS now allows the definition of so-called buddy mirror groups of storage targets to provide high availability and self-healing in case of a storage server failure. * BeeOND ("BeeGFS on-demand"): The new BeeOND package provides a tool that can create (and later destruct) complete BeeGFS instances on-the-fly on a given set of hosts with only a single command. It is typically called in pre-job and post-job scipts of a cluster batch system, but can also be used e.g. in cloud environments. * Access Control Lists (ACLs): BeeGFS now supports extended attributes and ACLs. * Setup tools: New setup tools allow basic configuration of BeeGFS services without manually editing the configuration files and also provide options to set custom node and target IDs. (See "/opt/beegfs/sbin/beegfs-setup-XY") * Target reachability states: The management service now keeps track of the reachability state (online/offline) of metadata and storage servers. This is used to avoid retry timeouts for applications when a server is unreachable for a longer time. See "beegfs-ctl --listtargets --state". Improvements: * Storage worker threads: The beegfs-storage service worker threads that handle parallel incoming client requests are now bound to the individual targets of a server to enable better balance and fairness in multi-target servers under high load. Thus, the option "tuneNumWorkers" in beegfs-storage.conf now defines the number of worker threads per storage target (and not the global number of worker threads as before). * Metadata performance: For some operations (e.g. file unlink), the metadata service can now respond to the client before all server-side work is done, resulting in lower latency. * Directory offsets: Clients now return native readdir() entry offsets from the underlying local file system on metadata servers, resulting in improved performance for applications that seek within a directory. Enhancements: * Per-user statistics: The Admon GUI can now show live I/O operation statistics for individual users. * Low-latency mode: For very latency critical applications, the metadata and storage servers can now be set to a new low latency mode with active polling at the cost of higher CPU usage. See option "tuneUseAggressiveStreamPoll" in beegfs-{meta,storage}.conf * Multiple listener threads: For systems with a very high number of disks and a high number of CPU cores, multiple stream listener threads that react on incoming client messages can now be enabled. See option "tuneNumStreamListeners" in beegfs-{meta,storage}.conf * Server IP address changes: Clients now update server IP addresses at runtime, so that clients do not need to be remounted when a server IP address changes. * Deny new target registration: To avoid unintended/accidental registration of new storage targets on a production system, the beegfs-mgmtd now has a new config file option "sysAllowNewTargets". * New target choosers: The metadata service now supports two new target chooser algorithms that select storage targets when a new file is created: "randominternode" (selects stripe targets from different servers) and "randomintranode" (selects stripe targets from within the same server group). See option "tuneTargetChooser" in beegfs-meta.conf. * Dynamic pool limits: Free space pool limits of storage targets are now adapted dynamically to improve new file placement if some targets are running out of space while other targets have significantly more free space available. General Changes: * Rebranding to BeeGFS: All services, tools, config files and log files now use "beegfs-" as name prefix instead of the old "fhgfs-" name prefix. * NodeIDs/TargetIDs: The management service now assigns sequential numeric IDs (1, 2, 3, ...) to new servers and storage targets instead of the previous random numeric IDs. * BeeGFS API: The BeeGFS client API files (that allow e.g. creation of a file with a user-defined number of storage targets for striping) has been moved to a separate new package "beegfs-client-devel". * BeeGFS pool listing: The output of the previous command line tool mode "fhgfs-ctl --listpools" has been merged into "beegfs-ctl --listtargets". * RDMA support: The command line tool that was used to enable userspace RDMA support for Infiniband/RoCE after initial package installation has been renamed to "beegfs-setup-rdma" (formerly "fhgfs-opentk-lib-update-ib"). === Compatibility Notes === * Supported distributions: RHEL 5/6/7 (and Fedora), SLES 11/12 (and OpenSuse), Debian 7/8 (and Ubuntu) * Supported Linux kernel versions (client): 2.6.18 - 4.0 * Servers and clients with different minor versions of the 2015.03 release series are compatible and can be used together in the same file system instance. * Servers and clients from different major release series are not compatible and cannot be used together in the same file system instance. * If you want to upgrade a 2014.01 release, see here for upgrade notes: http://www.beegfs.com/release/beegfs_2015.03/UpgradeNotes.txt