= FhGFS Changelog (2012.10 Release Series) = == Changes in 2012.10-r15 (released 2014-04-13) == Improvements: * fsck: Several performance improvements to reduce overall runtime. * general: Updated/added dependencies on other services (e.g. Infiniband) in init.d scripts. Enhancements: * client: Makefile now supports custom "strip" command for easier cross- compilation, e.g. for Intel Xeon Phi. Fixes: * admon: Fixed download of GUI jar file via webbrowser. * client: Fixed problem where root directory attributes were not correctly initialized if metadata server was unreachable during client mount. * client: Don't allow reuse of UDP socket port to avoid unintended sharing of the same UDP port for two different client mounts on the same machine. * client: Re-add additional local locks even if global POSIX locks are enabled, so that the Linux kernel does correct unlock calls on process exit. * fsck: Fixed problem of potential unclean shutdown due to invalid ModificationEventFlusher cleanup on termination. == Changes in 2012.10-r14 == * internal build, not released == Changes in 2012.10-r13 (released 2014-02-13) == Fixes: * client: Fixed compatibility with RHEL 5.10 kernel. [Thanks to RWTH Aachen for reporting.] * fsck: Fixed problem with error masking that resulted in false error reports in some cases. (Regression from r9.) [Thanks to Academy for Advanced Telecommunications and Learning Technology at Texas A&M University for reporting.] * admon: Fixed problem where admon daemon allowed unintended access to files on disk. [Thanks to University of Iowa for reporting.] == Changes in 2012.10-r12 (released 2014-01-09) == New Features: * Added support for user/group quota usage tracking. (See http://www.fhgfs.com/wiki/EnableQuota for details.) Enhancements: * client: Clearly show when connections were established over a fallback route in fhgfs-net. * storage/meta: New method to manually override reported amount of free disk space through a "free_space.override" file in a target directory. This can be used e.g. to report 0 disk space left for a certain target, so that it gets moved to the emergency pool and thus won't be used for new files: $ echo 0 > /free_space.override (No daemon restart is required when the file is created or removed.) [Thanks to Megware for suggestion during user meeting 2013.] * fhgfs-ctl: Mode "--setpattern" now accepts multiple directories. (See "fhgfs-ctl --setpattern --help" for usage example.) [Thanks to Academy for Advanced Telecommunications and Learning Technology for suggestion.] * client: Added fsync operation for directories (noop). [Thanks to University of Geneva DiSTIC department for reporting.] Improvements: * client: Skip local locking when global POSIX locks are used. * admon: Improved layout of server names in GUI status panel. General Changes: * client: Make module build a mutually exclusive section in init script. This avoids build problems when multiple concurrent processes (e.g. from a cluster management software) on the same machine try to start the client. [Thanks to CeMM of the Austrian Academy of Sciences for reporting.] Fixes: * fhgfs-ondemand: On stop, wait for processes to fully terminate before removing their data directories. * client: Flush file cache when locks are acquired or released via flock()/fcntl(). * mgmtd: Ignore "unreachable network" socket error instead of stopping daemon. * meta: Automatically select two targets for mirrored files in case parent directory is set to use only a single target per file. * fsck: Minor fixes of log messages on 32bit systems. * fsck: Fixed problem where fsck did not shutdown cleanly when network message decoding failed. * opentk-lib: Fixed rare problem with syslog logger cleanup during shutdown. (Syslog logger component was replaced by new implementation.) * servers: Perform clean shutdown when pidfile update fails during startup. * servers: Wait time between communication retries was very short in some cases. (Regression from r9.) * general: Change init script service requirement tags to avoid installation warnings on SLES11 systems. * client: Add workaround for kernel bug (linux-3.10+) where kernel's nsproxy variable was freed in do_exit() before all process files were closed, which is not expected by kernel's sock_create() method. == Changes in 2012.10-r11 == * internal build, not released == Changes in 2012.10-r10 (released 2013-11-21) == Fixes: * fsck: Disabled invalid chunk owner check. (Regression from r9.) [Thanks to Rise Visual Effects Studio for reporting.] == Changes in 2012.10-r9 (released 2013-10-29) == New Features: * general: Added support for hardlinked files in same directory. Enhancements: * fhgfs-ondemand: Added option to prefer local storage servers on clients. Improvements: * client: New faster algorithm to generate inode numbers. * fsck: Improved runtime of some checks. * fsck: Report file paths (not only IDs) in log messages when available. General Changes: * client: Updated to be compatible with linux-2.6.16 up to linux-3.12. Fixes: * admon: Fixed GUI-based installation of packages on SLES11. [Thanks to Medizinische Hochschule Hannover for reporting.] * client: Automatically generate 32bit inode numbers for 32bit programs using 32bit readdir() on 64bit systems. [Thanks to University of Oklahoma, Research Campus Computing Center for reporting.] * client: Fixed setgid bit handling of directories. [Thanks to Cambridge MRC Laboratory of Molecular Biology for reporting.] * servers: Fixed potential problem with cleanup of syslog logger on shutdown. * client: Package is now of architecture type "noarch" to allow installation on non-x86/x64 architectures. * meta: Fixed wrong modification event flusher error log message on shutdown. * client: Fixed missing file attributes revalidation after rename in certain cases. * meta: Fixed missing file size attribute update on file close in certain cases. * fsck: Fixed potentially wrong termination in certain error cases. * meta: Use random targets chooser if preferred targets are given by client (relevant for fhgfs-ctl --migrate), because other target choosers do not support preferred targets. == Changes in 2012.10-r8 (released 2013-08-23) == * admon: Added download support for Debian 7 repository. * client: Temporarily disable file create intent optimization due to problems with dangling symlinks. [Thanks to University of Iowa for reporting.] * general: Fixed potential problems during syslog logger initialization. == Changes in 2012.10-r7 (released 2013-07-25) == * fsck: Re-introducing online check (i.e. checking while users are accessing the file system), as known from the former 2011.04 release series. This mode is no longer experimental now. * meta/storage: New option to use per-user message queues for improved fairness in multi-user environments (experimental, see option tuneUsePerUserMsgQueues in /etc/fhgfs/fhgfs-XY.conf). [Thanks to TU Vienna for suggestion during user meeting 2013.] * meta/storage: New option to bind all threads to a certain NUMA CPU zone, e.g. because network adapter and storage controller are also attached to this zone (see option tuneBindToNumaZone in /etc/fhgfs/fhgfs-XY.conf). * meta/storage: Improved responsiveness in stiuations with frequent new connections (splitted former StreamListener into two components). * meta: Improved handling of metadata mirror messages by adding a dedicated new component for processing. * client: Added half-md4 as new inode number hash algorithm (not enabled by default yet). * client: Automatically generate 32bit inode numbers for 32bit programs using 32bit stat() on 64bit systems. * client: Several improvements and fixes for cache type "paged" (still experimental). * opentk-lib: Fixed problem when IB errors were detected during service startup. * packages: Added Debian 7 (Wheezy) packages, removed Debian 5 (Lenny) packages. == Changes in 2012.10-r6 (released 2013-07-05) == * client: Fixed comment in config file. [Thanks to Cambridge MRC Laboratory of Molecular Biology for reporting.] * admon: Shifted average graph in GUI throughput view. * admon: Several minor GUI changes, e.g. colors in storage throughput view. == Changes in 2012.10-r5 (released 2013-07-02) == * client: Fixed and re-enabled kernel NFS export and open_by_handle support. * client: Updated build system for compatibility RHEL 5.9 Infiniband drivers. == Changes in 2012.10-r4 (released 2013-05-23) == * client: Temorarily disabled kernel NFS export and open_by_handle support because of known problems: In some cases, files could have become inaccessible after moving until the file system was remounted. (Regression from r2.) == Changes in 2012.10-r3 (released 2013-05-17) == * client: Fixed internal file mds reference update after file move to another metadata server (regression from r2). * meta: Fixed error return code when unlink() and close() were racing together on the same file. == Changes in 2012.10-r2 (released 2013-05-10) == * client: Updated to be compatible with linux-2.6.16 up to linux-3.9. [Thanks to Taras Shevchenko National University of Kyiv for the preparatory work.] * client: Improved async buffer flushes (don't wait for locked buffers). * client: Always use 32bit inode numbers on 32bit systems. [Thanks to Dattatec for reporting.] * client: Updated comments regarding standard OFED kernel include path for new OFED releases (/usr/src/openib). * client: Added session check to detect server cache loss with read/write/fsync messages. (Only enabled if all servers run at least version 2012.10-r2.) * client: New config file option "sysSyncOnClose" to force buffer flush when a file is closed. * client: New config file option "sysSessionCheckOnClose" to perform an extra session check when a file is closed. * client: Fixed "CPU stuck for more than n seconds" kernel log message while a storage server was unresponsive. * client: Added support for open_by_handle syscall and kernel NFS server export (experimental, requires linux-2.6.29 or higher). * client: Init script updated to detect all currently active fhgfs mounts. * client: Fixed return code (EFAULT) when an invalid userspace buffer is passed to read()/write() in buffered mode. * client: Fixed file lookup+create problem on RHEL 5.x kernels. [Thanks to INRIA for reporting.] * client: Removed "dynamic" inode number style option from config. * client_opentk: Fixed Infiniband compile with RHEL 5.9 kernel (2.6.18-348). [Thanks to Clustervision for reporting.] * meta: Fixed a rare case where a warning was generated about a directory not being properly referenced. * meta: Several minor internal locking fixes/improvements. * storage: Fixed problem where benchmark files from fhgfs-ctl mode "storagebench" were not correctly removed from an underlying XFS file system. * storage: Fixed problem where fhgfs-ctl mode "clientstats" sometimes showed wrong values for read/write throughput. * fhgfs-ctl: Mirror targets now included in search of modes "find" and "migrate". * fhgfs-ctl: Fixed continue after certain errors in mode "migrate". [Thanks to University of California, Irvine for reporting.] * fhgfs-ctl: Mode "clientstats" now requires fewer network messages to query statistics. (Only if servers run at least version 2012.10-r2.) * fhgfs-ctl: Improved mode "getentryinfo" output for mirrored files. * fsck: Several minor fixes/improvements. * opentk_lib: Log Infiniband errors to syslog. * general: Fixed Debian binaries reporting wrong version numbers. == Changes in 2012.10-r1 (released 2013-01-24) == This is a major update with new features and improvements in all areas. If you are upgrading from a previous 2011.04 release, make sure to read the compatibility section below. === New & Noteworthy === * New metadata format with inlined inodes: The new format allows a more efficient internal handling and scales better to large numbers of files. The inode inlining also reduces disk access/seeks and required disk space. * Changes in metadata handling also include more general optimizations, e.g. internal locking optimizations that allow for higher parallelism, especially for modifying metadata operation types like file creation. * Intents for metadata operations are now enabled by default. * New optional metadata mirroring * Asynchronous mirroring of metadata can be enabled/disabled on a per-directory basis, so not all directories have to be mirrored. * The mirroring setting will be derived by new subdirs. * Mirror servers are chosen individually for each new directory, so there are no passive (i.e. mirror-only) servers in such a file system. * See "fhgfs-ctl --mirrormd --help" and http://www.fhgfs.com/wiki/AboutMirroring for more information. * New optional file contents mirroring * Synchronous mirroring of file contents can be enabled/disabled on a per-directory basis, so not all files have to be mirrored. * The mirroring setting will be derived by new subdirs. * Mirror targets are chosen individually for each new file, so there are no passive (i.e. mirror-only) targets in such a file system. * Mirroring also can be used with an odd number of targets. * See "fhgfs-ctl --setpattern --raid10 --help" and http://www.fhgfs.com/wiki/AboutMirroring for more information. * New numeric 16-bit server and target IDs: The new release internally uses and stores numeric 16-bit IDs (range 1..65535) instead of the previous string IDs from the 2011.04 series. * The new IDs significantly reduce metadata size. * The former string IDs still exist to provide human-readable names for servers and targets (e.g. in log messages). * As before, all IDs can be assigned automatically or manually. * Automatically assigned IDs are randomly chosen by fhgfs-mgmtd when a new server/target is registered. * As with previous string IDs, numeric IDs can be set manually by creating a file named "targetNumID" or "nodeNumID" in a storage directory. See here for more information on manual assignment: http://www.fhgfs.com/wiki/FAQ#force_nodeid * Redesigned file system check: The new fhgfs-fsck tool significantly reduces server-load and runtime for a full check by walking the file system only once and doing the analysis steps afterwards on the client which ran the tool. * Currently, clients may not write/modify the file system while the check is running. Online check/repair will be available in an upcoming minor release. * New fhgfs-ondemand script (/opt/fhgfs/sbin) allows very simple creation and destruction of a new FhGFS instance, e.g. on a per-job basis for clusters * The script takes a nodefile as argument and is comparable to running an MPI job. * On a per-job basis for clusters, the script would usually be called by the batch system. * New built-in benchmarking for disk throughput and network throughput: * The client network benchmark mode sends data over the network, but does not perform disk reads/writes on storage servers. * Network benchmarking is activated per-client by using: "echo 1 > /proc/fs/fhgfs//netbench_mode" * The storage targets benchmark mode measures server-side throughput of storage targets without any network data transfer. * See "fhgfs-ctl --storagebench --help" for more information. * New storage server config options were added to control explicit read-ahead and write flushing for sequential IO * See tuneFileReadAhead... and tuneFileWriteSyncSize in fhgfs-storage.conf. * Client logging can now optionally be done via syslog instead of via the fhgfs-helperd daemon. * See option "logType" in fhgfs-client.conf. * Client log level can now be changed at runtime * See /proc/fs/fhgfs//log_levels. * New client/server config option allows setting of Infiniband type-of-service to control quality-of-service (and other options) via opensm config * See config file option "connRDMATypeOfService". * New server config option allows binding of worker threads to NUMA areas * See option "tuneWorkerNumaAffinity" in server config files. * Command line parameters for the fhgfs-ctl tool are now specified in more the commonly known "--param" format with dashes. === Compatibility Notes === * The on-disk storage format has changed with this release series. Upgrading from a previous 2011.04 FhGFS version requires a data conversion. See here for upgrade instructions: http://wiki.fhgfs.com/ServerUpgrade201210 * Supported distributions: RedHat 5/6 (and Fedora), SLES 10/11 (and OpenSuse), Debian 5/6 (and Ubuntu) * Supported Linux kernel versions (client): 2.6.16 - 3.5 * Servers and clients from different major release series are not compatible and cannot be used together in the same file system instance. * Servers and clients with different minor versions of the 2012.10 release series are compatible and can be used together in the same file system instance.