$ show hpc/mpifileutils
mpiFileUtils is a suite of MPI-based command-line tools for managing large datasets on parallel file systems (Lustre, DAOS, GPFS, NFS). It provides parallel equivalents of common file operations: dcp (copy), dsync (sync), drm (remove), dcmp (compare), dtar (archive), dfind (find), dchmod (chmod), dbcast (broadcast), ddup (dedup), and others. The tools are built atop a common library (libmfu) that implements a distributed file list (mfu_flist) with collective operations, path manipulation, I/O wrappers with retry logic, progress reporting, and work distribution via libcircle. The architecture follows a walk->filter->act pattern where the file system is traversed in parallel, metadata is aggregated into a distributed list, and work is distributed across MPI ranks. The tools are designed for high-throughput operations on millions of files and are used in HPC environments. While the codebase shows signs of legacy (dcp1, Python 2 tests), it is actively maintained and includes advanced features like DAOS object-level copy and tar indexing for parallel extraction. The repository is primarily an application (CLI tool suite) with a strong library component (libmfu) that could be reused in other HPC data-movement projects.
The LLM identified 4 files worth highlighting for reuse:
A self-contained C library for manipulating file paths as a double-linked list of components. Provides operations like split, join, reduce (resolve . and ..), relative path computation, and comparison. It is a clean, dependency-light module that can be extracted into any C project dealing with path normalization or hierarchy traversal.
A generic AVL-tree-based string-to-string map with set, get, unset, merge, iteration, and pack/unpack operations. Self-balancing and fully contained, it is ideal for any application needing an ordered string map with O(log n) operations. The implementation is portable and demonstrates solid algorithmic craftsmanship.
A collection of low-level utilities: memory allocation wrappers (MFU_MALLOC, MFU_CALLOC) that abort on failure, byte/format conversion (mfu_abtoull, mfu_format_bytes), a Jenkins hash function, and mount info parsing. These are simple, well-tested building blocks that can be reused across C projects for robust error handling and common tasks.
I/O wrappers (mfu_open, mfu_read, mfu_write, mfu_stat, etc.) that unify POSIX and DAOS/DFS operations with automatic retry on EINTR/EIO and support for O_DIRECT and progress reporting. The abstraction layer is well-isolated and demonstrates a robust pattern for handling unreliable or high-latency file systems, making it reusable for other parallel I/O tools.