$ show martymac/fpart
Fpart is a high-performance C CLI tool that partitions filesystem trees into balanced subsets (called partitions) based on file count, size, or a fixed number of partitions. It uses a bin-packing algorithm to optimize space utilization. The tool supports live mode, where partitions are generated during filesystem crawling and hooks can trigger immediate actions (e.g., starting rsync transfers). Fpsync, a companion shell script, leverages fpart to parallelize data migrations using rsync, cpio, pax, or tar, with support for SSH workers, job queues, resume/replay, and email reports. The codebase is mature, well-structured, and portable across FreeBSD, Linux, Solaris, and macOS. It demonstrates strong C craftsmanship with careful memory management, error handling, and portability workarounds (e.g., embedded fts). The fpsync script is a sophisticated example of shell-based job scheduling and distributed synchronization.
The LLM identified 4 files worth highlighting for reuse:
Implements the core bin-packing algorithms for distributing file entries into partitions. Functions like dispatch_file_entry_p_by_size (largest-first greedy), dispatch_empty_file_entries (spreads zero-size files evenly), and dispatch_file_entries_by_limits (dynamic partition creation with size/count limits) are self-contained and could be extracted for any item-grouping problem. They only depend on types.h and partition.h, making them reusable in other C projects needing similar load-balancing logic.
Provides a double-linked list implementation for partition management, including add_partitions, remove_partition, find_smallest_partition_index, and get_partition_at. These functions are generic and could be reused in any project requiring a dynamic list of weighted buckets. The code is clean, uses assertions, and handles edge cases like empty partitions.
Contains a collection of reusable utility functions: str_to_uintmax (with human-friendly size multipliers), parent_path (robust path parent extraction), clone_env/push_env (safe environment manipulation for hooks), and valid_file (pattern matching with fnmatch). These functions are well-tested (see tests/test-parent_path.c) and could be extracted for use in other C programs needing similar string or path operations.
A sophisticated shell script that orchestrates parallel data migrations using fpart and external copy tools. It implements a disk-based job queue, manages concurrent workers (local or over SSH), supports resume/replay of runs, and provides real-time status via SIGINFO. The script demonstrates advanced shell patterns: FIFO-based logging, process group management, trap handlers, and robust error handling. It could serve as a reference for building similar distributed job schedulers in shell.