$ show liweiyi88/onedump
Onedump is a Go-based CLI tool for database administration, specializing in backup, restore, and migration workflows. It supports MySQL (with a zero-dependency native dumper and mysqldump) and PostgreSQL (via pg_dump), and can persist backups to local files, AWS S3, Google Drive, Dropbox, or SFTP endpoints. The architecture follows a clean pipeline pattern: YAML configuration drives concurrent job execution, each job uses a dumper interface to stream database content through gzip compression and fan-out to multiple storage backends. Beyond core backup, it includes advanced features: MySQL binlog streaming to S3 with checksum-based deduplication, point-in-time restore via binlog replay, a MySQL slow query log parser with query masking and deduplication, and a resumable/concurrent SFTP file transfer utility with exponential backoff. The codebase is well-structured, heavily tested, and uses modern Go idioms, making it suitable as a reference for building database-adjacent infrastructure tooling.
The LLM identified 5 files worth highlighting for reuse:
Provides a generic file synchronization helper that computes SHA-256 checksums and persists them to a state file to avoid re-transferring already-synced files. The `SyncFile` method wraps an arbitrary sync function with optional deduplication and state persistence. It is fully self-contained and can be extracted and reused in any Go project requiring idempotent file transfers.
Handles SSH client creation with flexible key input: supports raw PEM strings, base64-encoded keys, or filesystem paths. The `CreateSshClient` method returns a standard `*ssh.Client`. This component is small, isolated, and trivial to reuse in any Go project needing SSH connectivity, avoiding boilerplate key parsing logic.
Implements resumable SFTP file uploads with a progress bar, automatic retry with exponential backoff (configurable max attempts), and optional checksum-based deduplication. The `Save` method tracks bytes written and can resume from an offset, making it robust for large file transfers. This component demonstrates a solid pattern for building fault-tolerant network file operations and is easily extractable.
A dedicated MySQL slow query log parser that handles multi-line queries, extracts extensive performance metrics (query_time, lock_time, rows examined, etc.), deduplicates by query text keeping the slowest occurrence, and supports optional value masking via regex replacements. It reads log files using bufio.Reader to handle long lines gracefully. This component is self-contained (only standard library) and can be extracted as a library for parsing MySQL slow logs.
Handles MySQL binlog-based point-in-time recovery by creating a restore plan from sorted binlog files, optionally stopping at a given datetime. It supports batch execution of `mysqlbinlog` commands to avoid exceeding OS argument size limits, and pipes output to `mysql` for applying events. The `ParseBinlogFilePosition` utility extracts the binlog file and position from a mysqldump dump file. This component is a well-designed reusable pattern for any tool needing to orchestrate external commands for database recovery.