$ show influxdata/influxdb
InfluxDB 3 Core (IOx) is a production-grade, open-source time-series database written in Rust. It ingests data via line protocol (HTTP/gRPC), stores it in Apache Parquet on object storage (S3, Azure, GCP, local), and serves queries via SQL (DataFusion), InfluxQL, and FlightSQL. The architecture is a single binary (`influxdb3`) that combines an HTTP server (port 8181), gRPC FlightSQL endpoint, CLI, and an embedded Python processing engine for plugin-based triggers. The system is designed for low-latency last-value queries, high-throughput writes, and efficient columnar storage with catalog persistence as versioned binary logs and snapshots on object store. The codebase is a large Rust workspace (~40+ crates) with a clear separation of concerns: core libraries (`core/`) for query planning, Arrow utilities, authorization, and metrics; server crates (`influxdb3_*`) for catalog management, write buffering, processing engine, and system tables. Key entry points include `influxdb3/src/main.rs` (CLI), `influxdb3_server/src/unified_service/` (HTTP/gRPC routing), and `influxdb3_catalog/src/` (catalog operations). The query engine is built on a forked DataFusion with custom physical optimizers (gapfill, dedup, sort, union) and InfluxQL planning. External dependencies include object store backends, protobuf for gRPC, and Python (via PyO3) for plugins. The repository demonstrates high craftsmanship with several reusable patterns: a versioned binary record format for catalog persistence with forward-compatibility guarantees, a hook-based cache system with S3-FIFO eviction, a generic copy-on-write repository pattern, and a run-optimized merge sort for streaming sorted data. However, the large workspace, forked DataFusion pinned to a commit, and multiple catalog versions (v1–v4) introduce maintenance drift risks. The codebase is actively maintained with extensive tests, benchmarks, and CI/CD.
The LLM identified 5 files worth highlighting for reuse:
Implements `DisplaySourceChain`, a wrapper that walks an error's source chain and renders each message only once, avoiding duplication from third-party error types. It handles overlapping messages by splitting on known substrings. This is a reusable utility for any Rust project needing better error display, especially when integrating multiple error libraries.
Defines a versioned binary record format for catalog persistence. Records have a 16-byte header (CRC, version, type ID) and variable-length data. The module includes a registry of record types, roundtrip tests (`assert_roundtrip!`), and shape fingerprinting to ensure wire stability. This pattern is reusable for any system needing append-only, forward-compatible binary logs.
Implements a cache system with in-use tracking and early-access data for in-flight loads. Uses a hook chain (level triggers, limits, observers) and supports S3-FIFO eviction. The design avoids `strong_count` for in-use checks (relaxed ordering) and provides metrics hooks. This is a sophisticated caching pattern reusable for any object store or data caching layer.
Implements a run-optimized merge sort for streaming sorted data. It uses a builder pattern to construct merge streams, caches sort keys, and provides fallback comparators. The algorithm is designed for merging multiple sorted parquet files efficiently. This is a non-obvious, reusable pattern for any system needing sorted merge of columnar data.
Provides the `catalog_record` procedural macro that enforces wire-stable record shapes. It generates shape fingerprints (FNV-1a hash of field types), bitcode/serde encoding, and inventory registration. The macro ensures that once a record type is shipped, its serialized form cannot change without a version bump. This is a high-craftsmanship pattern for any system requiring binary backward compatibility.