$ show fede1024/rust-rdkafka
This is a Rust library that provides safe, idiomatic bindings over the C library librdkafka, enabling Rust applications to produce and consume messages from Apache Kafka. The architecture is a thin wrapper: it exposes a high-level typed API (producers, consumers, admin client, transactions) while delegating all network I/O, partition management, and protocol handling to librdkafka. Key entry points include `ClientConfig` for building clients, the `Consumer` and `Producer` traits, and concrete implementations (`BaseConsumer`, `StreamConsumer`, `BaseProducer`, `FutureProducer`, `ThreadedProducer`). Data flows through a polling loop that dispatches events (delivery reports, rebalance notifications, log messages) to user-provided `ClientContext` callbacks. External dependencies are the librdkafka native library (bundled via submodule or system) and optional async runtimes (Tokio, smol, async-std). The library is heavily used in production Rust Kafka deployments.
The LLM identified 4 files worth highlighting for reuse:
The `AsyncRuntime` trait and its implementations (Tokio, smol, async-std, naive) define a pluggable interface for async operations (spawn, delay, sleep). This is a clean abstraction that allows the library to support multiple async runtimes without coupling to any single one. The trait is isolated and could be extracted into a standalone crate for other libraries needing runtime-agnostic async support.
The `Statistics` struct and its sub-structs (`Broker`, `Topic`, `Partition`, `ConsumerGroup`, etc.) model the detailed metrics emitted by librdkafka's statistics callback. They derive `Serialize`, `Deserialize`, and `Debug`, making them ready for logging, monitoring, or export to time-series databases. This is a self-contained data model that could be reused in any Rust project that needs to ingest librdkafka statistics.
The `NativePtr<T>` wrapper and `KafkaDrop` trait provide a safe RAII pattern for managing pointers to C objects allocated by librdkafka. The wrapper ensures the C destructor is called when the Rust value goes out of scope, preventing memory leaks. This is a reusable pattern for any Rust crate that needs to wrap C objects with non-trivial lifetimes.
The `WakerSlab` data structure (likely in this file) implements a compact, efficient wake-up mechanism for `StreamConsumer`'s async interface. It stores a slab of registered wakers and wakes them all on new events, avoiding per-subscription allocations. This pattern is reusable for any Rust async stream that needs to notify multiple waiters efficiently.