$ show confluentinc/librdkafka
librdkafka is a high-performance, production-grade C (and C++) client library for Apache Kafka, developed and maintained by Confluent. It provides full protocol-level support for producing and consuming messages, including advanced features such as exactly-once semantics (idempotent producer, transactions), consumer groups with multiple rebalance protocols (classic and KIP-848), share consumers (KIP-932), and a comprehensive Admin API for cluster management (topics, configs, ACLs, SCRAM credentials). The library is designed for embedding into applications across many languages via bindings (Python, Go, .NET, Rust, etc.) and supports a wide range of authentication mechanisms (SSL, SASL/PLAIN, SCRAM, OAUTHBEARER, GSSAPI) and compression codecs (zstd, LZ4, snappy, gzip). It includes a robust mock cluster for integration testing, a highly configurable configuration system, and extensive unit and integration tests covering hundreds of scenarios. The architecture separates protocol handling, broker transport, partition assignment, and admin operations into modular internal components, while exposing a clean, callback-driven C API and an object-oriented C++ wrapper. The codebase is actively maintained with frequent releases addressing security fixes (CVE tracking), Kafka protocol evolution (e.g., KIP-848, KIP-932), and platform compatibility (Linux, macOS, Windows, s390x). Vendored dependencies like lz4, snappy, cJSON, and nanopb are periodically updated, but some platform-specific #ifdefs and legacy protocol support contribute to moderate technical drift.
The LLM identified 5 files worth highlighting for reuse:
Provides a general-purpose, segment-based byte buffer (rd_buf_t) with write/read operations, CRC32/CRC32C computation, and iovec integration. It abstracts memory allocation and supports slicing (rd_slice_t) for zero-copy access. This component is self-contained, has no internal librdkafka dependencies beyond basic types (rd.h, rdendian.h), and can be extracted into a standalone utility library for any project needing efficient, flexible byte buffering with optional checksums.
Implements a generic, type-agnostic dynamic list (rd_list_t) with operations for append, prepend, sort (qsort), binary search, deduplication, and shallow/deep copy. It uses a flexible element size and optional destroy callback, making it suitable for managing collections of any type. The implementation is compact and has no external dependencies beyond rd.h. It can be reused in C projects needing a lightweight, generic list with sorting support.
Implements a simulated Kafka cluster (rd_kafka_mock_cluster_t) that handles the full Kafka protocol (metadata, produce, fetch, join group, offset commit, etc.) in process, without external broker processes. It allows injection of errors, configuration of partition leaders/replicas, and simulation of network delays. This is an exceptional testing component that can be extracted into a separate library for testing any Kafka client in-memory. It demonstrates deep knowledge of the Kafka protocol and provides deterministic, high-speed testing.
A portable C11 threads emulation library that provides mutexes, condition variables, thread creation, thread-local storage, and once-flags across POSIX and Windows platforms. It is a standalone, well-tested, and widely used component (originally from TinyCThread project). It can be extracted and used in any C project needing cross-platform threading without depending on the full librdkafka library.
Implements the HdrHistogram data structure for recording and analyzing latencies with configurable precision and auto-resizing. It provides quantile calculations (e.g., p50, p99, p99.9) and is used internally for librdkafka's statistics. The implementation is self-contained (depends only on stdlib) and can be extracted as a reusable histogram library for any C project needing high-dynamic-range latency recording.