confluent-kafka-python is a production-grade Apache Kafka client library that wraps librdkafka via a C extension. It is not an application with user-facing endpoints; it exposes Python APIs for producing and consuming messages, admin operations, Schema Registry integration, and asyncio variants. Core entry points are confluent_kafka.Producer, confluent_kafka.Consumer, confluent_kafka.AdminClient, confluent_kafka.ShareConsumer, and confluent_kafka.aio.AIOProducer/AIOConsumer. Low-level broker communication happens in C, while the Python layer provides serialization, schema evolution, rules, and callback management.
This is a Rust library that provides safe, idiomatic bindings over the C library librdkafka, enabling Rust applications to produce and consume messages from Apache Kafka. The architecture is a thin wrapper: it exposes a high-level typed API (producers, consumers, admin client, transactions) while delegating all network I/O, partition management, and protocol handling to librdkafka. Key entry points include `ClientConfig` for building clients, the `Consumer` and `Producer` traits, and concrete implementations (`BaseConsumer`, `StreamConsumer`, `BaseProducer`, `FutureProducer`, `ThreadedProducer`). Data flows through a polling loop that dispatches events (delivery reports, rebalance notifications, log messages) to user-provided `ClientContext` callbacks. External dependencies are the librdkafka native library (bundled via submodule or system) and optional async runtimes (Tokio, smol, async-std). The library is heavily used in production Rust Kafka deployments.
This repository is Confluent's official JavaScript client for Apache Kafka, wrapping the high-performance C library librdkafka via native Node.js addons. It provides two API surfaces: a legacy callback-based API (node-rdkafka) and a modern promisified API compatible with the popular KafkaJS library. The client supports all core Kafka operations: producing and consuming messages (including transactional and exactly-once semantics), admin operations (topic management, consumer groups, offsets), and stream-based interfaces. Additionally, it includes a full Schema Registry client (TypeScript) with serializers/deserializers for Avro, JSON Schema, and Protobuf, along with a rule execution engine supporting CEL (Common Expression Language), JSONata, and field-level encryption via pluggable KMS drivers (AWS, GCP, Azure, HashiCorp Vault, local). The architecture is layered: a C++ binding layer handles the Kafka protocol, while pure JavaScript/TypeScript implements schema management, rule evaluation, and encryption. Key entry points are `lib/index.js` (node-rdkafka), `lib/kafkajs/index.js` (KafkaJS), and `schemaregistry/index.ts` (Schema Registry). External dependencies include librdkafka, protobuf, avsc, axios, and cloud KMS SDKs. The codebase is production-grade with extensive unit, integration, and e2e tests, performance benchmarks, and CI/CD pipelines across multiple platforms.
librdkafka is a high-performance, production-grade C (and C++) client library for Apache Kafka, developed and maintained by Confluent. It provides full protocol-level support for producing and consuming messages, including advanced features such as exactly-once semantics (idempotent producer, transactions), consumer groups with multiple rebalance protocols (classic and KIP-848), share consumers (KIP-932), and a comprehensive Admin API for cluster management (topics, configs, ACLs, SCRAM credentials). The library is designed for embedding into applications across many languages via bindings (Python, Go, .NET, Rust, etc.) and supports a wide range of authentication mechanisms (SSL, SASL/PLAIN, SCRAM, OAUTHBEARER, GSSAPI) and compression codecs (zstd, LZ4, snappy, gzip). It includes a robust mock cluster for integration testing, a highly configurable configuration system, and extensive unit and integration tests covering hundreds of scenarios. The architecture separates protocol handling, broker transport, partition assignment, and admin operations into modular internal components, while exposing a clean, callback-driven C API and an object-oriented C++ wrapper. The codebase is actively maintained with frequent releases addressing security fixes (CVE tracking), Kafka protocol evolution (e.g., KIP-848, KIP-932), and platform compatibility (Linux, macOS, Windows, s390x). Vendored dependencies like lz4, snappy, cJSON, and nanopb are periodically updated, but some platform-specific #ifdefs and legacy protocol support contribute to moderate technical drift.