$ show confluentinc/confluent-kafka-javascript
This repository is Confluent's official JavaScript client for Apache Kafka, wrapping the high-performance C library librdkafka via native Node.js addons. It provides two API surfaces: a legacy callback-based API (node-rdkafka) and a modern promisified API compatible with the popular KafkaJS library. The client supports all core Kafka operations: producing and consuming messages (including transactional and exactly-once semantics), admin operations (topic management, consumer groups, offsets), and stream-based interfaces. Additionally, it includes a full Schema Registry client (TypeScript) with serializers/deserializers for Avro, JSON Schema, and Protobuf, along with a rule execution engine supporting CEL (Common Expression Language), JSONata, and field-level encryption via pluggable KMS drivers (AWS, GCP, Azure, HashiCorp Vault, local). The architecture is layered: a C++ binding layer handles the Kafka protocol, while pure JavaScript/TypeScript implements schema management, rule evaluation, and encryption. Key entry points are `lib/index.js` (node-rdkafka), `lib/kafkajs/index.js` (KafkaJS), and `schemaregistry/index.ts` (Schema Registry). External dependencies include librdkafka, protobuf, avsc, axios, and cloud KMS SDKs. The codebase is production-grade with extensive unit, integration, and e2e tests, performance benchmarks, and CI/CD pipelines across multiple platforms.
The LLM identified 4 files worth highlighting for reuse:
Implements exact decimal arithmetic (add, sub, mul, div, sqrt, mod, round, trunc) for CEL rule execution, matching Java BigDecimal semantics with 38-digit HALF_UP rounding. Uses decimal.js with widened precision (1e9) for exact operations. This module is self-contained (depends only on decimal.js and protobuf types) and can be extracted for any CEL-based system requiring high-precision decimal operations.
Provides a binary codec for the Confluent Variant type (a self-describing format for semi-structured data, similar to Spark/Parquet Variant). Includes Variant (reader) and VariantBuilder (writer) classes with JSON parsing/serialization. Handles edge cases like non-finite numbers, BigInt, and precision loss. This module is isolated and reusable for any application needing a compact, self-describing binary format for dynamic data.
Implements a DEK (Data Encryption Key) registry client with LRU caches for KEKs and DEKs, thread-safe mutexes, and fallback to older API versions. Manages key lifecycle (retrieve, create, cache, rotate) and delegates to KMS drivers. This module is self-contained and can be reused in any encryption system requiring a key registry with caching and concurrency control.
Provides generic async concurrency primitives: a reentrant Lock (readers-writer lock), DeferredPromise (promise with external resolve/reject), and Timer (setTimeout wrapper). These utilities are used internally by the KafkaJS consumer and producer but are completely generic and can be extracted for any Node.js project needing fine-grained async control.