$ show DataDog/kafka-kit
This repository is a suite of Kafka cluster management tools developed by DataDog. It provides four primary tools: `topicmappr` for partition reassignment and rebalancing with storage-aware placement, `registry` for a gRPC/HTTP API that exposes Kafka metadata and custom tagging, `autothrottle` for dynamically adjusting replication throttles based on Datadog network metrics, and `metricsfetcher` for pulling partition size and broker storage metrics from Datadog into ZooKeeper. The architecture is a collection of independent CLI binaries and a gRPC server, all sharing well-factored Go libraries (`kafkaadmin`, `kafkazk`, `mapper`) that abstract Kafka admin operations, ZooKeeper interactions, and partition mapping logic. The codebase is production-grade, with extensive unit and integration tests, Docker-based development environments, and CI pipelines. It is tightly coupled to Datadog for metrics and ZooKeeper for metadata, which introduces some drift risk as Kafka evolves away from ZooKeeper. The last commit date (2026-09-16) appears anomalous and may indicate a timestamp issue, but the repository shows active maintenance with recent CI configurations and issue triage.
The LLM identified 4 files worth highlighting for reuse:
The `mapper` package is a self-contained library for computing optimal Kafka partition placements. It implements two rebuild strategies (count-based and storage-based), supports rack-aware constraints, substitution affinities, and leader-follower optimization. The core types (`PartitionMap`, `BrokerMap`, `Constraints`) are cleanly abstracted and could be extracted for use in any Kafka partition reassignment tool. The package includes thorough unit tests and test data, demonstrating non-trivial algorithmic skill.
`kafkaadmin` provides a clean Go interface (`KafkaAdmin`) for common Kafka administrative operations: creating/deleting topics, describing brokers and topics, setting and removing throttles, and managing configurations. It includes a stub implementation for unit testing and integration tests against a real Kafka cluster. This abstraction is reusable across any Go project that needs to interact with Kafka's admin API, and its design pattern (interface + stub + integration test) is a model for testable infrastructure code.
`kafkazk` abstracts ZooKeeper interactions for Kafka metadata, including reading/writing topic states, reassignment data, broker metrics, and configs. It provides a `Handler` interface with a real `ZKHandler` implementation and a stub for unit tests. The package handles JSON serialization, gzip compression, and ZooKeeper node management. This is a reusable pattern for any Go service that needs to read or write Kafka metadata in ZooKeeper, and its design decouples the application from direct ZooKeeper API calls.
The `cluster/zookeeper` package implements a distributed lock using ZooKeeper sequential ephemeral nodes with TTL support. It provides a fair queuing mechanism and is used internally by the registry server for coordinating tag cleanup. The implementation is self-contained, well-tested (including integration tests), and demonstrates a robust pattern for distributed coordination that can be reused in any Go application requiring ZooKeeper-based locking.