$ show go-graphite/carbon-clickhouse
carbon-clickhouse is a production-grade Graphite metrics receiver and storage gateway written in Go. It ingests time-series data over multiple protocols (plain text TCP, UDP, Pickle, gRPC, Prometheus Remote Write, Telegraf HTTP JSON) and persists it to ClickHouse using the efficient RowBinary format. The system is architecturally split into receivers, a writer that rotates and compresses data files, and uploaders that batch-insert data into ClickHouse tables. Receivers handle protocol-specific parsing and drop rules; the writer manages chunking, interval-based rotation, and optional LZ4 compression; uploaders support points, index (hierarchical metric trees), tagged metrics (sorting, dedup, and URL encoding), and points-reverse storage. External dependencies include ClickHouse (HTTP interface), Prometheus protobuf definitions, gRPC, and common Go libraries. The codebase is well-tested, includes end-to-end integration tests with actual ClickHouse containers, and is actively maintained with CI/CD pipelines producing DEB/RPM packages.
The LLM identified 5 files worth highlighting for reuse:
Implements a fixed-size WriteBuffer (524288 bytes) with sync.Pool for zero-allocation serialization of Graphite points in ClickHouse RowBinary format. Provides methods for writing varint-prefixed bytes, floats, uint16/32/64, reverse path, and tagged metrics. The buffer tracks usage and supports optional confirmation via WaitGroup. This component is fully self-contained and reusable in any Go system that needs to produce RowBinary data, and demonstrates careful memory management and performance tuning.
Implements a two-way tree-based blacklist matcher for metric name filtering. Patterns (with wildcard '*' support) are grouped by path length and stored in forward and reverse trees. The Contains method searches both exact and wildcard branches recursively, enabling O(depth) matching. This is a clean, reusable solution for any application needing efficient, pattern-based filtering of dot-separated hierarchical names.
A sharded concurrent map (CMap) for high-concurrency read/write access. Distributes keys across 1024 shards via FNV-32 hashing, each protected by a sync.RWMutex. Provides Exists, Add, Merge, Count, Clear, and Expire (TTL-based eviction with configurable worker). This is a textbook implementation of a scalable concurrent cache, easily extractable for any Go project requiring thread-safe key-value storage with optional expiration.
Parses Graphite tagged metric names (e.g., 'metric;key=value;key2=value2'), sorts tags by key, deduplicates, and encodes them into the URL-escaped format used by Carbon (e.g., 'metric?key=value&key2=value2'). Also supports converting plain metrics to tagged format using configurable templates (InfluxDB-like). The buffered variant (GraphiteBuffered) reuses memory to reduce allocations. This is a self-contained, well-optimized tag processing utility reusable in any Graphite-compatible metrics pipeline.
Provides a reusable lifecycle management struct with Start/Stop, context-based cancellation, and WaitGroup for goroutine tracking. The Go method launches goroutines that respect the context, and WithCtx provides context to synchronous callers. StartFunc and StopFunc allow custom initialization and cleanup. This is a clean, idiomatic pattern for managing concurrent workers in any Go service.