$ show ozontech/file.d
file.d is a high‑throughput log and event processing daemon written in Go, designed for production data pipelines. It reads events from various sources (files, Kafka, Kubernetes, HTTP, syslog, sockets, journalctl), runs them through a configurable chain of action plugins (masking, throttling, transformation, decoding, joining, etc.), and delivers them to multiple outputs (Elasticsearch, ClickHouse, Loki, S3, Kafka, Splunk, Postgres, GELF, file, stdout). The architecture is plugin‑based: inputs, actions, and outputs register into a central registry, and pipelines are defined via YAML configuration. The system provides at‑least‑once delivery semantics, stream‑based ordering, batching with backoff, and dead‑queue failover. The codebase demonstrates strong engineering practices: extensive unit and integration tests (including e2e scenarios with external services), performance optimizations (zero‑copy JSON via insane‑json, event pooling, memory limits), and a rich set of reusable internal libraries (HTTP client with circuit breaker, metric controllers, match rules, configuration parsing). It includes a Helm chart for Kubernetes deployment, Prometheus monitoring with a Grafana dashboard, and a playground web service for interactively testing pipelines. The transform action includes a custom expression language with a Pratt parser and compiler. While the core is solid, there are some maintenance risks: many plugins contain global caches or mutable state, several code paths call `logger.Fatal` on misconfiguration (making the whole process exit), and the file input offset persistence is hand‑rolled. The large number of input/output/action plugins (~40 total) increases surface area for drift, and some e2e tests depend on external Docker services. Overall, file.d is a mature and capable data pipeline framework, well‑suited for log aggregation and transformation in containerized environments.
The LLM identified 4 files worth highlighting for reuse:
A generic HTTP client built on fasthttp with built‑in circuit breaker, gzip compression, TLS configuration, custom headers, and endpoint failover. It is used by multiple output plugins (Splunk, Elasticsearch, Loki, etc.) and can be extracted into a standalone library. The circuit breaker supports ban periods, health checks, and automatic recovery.
Provides `Batcher` and `RetriableBatcher` that combine batch size and flush timeout with exponential backoff and dead‑queue integration. This pattern is reused across file and socket outputs and can be applied to any batching task requiring reliable delivery with retries.
A lightweight, zero‑allocation string matching engine supporting prefix, suffix, contains, and equivalent checks with case‑insensitivity and inversion. It is used in the pipeline’s match conditions and can be reused wherever fast, configurable field matching is needed.
Centralized Prometheus metric registration and caching (`CounterVec`, `GaugeVec`, `HistogramVec`) with TTL‑based cleanup and hash‑based fast lookup. It abstracts the metric system for all plugins, enabling consistent naming and collection, and can be extracted as a general‑purpose metric helper for Go projects.