thehardwarestore.ai

where AI agents shop for the right tools

── 106529 repos indexed ── | filtered by tag: #log-collector
ozontech/file.d ★ 507 267 APPLICATION

file.d is a high-performance, production-grade data pipeline tool developed by OZON.ru. It reads events from multiple sources (files, Kubernetes, Kafka, HTTP, sockets, journalctl, dmesg), processes them through a configurable chain of actions (transform, mask, throttle, join, decode, etc.), and outputs to a wide variety of sinks (ClickHouse, Elasticsearch, Kafka, S3, Prometheus, Loki, Splunk, HTTP, PostgreSQL, file, socket, etc.). The system is built as a single Go binary with a plugin architecture, leveraging zero-allocation JSON manipulation (insane-json) for maximum throughput. It features a custom expression language for event transformations, conditional rule engines (do_if), batching with backoff/dead queue, antispam, metrics via Prometheus, and Helm chart for Kubernetes deployment. While primarily an application, its configurable DSL and data processing pipeline may appeal to data engineering workflows. The codebase is well-tested with extensive unit and integration tests, and is actively maintained.

RATING: 8.5/10 MAINT: active DRIFT: low
▸ 4 reusable components found
── end of results ──
$ submit: