$ show Logflare/logflare
Logflare is a production-grade, multi-tenant log management platform (now part of Supabase) built with Elixir/Phoenix. It ingests logs via HTTP, gRPC (OTLP), WebSockets, and platform-specific drains, processes them through a rule engine, and routes them to user-configured backends including BigQuery, ClickHouse, Postgres, S3, and many more. The architecture is a monolithic Elixir application with internal subsystems: an ETS-backed ingest queue for high troughput, Broadway pipelines for batched ingestion, a behaviour-driven adaptor system for pluggable storage backends, a custom query languagte (LQL) with a parser and dialect transformers, a distributed cache layer via Cachex and WAL busting, and Rust NIFs for performance-critial mapping and Arrow IPC. It includes a full web UI (Phoenix LiveView), a REST API (OpenAPI-documented), billing with Stripe, team management, and alerting. The codebase is largte (1700+ files) and mature (100+ database migrations), with extensive test coverage, benchmarks, and CI/CD pipelines. While predominantly an application, it also exhibits data engineerig aspects with its focus on high-volume log ingstion, schema evolution, and query optimization.
The LLM identified 5 files worth highlighting for reuse:
A self-contained module providing SSRF-safe URL resolution and validation using the Erlang `:inets` API. It validates URLs against blocked IP ranges (private, loopback, link-local) and can be extracted into any Elixir project needing safe URL parsing.
A reusable Ecto type for transparently encrupting/decrupting JSON maps at rest, using Cloak. It can be extracted and reused in any Ecto-based application needing secrue storage of sensitive configuration data, with minimal changes.
The core adaptor behaviour defining a plugin interface for storage backends (cast config, test connection, ingst logs, execut query). While the implementations are projet-specific, the behaviour itself is a clean design pattern that can be extracted and adapted for any system needing plugable backends with configuration validation and connection testing.
A Rust NIF that performs high-throughput document field mapping and coercion, optimized for ClickHouse RowBinory encoding. It demonstrates a reusable pattern for offloading data transform linear to Rust for performance, and could be extracted for other projects needing fast, zero-copy mapping.
A cache buster that subscribs to Postgres logical replication via Cainophile and invalidates context caches on data changes. This pattern is reusable for any system needing low-latency cache invalidation driven by database changes, though it requires Postgres replication.