Watermill is a Go library for building message-driven and event-driven applications. It is not a standalone service; applications embed it and use the core Publisher/Subscriber interfaces to communicate with brokers such as Kafka, RabbitMQ, NATS, Redis Streams, Google Cloud Pub/Sub, AWS SNS/SQS, SQL databases, or an in-memory Go channel. This repository contains the core library, reusable components, runnable examples, and small CLI tools, while broker-specific adapters and some integrations live in separate modules. The architecture is layered. At the bottom are the low-level Pub/Sub abstractions and the Message object with UUID metadata, payload, and Ack/Nack lifecycle semantics. The middle layer is the Router, which wires handlers into consumption loops and supports middleware and publisher/subscriber decorators. On top of that, optional components implement CQRS event/command buses and processors, topic forwarding, request-reply, delayed publishing, metrics, fan-in/fan-out, and requeuing. Middleware covers retry with exponential backoff, poison queues, deduplication, throttling, circuit breaking, correlation IDs, timeout enforcement, and panic recovery. The repository is extensively tested, including a universal pub/sub conformance suite intended for adapter authors to reuse, and GoChannel serves as a complete in-memory reference implementation. The examples demonstrate real production patterns: CQRS with protobuf, ordered events, exactly-once delivery counters, outbox/transactional event forwarding, server-sent event feeds, webhook ingestion, and consumer groups. There is no data research component; this is purely a software library/application framework.
Ahoy is a Rails engine that provides first-party analytics for Ruby on Rails applications. It tracks visits and events, storing data in the application's database by default while supporting custom data stores. The architecture follows Rails engine patterns with controllers, models, and concerns. Key features include automatic visit tracking, event tracking, geocoding via background jobs, IP masking for GDPR compliance, and associating visits with application models. The system provides both server-side Ruby tracking and client-side JavaScript/AMP/native app tracking through a configurable API.
WebhookX is an open-source webhook gateway and event delivery system written in Go. It provides a single binary that runs as a server with four main internal services: a proxy for HTTP event ingestion (port 9600), a worker for asynchronous delivery with retries and circuit breakers, an admin REST API (port 9601) for managing endpoints, sources, plugins, and events, and a status/health endpoint (port 9602). Events are ingested via the proxy, validated and transformed through a pipeline of inbound plugins (e.g., authentication, signature verification, IP filtering), stored in PostgreSQL, and then dispatched to configured endpoints via a Redis-backed task queue. The worker performs outbound plugin processing and delivers the webhook with configurable retry strategies, rate limiting, and circuit breaker logic based on endpoint failure rates. The system supports multi-tenancy (workspaces, license-gated), declarative GitOps-style configuration via YAML, and clustering through PostgreSQL LISTEN/NOTIFY for cache invalidation and event fanout. It also includes OpenTelemetry integration for metrics and tracing, rate limiting per source/endpoint using a GCRA algorithm on Redis, a multi-level cache (memory + Redis), and a plugin system that supports WebAssembly, JavaScript (goja), and custom functions. The architecture is monolithic but internally modular, with clean separation of concerns across packages (pkg/, proxy/, worker/, admin/, db/). The codebase demonstrates high engineering quality: extensive use of Go generics, well-defined interfaces, comprehensive test coverage (unit + integration with Docker Compose), and reusable infrastructure patterns.
Quamina is a high-performance Go library for matching patterns against JSON events. It uses a custom NFA/DFA automaton approach internally, supporting exact strings, shell-style wildcards, and a subset of I-Regexp (RFC 9485). Patterns are defined as JSON objects with field/value constraints, and the library provides a concise API to add patterns and efficiently retrieve matches for events. The core is a compact `smallTable` data structure for byte-transition tables, with optimizations like incremental epsilon closures and table-pointer deduplication to keep matching near O(1) for exact patterns. Quamina is version 2.0.1, has zero runtime dependencies, and includes extensive benchmarks and a research subdirectory for performance analysis. It is intended as an embedded library for real-time event classification (e.g., routing, log analysis) rather than a standalone service.
file.d is a high‑throughput log and event processing daemon written in Go, designed for production data pipelines. It reads events from various sources (files, Kafka, Kubernetes, HTTP, syslog, sockets, journalctl), runs them through a configurable chain of action plugins (masking, throttling, transformation, decoding, joining, etc.), and delivers them to multiple outputs (Elasticsearch, ClickHouse, Loki, S3, Kafka, Splunk, Postgres, GELF, file, stdout). The architecture is plugin‑based: inputs, actions, and outputs register into a central registry, and pipelines are defined via YAML configuration. The system provides at‑least‑once delivery semantics, stream‑based ordering, batching with backoff, and dead‑queue failover. The codebase demonstrates strong engineering practices: extensive unit and integration tests (including e2e scenarios with external services), performance optimizations (zero‑copy JSON via insane‑json, event pooling, memory limits), and a rich set of reusable internal libraries (HTTP client with circuit breaker, metric controllers, match rules, configuration parsing). It includes a Helm chart for Kubernetes deployment, Prometheus monitoring with a Grafana dashboard, and a playground web service for interactively testing pipelines. The transform action includes a custom expression language with a Pratt parser and compiler. While the core is solid, there are some maintenance risks: many plugins contain global caches or mutable state, several code paths call `logger.Fatal` on misconfiguration (making the whole process exit), and the file input offset persistence is hand‑rolled. The large number of input/output/action plugins (~40 total) increases surface area for drift, and some e2e tests depend on external Docker services. Overall, file.d is a mature and capable data pipeline framework, well‑suited for log aggregation and transformation in containerized environments.