$ show taskforcesh/bullmq
BullMQ is a high-performance, distributed job queue library. It manages the lifecycle of background jobs (add, process, retry, fail, complete) using Redis as the primary backend, with an optional PostgreSQL backend. The library supports advanced features including delayed jobs, prioritized queues, recurring jobs via schedulers, parent-child job flows, deduplication, rate limiting, concurrency control, stalled job recovery, and real-time event streaming. The architecture is driven by a set of Lua scripts (for Redis) and SQL functions (for PostgreSQL) that act as the single source of truth for all atomic state transitions. The core Node.js/TypeScript implementation features a clean backend abstraction (`IQueueBackend`) allowing transparent swapping of storage backends. Official ports exist for Python, Elixir, Rust, .NET, and PHP, all sharing the same underlying Lua/SQL semantics. The codebase demonstrates high craftsmanship with several safely reusable components, including a generic Lua script loader with include resolution, a standalone backoff strategy calculator, a cooperative cancellation token, and a batch lock renewal manager. The project is actively maintained with extensive CI/CD, semantic versioning, and a large community. The primary maintenance risk is the inherent complexity of keeping six language ports in perfect feature parity.
The LLM identified 6 files worth highlighting for reuse:
A sophisticated Lua script loader that resolves `@include` directives, supports path mappings, globs, and circular dependency detection. This component is safely reusable in any Node.js project requiring complex Lua script management, demonstrating high craftsmanship in build-time code generation.
Implements fixed, exponential, and custom backoff strategies using the Strategy pattern. Completely isolated from the queue logic, it is a drop-in component for any retry system in a new project.
A standalone cooperative cancellation token implementation providing a clean signal/callback API. Easily extractable for managing cancellation in any async workflow or long-running operation.
Loads SQL files at build time and inlines them into the codebase, enabling single-file distribution without runtime filesystem access. Solves a common problem for libraries that need to bundle SQL commands.
Centralizes the generation of all Redis key names for the queue system. While the specific keys are domain-specific, the pattern of a dedicated key generator is critical for consistency in distributed systems and is replicated across all language ports.
Manages periodic lock renewal for multiple jobs in efficient batches. Implements a robust heartbeat pattern for distributed locks, handling failures gracefully. The batch renewal algorithm is a non-obvious optimization for high-throughput systems.