$ show openclaw/lobster
Lobster is a local-first workflow runtime for AI agents, implemented as a TypeScript CLI and embeddable SDK. It executes deterministic pipelines defined in JSON, YAML, or .lobster files: steps are shell commands or built-in pipeline commands (where, pick, map, sort, groupBy, table, approve, ask, llm.invoke, openclaw.agent, etc.) connected through typed async-iterator data flow rather than text pipes. Pipelines support approval gates, interactive input, parallel branches, for_each loops, retry/timeout policies, on_error handlers, cost limits, and persistent state that can be suspended and resumed via base64url resume tokens. Architecturally it is a modular Node.js application: a core runtime (src/runtime.ts, src/core/tool_runtime.ts) with a command registry, a workflow file loader/executor, an atomic-file state store, an LLM cost-tracking and accounting layer, and a small SDK (src/sdk/index.ts). CLI entry points are bin/lobster.js, bin/openclaw.invoke.js, and bin/clawd.invoke.js. The package also ships GitHub and Gmail recipes that shell out to the gh and gog CLIs, and can delegate agent turns to OpenClaw. External dependencies are limited to those CLIs, the OpenClaw gateway/LLM adapters, and Node.js >= 22. Notable engineering strengths are crash-safe state persistence, idempotent LLM spend accounting, retry/backoff, abortable process-tree cancellation, and extensive tests covering resume, cancellation, concurrency, and billing edge cases. The repository is unambiguously an application; it has no data-research pipelines, notebooks, or analytics assets. It is a well-maintained, thoughtfully structured codebase with several independently reusable Node.js components.
The LLM identified 5 files worth highlighting for reuse:
Configurable retry helper with fixed/exponential backoff, jitter, max attempts, and abort-signal awareness. It is generic and self-contained, usable in any Node/TypeScript project that needs resilient I/O or remote calls.
Implements crash-safe writeFileAtomic (temp file + rename), exclusive create via hard-link, parent-directory fsync, and file-lock acquisition with owner metadata. Can be extracted for any project needing durable local state or cache storage.
LlmSpendLedger tracks LLM usage across pipeline runs, retries, parallel branches, and resume sessions using provenance symbols and a spend ledger; it prevents double-billing replayed items and supports session restore. This demonstrates a non-obvious accounting pattern for LLM orchestration tools.
Child-process spawner that groups processes, propagates abort signals, tracks descendants, enforces output limits, and force-kills after timeout. Reusable for any CLI/automation tool that must cleanly terminate process trees.
Memoized Ajv schema compilation keyed by stable JSON stringify, preventing SchemaEnv memory leaks in long-lived processes. A small, generic utility for validating untrusted JSON envelopes.