$ show exo-explore/exo
exo is a distributed inference system for running large language models (LLMs) and image generation models across clusters of devices, primarily targeting Apple Silicon hardware via the MLX framework. It enables running models larger than a single device's memory by sharding them across multiple nodes using pipeline and tensor parallelism. The system provides an OpenAI-compatible API, a web dashboard, and a native macOS application. The architecture is event-sourced, with a master node managing state and workers executing inference tasks. Communication is handled by a custom Rust networking layer built on Zenoh, supporting peer discovery and RDMA over Thunderbolt. The codebase demonstrates strong engineering in several areas, including immutable state models (FrozenModel), typed pub/sub routing, and sophisticated KV-cache management for LLM inference. While the core architecture is sound, the project carries significant technical debt through custom forks of critical dependencies (zenoh, mlx-lm), extensive monkey-patching to work around upstream library limitations, and a highly complex Nix-based build system. The codebase is tightly coupled to the Apple Silicon ecosystem, limiting portability but enabling deep optimization. It is a high-quality, ambitious project under active development, but its complexity and ecosystem coupling present challenges for long-term maintainability and external contribution.
The LLM identified 6 files worth highlighting for reuse:
Provides FrozenModel (immutable Pydantic base with camelCase aliases) and TaggedModel (discriminated union serialization via a tag field). This is a foundational pattern for building event-sourced systems with strict typing and clean serialization. Highly reusable in any Python project needing robust state management.
Implements an OrderedBuffer that resequences out-of-order events based on a sequence number. Essential for distributed systems where events arrive from multiple sources. The implementation is generic and can be extracted for any project dealing with unordered streams.
A utility for managing exponential backoff on a per-key basis. Useful for rate-limiting retries to different resources (e.g., model downloads, API calls). The implementation is clean, generic, and immediately reusable.
Implements a prefix-aware KV cache for LLM inference. It handles exact and partial prefix matches, mutation-protected copies, and LRU eviction. This is a non-trivial optimization for reducing prefill latency in multi-turn conversations or shared prompt scenarios. The core logic is decoupled from the MLX backend and represents a reusable pattern for any LLM serving system.
A typed publish-subscribe router that abstracts over the underlying network transport (Zenoh). It allows components to subscribe to specific message types and topics, decoupling senders from receivers. The pattern is cleanly separated from the networking implementation, making it a reusable architectural pattern for event-driven applications.
Implements a clock-based leader election algorithm with seniority and commands_seen metrics. This is a core distributed systems pattern, and the implementation is self-contained and well-structured, making it a good reference or reusable component for projects needing simple leader election.