$ show xorbitsai/inference
Xinference is a production-grade model serving platform that provides OpenAI-compatible REST APIs for a wide range of AI models including LLMs, embeddings, rerank, image generation, video, audio/TTS, and world models. It supports multiple inference backends (vLLM, SGLang, transformers, MLX, llama.cpp) and can be deployed as a single process or a distributed cluster using an actor-based supervisor/worker architecture (xoscar). The system includes a web UI for model management, authentication/authorization with OAuth2 scopes, token-aware routing for load balancing, and monitoring capabilities. It bundles many third-party model implementations for turnkey deployment, which reduces setup friction but introduces maintenance overhead from vendored code. The architecture is a hybrid of a monolithic FastAPI server and a distributed actor system. Key entry points are CLI commands (xinference, xinference-supervisor, xinference-worker) and the REST API. The frontend is a Next.js application statically exported and served by the backend. The codebase demonstrates strong patterns in modular route registration, plugin-style model registries, and streaming response handling, but also carries significant technical debt from heavy vendoring, duplicated code across third-party integrations, and version-sensitive patches to upstream libraries like vLLM.
The LLM identified 5 files worth highlighting for reuse:
Implements a decorator-based registry for model-specific tool call parsers. New parsers can be registered with @register_tool_parser, enabling clean separation of parsing logic per model family. This pattern is reusable for any plugin system where different implementations need to be discovered and selected at runtime.
Provides streaming_response_iterator and async_streaming_response_iterator functions that parse Server-Sent Events (SSE) from OpenAI-compatible endpoints. These are self-contained utilities that can be extracted into any Python project needing to consume streaming API responses, handling chunk parsing, error propagation, and graceful termination.
Maps legacy OAuth2 scope names to their current equivalents, allowing gradual migration of access tokens without breaking existing clients. This pattern is applicable to any system that needs to rename or restructure permission scopes while maintaining backward compatibility.
Reuses the LLM cache manager via composition to provide caching for embedding models. Demonstrates a clean pattern for sharing complex infrastructure (download, caching, lifecycle) across different model types without inheritance, making it easy to add caching to new model families.
Implements a modular route registration system where each domain (llm, images, audio, etc.) exposes a register_routes(api) function. This allows the API surface to be extended by simply adding new route modules, without modifying core server code. The pattern is applicable to any FastAPI or similar framework needing clean separation of endpoint groups.