vLLM is a high-throughput, memory-efficient inference and serving engine for large language models (LLMs). It implements PagedAttention for near-zero waste KV cache management, continuous batching, chunked prefill, prefix caching, CUDA/HIP graphs, speculative decoding, and torch.compile. The system serves OpenAI-compatible APIs (chat completions, completions, embeddings, rerank, speech-to-text), Anthropic Messages API, Cohere API, and custom pooling/classification endpoints. It supports 200+ model architectures including dense, MoE, hybrid SSM/attention, and multimodal models (vision, audio, video). The architecture is a hybrid Python/Rust system: a Python core handles model execution, scheduling, and kernel dispatch, while a Rust frontend provides a high-performance HTTP/gRPC server with middleware for auth, metrics, and load management. Key entry points include the `vllm` CLI (serve, bench, launch), the `LLM` class for offline inference, and the OpenAI-compatible API server. External dependencies include PyTorch, HuggingFace Transformers, FlashAttention, CUTLASS, Triton, NCCL/RCCL, and various quantization libraries. The project is production-grade with extensive CI/CD, Docker images, Helm charts, and deployment integrations for Kubernetes, Ray, SageMaker, and more.
A production-grade AI coding agent (the Codex CLI) that operates as a terminal-based assistant. It provides a multi-modal, extensible platform for thread-based conversations with language models (OpenAI, Bedrock, Ollama), including sandboxed code execution, MCP tool integration, plugin/skills/hooks extension system, and state persistence via SQLite. The architecture is a Rust monorepo with a client-server split: a high-level TUI or CLI sending JSON-RPC messages to an app-server that manages session state, model routing, and execution policies. The system supports cross-platform sandboxing (Linux bubblewrap, Windows managed sandbox, macOS seatbelt) and includes comprehensive telemetry, rollout tracing, and guardian approval pipelines. It is a fork of OpenAI's Codex CLI, with significant Rust rewrite and enhanced extensibility. While the codebase exhibits strong engineering practices (extensive testing, layered configuration, well-defined protocol types, and platform-agnostic abstractions), it is a large monorepo with many interdependent crates, some legacy modules, and platform-specific workarounds that reduce overall cohesion. Key reusable components include the protocol type definition crate (with clever no-op macro optimizations for build speed), a RAII OpenTelemetry timer, a RedactedString wrapper for secret handling, and token-aware text truncation utilities.
Xinference is a production-grade model serving platform that provides OpenAI-compatible REST APIs for a wide range of AI models including LLMs, embeddings, rerank, image generation, video, audio/TTS, and world models. It supports multiple inference backends (vLLM, SGLang, transformers, MLX, llama.cpp) and can be deployed as a single process or a distributed cluster using an actor-based supervisor/worker architecture (xoscar). The system includes a web UI for model management, authentication/authorization with OAuth2 scopes, token-aware routing for load balancing, and monitoring capabilities. It bundles many third-party model implementations for turnkey deployment, which reduces setup friction but introduces maintenance overhead from vendored code. The architecture is a hybrid of a monolithic FastAPI server and a distributed actor system. Key entry points are CLI commands (xinference, xinference-supervisor, xinference-worker) and the REST API. The frontend is a Next.js application statically exported and served by the backend. The codebase demonstrates strong patterns in modular route registration, plugin-style model registries, and streaming response handling, but also carries significant technical debt from heavy vendoring, duplicated code across third-party integrations, and version-sensitive patches to upstream libraries like vLLM.