$ show kaito-project/kaito
KAITO (Kubernetes AI Toolchain Operator) is a production-grade Kubernetes operator that automates the full lifecycle of large language model (LLM) deployments. It reconciles custom resources—Workspace, InferenceSet, RAGEngine, MultiRoleInference, and ModelMirror—to provision GPU nodes, download or stream model weights from HuggingFace or Azure Blob, run inference servers (vLLM or HuggingFace Transformers), perform fine-tuning (LoRA/QLoRA), and manage RAG pipelines with vector stores (FAISS/Qdrant) and output guardrails. The operator supports multi-cloud node provisioning (Azure GPU provisioner, Karpenter, BYO) and features like prefill-decode disaggregation, model mirroring, and benchmark metrics collection. The system architecture is a collection of Go controllers using controller-runtime, with admission webhooks for CRD validation and a set of Python services for inference, tuning, and RAG. Key entry points: cmd/workspace/main.go (workspace controller), cmd/ragengine/main.go (RAG engine controller), presets/workspace/inference/vllm/inference_api.py (vLLM inference service), and presets/ragengine/main.py (RAG FastAPI service). Data flows: user creates a Workspace CR → controller creates NodeClaims, provisions GPU nodes, downloads/streams model weights → creates StatefulSet with inference service → exposes Service. For RAG: user creates RAGEngine CR → controller deploys vector store and inference service → user indexes documents via API → chat completions with retrieval augmentation. Notable engineering patterns include a plugin-based model registry (KaitoModelRegister), a generator pattern (WorkspaceGeneratorContext with chainable manifest modifiers), an abstract NodeProvisioner interface with multiple implementations, and a hot-reloadable guardrails system with Prometheus metrics. The codebase has extensive test coverage: unit tests with mock clients, integration tests for controllers, and end-to-end tests using Ginkgo that validate real cluster behavior. The project is actively maintained with frequent commits, CI/CD workflows, and versioned documentation.
The LLM identified 3 files worth highlighting for reuse:
Defines the NodeProvisioner interface (ProvisionNodes, EnsureNodesReady, DeleteNodes, CollectNodeStatusInfo, BuildNodeSelector) that abstracts GPU node provisioning across cloud providers. Implementations exist for Karpenter, Azure GPU provisioner, and BYO nodes. This pattern allows the workspace controller to remain cloud-agnostic while supporting different provisioning strategies. Reusable because any project needing multi-cloud node management can adopt this interface pattern with minimal adaptation.
Implements a chainable manifest generator using a WorkspaceGeneratorContext and TypedManifestModifier functions. This functional options pattern constructs Kubernetes objects (StatefulSet, Service, Job, etc.) by composing small, testable modifier functions. It separates object generation from controller logic and enables reuse across different workloads (inference, tuning, RAG). The pattern is self-contained and can be extracted for any project that needs to build complex K8s manifests from a common context.
Provides a hot-reloadable output guardrails system for RAG chat completions. The GuardrailsReloader watches policy YAML files on disk, builds scanners (ban_substrings, regex, secrets, sensitive data, invisible text, token limit, JSON, reading time) from configuration, and applies them to model outputs with block/redact actions. Includes Prometheus metrics for scanner hits and errors. This pattern solves a recurring data safety problem and can be extracted for any LLM serving system needing dynamic output filtering without restarts.