$ show BerriAI/litellm
LiteLLM is a production-grade, open-source proxy/gateway for over 200 large language model (LLM) providers, including OpenAI, Anthropic, Azure, Bedrock, Gemini, and many more. It presents a unified OpenAI-compatible API, handling request routing, authentication, rate limiting, cost tracking, caching, guardrails (content filtering, PII detection), and audit logging. The system is architected as a FastAPI Python server with a Next.js admin dashboard, Rust extensions for performance-critical paths (token counting, caching, auth, cost calculation), and optional PostgreSQL/Redis backends. It supports streaming, MCP (Model Context Protocol), A2A (Agent-to-Agent) protocol, and enterprise features like SSO, secret management, and Terraform deployment modules. Key entry points include the proxy server (litellm/proxy/proxy_server.py), CLI commands (litellm, litellm-proxy), and the admin UI at /ui/litellm-dashboard. The primary data flow is from client requests (chat completions, embeddings, audio, images) received by the proxy, which authenticates, applies guardrails, routes to the configured LLM provider(s), logs usage, and returns a standardized response. Caching is multi-tier (Redis, disk, S3, GCS, Azure Blob, semantic). Spend tracking is aggregated into daily tables and exposed via the management API and UI. External dependencies include PostgreSQL (via Prisma ORM), Redis (for distributed caching, rate limiting, and queuing), and cloud-specific secret managers (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, HashiCorp Vault). The project uses uv for Python dependency management, Cargo for Rust crates, and npm/pnpm for the dashboard. Extensive CI/CD pipelines (CircleCI, GitHub Actions) run unit, integration, e2e, mutation, and Rust-Python parity tests.
The LLM identified 2 files worth highlighting for reuse:
Provides a shared test harness (test_cache_impl function) that validates any cache implementation against a common contract. Tests include get/set roundtrips, TTL expiration, overwrite behavior, and deletion. Any cache backend (Redis, Disk, S3, etc.) that implements the Cache trait can reuse these tests with minimal setup. This pattern prevents drift between cache implementations and ensures consistent behavior across backends.
A fully generic, reusable table component built on TanStack Table with built-in sorting, client/server pagination, column visibility toggling, row selection, and customizable empty states. It supports async data loading via a 'fetchData' callback and integrates with the dashboard's filtering/sorting URL parameters. The component is used across multiple admin pages (keys, teams, budget, guardrails, MCP servers) and can be extracted as a standalone library with minimal modification.