LiteLLM is a production-grade, open-source proxy/gateway for over 200 large language model (LLM) providers, including OpenAI, Anthropic, Azure, Bedrock, Gemini, and many more. It presents a unified OpenAI-compatible API, handling request routing, authentication, rate limiting, cost tracking, caching, guardrails (content filtering, PII detection), and audit logging. The system is architected as a FastAPI Python server with a Next.js admin dashboard, Rust extensions for performance-critical paths (token counting, caching, auth, cost calculation), and optional PostgreSQL/Redis backends. It supports streaming, MCP (Model Context Protocol), A2A (Agent-to-Agent) protocol, and enterprise features like SSO, secret management, and Terraform deployment modules. Key entry points include the proxy server (litellm/proxy/proxy_server.py), CLI commands (litellm, litellm-proxy), and the admin UI at /ui/litellm-dashboard. The primary data flow is from client requests (chat completions, embeddings, audio, images) received by the proxy, which authenticates, applies guardrails, routes to the configured LLM provider(s), logs usage, and returns a standardized response. Caching is multi-tier (Redis, disk, S3, GCS, Azure Blob, semantic). Spend tracking is aggregated into daily tables and exposed via the management API and UI. External dependencies include PostgreSQL (via Prisma ORM), Redis (for distributed caching, rate limiting, and queuing), and cloud-specific secret managers (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, HashiCorp Vault). The project uses uv for Python dependency management, Cargo for Rust crates, and npm/pnpm for the dashboard. Extensive CI/CD pipelines (CircleCI, GitHub Actions) run unit, integration, e2e, mutation, and Rust-Python parity tests.
MLflow is an open-source ML lifecycle platform that provides experiment tracking, model registry, model deployment, evaluation, and observability for both classical ML and GenAI/LLM workflows. In production, it operates as a modular monolith: a Python backend (Flask/FastAPI) with SQLAlchemy-backed storage (SQLite, PostgreSQL, MySQL, MSSQL), a React/TypeScript web UI, and client SDKs for Python, R, Java, and TypeScript. The system exposes a REST API and CLI for logging experiments, registering models, collecting traces via OpenTelemetry, and managing an AI Gateway that routes requests to dozens of LLM providers with budget controls, guardrails, and traffic splitting. The architecture includes a plugin system for custom store backends, deployment targets, and authentication providers, as well as a custom linting tool (clint) and extensive CI/CD pipelines. Key entry points are the `mlflow` CLI, `MlflowClient` Python API, tracking server (`mlflow server`), gateway server (`mlflow gateway`), and model scoring server (`mlflow models serve`). Data flows from user instrumentation through asynchronous logging to the backend store, with traces and evaluations queryable via the UI or API. External dependencies span ML frameworks (scikit-learn, PyTorch, TensorFlow, LangChain, etc.), LLM provider SDKs, and infrastructure tools (Docker, Kubernetes, Helm).
CPA Manager Plus is a full-stack, self-hosted management panel for AI gateway proxies (CPA / CLIProxyAPI). It provides a rich React dashboard to monitor usage across multiple AI providers, manage credentials and API keys, configure provider-specific settings, automate quota cooldowns and account health actions, and inspect credential health via Codex/xAI probes. The Go backend persists all data in SQLite, runs background workers for data rollups, migrations, and collector polling, and exposes a comprehensive REST API. The codebase is well-structured with a clean layered architecture (HTTP handlers, services, repositories), extensive unit and integration tests in both Go and TypeScript, a demo mode with synthetic data, and i18n support. It is actively maintained with a rigorous release process and frequent updates.
aiproxy is a production-grade AI API gateway written in Go, acting as a unified proxy between client applications and over 40 AI provider APIs (OpenAI, Anthropic, Gemini, etc.). It converts protocol formats (OpenAI, Anthropic, Gemini), handles authentication, rate limiting, load balancing, request caching, and detailed usage billing. The system includes a React admin dashboard for managing channels, tokens, models, groups, logs, and consumption. It also supports the Model Context Protocol (MCP) for integrating AI tool servers, both hosted and proxied. The architecture is monolithic with a plugin system for extensibility, storing state in PostgreSQL and Redis. It demonstrates strong engineering practices including comprehensive test coverage, efficient caching (multi-layer with keyed locks), sliding window rate limiting, and a clean adaptor pattern for provider abstraction.
Higress is a cloud-native, AI-native API gateway built on Istio and Envoy. In production, it operates as a Kubernetes ingress controller, an AI/LLM proxy gateway, and a Wasm plugin host. The system ingests Kubernetes resource definitions (Ingress, Gateway API, custom CRDs like WasmPlugin, Http2Rpc, McpBridge) and translates them into Envoy xDS configurations, managing the full lifecycle of gateway routes, policies, and plugins. It provides a rich set of Wasm plugins (Go, Rust, C++, AssemblyScript) for authentication, rate limiting, request/response transformation, AI proxy (with 15+ LLM providers), semantic caching, RAG, MCP server hosting, and security (WAF, bot detection, IP restriction). The architecture is microservices-oriented: a control plane (Go-based ingress controller) manages Kubernetes resources and pushes config to Envoy data planes; a CLI tool (hgctl) handles installation, plugin development, and debugging; and a plugin release pipeline manages OCI image distribution with provenance verification. External dependencies include Kubernetes (client-go, controller-runtime), Istio (pilot, networking APIs), Envoy (Go filter API), and various AI/vector DB providers (OpenAI, Milvus, Pinecone, etc.).
This repository implements a production-grade semantic router for LLM inference, functioning as an Envoy ext_proc filter that intercepts chat completion requests, classifies them via multiple signals (intent, PII, jailbreak, domain, embeddings, etc.), and routes to the most appropriate model or model composition. The system supports Mixture-of-Models (MoM) with configurable algorithms (fusion, cascading, workflows) and provides advanced features like semantic caching, context compression, rate limiting, and observability. It includes a full-stack dashboard (React + Go backend) for configuration management, evaluation/experimentation (campaigns, A/B tests, capacity SLOs), and monitoring. A Python CLI and Kubernetes operator are available for deployment. The codebase demonstrates high engineering quality with extensive validation, testing, and a modular architecture, though its polyglot nature (Go, Rust, Python, TypeScript) and numerous optional backends introduce maintenance complexity. While research elements (training scripts, benchmarks) exist, the primary character is a deployable application.
New API is a full-stack AI gateway/proxy service that provides a unified interface for multiple AI model providers (OpenAI, Claude, Gemini, etc.). It features user authentication and authorization, API key/token management with quota tracking and billing, administrative dashboard for configuration and monitoring, intelligent routing between providers, format conversion between different AI APIs, payment integrations, webhook support, and a playground for testing models. The backend is built with Go using the Gin web framework and GORM ORM, while the frontend is a React/TypeScript application built with Bun. The system implements a modular architecture with clear separation of concerns across controller, service, model, and relay layers.
Kong is a cloud-native, platform-agnostic API gateway built on OpenResty/Nginx. It provides traffic management, security, observability, and AI/LLM proxying capabilities through a plugin architecture. The system supports traditional, DB-less, and hybrid deployment modes, with an Admin API for configuration and Kong Manager for GUI-based management. Key features include load balancing, authentication, rate limiting, request/response transformation, and extensive observability integrations.
gpt4free (g4f) is a large-scale, community-driven Python library that aggregates dozens of free AI providers (LLMs, image, audio, video) into a unified, OpenAI-compatible interface. It provides multiple access modes: a Python client library (sync/async), a FastAPI-based API server, a Flask web GUI, a CLI tool, a Model Context Protocol (MCP) server for tool calling, a browser extension, a Discord bot, an Android app, and a Go launcher. The architecture is provider-based: each AI service (e.g., ChatGPT, Gemini, DeepSeek, Copilot, Grok, Perplexity) is wrapped in an adapter class implementing a common `AsyncGeneratorProvider` interface. The system handles authentication via cookies, OAuth flows, HAR files, and browser automation (CDP/nodriver), and supports streaming, tool/function calling, image generation, audio synthesis, and web search. It is primarily an application for consuming free AI models, not a research or data pipeline project, though it includes some data-oriented utilities like token optimization and usage tracking.
AIBrix is a Kubernetes-native control plane and intelligent gateway for serving production LLM workloads. The repository implements two intertwined planes: a data plane composed of Envoy ext_proc gateway plugins (cmd/plugins, pkg/plugins/gateway) that intercept OpenAI-compatible inference requests and route them to vLLM, SGLang, or TensorRT-LLM replicas using algorithms such as least-load, prefix-cache-aware routing, SLO queueing, token-load tracking, and prefill/decode disaggregation; and a control plane of Kubernetes controllers (cmd/controllers) that reconcile custom resources for autoscaling (PodAutoscaler), LoRA adapters, model warmup/claims, KV caches, RayCluster fleets, and StormService/RoleSet orchestration. Supporting runtime code includes a Python metadata and batch-job service, a distributed KV-cache package, a GPU optimizer, and an OpenAI-compatible frontend.
ClawRouter is a production-grade AI/LLM proxy gateway deployed on Cloudflare Workers. It provides a unified API endpoint for multiple AI providers (OpenAI, Anthropic, Google Gemini, etc.) with policy-based access control, per-user and per-provider budget accounting, usage tracking, and a comprehensive admin console (React/TypeScript). The system authenticates users via Cloudflare Access, local auth, or API keys, then routes requests through configurable policies that enforce budget limits, grant selection, and content retention. It supports streaming, WebSocket-based Responses API, and a Fusion mixture-of-agents feature for multi-model orchestration. The worker uses Durable Objects for stateful budget ledgers, usage ledgers, credential stores, and background job recovery, all backed by SQLite. A rich admin UI (Patchboard) allows operators to manage policies, credentials, grants, users, and view analytics. The codebase includes extensive unit, integration, and browser (Playwright) tests, along with CI/CD pipelines for Cloudflare deployment and self-hosting via Docker/workerd. It is a well-architected, thoroughly tested system that demonstrates high craftsmanship in state management, error handling, and idempotent accounting patterns.
This repository is the `luci-app-openclaw` package for OpenWrt routers, providing a complete web UI (via LuCI) and command-line management surface for the OpenClaw AI gateway. The system handles installation and lifecycle of a Node.js runtime and the OpenClaw npm package, configures AI model and messaging channel settings (Telegram, WeChat, etc.), manages device pairing approvals, offers an interactive terminal (xterm.js) via WebSocket, and supports backup/restore with atomic writes. The architecture is a monolithic OpenWrt plugin that integrates with procd for service management, UCI for configuration, and relies on external Node.js binaries plus the OpenClaw gateway software. The codebase exhibits strong engineering practices: a comprehensive contract-test suite with mutation testing, atomic configuration writes with pre-write backups, CSRF protection on all state-changing endpoints, a shell/JS menu engine for interactive configuration, and an upgrade state machine with rollback. It is actively maintained with frequent adjustments to upstream OpenClaw versions and security hardening.
This repository is a full-stack, multi-tenant AI API gateway platform (named 'sub2api') that proxies requests to a wide range of AI model providers including OpenAI, Anthropic, Gemini, Grok, DeepSeek, and many others. It provides authentication (JWT, OAuth, API keys), rate limiting, token-level billing, subscription management, user and admin management, payment processing (Alipay, WeChat Pay, Stripe, Airwallex, EasyPay), content moderation, channel monitoring, and operational dashboards. The backend is written in Go using the Gin framework and Ent ORM, with PostgreSQL and Redis for data storage and caching. The frontend is built with Vue 3, TypeScript, and Pinia stores, offering both admin and user interfaces. The system is designed for production deployment with Docker, includes extensive unit and integration tests, and supports a plugin architecture for extensible OAuth transport. Despite its large codebase and many features, the architecture follows a clear service-repository pattern with well-isolated packages, making it a solid example of a complex API gateway implementation.
Claude Code Router (CCR) is a cross-platform desktop application built with Electron and React that functions as a local gateway and router for AI agent traffic. It intercepts requests from local AI agents (Claude Code, Codex, ZCode) and routes them to various LLM providers (OpenAI, Anthropic, DeepSeek, etc.) through a configurable proxy. The application manages API keys, routing rules, fallback strategies, and provides detailed usage observability via a system tray and dashboard. It supports complex features like Fusion models (combining base models with tools), IM bot relay integrations (WeChat, Slack, etc.), and deep linking for provider provisioning. The architecture follows a standard Electron pattern with a Main process handling system-level tasks (proxy, SQLite storage, process management) and a Renderer process for the React UI. Data persistence relies on SQLite for configuration and usage logs. The codebase demonstrates significant engineering effort in handling cross-platform differences (macOS/Windows/Linux) for proxy installation, certificate management, and agent profile injection. While the code is well-structured with clear separation of concerns, it carries medium technical drift due to heavy reliance on native dependencies (better-sqlite3) and OS-specific commands for proxy and certificate management.
CoAI (formerly ChatNio) is a full-stack AI chat platform and LLM gateway that provides a unified chat interface and OpenAI-compatible API proxy for 16+ AI model providers (OpenAI, Anthropic Claude, Google Gemini, Midjourney, DeepSeek, etc.). The system is built with a Go backend (Gin framework) and a React/TypeScript frontend (Redux, Tailwind CSS, Radix UI), with MySQL and Redis for persistence and caching. It supports real-time WebSocket chat, streaming responses, conversation management, file upload, web search, image generation, and a plugin system (MCP). The platform includes a comprehensive billing and subscription system with elastic per-token pricing and tiered plans (Basic/Standard/Pro), along with an admin dashboard for user management, channel configuration, analytics (model usage, request volume, revenue, errors), and system settings. It also features invitation codes, redeem codes, affiliate marketing, and multiple payment gateway integrations (Stripe, WeChat Pay, etc.). Architecturally, the backend employs an adapter pattern for AI providers, a channel manager for load balancing and retries, and a WebSocket connection manager with rate limiting. The frontend is heavily componentized with shadcn/ui and includes i18n support (English, Chinese, Japanese, Russian). Deployment is supported via Docker Compose, Zeabur, and a Tauri desktop app. While the codebase is well-structured and feature-rich, it shows signs of technical drift such as copy-pasted processor files across adapters, hardcoded model constants, and a large dependency surface. The open-source version is Apache 2.0 licensed, with a commercial 'CoAI Pro' tier offering additional features.
AIClient2API (A2) is a production-grade API proxy/gateway that converts proprietary AI provider APIs (Gemini, Claude/Kiro, Qwen, Codex, Grok, OpenAI, and more) into standard OpenAI-compatible endpoints (`/v1/chat/completions`, `/v1/messages`, `/v1beta/models`). It supports multi-account OAuth credential pools with automatic token refresh, health checks, and fallback routing, enabling clients to break free-tier restrictions. The server is built on Node.js with a master-worker architecture for process management, a plugin system for extensibility (authentication, monitoring, usage statistics, API key sharing), and a Go-based TLS sidecar for JA3/JA4 bypass. A full web management console (static HTML/CSS/JS) provides configuration, provider management, credential file upload, interactive playground, and usage monitoring. The codebase demonstrates strong patterns (strategy, factory, adapter, converter) but suffers from some monolithic utility files, hardcoded provider mappings, and high maintenance overhead from multiple OAuth implementations.
AgentTeams is a production-grade multi-agent orchestration platform that manages teams of LLM-powered worker agents on Kubernetes. It uses a central Manager controller that reconciles Custom Resource Definitions (Worker, Team, Human, Manager) with container backends (Docker, K8s), a Matrix chat layer for all agent communication, and an AI Gateway (Higress) for credential isolation. The system supports multiple worker runtimes (OpenClaw, QwenPaw, Hermes, DeepSeek Harness) through a plugin architecture, provides a CLI tool (`agt`) for resource management, and includes human-in-the-loop oversight via Matrix rooms. Key features include STS token-based credential management, role-based authorization, audit logging, file synchronization via MinIO/OSS, and a skill discovery and distribution system. The codebase is heavily tested with unit, integration, and end-to-end tests, but shows moderate technical drift due to legacy runtime coexistence (CoPaw EOL migration) and some duplicated shell scripts across runtimes.
ThinkWatch is a Rust workspace implementing an AI gateway platform that fronts multiple model providers and MCP (Model Context Protocol) servers. It exposes OpenAI-compatible chat/completion endpoints, streaming, MCP tool-call routing, content filtering, and provider failover. The backend is split into crates: `common` (shared lifecycle/limits/access), `auth` (PoW login, API keys), `gateway` (model routing), `mcp-gateway` (MCP server proxying), `server` (REST admin/audit/trace endpoints), and `test-support` (integration test client). A React/TypeScript frontend in `web/` provides an admin console, setup wizard, limits panels, and MCP server management. Data flows: requests enter the gateway, pass through `check_limits`, `check_budget`, `check_access`, then `run_post_invoke`; `trace_id` is propagated across gateway, MCP, audit, and app logs stored in ClickHouse; Postgres holds users, API keys, budgets, and MCP server configs; Redis/Lua scripts enforce rate limits.