vLLM is a high-throughput, memory-efficient inference and serving engine for large language models (LLMs). It implements PagedAttention for near-zero waste KV cache management, continuous batching, chunked prefill, prefix caching, CUDA/HIP graphs, speculative decoding, and torch.compile. The system serves OpenAI-compatible APIs (chat completions, completions, embeddings, rerank, speech-to-text), Anthropic Messages API, Cohere API, and custom pooling/classification endpoints. It supports 200+ model architectures including dense, MoE, hybrid SSM/attention, and multimodal models (vision, audio, video). The architecture is a hybrid Python/Rust system: a Python core handles model execution, scheduling, and kernel dispatch, while a Rust frontend provides a high-performance HTTP/gRPC server with middleware for auth, metrics, and load management. Key entry points include the `vllm` CLI (serve, bench, launch), the `LLM` class for offline inference, and the OpenAI-compatible API server. External dependencies include PyTorch, HuggingFace Transformers, FlashAttention, CUTLASS, Triton, NCCL/RCCL, and various quantization libraries. The project is production-grade with extensive CI/CD, Docker images, Helm charts, and deployment integrations for Kubernetes, Ray, SageMaker, and more.
LiteLLM is a production-grade, open-source proxy/gateway for over 200 large language model (LLM) providers, including OpenAI, Anthropic, Azure, Bedrock, Gemini, and many more. It presents a unified OpenAI-compatible API, handling request routing, authentication, rate limiting, cost tracking, caching, guardrails (content filtering, PII detection), and audit logging. The system is architected as a FastAPI Python server with a Next.js admin dashboard, Rust extensions for performance-critical paths (token counting, caching, auth, cost calculation), and optional PostgreSQL/Redis backends. It supports streaming, MCP (Model Context Protocol), A2A (Agent-to-Agent) protocol, and enterprise features like SSO, secret management, and Terraform deployment modules. Key entry points include the proxy server (litellm/proxy/proxy_server.py), CLI commands (litellm, litellm-proxy), and the admin UI at /ui/litellm-dashboard. The primary data flow is from client requests (chat completions, embeddings, audio, images) received by the proxy, which authenticates, applies guardrails, routes to the configured LLM provider(s), logs usage, and returns a standardized response. Caching is multi-tier (Redis, disk, S3, GCS, Azure Blob, semantic). Spend tracking is aggregated into daily tables and exposed via the management API and UI. External dependencies include PostgreSQL (via Prisma ORM), Redis (for distributed caching, rate limiting, and queuing), and cloud-specific secret managers (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, HashiCorp Vault). The project uses uv for Python dependency management, Cargo for Rust crates, and npm/pnpm for the dashboard. Extensive CI/CD pipelines (CircleCI, GitHub Actions) run unit, integration, e2e, mutation, and Rust-Python parity tests.
Nanobot is a self-hosted, open-source personal AI assistant framework that operates as a full-stack application. The Python backend provides an agent loop with tool execution, session management, multi-provider LLM orchestration (OpenAI, Anthropic, etc.), and a plugin system for 18+ messaging channels (Telegram, Discord, Slack, WeChat, Feishu, etc.). It exposes a WebSocket-based gateway, an OpenAI-compatible API, and a CLI. Two frontends exist: a rich WebUI built with React/TypeScript/Vite/Tailwind and a terminal TUI built with Bun/OpenTUI. The architecture is event-driven via a message bus, with decoupled channel adapters, provider abstraction, and a config system supporting hot-reload. It includes security features like workspace sandboxing, credential redaction, and SSRF protection. The codebase is extensively tested (hundreds of unit/integration tests) and demonstrates strong engineering practices with clean abstractions, though its large dependency surface and multiple frontends introduce some maintenance surface area.
OGX (formerly Llama Stack) is a production-grade, open-source AI inference platform that provides a drop-in replacement for the OpenAI API. It is a multi-provider API gateway supporting chat completions, embeddings, the Responses API (agentic turns with tool use), vector stores, file processing, conversations, prompts, skills, and connectors. The system is built on a clean provider plugin architecture where inline (built-in) and remote providers (30+ including Anthropic, Bedrock, Gemini, Ollama, vLLM, NVIDIA) are registered via ProviderSpec and dynamically routed. The FastAPI server auto-discovers API protocols and generates routes with telemetry (OpenTelemetry), authentication, multi-tenancy, and ABAC access control. The repository includes a Next.js admin UI, a Python CLI, OpenAPI-generated client SDKs (Python, TypeScript), benchmarking tools (RAG eval, vertical scaling), and comprehensive integration tests with recording/replay. It is designed for self-hosted or cloud deployment with Kubernetes and Docker support.
CPA Manager Plus is a full-stack, self-hosted management panel for AI gateway proxies (CPA / CLIProxyAPI). It provides a rich React dashboard to monitor usage across multiple AI providers, manage credentials and API keys, configure provider-specific settings, automate quota cooldowns and account health actions, and inspect credential health via Codex/xAI probes. The Go backend persists all data in SQLite, runs background workers for data rollups, migrations, and collector polling, and exposes a comprehensive REST API. The codebase is well-structured with a clean layered architecture (HTTP handlers, services, repositories), extensive unit and integration tests in both Go and TypeScript, a demo mode with synthetic data, and i18n support. It is actively maintained with a rigorous release process and frequent updates.
aiproxy is a production-grade AI API gateway written in Go, acting as a unified proxy between client applications and over 40 AI provider APIs (OpenAI, Anthropic, Gemini, etc.). It converts protocol formats (OpenAI, Anthropic, Gemini), handles authentication, rate limiting, load balancing, request caching, and detailed usage billing. The system includes a React admin dashboard for managing channels, tokens, models, groups, logs, and consumption. It also supports the Model Context Protocol (MCP) for integrating AI tool servers, both hosted and proxied. The architecture is monolithic with a plugin system for extensibility, storing state in PostgreSQL and Redis. It demonstrates strong engineering practices including comprehensive test coverage, efficient caching (multi-layer with keyed locks), sliding window rate limiting, and a clean adaptor pattern for provider abstraction.
Jan is an open-source desktop application that provides a ChatGPT-like interface for interacting with both local and remote AI models. Built with Tauri (Rust backend) and React (TypeScript frontend), it allows users to download, manage, and run LLMs locally via backends like llama.cpp and MLX, while also supporting cloud APIs from OpenAI, Anthropic, and others. The application features an OpenAI-compatible API server, custom assistants, agent capabilities ('Cowork' mode), Retrieval-Augmented Generation (RAG), web search, and Model Context Protocol (MCP) integration for extensibility. It emphasizes privacy and local processing while offering cross-platform support for Windows, macOS, Linux, iOS, and Android.
llama-swap is a production-grade reverse proxy and model router for local large language model (LLM) inference servers. It manages the lifecycle of upstream model processes (e.g., llama.cpp, vLLM), routes incoming OpenAI-compatible HTTP requests to the appropriate model based on configurable groups, matrix constraints, and peer definitions, and supports hot-swapping models without restarting the proxy. The system provides a rich web UI (Svelte 5) for monitoring activity, managing models, and running interactive playground tools including a documentation agent that answers questions via MCP tools. It also includes auxiliary utilities like a Wake-on-LAN proxy and a vLLM sleep mode wrapper. The architecture is a single Go binary with embedded static assets, internally modular with distinct packages for configuration (YAML with macros and deep merge), process management (state machine with health checks and TTL), routing (group and matrix DSL), hardware detection, performance monitoring, and activity storage (SQLite). External dependencies are limited to upstream model servers and optional remote peers. The codebase is actively maintained, well-tested, and demonstrates advanced engineering patterns such as a generic event pub/sub system, a ring buffer, composable HTTP middleware chains, a configurable matrix-based model selection DSL, and a poll-based directory watcher suitable for containerized environments.
Windows Copilot API is an unofficial Python library and OpenAI-compatible HTTP server that bridges the consumer Microsoft Copilot chat service. It uses a pure-HTTP driver with curl_cffi to impersonate Chrome and speak Copilot's WebSocket protocol, handling authentication, proof-of-work challenges, and Cloudflare Turnstile clearance in-process. A Playwright-based browser module enables interactive sign-in and scheduled clearance refresh. The server wraps the library behind FastAPI endpoints exposing /v1/chat/completions and /v1/models with full streaming support, rate limiting, and concurrency serialization to cope with Copilot's single-account constraints. Notable features include transparent token refresh, conversation persistence, and a diagnostic tool for session debugging. The codebase is well-structured, with clear separation of concerns across protocol constants, challenge solvers, models, and drivers. It demonstrates advanced reverse-engineering and browser-emulation skills.
gpt4free (g4f) is a large-scale, community-driven Python library that aggregates dozens of free AI providers (LLMs, image, audio, video) into a unified, OpenAI-compatible interface. It provides multiple access modes: a Python client library (sync/async), a FastAPI-based API server, a Flask web GUI, a CLI tool, a Model Context Protocol (MCP) server for tool calling, a browser extension, a Discord bot, an Android app, and a Go launcher. The architecture is provider-based: each AI service (e.g., ChatGPT, Gemini, DeepSeek, Copilot, Grok, Perplexity) is wrapped in an adapter class implementing a common `AsyncGeneratorProvider` interface. The system handles authentication via cookies, OAuth flows, HAR files, and browser automation (CDP/nodriver), and supports streaming, tool/function calling, image generation, audio synthesis, and web search. It is primarily an application for consuming free AI models, not a research or data pipeline project, though it includes some data-oriented utilities like token optimization and usage tracking.
This repository is a Home Assistant custom component that adds OpenAI-compatible text-to-speech to a HA instance. It is distributed as an integration for HACS and registered as a config-flow service integration. In production it exposes TTS entities, an `openai_tts.say` service that announces text to media players with automatic volume save/restore and optional chimes, and an `openai_tts.set_api_key` service for credential rotation. The architecture is a HA integration split into focused modules: config flow and profile subentries, a streaming TTS engine, a sentence-level audio pipelining layer, an ffmpeg-based loudness pipeline, push-driven API health tracking, cache and validation logic, repairs, diagnostics, and a large announcement orchestration module. Provider support is driven by declarative presets for OpenAI, Mistral, Groq, Lemonfox, Kokoro, Chatterbox, and custom OpenAI-compatible endpoints, and the code intentionally modern-targets HA's streaming TTS contract rather than retaining legacy fallback paths.
DS2API is a production-grade API proxy/gateway that translates DeepSeek's web chat API into OpenAI, Claude, Gemini, and Ollama-compatible endpoints. It provides a full-featured web UI for administration, account management, proxy configuration, and API testing. The system handles authentication (JWT, API keys, managed accounts), streaming responses with tool call detection, and multi-protocol request normalization. Architecturally, it is a Go backend with a React frontend (Vite + Tailwind). The backend is modular: protocol-specific handlers (OpenAI chat/responses, Claude messages, Gemini generate, Ollama chat) share a common prompt compatibility layer that normalizes requests to DeepSeek's internal format. A custom DeepSeek client manages authentication, PoW challenges, session lifecycle, and streaming. The SSE parsing infrastructure, tool call sieve, account pool, and config store are reusable components with extensive test coverage. Key data flows: incoming API requests are parsed and normalized to a unified prompt representation (role-tagged messages, tool descriptions, file references). The prompt is sent to DeepSeek's web chat API via a streaming or non-streaming completion runtime. Responses are parsed from SSE chunks, transformed (thinking extraction, tool call normalization, citation handling), and streamed back in the original protocol format. The system supports retry logic, account switching, and current input file injection for large contexts. External dependencies include DeepSeek's web chat API (reverse-engineered), Go standard library, chi router, utls for TLS fingerprinting, Node.js for Vercel streaming and WebUI build, and React/Vite for the frontend. The project is actively maintained with CI/CD pipelines, Docker images, and Vercel deployment support.
Mini-SGLang is a lightweight, high-performance inference engine for Large Language Models, designed as a compact reference implementation of the larger SGLang project. It provides an OpenAI-compatible API server, an interactive shell, and a programmatic Python interface. The engine supports advanced optimizations including tensor parallelism (via PyNCCL or NCCL), chunked prefill, radix cache for KV reuse, CUDA graph capture for reduced launch overhead, and multiple attention backends (FlashAttention, FlashInfer, TRT-LLM). It uses a distributed architecture with separate processes for the API server, tokenizer/detokenizer, and per-GPU scheduler workers, communicating via ZeroMQ for control messages and NCCL for tensor data. The codebase is ~5,000 lines of Python with custom CUDA kernels JIT-compiled via TVM-FFI, making it both performant and educational.
This repository is a full-stack, multi-tenant AI API gateway platform (named 'sub2api') that proxies requests to a wide range of AI model providers including OpenAI, Anthropic, Gemini, Grok, DeepSeek, and many others. It provides authentication (JWT, OAuth, API keys), rate limiting, token-level billing, subscription management, user and admin management, payment processing (Alipay, WeChat Pay, Stripe, Airwallex, EasyPay), content moderation, channel monitoring, and operational dashboards. The backend is written in Go using the Gin framework and Ent ORM, with PostgreSQL and Redis for data storage and caching. The frontend is built with Vue 3, TypeScript, and Pinia stores, offering both admin and user interfaces. The system is designed for production deployment with Docker, includes extensive unit and integration tests, and supports a plugin architecture for extensible OAuth transport. Despite its large codebase and many features, the architecture follows a clear service-repository pattern with well-isolated packages, making it a solid example of a complex API gateway implementation.
fastllm is a high-performance LLM inference engine written in C++ with CUDA/CPU backends, designed for local and server deployment of large language models. In production, it loads models from HuggingFace, GGUF, and custom formats, then executes inference with advanced features: quantization (FP8, INT4, NVFP4, GGUF), speculative decoding, multi-token prediction (MTP), Mixture-of-Experts (MoE), paged KV cache, tensor parallelism, NUMA-aware execution, and disk offloading. It serves models through multiple interfaces: a C++ CLI, a Python CLI (ftllm), an OpenAI-compatible HTTP API server (Chat Completions, Responses, Anthropic Messages), a WebUI, a desktop launcher with agent integrations (DeepSeek Harness, OpenCode, Codex, Claude Code), and an Android app. The system also includes model conversion/export tools, a launcher with hardware auto-recommendation, and a plugin system for UI customization.
This repository provides a TypeScript SDK for the Dedalus API and an MCP (Model Context Protocol) server that exposes the API as MCP tools. The SDK is generated from an OpenAPI specification using Stainless, offering type-safe access to AI model functionalities including chat completions, embeddings, image generation, and audio processing. It includes features such as automatic retries with exponential backoff, streaming responses, file upload handling, and structured output parsing via Zod schemas. The MCP server implementation allows Dedalus API endpoints to be used as tools by MCP-compatible clients like Claude and Cursor. It includes a compatibility layer that transforms JSON schemas to accommodate various client limitations (e.g., handling unions, inlining references, truncating tool names). The server supports both stdio and HTTP transports and can be extended with custom tools. Key architectural elements include a resource-based client structure, a robust request/response transformation system, and a modular MCP server design. The codebase demonstrates strong engineering practices with comprehensive testing, active maintenance, and clear separation of concerns between the SDK and server components.
free-claude-code is a local proxy and developer tool that connects coding agents (Claude Code and similar harnesses) to OpenAI-compatible AI providers. It is a Python package (`free_claude_code`) with a CLI, a local API server, an admin web UI, and a substantial configuration subsystem. The system abstracts over a dozen AI providers (including NVIDIA NIM, Cloudflare, HuggingFace, and others) with provider-specific workarounds for tool parameter aliasing, thinking templates, and reasoning replay. It persists sessions, messages, and model metadata in SQLite, manages a tree-based conversation model with Telegram platform support, and renders transcripts with segment-based truncation and subagent state management. The repo is a shipped developer tool with installer/update scripts, GitHub issue templates, dependabot config, and a diagnostic `fcc-doctor` command; it is not a research or data project.
This repository provides a lightweight CLI tool (`vision.js`) that enables text-only AI models (like Claude Code without native vision) to 'see' images by converting them to text via an external vision API. The core script reads images from local files, URLs, or the system clipboard (via platform-specific helpers for macOS and Windows), encodes them as base64, and sends them to any OpenAI-compatible vision API (defaulting to Alibaba Cloud's DashScope). It features automatic clipboard fallback, configurable API credentials via environment variables, and clear error handling. The repo also includes documentation files (`AGENTS.md`, `CLAUDE.md`, `SKILL.md`) that instruct AI agents on when and how to invoke the script, plus integration instructions for the cyberboss chatbot framework. It is a practical, well-structured utility that solves a common limitation of text-only AI models.
GCLI2API is a production-grade API proxy that converts Google Gemini CLI and Antigravity credentials into OpenAI, Gemini, and Claude compatible API endpoints. It manages OAuth credentials with automatic rotation, error-based cooldowns, and load balancing, providing a unified interface for multiple backend services. The system includes a web-based control panel for credential management, real-time logs, configuration editing, and batch operations. The architecture is a FastAPI web application with modular routers for each API format (OpenAI, Gemini, Claude) and two backend modes (geminicli, antigravity). Format converters handle bidirectional translation between Anthropic, OpenAI, and Gemini schemas, including tool calls, streaming, and thinking blocks. Storage is abstracted via an adapter pattern supporting SQLite, MongoDB, and PostgreSQL, with optional Redis caching for credential pools. Key features include streaming support with fake streaming fallback, anti-truncation via synthetic tool calls, thought signature preservation across format conversions, and a keepalive service. The system relies on httpx for async HTTP, Pydantic for data models, and external Google Cloud APIs for OAuth and quota management. Deployment is supported via Docker, docker-compose, and platform-specific install scripts for Linux, macOS, Windows, and Termux. This is a pure application with no data research or ML components. It demonstrates strong engineering practices in credential management, format conversion, and async I/O, though it suffers from some code duplication between backend modes and limited test coverage.
This Rust codebase is an LLM inference proxy and tool-routing gateway. It exposes an OpenAI-compatible HTTP API (`/v1/chat/completions`, `/v1/models`) and forwards requests to multiple upstream LLM providers (OpenAI, Anthropic via direct/Bedrock, Vertex AI, vLLM DeepSeek, TokenHub) through a unified Provider trait. A QUIC server (s2n-quic, TLS 1.3) accepts connections from serverless tools written in Go or Node.js; the agent loop performs multi-round tool calling, invokes registered tools, and streams results back in SSE format. A separate model API proxy layer handles non-chat endpoint shapes (messages, responses, generateContent) with usage extraction and injection. Architecturally, the system is a Rust CLI binary (`yomo`) with three subcommands: `serve` starts the HTTP API and QUIC zipper, `init` scaffolds serverless tool projects, and `run` launches a tool subprocess and connects it to the zipper. Tool registration is done at runtime over QUIC and stored in-memory, so state is lost on restart. The HTTP layer is Axum, auth is bearer-token, and observability uses OpenTelemetry. The serverless execution model is subprocess-based: it creates temporary directories, runs `npm install`/`go mod tidy`, and exchanges metadata over stdout. Reusable engineering exists in the framing protocol (`io`), streaming SSE mappers, usage aggregation logic, TLS configuration, and provider error separation. However, the provider implementations are duplicated and brittle to upstream API drift; hard-coded model limits, ephemeral tool state, and near-empty serverless stubs reduce long-term maintainability. No integration tests against real upstream APIs were found in the reviewed surface.