LiteLLM is a production-grade, open-source proxy/gateway for over 200 large language model (LLM) providers, including OpenAI, Anthropic, Azure, Bedrock, Gemini, and many more. It presents a unified OpenAI-compatible API, handling request routing, authentication, rate limiting, cost tracking, caching, guardrails (content filtering, PII detection), and audit logging. The system is architected as a FastAPI Python server with a Next.js admin dashboard, Rust extensions for performance-critical paths (token counting, caching, auth, cost calculation), and optional PostgreSQL/Redis backends. It supports streaming, MCP (Model Context Protocol), A2A (Agent-to-Agent) protocol, and enterprise features like SSO, secret management, and Terraform deployment modules. Key entry points include the proxy server (litellm/proxy/proxy_server.py), CLI commands (litellm, litellm-proxy), and the admin UI at /ui/litellm-dashboard. The primary data flow is from client requests (chat completions, embeddings, audio, images) received by the proxy, which authenticates, applies guardrails, routes to the configured LLM provider(s), logs usage, and returns a standardized response. Caching is multi-tier (Redis, disk, S3, GCS, Azure Blob, semantic). Spend tracking is aggregated into daily tables and exposed via the management API and UI. External dependencies include PostgreSQL (via Prisma ORM), Redis (for distributed caching, rate limiting, and queuing), and cloud-specific secret managers (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, HashiCorp Vault). The project uses uv for Python dependency management, Cargo for Rust crates, and npm/pnpm for the dashboard. Extensive CI/CD pipelines (CircleCI, GitHub Actions) run unit, integration, e2e, mutation, and Rust-Python parity tests.
MLflow is an open-source ML lifecycle platform that provides experiment tracking, model registry, model deployment, evaluation, and observability for both classical ML and GenAI/LLM workflows. In production, it operates as a modular monolith: a Python backend (Flask/FastAPI) with SQLAlchemy-backed storage (SQLite, PostgreSQL, MySQL, MSSQL), a React/TypeScript web UI, and client SDKs for Python, R, Java, and TypeScript. The system exposes a REST API and CLI for logging experiments, registering models, collecting traces via OpenTelemetry, and managing an AI Gateway that routes requests to dozens of LLM providers with budget controls, guardrails, and traffic splitting. The architecture includes a plugin system for custom store backends, deployment targets, and authentication providers, as well as a custom linting tool (clint) and extensive CI/CD pipelines. Key entry points are the `mlflow` CLI, `MlflowClient` Python API, tracking server (`mlflow server`), gateway server (`mlflow gateway`), and model scoring server (`mlflow models serve`). Data flows from user instrumentation through asynchronous logging to the backend store, with traces and evaluations queryable via the UI or API. External dependencies span ML frameworks (scikit-learn, PyTorch, TensorFlow, LangChain, etc.), LLM provider SDKs, and infrastructure tools (Docker, Kubernetes, Helm).
Langfuse is an open-source LLM observability and evaluation platform. It provides tracing, prompt management, evaluations, datasets, experiments, monitors, and a public API for integrating with AI applications. The architecture is a monorepo with pnpm workspaces and Turborepo, consisting of a Next.js frontend (web), a background worker (Express + BullMQ), a shared package with types and server utilities, and an enterprise edition. Data flows from ingestion (REST API, OpenTelemetry, SDKs) into ClickHouse for high-volume analytics and PostgreSQL for relational metadata. Background jobs handle evaluations, batch exports, cloud billing, and integrations. The codebase is well-structured with extensive testing, a custom ESLint plugin, and reusable UI components. It is actively maintained with frequent migrations and feature additions.
Composio is a production-grade, dual-language SDK monorepo that connects AI agents to external services (Gmail, GitHub, Slack, Notion, Salesforce, and hundreds more) through a unified tool abstraction. In production it operates as the client layer of the Composio platform: applications create a Composio session bound to a user_id, discover toolkits, manage OAuth auth-configs and connected accounts, and execute tool calls either remotely or via a session-scoped Tool Router that also exposes hosted MCP endpoints. The repository ships a TypeScript SDK (@composio/core), a Python SDK (composio), a standalone Effect-TS-based CLI, 15+ framework provider adapters (OpenAI, Anthropic, LangChain, LlamaIndex, CrewAI, AutoGen, Mastra, Vercel AI, Google/ADK, etc.), and a Next.js documentation site. It is unambiguously an application/developer-platform repo rather than data research, although it contains data-adjacent engineering assets: a cross-SDK JSON Schema conversion corpus, OpenAPI dereferencing tooling, a knowledge-base hybrid-search index, and telemetry redaction utilities.
BentoML is a machine-learning model serving framework and deployment platform. In production it lets developers define inference services via a decorator-based SDK (`@bentoml.service`, `@bentoml.api`), package those services into versioned "Bentos" with locked Python dependencies, and build OCI container images or deploy to Kubernetes, EC2, or BentoCloud. The serving layer supports both HTTP (Starlette/Uvicorn) and gRPC, with adaptive batching, model runners, Prometheus metrics, OpenTelemetry tracing, and pluggable monitoring. A comprehensive CLI (`bentoml serve/build/containerize/deploy`) is the primary operational surface, while the `_bentoml_sdk` and `_bentoml_impl` packages provide the user-facing and internal implementation layers respectively. Architecturally, the project separates concern into model registry, service definition, packaging, server workers, and cloud clients. Model artifacts are persisted in a versioned local store via framework adapters for PyTorch, TensorFlow, Transformers, ONNX, scikit-learn, MLflow, XGBoost, LightGBM, and others. Data flows from trained model artifacts into the registry, get referenced by service code, get bundled by `bentoml build` into self-contained Bento packages, and are served by Circus-managed worker processes. The core request path involves serialization, batching, resource allocation, and process supervision, all implemented in internal modules with generally clean abstractions. The repository is mid-migration from a legacy `bentoml.legacy.Service` API to a newer `_bentoml_sdk`, which creates visible duplication: two HTTP client stacks, multiple protobuf versions, and deprecation shims. This is a real maintenance cost but not a functional blocker; the modern SDK is actively developed, documented, and tested. The repo is clearly an application/framework rather than a data-research project; research-like artifacts are limited to test fixtures, training examples, and ML framework integration tests.
This repository is a collection of deployable templates and educational cookbooks for building real-time LLM applications using the Pathway framework. It provides ready-to-deploy RAG (Retrieval-Augmented Generation) solutions with live data synchronization from various sources (files, Google Drive, SharePoint, etc.), built-in vector and hybrid search, and integration with LangGraph for agentic workflows. The templates are production-oriented with Docker support, while the cookbooks demonstrate advanced patterns like Self-RAG with self-reflection and query rewriting. The codebase is well-structured but exhibits some duplication across templates, and the core logic relies heavily on the external Pathway library.
llm-action is a comprehensive resource hub for large language model engineering that combines theoretical explanations with practical implementation code. It covers the full LLM lifecycle including data engineering, training (full fine-tuning and parameter-efficient methods like LoRA/QLoRA), model compression (quantization, pruning, distillation), inference serving, deployment, and evaluation. The repository contains both educational documentation and runnable code examples, with particular emphasis on Chinese-language resources and domestic AI hardware adaptation (especially Huawei Ascend NPUs). While it serves as a knowledge base, its inclusion of actual training scripts, inference servers, and utility tools makes it primarily application-oriented rather than purely research-focused.