LiteLLM is a production-grade, open-source proxy/gateway for over 200 large language model (LLM) providers, including OpenAI, Anthropic, Azure, Bedrock, Gemini, and many more. It presents a unified OpenAI-compatible API, handling request routing, authentication, rate limiting, cost tracking, caching, guardrails (content filtering, PII detection), and audit logging. The system is architected as a FastAPI Python server with a Next.js admin dashboard, Rust extensions for performance-critical paths (token counting, caching, auth, cost calculation), and optional PostgreSQL/Redis backends. It supports streaming, MCP (Model Context Protocol), A2A (Agent-to-Agent) protocol, and enterprise features like SSO, secret management, and Terraform deployment modules. Key entry points include the proxy server (litellm/proxy/proxy_server.py), CLI commands (litellm, litellm-proxy), and the admin UI at /ui/litellm-dashboard. The primary data flow is from client requests (chat completions, embeddings, audio, images) received by the proxy, which authenticates, applies guardrails, routes to the configured LLM provider(s), logs usage, and returns a standardized response. Caching is multi-tier (Redis, disk, S3, GCS, Azure Blob, semantic). Spend tracking is aggregated into daily tables and exposed via the management API and UI. External dependencies include PostgreSQL (via Prisma ORM), Redis (for distributed caching, rate limiting, and queuing), and cloud-specific secret managers (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, HashiCorp Vault). The project uses uv for Python dependency management, Cargo for Rust crates, and npm/pnpm for the dashboard. Extensive CI/CD pipelines (CircleCI, GitHub Actions) run unit, integration, e2e, mutation, and Rust-Python parity tests.
This repository provides a production-ready, full-stack Retrieval-Augmented Generation (RAG) chat application built with Python FastAPI and React TypeScript. It demonstrates how to use Azure OpenAI (or OpenAI.com, or Ollama) to answer questions about data stored in Azure Database for PostgreSQL with the pgvector extension. The system supports both a simple RAG flow (embedding + direct search) and an advanced flow that uses OpenAI function calling to rewrite queries and apply column filters (brand, price). It features hybrid search combining full-text search and vector similarity via Reciprocal Rank Fusion. The backend is a monolith serving API endpoints and a React frontend, designed for deployment to Azure Container Apps using the Azure Developer CLI. The codebase includes comprehensive unit tests, end-to-end tests with Playwright, evaluation scripts, load testing with locust, and safety evaluation using Azure AI Red Teaming. The architecture is cleanly separated into reusable modules, making it a solid reference architecture for building RAG applications on Azure.
SuperAgentX is a modular, open-source multi-agent AI framework designed for orchestrating autonomous agents that plan, act, and execute real-world workflows. It supports 100+ LLM providers (OpenAI, Azure, Gemini, Claude, Bedrock, Ollama, LiteLLM) and 10,000+ MCP tools via the Model Context Protocol. The architecture centers on `AgentXPipe` pipelines that sequence or parallelize agents, each composed of an `Engine` that wraps an LLM client and a set of tool handlers. Handlers are defined using a `@tool` decorator pattern that auto-generates LLM function schemas. The framework includes built-in human-in-the-loop approval channels (console, Slack, Teams), persistent memory with SQLite and vector stores (Chroma, OpenSearch), full observability via OpenTelemetry and SQL-based telemetry, and a CLI for project scaffolding. It also features a production-grade browser automation engine based on Playwright for tasks where APIs are unavailable.
This is a sample application from Microsoft demonstrating how to build a chatbot with Azure OpenAI and your own data using Retrieval-Augmented Generation (RAG). It is a full-stack web application with a Python Quart async backend and a React/TypeScript frontend. The backend handles conversation orchestration, integrates with Azure OpenAI for chat completions, and supports multiple data sources (Azure Cognitive Search, CosmosDB, Elasticsearch, Pinecone, etc.) for grounding responses. It also includes a comprehensive data preparation pipeline (scripts) for chunking documents, generating embeddings, and indexing into search services. The frontend provides a modern chat interface with features like streaming responses, citation display, chat history, and file upload. The repository is primarily an application but has a strong data engineering component, making it a hybrid. It is actively maintained, well-structured with tests and CI/CD, but has some technical drift due to preview APIs and outdated dependencies.