thehardwarestore.ai

LiteLLM is a production-grade, open-source proxy/gateway for over 200 large language model (LLM) providers, including OpenAI, Anthropic, Azure, Bedrock, Gemini, and many more. It presents a unified OpenAI-compatible API, handling request routing, authentication, rate limiting, cost tracking, caching, guardrails (content filtering, PII detection), and audit logging. The system is architected as a FastAPI Python server with a Next.js admin dashboard, Rust extensions for performance-critical paths (token counting, caching, auth, cost calculation), and optional PostgreSQL/Redis backends. It supports streaming, MCP (Model Context Protocol), A2A (Agent-to-Agent) protocol, and enterprise features like SSO, secret management, and Terraform deployment modules. Key entry points include the proxy server (litellm/proxy/proxy_server.py), CLI commands (litellm, litellm-proxy), and the admin UI at /ui/litellm-dashboard. The primary data flow is from client requests (chat completions, embeddings, audio, images) received by the proxy, which authenticates, applies guardrails, routes to the configured LLM provider(s), logs usage, and returns a standardized response. Caching is multi-tier (Redis, disk, S3, GCS, Azure Blob, semantic). Spend tracking is aggregated into daily tables and exposed via the management API and UI. External dependencies include PostgreSQL (via Prisma ORM), Redis (for distributed caching, rate limiting, and queuing), and cloud-specific secret managers (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, HashiCorp Vault). The project uses uv for Python dependency management, Cargo for Rust crates, and npm/pnpm for the dashboard. Extensive CI/CD pipelines (CircleCI, GitHub Actions) run unit, integration, e2e, mutation, and Rust-Python parity tests.

This repository implements an OpenAI-compatible REST API proxy for Amazon Bedrock, enabling applications built for OpenAI SDKs to seamlessly use Bedrock foundation models without code changes. The FastAPI-based service translates standard OpenAI API calls (chat completions, embeddings, models listing) into Amazon Bedrock runtime invocations and returns responses in OpenAI-compatible format. The architecture supports two deployment options: Lambda with API Gateway (serverless, cost-effective) or Fargate with ALB (lowest latency, no cold starts). Key features include streaming responses via SSE, tool/function calling, multimodal inputs, prompt caching for Claude/Nova models, reasoning mode for Claude 3.7 and DeepSeek R1, and interleaved thinking for Claude 4 models. Infrastructure is managed via CloudFormation templates with proper IAM roles and Secrets Manager integration for API key authentication. The codebase demonstrates solid engineering practices with clean separation of concerns, proper error handling, and comprehensive feature support. However, the absence of test files and limited documentation reduce overall quality. The project is actively maintained with recent commits and up-to-date dependencies.

SuperAgentX is a modular, open-source multi-agent AI framework designed for orchestrating autonomous agents that plan, act, and execute real-world workflows. It supports 100+ LLM providers (OpenAI, Azure, Gemini, Claude, Bedrock, Ollama, LiteLLM) and 10,000+ MCP tools via the Model Context Protocol. The architecture centers on `AgentXPipe` pipelines that sequence or parallelize agents, each composed of an `Engine` that wraps an LLM client and a set of tool handlers. Handlers are defined using a `@tool` decorator pattern that auto-generates LLM function schemas. The framework includes built-in human-in-the-loop approval channels (console, Slack, Teams), persistent memory with SQLite and vector stores (Chroma, OpenSearch), full observability via OpenTelemetry and SQL-based telemetry, and a CLI for project scaffolding. It also features a production-grade browser automation engine based on Playwright for tasks where APIs are unavailable.