OGX (formerly Llama Stack) is a production-grade, open-source AI inference platform that provides a drop-in replacement for the OpenAI API. It is a multi-provider API gateway supporting chat completions, embeddings, the Responses API (agentic turns with tool use), vector stores, file processing, conversations, prompts, skills, and connectors. The system is built on a clean provider plugin architecture where inline (built-in) and remote providers (30+ including Anthropic, Bedrock, Gemini, Ollama, vLLM, NVIDIA) are registered via ProviderSpec and dynamically routed. The FastAPI server auto-discovers API protocols and generates routes with telemetry (OpenTelemetry), authentication, multi-tenancy, and ABAC access control. The repository includes a Next.js admin UI, a Python CLI, OpenAPI-generated client SDKs (Python, TypeScript), benchmarking tools (RAG eval, vertical scaling), and comprehensive integration tests with recording/replay. It is designed for self-hosted or cloud deployment with Kubernetes and Docker support.
This repository provides a high-performance, production-grade inference server for text embeddings and sequence classification models, known as Text Embeddings Inference (TEI). The system is implemented primarily in Rust with a modular architecture comprising a router (HTTP/gRPC), a core layer for batching, tokenization, and inference orchestration, and multiple backends (Candle, ONNX Runtime, Python/PyTorch) that can be selected at compile time. The router exposes a comprehensive REST API (including OpenAI-compatible endpoints) and a gRPC service, with support for dynamic token-based batching, Prometheus metrics, OpenTelemetry tracing, and authentication. The Candle backend is heavily optimized with custom CUDA kernels (flash attention variants, cuBLASLt linear layers, fused norms) and supports a wide range of transformer architectures (BERT, ModernBERT, Mistral, Qwen2/3, Nomic, GTE, Jina, etc.). The codebase demonstrates strong engineering practices with snapshot testing, integration tests, load tests, Docker multi-arch builds, and CI/CD pipelines. The primary data flow is: client request -> router -> core inference queue (token-based batching) -> backend (model forward pass) -> pooled/raw embeddings or classification scores returned. No data research aspects are present; this is a pure application deployment of inference models.
LMFlow is a comprehensive toolkit for fine-tuning, aligning, and deploying large foundation models. It provides a modular library with support for various efficient fine-tuning methods (LoRA, QLoRA, LISA), alignment techniques (DPO, RAFT), and inference backends (vLLM, SGLang). The architecture separates concerns into models, pipelines, optimizers, and utilities, with factory patterns for extensibility. Key features include conversation templating for multiple model families, custom optimizers, and distributed training integration. The toolkit is actively maintained with recent commits and addresses current research in model alignment and efficiency.
PMetal is a comprehensive machine learning framework specifically optimized for Apple Silicon hardware. It provides a complete suite for LLM training, fine-tuning, inference, and deployment through multiple interfaces (CLI, TUI, GUI, Python SDK, and REST API). The framework leverages Apple-specific technologies like Metal GPU, ANE, and UltraFusion for performance optimization while supporting standard ML workflows including LoRA/QLoRA fine-tuning, knowledge distillation, model merging, and quantization to formats like GGUF. The architecture follows a modular workspace design with over 20 specialized crates handling different aspects of the ML lifecycle, unified through a job specification system and configuration framework.
RUMUS is a native-Rust deep learning framework with a modular architecture including core tensor/autograd library, GPU backend via WebGPU (wgpu), neural network modules, optimizers, data loading, ONNX export, JIT kernel fusion, and subcrates for vision (CNN ops, INT4 quantization), graph neural networks (SpMM), distributed training (3D parallelism), and an inference server with continuous batching. The framework emphasizes memory safety, zero-copy view operations, borrow-safe optimizers (gradient draining mechanism), and supports both CPU and GPU computation. It includes advanced features like FlashAttention, block-quantized INT8/INT4 inference, deterministic fixed-point inference for zero-knowledge backend fixtures, and a custom ops plugin API for external WGSL kernels. The codebase is well-organized with clean abstractions (Backend trait, Module derive macro), but shows signs of incomplete pieces (distributed training may be partial, inference server model is mock) and minor code-README discrepancies (unsafe usage in mmap despite claims of zero unsafe).