MLflow is an open-source ML lifecycle platform that provides experiment tracking, model registry, model deployment, evaluation, and observability for both classical ML and GenAI/LLM workflows. In production, it operates as a modular monolith: a Python backend (Flask/FastAPI) with SQLAlchemy-backed storage (SQLite, PostgreSQL, MySQL, MSSQL), a React/TypeScript web UI, and client SDKs for Python, R, Java, and TypeScript. The system exposes a REST API and CLI for logging experiments, registering models, collecting traces via OpenTelemetry, and managing an AI Gateway that routes requests to dozens of LLM providers with budget controls, guardrails, and traffic splitting. The architecture includes a plugin system for custom store backends, deployment targets, and authentication providers, as well as a custom linting tool (clint) and extensive CI/CD pipelines. Key entry points are the `mlflow` CLI, `MlflowClient` Python API, tracking server (`mlflow server`), gateway server (`mlflow gateway`), and model scoring server (`mlflow models serve`). Data flows from user instrumentation through asynchronous logging to the backend store, with traces and evaluations queryable via the UI or API. External dependencies span ML frameworks (scikit-learn, PyTorch, TensorFlow, LangChain, etc.), LLM provider SDKs, and infrastructure tools (Docker, Kubernetes, Helm).
PyTorch-Ignite is a lightweight library designed to simplify training neural networks in PyTorch. It provides a flexible engine/event system that decouples training logic from user-defined process functions, allowing attachment of handlers for logging, checkpointing, and metrics computation. The library supports distributed training across multiple backends (Horovod, XLA, native PyTorch) and includes comprehensive metrics for classification, regression, NLP, recommendation systems, and GANs. Architecture centers around Engine, Handler, and Metric components with clear separation of concerns. Key entry points include ignite.engine.Engine, ignite.handlers.* loggers, ignite.metrics.* metrics, and ignite.distributed utilities for distributed training abstraction.
H2O LLM Studio is a web-based application for fine-tuning large language models (LLMs) without requiring coding experience. It provides a GUI built on H2O Wave and CLI tools for dataset import, experiment configuration, model training, evaluation, and export to Hugging Face Hub. The system supports multiple problem types including causal language modeling, classification, regression, sequence-to-sequence, and Direct Preference Optimization (DPO). The architecture centers around a Wave server handling both UI and backend logic, with distributed training capabilities via DeepSpeed. Key components include a configuration system using dataclasses, utilities for conversation chain handling, model loading abstractions, and factory patterns for training components.
LUML is an agent-native MLOps platform that orchestrates the full machine learning lifecycle: experiment tracking, model packaging, registry, deployment, monitoring, and AI-mediated code generation. The monorepo contains multiple services: a FastAPI backend (multi-tenant platform API with PostgreSQL), a Vue 3 frontend (SPA with Pinia, PrimeVue), a satellite runtime for edge deployment and monitoring (Docker/Kubernetes), an SDK for experiment tracking and model packaging (supporting sklearn, XGBoost, LangGraph, LLM evaluation), a Prisma service for agent-driven code generation (Claude/Codex orchestrator with git worktree isolation), a local experiment tracker (lumlflow) with both a Textual TUI and web interface, and WASM-based in-browser ML workers (prompt optimization, tabular forecasting). The architecture is microservice-oriented with a shared SDK and wire contracts. The platform is production-grade with comprehensive testing (unit, integration, E2E), CI/CD pipelines, OpenAPI contracts, and a well-defined RBAC permission matrix. It demonstrates high craftsmanship in areas like monitoring drift computation (PSI, multivariate), WebWorker RPC proxies, and LLM judge scoring, though the sheer size and number of packages introduce some coordination complexity.
Triage is a Python framework for end-to-end machine learning workflows focused on temporal prediction problems, particularly in social good domains. It provides modular components for feature generation (architect), model training/evaluation (catwalk), experiment orchestration, model selection (audition), and post-modeling analysis. The system uses PostgreSQL for storing experiment metadata, feature matrices, and model predictions, with SQLAlchemy ORM and Alembic migrations managing the schema. Key entry points include the CLI (`triage experiment`, `triage audition`) and `manage.py` for development tasks. The architecture emphasizes reproducibility through configuration-driven experiments, temporal validation to prevent lookahead bias, and metadata tracking. While powerful, the system has complexity from multiple interdependent components and external service dependencies (PostgreSQL, Redis for RQ).
This repository contains three distinct research projects focused on language model development: pre-training (Autolab large-scale autoregressive model training), post-training (NanoChat synthetic task fine-tuning), and inference (llama.cpp optimization benchmarking). Each project follows a disciplined experimentation framework with Hugging Face Jobs integration, experiment tracking via TSV ledgers, and isolated variable modification policies. The pre-training project trains custom GPT architectures with value embeddings and MuonAdamW optimizer on climbmix-400b dataset, evaluating via bits-per-byte. Post-training uses a fixed NanoChat architecture for synthetic math/tool-use/multiple-choice tasks with staged SFT+RL training. Inference benchmarks GGUF model throughput via llama-server OpenAI-compatible API. All projects share common patterns for experiment workflow, HF job management, and results tracking while maintaining strict separation between research domains.