This repository is a specialized library of distributed orthonormal optimizers for large-scale language model training, implementing the Dion family of algorithms (Dion, Dion2, Dion3, NorMuon, Muon) alongside traditional optimizers (AdamW, Lion). It provides a complete training pipeline including a GPT model, data loaders for FineWeb datasets, and configurable parallelism strategies (FSDP2, TP, DDP). The core innovation lies in communication-efficient orthogonal gradient updates via submatrix selection, Newton-Schulz iterations, and polar decomposition, all wrapped in an async megabatch framework that overlaps collectives with computation. The codebase is well-structured with modular optimizer implementations, extensive tests (including distributed process spawning), and CUDA graph capture support. It is primarily an application library for ML practitioners, but retains a research character through its multiple algorithm variants and academic references. The high engineering quality and active maintenance make it suitable for both production use and further research extensions.
FlexMeasures is a Flask-based energy flexibility management platform that ingests time-series sensor data, forecasts future values, schedules flexible assets (batteries, heat pumps, industrial processes) using linear optimization, and reports on performance. It provides a REST API (v3.0), a web UI with interactive charts, a CLI for data management, and background job queues (RQ) for forecasting, scheduling, and automation. The architecture is a multi-tenant web application with PostgreSQL, Redis, and optional Sentry monitoring. It features a plugin system for custom data generators (forecasters, schedulers, reporters) and automation handlers. The codebase is well-tested with extensive unit and integration tests, and follows modern Python practices (type hints, Marshmallow schemas, Click CLI).
This repository is the official companion code for the book *Build a Large Language Model (From Scratch)* by Sebastian Raschka (105k+ stars). It provides a step-by-step, educational implementation of GPT-2 and its modern variants (Llama 3, Qwen3, Gemma 3/4, Olmo3) entirely in PyTorch, organized by book chapters (2–7 plus appendices). The codebase is a mix of Jupyter notebooks for interactive learning and Python modules that form a reusable pip package (`llms-from-scratch`), accompanied by extensive CI/CD pipelines, unit tests, and performance analysis scripts. Architecturally, the core is a modular transformer library that includes tokenization (BPE, tiktoken, custom), multi-head attention with multiple variants (GQA, MLA, SWA, DSA, cross-layer KV sharing), feedforward blocks, normalization (LayerNorm, RMSNorm), and LoRA. The code supports pretraining from scratch, classification finetuning (spam, IMDB), instruction finetuning (with/without LoRA), and preference tuning (DPO). It can load pretrained weights from OpenAI, Hugging Face, and local checkpoints. The package exposes clean interfaces (`GPTModel`, `generate`, `train_model_simple`) and includes bonus materials for KV cache implementations, mixture-of-experts, memory estimators, and user interfaces (Chainlit). While primarily educational, the code exhibits production-quality patterns: isolated attention modules, configurable model architecture, well-structured training loops, and robust weight loading with shape validation. The repository is actively maintained with recent commits, cross-platform support (Linux, macOS, Windows), and a strong emphasis on reproducibility (fixed seeds, deterministic settings, environment definitions via pixi/uv/conda). Duplication exists across bonus notebooks and a few hardcoded paths, but the core package and test suite are well-coordinated.
This repository implements a low-latency, real-time speech-to-speech conversational AI system. It provides an OpenAI Realtime API-compatible server over WebSocket and WebRTC transports, along with a CLI tool for local, talk, and serve modes. The architecture is modular, with separate handler stages for Voice Activity Detection (VAD), Speech-to-Text (STT), Large Language Model (LLM) reasoning, and Text-to-Speech (TTS), connected via typed queues. Multiple backends are supported for each stage, including local models (Whisper, Parakeet, Kokoro, Qwen3-TTS, MLX) and cloud APIs (OpenAI, vLLM, Hugging Face Inference Providers). A demo web application with HF OAuth, rate limiting, and audio worklets is included for browser-based interaction. The system is designed for production use, with emphasis on low latency through speculative turn handling, cancellation scopes, and turn latency instrumentation. Extensive test suites cover unit, integration, and protocol contract tests, including an evaluation harness for Big Bench Audio. While primarily an application, the repo also includes data/research components such as benchmarking scripts and a custom speech evaluation dataset.
Gorse is a production-grade, open-source recommender system engine written in Go. It implements a master/server/worker distributed architecture: the master coordinates training and cluster state, workers generate offline recommendations, and servers expose a REST API, dashboard, and gRPC endpoints. The system supports collaborative filtering via BPR and ALS matrix factorization, click-through rate prediction via an Attentional Factorization Machine, item-to-item/user-to-user similarity based on embeddings, non-personalized leaderboards, external script recommenders, and LLM-based reranking/chat recommenders. It includes a complete data model for users, items, feedback, and labels, with storage backends for SQL databases, MongoDB, Redis, and vector stores (xvec, Qdrant, Weaviate, Milvus), plus blob storage for model artifacts. The codebase is organized around strong abstractions: storage interfaces, protobuf-based services, a recommendation logic layer, model training/evaluation packages, and a common utility layer containing an autograd tensor library, SIMD-optimized float kernels, HNSW vector search, and concurrency helpers. It exposes admin/client CLIs, JSONL import/export, hyperparameter optimization with Optuna, monitoring metrics, and OpenTelemetry tracing. Although this repository contains ML training, benchmarks, and data-loading code, it is fundamentally an application platform rather than a data-research repository. The engineering is strong but carries complexity from supporting many storage backends, architecture-specific SIMD generated code, and a large monolithic master task file; some deprecated APIs and generated-code drift are present.
This is the official PyTorch tutorials repository, a Sphinx-Gallery documentation site that publishes to pytorch.org/tutorials. It contains hundreds of executable tutorial scripts covering beginner to advanced PyTorch topics (neural networks, distributed training, torch.compile, performance profiling, custom operators, etc.). The repository is not a standalone application but a documentation generator with sophisticated CI/CD pipelines for building, sharding, and testing tutorials across multiple GPU configurations. It includes extensive build tooling (Docker, Jenkins, GitHub Actions), linters, spell checking, link validation, and redirect management. The code within tutorial files is pedagogical and not production-ready; however, several utility scripts and patterns are reusable for documentation projects and PyTorch extension development.
BentoML is a machine-learning model serving framework and deployment platform. In production it lets developers define inference services via a decorator-based SDK (`@bentoml.service`, `@bentoml.api`), package those services into versioned "Bentos" with locked Python dependencies, and build OCI container images or deploy to Kubernetes, EC2, or BentoCloud. The serving layer supports both HTTP (Starlette/Uvicorn) and gRPC, with adaptive batching, model runners, Prometheus metrics, OpenTelemetry tracing, and pluggable monitoring. A comprehensive CLI (`bentoml serve/build/containerize/deploy`) is the primary operational surface, while the `_bentoml_sdk` and `_bentoml_impl` packages provide the user-facing and internal implementation layers respectively. Architecturally, the project separates concern into model registry, service definition, packaging, server workers, and cloud clients. Model artifacts are persisted in a versioned local store via framework adapters for PyTorch, TensorFlow, Transformers, ONNX, scikit-learn, MLflow, XGBoost, LightGBM, and others. Data flows from trained model artifacts into the registry, get referenced by service code, get bundled by `bentoml build` into self-contained Bento packages, and are served by Circus-managed worker processes. The core request path involves serialization, batching, resource allocation, and process supervision, all implemented in internal modules with generally clean abstractions. The repository is mid-migration from a legacy `bentoml.legacy.Service` API to a newer `_bentoml_sdk`, which creates visible duplication: two HTTP client stacks, multiple protobuf versions, and deprecation shims. This is a real maintenance cost but not a functional blocker; the modern SDK is actively developed, documented, and tested. The repo is clearly an application/framework rather than a data-research project; research-like artifacts are limited to test fixtures, training examples, and ML framework integration tests.
Deep Java Library (DJL) is a comprehensive, engine-agnostic deep learning framework for Java, providing a unified API for model loading, training, inference, and deployment across multiple backends (PyTorch, TensorFlow, MXNet, ONNX Runtime, XGBoost, LightGBM, and a Rust-based engine). Its architecture is modular: a core API defines abstractions like `NDArray`, `NDManager`, `Model`, `Predictor`, `Trainer`, and `Translator`; engine-specific implementations adapt these to native libraries; model zoos and repositories supply pre-trained models; and extensions add capabilities for generative AI, audio, timeseries, Spark integration, and more. The codebase is well-organized with extensive use of the builder pattern, ServiceLoader-based plugin discovery, and a rich set of training utilities (losses, optimizers, trackers, evaluators, listeners). It supports both training and inference workflows and includes runnable examples for CV, NLP, audio, and tabular tasks. While the project is actively maintained with frequent commits and CI, it exhibits medium technical drift due to some outdated engine dependencies (e.g., MXNet) and the need to synchronize multiple native library versions.
TensorFlow Datasets (TFDS) is a library providing ready-to-use public datasets for machine learning, primarily integrated with TensorFlow but also supporting JAX and PyTorch. It offers a standardized API (tfds.load, tfds.builder) for downloading, preparing, and loading datasets as tf.data.Dataset objects or numpy arrays. The architecture consists of core components (dataset builders, feature connectors, download management) and hundreds of dataset-specific implementations following a consistent pattern. Key entry points include the tfds CLI for dataset operations and the main tfds module for programmatic access. Data flows involve downloading raw data, generating TFRecord files via builders, and yielding examples through tf.data pipelines. External dependencies include TensorFlow, absl-py, numpy, pandas, and dataset-specific extras like apache-beam for large-scale processing or huggingface_hub for community datasets.
Langchain.rb is a Ruby gem for building LLM-powered applications with a unified interface to multiple LLM providers and vector databases. It provides a modular architecture supporting Retrieval Augmented Generation (RAG), AI agents with tool usage, document processing, and prompt templating. The gem includes a Rails engine for deep Rails integration and is actively maintained with frequent updates. The system enables developers to create interactive AI applications by combining LLMs, external tools, and vector search capabilities.
MLJ.jl is the umbrella entry point and coordination layer for the MLJ machine-learning ecosystem in Julia. The repository's own source is a thin facade module (`src/MLJ.jl`) that re-exports the public API from satellite packages — MLJBase, MLJModels, MLJTuning, MLJEnsembles, MLJIteration, and StatisticalMeasures — plus a small amount of scitype-related code. The actual implementations of fit/predict/transform, learning networks, tuning, ensembling, model composition, and evaluation live primarily in those dependencies. This repo is therefore best understood as the distribution, documentation, integration-testing, and governance hub for the larger MLJ framework rather than a standalone implementation of ML algorithms. In production, users enter through `using MLJ`, load concrete models via `@load`, wrap data and models in `machine(...)`, and drive the system with `fit!`, `predict`, `transform`, `evaluate`, `TunedModel`, `IteratedModel`, `Pipeline`, and related APIs. The system is designed around scientific types and the Tables.jl interface, allowing tabular data to flow uniformly into 200+ models across Julia and other languages, including scikit-learn interop. It supports complex composite workflows: linear pipelines, learning networks/DAGs, target transformations, homogeneous ensembles, stacking, hyperparameter optimization, and performance evaluation. The repository also contains an extensive documentation site, runnable examples, a JOSS paper, and a registry-driven integration test suite. Maintenance is active: the latest commit is recent, CI runs on GitHub Actions, dependabot is configured, and release automation via TagBot is present. Some documentation artifacts show age — for example, the synthetic-data notebook records Julia 1.4.2 — and a few APIs are explicitly experimental, but the core package remains current with the Julia ecosystem and demonstrates disciplined package management across a large dependency graph.
nnsight is a Python library for interpreting and intervening in neural network models during execution. It provides a tracing API that allows users to access and modify internal activations, compute gradients, and perform interventions like activation patching and steering. The system uses a deferred execution model with thread-based synchronization to run user intervention code alongside model forward passes. Key features include support for Hugging Face models, vLLM integration for high-performance inference, remote execution via NDIF infrastructure, and persistent model editing. The architecture centers around Envoy wrappers for modules, a tracer/interleaver system for coordinating execution, and a batching mechanism for efficient multi-input processing.
onnx2tf is a production-ready ONNX to TensorFlow/TFLite conversion framework that employs a sophisticated pass-based optimization pipeline. The system converts ONNX models through preprocessing, lowering to an intermediate representation (ModelIR), applying 50+ optimization passes (layout transformation, operator fusion, quantization), and exporting to target formats including TFLite, SavedModel, PyTorch nn.Module, and TorchScript. Its architecture features clean separation between operator implementations, IR transformation passes, and backend-specific exporters, with the flatbuffer_direct backend providing 100x+ faster conversion by bypassing TensorFlow graph construction. The framework includes comprehensive accuracy validation tools, dynamic shape handling, and model surgery capabilities for intermediate layer extraction.
KerasHub is a production-grade Keras 3 library providing pretrained model architectures, layers, tokenizers, metrics, and sampling utilities for NLP, vision, multimodal, and audio tasks. The codebase is organized around a consistent Backbone → Task → Preprocessor pattern, where each model family (e.g., BERT, Gemma, ViT, Whisper) defines a backbone, a preprocessor (tokenizer + optional media converter), and task-specific heads (CausalLM, ImageClassifier, etc.). All models expose a unified `from_preset()` API for loading pretrained weights hosted on Kaggle Models, and `save_to_preset()` for export. The library includes extensive checkpoint conversion tooling from HuggingFace and TIMM, export to HuggingFace format and OpenVINO, and integration with vLLM for serving. While predominantly an application library, it has strong data/research character through its pretrained weight infrastructure, benchmark scripts, and research-oriented architectures (e.g., FNet, RWKV7, BLIP-2).
AlbumentationsX is a production-grade Python library for fast, flexible image augmentation in deep learning and computer vision pipelines. It provides a comprehensive catalog of 70+ transforms covering pixel-level (blur, noise, color, weather effects), geometric (flip, rotate, crop, resize, distortion, perspective), dropout (coarse, grid, mask, pixel), mixing (mosaic, copy-paste, overlay), and domain adaptation (FDA, histogram matching, stain normalization). The library supports 2D and 3D data, including images, masks, bounding boxes (HBB and OBB), keypoints, and volumes, with seamless integration into PyTorch via tensor transforms. Its architecture is built on a clean inheritance hierarchy (BasicTransform → DualTransform/ImageOnlyTransform/Transform3D) with Pydantic-based parameter validation, serialization to JSON/YAML, and deterministic replay via ReplayCompose. The codebase is backed by an extensive test suite (property-based, regression, contract, and golden vector tests), ASV benchmarks, and a sophisticated CI/CD system with path-aware planning, performance budgets, and legal integrity checks. Telemetry (opt-out) collects anonymous usage statistics to guide development. While primarily a software library, it incorporates research-level transforms (e.g., PlanckianJitter, HEStain, dithering) and is used in both academic and production settings.
Netron is a cross-platform neural network model visualizer that supports a wide range of deep learning frameworks (ONNX, TensorFlow, PyTorch, CoreML, etc.). It operates as an Electron desktop application, a web app, and a Python-based CLI/server. The architecture is modular: each model format is handled by a dedicated `ModelFactory` class implementing `match()` and `open()` methods, which parse binary or protobuf/flatbuffers serialized models into a uniform internal graph representation (Model -> Graph -> Node -> Value/Tensor). The application provides interactive graph visualization with SVG rendering, attribute inspection, and export capabilities. While primarily an application, it contains substantial data-processing code for format parsing, tensor decoding, and schema-driven metadata handling. The codebase is well-structured with consistent patterns, though it exhibits some duplication across format parsers and experimental/partial support for minority formats. The Python server (`server.py`) and the core binary parsing libraries (`base.js`, `protobuf.js`, `flatbuffers.js`) are independently reusable.
Cog is an open-source CLI and runtime for packaging machine learning models into production-ready Docker containers. It automatically generates HTTP inference servers with OpenAPI schema, handles GPU/CUDA/cuDNN compatibility, and provides a full weight management pipeline for large model artifacts. The project consists of a Go CLI (build, push, serve, predict, doctor, weights management), a Rust/PyO3 prediction server (coglet) with slot-based concurrency, a Python SDK for defining model interfaces (BaseRunner, Input, streaming), and a TypeScript playground UI. Key workflows include building OCI images from cog.yaml, pushing to registries, serving models via HTTP with SSE streaming, and managing model weights via content-addressed storage and lockfiles. The architecture is multi-language but cleanly separated, with well-defined protocols between components.
ML.NET is a cross-platform, open-source machine learning framework for .NET developers. It provides a comprehensive set of tools for building, training, and deploying custom machine learning models without requiring expertise in Python or R. The framework covers traditional ML algorithms (classification, regression, clustering, ranking, time series), deep learning via TensorFlow and ONNX integration, automated machine learning (AutoML) with hyperparameter sweeping and pipeline search, and generative AI (GenAI) support for large language models like LLaMA, Phi, and Mistral. It also includes a DataFrame API for data analysis and a code generator for producing prediction code. The architecture is modular, with core abstractions like IDataView, IEstimator, and ITransformer forming a composable pipeline pattern. The repository is actively maintained by Microsoft and has a large community.
Linfa is a comprehensive machine learning library for Rust, providing implementations of classical ML algorithms including supervised learning (regression, classification, SVM, naive Bayes, decision trees, ensemble methods) and unsupervised learning (clustering, dimensionality reduction, ICA). The library features a modular architecture with core traits defining standard ML interfaces (Fit, Predict, Transformer), a flexible Dataset abstraction for data handling, and algorithm-specific crates following consistent patterns. It emphasizes performance through optional BLAS backends and parallel processing, while maintaining usability via builder-pattern hyperparameter configuration and comprehensive example code.
MLX Swift is a Swift language binding for the MLX machine learning framework, providing a comprehensive library for building and training neural networks on Apple Silicon (Metal) and NVIDIA GPUs (CUDA). It offers automatic differentiation, a wide range of neural network modules (MLXNN), optimizers (MLXOptimizers), and support for custom JIT-compiled GPU kernels. The architecture is a Swift frontend wrapping a C/C++ backend, with auto-generated GPU kernels for high performance. It is primarily a library, not a standalone application, designed for integration into Swift projects via Swift Package Manager or Xcode. The codebase demonstrates high craftsmanship in Swift metaprogramming and GPU kernel optimization, though it carries some technical debt from vendored dependencies and dual build systems.