thehardwarestore.ai

Kineto is a production-grade C++ library for GPU and CPU performance profiling, primarily embedded in PyTorch's profiler. It provides low-overhead timeline tracing of GPU kernels, memory operations, synchronization events, and CPU-side activities, with support for CUDA (CUPTI), ROCm (rocprofiler-sdk), and Intel XPU (PTI) backends. The library supports both synchronous (manual start/stop) and asynchronous (on-demand, daemon-triggered) profiling modes, and emits Chrome Trace JSON and in-memory trace buffers for downstream analysis. It is not a standalone application; it is a linkable library with a public C API and plugin interfaces for custom profilers and loggers. Architecturally, Kineto is organized around a central ActivityProfilerController that drives a warmup/collect/process state machine, delegating platform-specific capture to backend profilers derived from GenericActivityProfiler. Configuration flows through a ConfigLoader that polls files or an IPC fabric daemon, and activity data flows from vendor callback/activity APIs through correlation-ID-based flow tracking into typed metadata and trace spans. The codebase demonstrates strong software engineering practices: extensive unit tests, CI matrices for CPU/CUDA/ROCm/XPU, clean interface abstractions, and reusable patterns such as observer-based logging, factory-based output registration, and type-safe metadata visitors. However, heavy conditional compilation, vendor SDK version guards, and platform-specific ifdefs introduce maintenance complexity and moderate technical drift risk.

NanoFlow is a high-throughput LLM serving framework designed to optimize inference through intra-device parallelism and efficient resource utilization. It employs nano-batching (splitting requests at operation granularity) and asynchronous CPU scheduling to overlap compute, memory, and network operations within a single device. The framework supports tensor parallelism for multi-GPU deployment and includes advanced KV-cache management strategies, including offloading to SSDs to avoid recomputation. It is evaluated on models such as Llama2, Llama3, and Qwen2, demonstrating significant throughput improvements over baselines like TensorRT-LLM. The architecture is modular, comprising core components for execution and worker management, model-specific implementations, operation kernels with multiple backends (Torch, CUDA, etc.), KV-cache mechanisms, and utility functions for profiling and resource management. Key entry points include `run_llama3.py` for launching the service and various test scripts for multi-GPU scenarios. Data flows involve request processing through transformer layers, with overlapping of computation and communication via streams and asynchronous execution, managed by a topological executor. The framework relies on a complex dependency stack including NVIDIA CUDA, NCCL, CUTLASS, FlashInfer, MSCCL++, and Gurobi, which introduces maintenance challenges. However, it demonstrates strong engineering practices through reusable components such as the IOWrapper for tensor dependency management, a cross-platform profiler, and a linear programming-based buffer allocator. While active development continues, the repository faces challenges in dependency management and platform fragmentation, indicating a need for careful version control to maintain stability.

This repository implements a system to use NVIDIA GPU VRAM as swap space on Linux via the NBD (Network Block Device) protocol. The core component is a userspace daemon (nbd-vram) that allocates VRAM using the CUDA driver API and serves it as a block device over a Unix socket. The Linux kernel's NBD driver connects to this socket, exposing /dev/nbdX as a swap device. The system includes scripts for connection/disconnection, power management (automatically disabling swap on battery), and proper handling of suspend/resume to avoid GPU-related deadlocks. Key features include memory locking to prevent the daemon from being swapped out, using PR_SET_IO_FLUSHER to avoid deadlocks under memory pressure, and multi-threaded request batching to amortize CUDA synchronization overhead. The solution works with any CUDA-capable NVIDIA GPU without requiring kernel modifications or NVIDIA's P2P API, which is restricted to professional GPUs.

TurboQuant Model is a Python library and CLI tool for near-lossless LLM weight quantization, implementing the TurboQuant algorithm from Zandieh et al. (2025). It provides a drop-in replacement for nn.Linear (TurboQuantLinear) that stores weights in packed 4-bit indices and dequantizes on-the-fly during matrix multiplication using fused GPU kernels (CuTile, Triton, Metal). The library supports single-pass 4-bit, multi-pass residual (e.g., 4+4), polar decomposition, and hash-based compression, achieving 4x compression vs bf16 with KLD as low as 0.002 nats. It includes a CLI for quantizing, evaluating, generating, and benchmarking HuggingFace models, with CPU offload for residual passes and advanced techniques like norm compression and entropy coding. The codebase is well-structured with comprehensive tests and benchmarks, though it targets cutting-edge CUDA 13.1+ and may have limited hardware compatibility.

Tilus is a tile-level GPU kernel programming language and compiler framework, primarily targeting NVIDIA architectures (SM80, SM90, SM100). It provides a Python DSL (`tilus.Script`) for writing high-performance CUDA kernels with explicit management of shared/register memory, tensor cores (WGMMA, tcgen05), and advanced synchronization primitives (mbarrier, cluster sync, TMA). The compiler stack includes a custom IR with layout inference (register/shared/tensor-memory), a pass pipeline (dead code elimination, bound-aware simplification, sub-byte type lowering), and backend emitters that generate PTX/CUDA code. Autotuning over tile sizes, warp counts, and pipeline stages is built into the framework. The architecture separates concerns into three layers: the user-facing `lang` package (Script, autotune, instructions), the `ir` package (tensors, layouts, instructions, inference rules), and the `backends` package (emitters, contexts, code generation). The `hidet` subpackage integrates a vendored copy of the Hidet IR and transforms, which provides the low-level expression/statement IR and a comprehensive set of optimization passes. Examples cover progressive matmul implementations (V0–V6) for Ampere, Hopper, and Blackwell, as well as flash attention, layer norm, and quantization kernels. This is a research project (ASPLOS '26 paper) with a strong emphasis on correctness and performance, evidenced by CUDA-level validation against PyTorch and cuBLAS. While the codebase is well-structured and demonstrates advanced compiler design patterns, it remains in beta (version 0.2.0) and carries some duplication between vendored Hidet code and native Tilus code, as well as hardware-specific constraints (NVGPU only). The layout inference system and autotuning infrastructure are particularly notable as reusable components.

NVIDIA's official CUDA Samples repository is a large catalog of self-contained, compilable example programs demonstrating the CUDA platform across C++ and Python. The C++ collection spans hundreds of samples organized into ten categories: fundamentals (vector add, matrix multiply), utilities (deviceQuery), concepts and techniques (reduction, scan, convolution, histograms), feature showcases (CUDA graphs, cooperative groups, Tensor Core GEMMs, dynamic parallelism, NVRTC JIT), CUDA libraries (cuFFT, cuRAND, cuBLAS, cuSOLVER, CUB), domain-specific applications (Black-Scholes, N-body, optical flow, fluid simulation, volume rendering, Monte Carlo), performance studies, libNVVM/PTX generation, platform-specific Tegra/embedded examples, and CUDA Tile C++ microbenchmarks. A parallel Python collection uses the modern cuda.core API with CuPy/NumPy interop and framework integrations (PyTorch, TensorFlow, RAPIDS). Each sample is typically a standalone CLI or interactive graphics program with its own main() entry point; most include a CPU 'gold' reference implementation and automated pass/fail verification, making the collection both educational and regression-tested.

NVIDIA cuOpt is a GPU-accelerated optimization solver library covering linear programming (LP), mixed-integer programming (MIP), quadratic programming (QP), second-order cone programming (SOCP), and vehicle routing problems (VRP, PDP, TSP). The codebase is predominantly an application: it provides multiple deployment forms—a C++ shared library, Python bindings (Cython), a Java JNI interface, a CLI tool (`cuopt_cli`), a gRPC server for remote solves, and a FastAPI-based HTTP server (`cuopt_server`). It also includes comprehensive CI/CD pipelines, benchmarks, and documentation. However, it exhibits a hybrid nature due to its strong research character: advanced algorithmic innovations (e.g., primal-dual hybrid gradient with adaptive step sizing, novel MIP presolve techniques, lattice propagation, deterministic parallel branch-and-bound, SIMD-accelerated MPS parsing, and diversity-based routing heuristics) are deeply embedded in the core solvers. The system is architected as a layered monolith: common utilities and linear algebra form the base, solver-specific modules (PDLP, dual simplex, barrier, branch-and-bound, routing heuristics) sit above, and language bindings and server endpoints wrap the C++ core. Key data flows include parsing MPS/LP files or building problems programmatically, dispatching to the appropriate GPU solver, and retrieving solutions via C structs, Python objects, or protobuf messages. External dependencies include CUDA, cuSPARSE, RMM, NCCL, gRPC, Protobuf, and optionally PaPILO for presolve. The repository is actively maintained with frequent commits and nightly CI, though the presence of deprecated modules and experimental features indicates moderate technical drift.

A ROS 2 node that performs real-time metric depth estimation and point cloud generation using Depth Anything V3 accelerated with TensorRT. The system subscribes to camera image and camera info topics, synchronizes them via approximate time, runs GPU-accelerated preprocessing and inference, then publishes a 32-bit float depth image and a colorized or monochrome point cloud. The pipeline includes GPU-side bicubic upscaling, focal-length-based metric scaling, sky detection with percentile-based fill, and optional debug visualization with FPS overlay. The repository provides a complete build system (ament_cmake, CUDA, TensorRT), a standalone engine generation tool, a Docker image, and an ONNX export script for the underlying PyTorch model. It is designed for autonomous driving and robotics perception applications requiring high frame rates (claimed 50 FPS on Quadro RTX 6000). The architecture is a monolithic ROS 2 component with a clean separation between the node, the TensorRT wrapper, and the GPU kernels.

This repository is a C++/CUDA header-only library implementing Model Predictive Path Integral (MPPI) control, a sampling-based trajectory optimization method for real-time robotic control. It provides multiple controller variants including Vanilla MPPI, Tube-MPPI, Robust-MPPI, and Colored-MPPI, each designed to handle different robustness and noise characteristics. The architecture uses CRTP (Curiously Recurring Template Pattern) to achieve device-side polymorphism without virtual functions, enabling efficient GPU execution. The library includes a broad set of dynamics models (cartpole, double integrator, quadrotor, dubins car, autorally neural network models, racer dubins with elevation and suspension) and corresponding cost functions, along with Gaussian, colored noise, and log-normal sampling distributions. It also integrates DDP and CCM feedback controllers, and provides utilities for neural network inference (LSTM, FNN) and texture-based costmap queries on GPU. Typical use cases are autonomous racing and off-road vehicle control, where the library's GPU acceleration allows thousands of trajectory rollouts to be evaluated in real-time. The codebase is well-structured with extensive tests using Google Test and Google Mock, covering kernel correctness, dynamics, cost functions, and full controller pipelines. It includes real-time control loop abstractions (BasePlant, BufferedPlant) and supports integration with ROS via state interpolation and parameter updates. The library is research-grade, citing Williams et al. 2017 and including implementations of concepts like robust free-energy bounds and colored noise shaping. While the overall architecture is sound and many components are reusable, the codebase shows signs of research-driven growth: some modules contain commented-out code, skipped tests, and conditional compilation flags. Manual CUDA memory management and complex template hierarchies increase the learning curve. Despite this, the library is actively maintained and offers high-quality reusable patterns for GPU-accelerated control, making it a strong reference for MPPI implementations in robotics.

OpenMM is a high-performance molecular dynamics simulation library that provides a unified API across CPU (with SIMD and JIT compilation via AsmJit) and GPU backends (CUDA, OpenCL, HIP). The codebase is structured into several layers: the public API (`openmmapi`) defines `System`, `Force`, `Integrator`, and `Context` classes; the OLLA layer abstracts kernel dispatch; platform modules (`reference`, `cpu`, `cuda`, `opencl`, `hip`) provide concrete kernel implementations; and plugins extend functionality with AMOEBA polarizable force fields, Drude oscillators, and ring-polymer MD. The project supports standard force fields (AMBER, CHARMM, GROMACS, Tinker), advanced algorithms (PME, GB/SA, custom forces via the Lepton expression parser), and features like metadynamics, replica exchange, checkpointing, and XML serialization. It is primarily a library for computational chemistry and biophysics, designed to be used through Python (with extensive application-level tools in `openmm.app`) or C++.

YaFSDP is a specialized PyTorch library designed to optimize Fully Sharded Data Parallel (FSDP) training for Large Language Models (LLMs). It implements custom sharding strategies that claim up to 20% performance improvements over standard FSDP by reducing communication overhead and optimizing memory usage. The library provides two integration points: a modern composable API via the `fully_shard` function and a legacy-style `YaFSDP` wrapper class that integrates with the `accelerate` ecosystem. It supports advanced features like ragged parameter sharding, CUDA stream-based communication/computation overlap, and optional integration with the Yandex Collective Communication Library (YCCL) for high-performance collectives. The codebase demonstrates high engineering craftsmanship, particularly in its handling of distributed tensor operations and memory management. It utilizes custom tensor subclasses (`RaggedShardDTensor`) to manage non-uniform parameter sizes within sharded groups and implements low-level CUDA stream synchronization to hide communication latency. However, the library relies on internal PyTorch APIs (e.g., `torch._C`, `torch.distributed.fsdp.wrap._...`) and patches external libraries like `accelerate` and `transformers`, which introduces maintenance risks as upstream frameworks evolve.

This repository implements `cuda_tile`, an MLIR-based compiler dialect and toolchain for GPU tile programming, primarily targeting NVIDIA CUDA architectures. It defines a rich set of operations for tile-level computation (arithmetic, matrix multiply, memory views, atomics, control flow) with explicit memory ordering and synchronization. The project includes a custom bytecode serialization format (.tileirbc) with versioned backward compatibility, allowing efficient IR transport and JIT compilation. Key tools are `cuda-tile-opt` (dialect testing), `cuda-tile-translate` (MLIR-to-bytecode conversion), `cuda-tile-optimize` (bytecode-to-bytecode optimization passes), and `cuda-tile-tblgen` (TableGen-based code generation for bytecode readers/writers). Python bindings enable programmatic IR construction. The architecture is a set of modular libraries (dialect, bytecode, optimizer, C API) that can be integrated into larger compiler pipelines. External dependencies are LLVM/MLIR at a pinned commit and CUDA. The codebase demonstrates high engineering rigor with extensive test coverage (bytecode stability, version compatibility, round-trip, transformation passes) and careful version management for evolving the IR format. While primarily a compiler infrastructure application, the project's focus on low-precision matrix operations (FP8, FP4) and tensor core optimizations touches on ML research areas.

Falkon is a Python library for large-scale kernel ridge regression and logistic regression, leveraging Nyström approximation and preconditioned conjugate gradient to scale to billions of points. It provides a scikit-learn-like API (fit/predict) with GPU acceleration via custom CUDA/cuBLAS/cuSOLVER kernels and optional KeOps integration for on-the-fly kernel-vector products. The library includes hyperparameter optimization through differentiable objectives (LOOCV, GCV, Nyström compression) with gradient-based solvers, out-of-core operations for GPU memory overflow, and sparse tensor support. Benchmark infrastructure compares Falkon against GPyTorch, GPflow, and EigenPro on datasets like Higgs, TIMIT, MillionSongs. The codebase is well-tested with CI/CD for CPU/GPU builds and pre-built wheels for multiple PyTorch/CUDA combinations. While the core library is production-grade, some benchmark scripts are one-off and a few modules have TODO comments or limited platform support (Linux-only, Windows untested).

nvml-wrapper is a Rust workspace providing safe, high-level bindings to the NVIDIA Management Library (NVML). It is a library crate, not an application: there are no binaries, servers, or CLIs. The workspace is split into `nvml-wrapper-sys` (low-level FFI bindings, including runtime loading of `libnvidia-ml.so`/`nvml.dll` via `libloading`) and `nvml-wrapper` (a safe, typed API). The primary entry point is `Nvml::init()`, which loads the NVML library and returns a handle; from there, `Device` exposes 100+ methods for querying GPU state such as memory, clocks, temperature, power, ECC, NvLink, vGPU, MIG, and GPM metrics. The crate also supports mutating operations like setting application clocks and fan speed, and includes a `legacy-functions` feature for older NVML versions.

TorchFort is a library that bridges PyTorch with Fortran and C/C++ programs, primarily targeting HPC/scientific computing applications. It provides a C API that allows users to load PyTorch models (via TorchScript or built-in MLP), train them, and perform inference directly on user-provided arrays without data copying. The library supports supervised learning as well as reinforcement learning (RL) with several state-of-the-art algorithms: DDPG, TD3, SAC, and PPO. It includes comprehensive RL infrastructure such as replay buffers, rollout buffers, online running normalizers, and distributed training via MPI. Integration with Weights & Biases is available for experiment tracking. Architecturally, TorchFort consists of a thin C API layer, a set of C++ implementations using LibTorch, and Fortran bindings generated via ISO_C_BINDING. The core C++ code is organized into modules for model management, training, distributed communication, CUDA utilities, and RL subsystems. Configuration is externalized via YAML files, and the library is designed to be used as a static or shared library linked into user applications. Notable components include a prioritized replay buffer with sum-tree implementation (replay_buffer.h), an online running normalizer using Welford's algorithm with MPI synchronization (running_normalizer.h), and a tensor list abstraction for handling multiple typed arrays (tensor_list.h). The codebase demonstrates good engineering practices with extensive test coverage, error handling via return codes, and support for both CPU and GPU execution. While primarily an application/library for integrating PyTorch into HPC codes, the repository also contains research-quality RL algorithm implementations and data preprocessing components, giving it a hybrid character.

This repository is an offline speech-to-text application that uses the faster-whisper model to transcribe audio and video files into text, JSON, or SRT formats. It provides a web interface built with LayUI and a REST API (including an OpenAI-compatible endpoint) for easy integration. The application supports CPU and CUDA acceleration, caches loaded models for efficiency, and handles media conversion via FFmpeg. The architecture consists of a Flask web server served by gevent, with a background thread processing transcription tasks from a queue. Configuration is managed via an INI file, and utilities for FFmpeg execution, update checking, and browser launching are encapsulated in a helper module. The model caching mechanism avoids reloading large models for multiple requests, improving performance in a server setting.

PopSift is a CUDA-accelerated SIFT (Scale-Invariant Feature Transform) feature extraction library. It provides a C++ library (libpopsift) with an asynchronous API for enqueuing images and retrieving features, and two command-line demo applications (popsift-demo, popsift-match). The library implements the full SIFT pipeline (Gaussian pyramid, extrema detection, orientation assignment, descriptor computation) on NVIDIA GPUs, supporting multiple algorithmic variants (PopSift, VLFeat, OpenCV) for research comparison. It is designed for high-performance real-time feature extraction and can be integrated via CMake. Despite its research origins (Simula Research Laboratory), the codebase is well-structured with modular CUDA kernels, debug utilities, and parallel primitives. The library is actively used in computer vision projects and maintains compatibility with recent CUDA versions while preserving legacy fallbacks.

FaithC is a Python library (with CUDA acceleration) for near-lossless 3D mesh encoding and decoding using Faithful Contour Tokens (FCT), as accepted at CVPR 2026 (Oral). It implements an encoder that transforms a mesh into compact per-voxel tokens (anchors, normals, edge flux signs) via octree traversal, SAT-based triangle clipping, QEF optimization, and edge flux computation. The decoder reconstructs the mesh from tokens using edge-based quad extraction and triangulation. The library is not a standalone application but a research-oriented tool intended for downstream tasks like VAE and diffusion models. It provides both a modern class-based API (FCTEncoder/FCTDecoder) and an older functional API. The codebase is well-structured with modular components, custom CUDA kernels for performance, and a PyTorch-centric design. However, it has some maintenance risks including dual code paths, hardcoded CUDA dependencies, and a custom 'atom3d' package without version pinning.

FlexFlow Train is a deep learning framework that automatically parallelizes and distributes neural network training across GPU/CPU clusters. The system ingests models from PyTorch, Keras, ONNX, or its own C++ API, converts them into a computation graph, and then applies a multi-stage optimization pipeline: graph-level substitutions (pattern matching and rewriting), series-parallel decomposition, cost estimation, and machine mapping via MCMC or dynamic programming. The optimized parallel graph is executed on the Legion/Realm runtime, coordinating distributed tasks across devices. The codebase is organized as a C++ library with Python bindings, comprising ~3500 files including kernels, a compiler, graph utilities, and model definitions. While the core architecture demonstrates sophisticated engineering with extensive tests and reusable graph abstractions, the repository carries significant complexity from heavy template metaprogramming, two execution backends, and partially implemented CPU/HIP kernels, making it more suitable for integration by teams with C++ and distributed systems expertise.