thehardwarestore.ai

ncnn is a high-performance neural network inference framework designed for mobile, embedded, and desktop deployment. It provides a self-contained C++ library with no mandatory external dependencies, supporting CPU inference with extensive SIMD optimizations (ARM NEON, x86 SSE/AVX/AVX512, RISC-V Vector, MIPS MSA, LoongArch LSX/LASX) and GPU inference via Vulkan. The framework includes a comprehensive set of neural network layers (convolution, pooling, normalization, activation, recurrent, attention, etc.) and supports multiple data types (float32, float16, bfloat16, int8). The companion pnnx tool converts models from PyTorch (TorchScript) and ONNX into ncnn format using a sophisticated graph rewriting pipeline. Python bindings and a model zoo are provided for rapid prototyping. The codebase is production-grade, used in Tencent applications like QQ and WeChat, and is actively maintained with support for modern architectures and operations (e.g., scaled dot-product attention, rotary embeddings).

Rembg is a production-grade tool for removing image backgrounds, usable as a Python library, CLI, HTTP server, or Docker container. It wraps multiple ONNX-based segmentation models (U2Net, BiRefNet, SAM, BRIA-RMBG, etc.) and provides several cutout strategies: naive, alpha matting, color decontamination, and ViTMatte refinement. The architecture is modular: a session abstraction handles model download and inference, while the `remove()` function in `rembg/bg.py` orchestrates preprocessing, mask prediction, and post-processing. The CLI offers subcommands for single files (`i`), batch processing (`p`), HTTP serving (`s`), binary stream input (`b`), model download (`d`), and legacy migration (`m`). The HTTP server uses FastAPI with a Gradio UI and includes security measures like DNS rebinding protection. The codebase is well-tested, with synthetic fixtures for color decontamination and integration tests using perceptual hashing. It demonstrates strong engineering practices: idempotent migration, session caching, and careful handling of edge cases like oversized uploads and failed alpha matting convergence.

clpeak is a cross-platform synthetic micro-benchmark that measures peak achievable compute performance (FLOPS/TOPS), bandwidth, latency, and AI inference throughput across CPUs, GPUs, and NPUs. It supports eight hardware backends: OpenCL, Vulkan, CUDA, ROCm/HIP, Metal, oneAPI/SYCL, native CPU (with per-ISA dispatch), and ONNX Runtime. The tool is architected as a set of backend-specific static libraries orchestrated by a common framework, with two front-ends: a CLI tool (`clpeak`) and a Flutter desktop/mobile GUI. The GUI communicates with the native library via a C-ABI FFI bridge, streaming structured events for live display. Results are serialized to a versioned JSON schema (v3) and can be compared against baselines. The codebase is notable for its meticulous anti-optimization engineering (MAD chains, compiler barriers, calibration probes) and its protobuf-free ONNX model generator. It is a mature, actively maintained project with extensive CI coverage and packaging for multiple platforms.

PDFMathTranslate is a production-grade CLI and GUI application for translating PDF documents while preserving scientific formatting (formulas, charts, tables, and fonts). It uses an ONNX-based layout detection model (DocLayout-YOLO) to understand document structure, supports OCR for scanned PDFs, and integrates with numerous translation services (Google, DeepL, OpenAI, Ollama, etc.). The system operates as a monolithic package with multiple entry points: a command-line tool, a Gradio web interface, a Flask+ Celery HTTP API, and an MCP server for AI assistant integration. Internally, it follows a pipeline architecture: input PDF → optional OCR → layout analysis → text extraction → translation (with caching) → typesetting → bilingual output PDF. Notable engineering patterns include a pluggable translation backend via a kernel registry, a configurable translator base class with retry and caching, and a singleton configuration manager. The code is well-structured with test coverage and CI/CD pipelines, though the dependency on a Git submodule for the 'precise' kernel introduces minor coupling risk.

YOLOv5 is a production-oriented PyTorch framework for object detection, instance segmentation, and image classification, with a hybrid research/educational flavour from its tutorial notebooks. The dominant character is application: the repository provides CLI tools (`train.py`, `val.py`, `detect.py`, `export.py`, plus `segment/` and `classify/` variants), a Flask REST API, Docker images, PyTorch Hub support, CI smoke tests, benchmarking scripts, and experiment-tracking integrations for W&B, ClearML, and Comet. It is not just an experiment repo; it is designed to be deployed and exported to edge and cloud inference targets. Core architecture is config-driven: model YAMLs are parsed by `models/yolo.py` into `DetectionModel`, `SegmentationModel`, or `ClassificationModel` geometries, while `models/common.py` holds reusable layers and the central `DetectMultiBackend` inference wrapper for PyTorch, TorchScript, ONNX Runtime, OpenVINO, TensorRT, CoreML, TensorFlow, TFLite/Edge TPU, Paddle, and Triton. The primary data flow is dataset YAML -> dataloaders with augmentation (`utils/augmentations.py`) -> model forward -> task-specific loss (`utils/loss.py`, `utils/segment/loss.py`) -> train/val loops. `export.py` converts trained checkpoints into 12+ deployment formats using a decorated `@try_export` pattern, and `data/` embeds dataset download/conversion scripts for COCO, ImageNet subsets, VOC, VisDrone, and others. Maintenance is active, but the codebase sits in a legacy/support position relative to the unified `ultralytics` package. Technical drift is medium: some loggers are deprecated, many `models/hub/` configs are experimental, and the code depends heavily on the external `ultralytics` package, which creates version-compatibility risk. The code is generally clean and modular, though it has task-level duplication across detection, segmentation, and classification paths and limited unit test coverage outside smoke tests.

Deep-Live-Cam is a real-time face swapping desktop application that uses ONNX Runtime with insightface models to swap faces in live webcam feeds or pre-recorded images/videos. The core pipeline involves face detection (RetinaFace), face recognition (ArcFace), and a face swap model (inswapper_128), with optional face enhancement via GFPGAN or GPEN-BFR ONNX models. The application provides a PySide6 GUI for selecting source/target faces, toggling options (mouth mask, many faces, Poisson blending, sharpening, temporal interpolation), and managing camera selection. It supports multiple execution providers: CUDA (NVIDIA), CoreML (Apple Silicon), DirectML (AMD/Intel), OpenVINO (Intel), and CPU fallback. Video processing can run in-memory via FFmpeg pipes (avoiding disk I/O) or via traditional frame extraction to PNG files. The architecture is modular: frame processors (face swapper, enhancers) follow a common interface, and the codebase includes optimizations for Apple Silicon (CoreML subgraph optimization, Pad decomposition, Shape/Gather folding) and CUDA (IO binding, CUDA graphs). The project also includes a benchmark pipeline and unit tests for core components.

Deep Java Library (DJL) is a comprehensive, engine-agnostic deep learning framework for Java, providing a unified API for model loading, training, inference, and deployment across multiple backends (PyTorch, TensorFlow, MXNet, ONNX Runtime, XGBoost, LightGBM, and a Rust-based engine). Its architecture is modular: a core API defines abstractions like `NDArray`, `NDManager`, `Model`, `Predictor`, `Trainer`, and `Translator`; engine-specific implementations adapt these to native libraries; model zoos and repositories supply pre-trained models; and extensions add capabilities for generative AI, audio, timeseries, Spark integration, and more. The codebase is well-organized with extensive use of the builder pattern, ServiceLoader-based plugin discovery, and a rich set of training utilities (losses, optimizers, trackers, evaluators, listeners). It supports both training and inference workflows and includes runnable examples for CV, NLP, audio, and tabular tasks. While the project is actively maintained with frequent commits and CI, it exhibits medium technical drift due to some outdated engine dependencies (e.g., MXNet) and the need to synchronize multiple native library versions.

JSTprove is a production-grade zero-knowledge proof system for verifying ONNX neural network inference. It compiles ML models into arithmetic circuits using the Expander Compiler Collection (ECC) frontend, generates witnesses, and produces proofs via a GKR/sumcheck-based prover with multiple polynomial commitment schemes (KZG, Hyrax, Orion, Basefold, WHIR). The system supports multiple efficient field arithmetics (Mersenne31, BabyBear, Goldilocks, BN254, GF2) with SIMD/AVX/NEON optimizations, and provides hardware acceleration through Metal and CUDA backends. The codebase is organized as a Rust workspace with ~30 crates, offering a CLI (`jstprove`, `jstprove-remainder`), Python bindings via PyO3, and a zkVM guest (Jolt). It is actively maintained by Inference Labs with regular commits, CI/CD pipelines, and e2e tests covering full model pipelines (LeNet, CIFAR CNN, MiniResNet). While primarily an application toolkit, it contains deep cryptographic research contributions in the form of advanced polynomial commitment constructions (sparse WHIR, hyper-bivariate KZG) and circuit optimizations (cross-layer GKR, LogUp range checks). Some hints (e.g., sigmoid, gridsample) carry documented soundness caveats, and the system relies on nightly Rust and platform-specific intrinsics, which may limit portability.

onnx2tf is a production-ready ONNX to TensorFlow/TFLite conversion framework that employs a sophisticated pass-based optimization pipeline. The system converts ONNX models through preprocessing, lowering to an intermediate representation (ModelIR), applying 50+ optimization passes (layout transformation, operator fusion, quantization), and exporting to target formats including TFLite, SavedModel, PyTorch nn.Module, and TorchScript. Its architecture features clean separation between operator implementations, IR transformation passes, and backend-specific exporters, with the flatbuffer_direct backend providing 100x+ faster conversion by bypassing TensorFlow graph construction. The framework includes comprehensive accuracy validation tools, dynamic shape handling, and model surgery capabilities for intermediate layer extraction.

OnnxSlim is a Python library and CLI tool for optimizing ONNX models through graph rewriting. It applies elimination of redundant operations (e.g., no-op reshapes, slices, unsqueezes) and fusion of common patterns (Conv+BN, Conv+Add, GELU, etc.), combined with dead node elimination and shape inference. The architecture is modular: a core optimization pipeline orchestrates pattern matchers, a shape inference engine, and a vendored onnx_graphsurgeon for graph manipulation. The tool is used in production by NVIDIA TensorRT-Model-Optimizer, HuggingFace Optimum, and Ultralytics, confirming its practical value. The codebase demonstrates strong engineering with comprehensive tests, a clean CLI, and a well-organized package structure.

Netron is a cross-platform neural network model visualizer that supports a wide range of deep learning frameworks (ONNX, TensorFlow, PyTorch, CoreML, etc.). It operates as an Electron desktop application, a web app, and a Python-based CLI/server. The architecture is modular: each model format is handled by a dedicated `ModelFactory` class implementing `match()` and `open()` methods, which parse binary or protobuf/flatbuffers serialized models into a uniform internal graph representation (Model -> Graph -> Node -> Value/Tensor). The application provides interactive graph visualization with SVG rendering, attribute inspection, and export capabilities. While primarily an application, it contains substantial data-processing code for format parsing, tensor decoding, and schema-driven metadata handling. The codebase is well-structured with consistent patterns, though it exhibits some duplication across format parsers and experimental/partial support for minority formats. The Python server (`server.py`) and the core binary parsing libraries (`base.js`, `protobuf.js`, `flatbuffers.js`) are independently reusable.

Silero VAD is a pre-trained enterprise-grade Voice Activity Detector library that provides speech detection capabilities for audio signals. The system consists of a core PyTorch model (also available in ONNX format) with Python utilities for audio processing, speech timestamp extraction, and streaming applications. It supports 8kHz and 16kHz sampling rates and includes a VADIterator class for real-time processing. The repository offers multi-language examples (C++, C#, Go, Haskell, Java, Rust) demonstrating integration, along with tools for model tuning and sequence-based ONNX optimization for improved performance on long audio files.

A comprehensive tool for removing watermarks from Gemini AI-generated images and videos. It provides multiple deployment options: a Chrome extension, a Tampermonkey userscript, an npm package (SDK), a CLI, and an AI agent skill. The core algorithm uses mathematically exact reverse alpha blending with pre-calibrated alpha maps to remove the Gemini logo without degrading underlying image quality. The system includes a sophisticated pipeline for detection, candidate selection, alpha trial, repair, and quality evaluation, with support for both image and video watermark removal. The repository is hybrid: while primarily an application, it contains an extensive set of data/research scripts for benchmarking, alpha profile analysis, video residual analysis, and release quality gating, reflecting a data-driven development approach. The codebase is well-structured with clear separation of concerns, extensive test coverage, and modular components that can be reused in other projects.

VieNeu-TTS is an advanced on-device Vietnamese text-to-speech library with instant voice cloning, supporting bilingual (Vietnamese/English) synthesis. The system provides a Python SDK (`vieneu`) with multiple backends: v3 Turbo (default, 48kHz, ONNX or PyTorch), v3 Nano (lightweight 24kHz, ONNX only), and legacy v1/v2 (deprecated). It includes a Gradio web UI (`vieneu-web`), a FastAPI streaming server, and a browser-based HTML/JS client. The core architecture is a factory pattern (`Vieneu(mode=...)`) that returns the appropriate engine, all inheriting from a shared base class. Inference can run entirely torch-free on CPU via ONNX Runtime, with optional GPU acceleration (CUDA) for batched generation and lower latency. Feature highlights include instant voice cloning from a 3–8 second reference clip using a speaker encoder (ONNX) and denoiser, inline emotion cues (`[cười]`, `[thở dài]`), frame-level streaming, SRT subtitle-to-speech, and LoRA fine-tuning for custom voices (requires ~6GB GPU). The system is deployed via Docker with CPU and GPU profiles, and has CI/CD pipelines for testing and PyPI publishing. The codebase is well-structured with clear separation of concerns, though it carries some legacy weight from older model versions and a few deprecated parameters. Other notable components include a sliding-window repetition penalty, a babble guard heuristic for short utterances, and a voice persistence system for user-defined presets.

Fonnx is a Flutter plugin that enables cross-platform, on-device machine learning inference by wrapping the ONNX Runtime engine. It provides a unified Dart API that abstracts away the complexities of native FFI (for iOS, Android, macOS, Windows, Linux) and WebAssembly (for web), allowing developers to run models like OpenAI Whisper, Silero VAD, Magika, and various text embedding models without cloud dependencies. The codebase exhibits high software craftsmanship. Its architecture uses conditional Dart imports to cleanly separate native and web backends, employs Dart isolates for offloading heavy inference tasks, and implements a custom RPC protocol for web workers. A standout feature is its robust native memory management, including a C finalizer tested with sanitizers to prevent leaks during Flutter hot-restarts. While primarily a software library, it bridges the gap between ML research and production by bundling specific, optimized ONNX models and providing a deterministic test corpus with golden outputs. It is an excellent resource for Flutter developers building privacy-focused, offline-first AI applications, though its reliance on pinned native assets and specific build tooling introduces a moderate technical drift risk.

Sunone Aimbot is an AI-powered aimbot for first-person shooter games that uses YOLO models to detect players and heads in real-time screen captures. The system captures frames via MSS, BetterCam, or OBS, processes them with Ultralytics YOLO, identifies targets (prioritizing heads unless disabled), calculates mouse movement to aim at targets, and optionally shoots automatically. It features a modular architecture with separate threads for capture, visualization, overlay, shooting, and hotkey monitoring. A helper API and React-based UI provide configuration management, model training/exporting, and testing capabilities. The aimbot supports multiple input methods including standard mouse, GHUB, Razer, and Arduino devices.

dsperse is a distributed Zero-Knowledge Machine Learning (zkML) toolkit that decomposes ONNX neural network models into circuit-compatible segments for verifiable AI inference. The core Rust library provides model slicing, circuit compilation (using JSTprove backend), witness generation, and proof verification. It exposes both a CLI interface and Python bindings via PyO3/Maturin for integration into ML pipelines. The architecture follows a clear separation: the slicer module handles ONNX graph decomposition and tiling strategies (channel split, dimension split, spatial tiling), the pipeline orchestrates execution with memory-efficient tensor caching, and the backend trait enables pluggable ZK proving systems. Content-addressed storage with SHA256 hashing ensures artifact integrity across distributed execution. Notable constraints: the license explicitly restricts operational use without written permission from the IP owner, and the codebase relies on nightly Rust toolchain with specific git dependencies (tract-onnx, jstprove_circuits) that create version coupling risks. Security advisories are partially ignored in the audit configuration.

This repository is a collection of C++ sample programs for deploying deep learning models on AXERA AI SoCs (NPU chips). It provides ready-to-run inference examples for a wide range of computer vision tasks including object detection (YOLOv5/6/7/8/9/10/11/26, YOLOX, YOLO-World, PPYOLOE+, RT-DETR, SCRFD), instance segmentation, oriented object detection (OBB), pose estimation, depth estimation, semantic segmentation, stereo matching, and more. Each example is a standalone command-line executable that loads a pre-compiled .axmodel, runs inference via the AX_ENGINE API, post-processes the outputs, and saves a result image. The code is organized per target chip (ax650, ax637, ax620e, ax615) with near-identical copies of examples across platforms, differing only in system initialization and minor chip-specific logic. The architecture is not a library or framework but a set of reference implementations demonstrating the integration of AXERA's NPU SDK. The primary data flow is: load image → preprocess (letterbox resize, color conversion) → push to engine → run synchronous inference → decode model outputs (e.g., anchor-based proposals, DFL, NMS) → draw bounding boxes/keypoints/masks → save output. External dependencies include the AX_ENGINE runtime, OpenCV for image processing, and standard C++ libraries. The build system uses CMake with per-chip toolchain files and GitHub Actions CI for cross-compilation verification. While the code is functional and covers a broad model zoo, it exhibits significant duplication across chip directories and hardcoded configuration (thresholds, image sizes, class names). The monolithic detection.hpp (~3500 lines) bundles many model-specific post-processing functions, increasing maintenance burden. There are no automated tests, and manual memory management is prevalent. Despite these quality issues, the repository serves its purpose as a practical reference for developers integrating AXERA hardware, and it stays current with recent model releases (YOLOv13, YOLO26, Depth Anything V2).

This repository is a comprehensive toolkit for analyzing, optimizing, and transforming ONNX models. It provides a core library (onnx_tool) for loading models, performing graph analysis (shape/value inference), profiling computational complexity (MACs, parameters, memory), applying optimizations (operator fusion, constant folding), quantizing weights, and managing memory for efficient inference. The toolkit supports a wide range of model architectures including LLMs (BERT, GPT, LLaMa, MPT, Qwen), diffusion models (Stable Diffusion), computer vision (ResNet, EfficientNet), and multimodal models. Key features include a custom inference engine (GraphInfer) with memory pooling for zero-copy tensor views, a plugin-based node registry system for extending ONNX operator support, and utilities for model conversion and evaluation. The architecture centers around the onnx_tool package, which defines fundamental abstractions like Graph, Tensor, and Node. The Graph class handles model loading, shape inference, and transformation pipelines. The Node registry system allows dynamic registration of operation-specific handlers for shape/value inference and profiling. Optimization modules (fusion, quantization) work on the graph representation, while the inference engine (inference/infer.py) executes optimized graphs using a memory pool to minimize allocation overhead. Additional components include benchmarking scripts for performance analysis, model configuration management, and specialized support for LLM KV-cache handling. While the core library demonstrates solid engineering with reusable patterns like the node registry and graph rewrite framework, there are areas of technical drift due to loose dependency versioning and the broad scope requiring continuous maintenance to keep pace with evolving model architectures and ONNX specifications. The project appears maintained based on recent commit activity and community engagement (stars, forks, issue templates), though dependency management without version constraints poses a risk for long-term stability.