Lemonade is a local AI inference server and desktop application that provides OpenAI/Anthropic/Ollama-compatible APIs with support for multiple backends (llama.cpp, vLLM, whisper.cpp, Kokoro, etc.). The C++ server core handles model lifecycle, routing policies, multi-modal inference, and telemetry, while the React/TypeScript frontend (Tauri desktop app and web build) offers a full GUI for chat, image/audio generation, transcription, and model management. The repository also includes a CLI client, tray application, extensive integration tests, and automated documentation generation. Architecture follows a clean backend-descriptor registry pattern with generated boilerplate, platform abstraction for process/archive/path operations, and a sophisticated routing policy engine for model selection.
HailoRT is a production runtime SDK for Hailo AI accelerators. It provides a C/C++ library (libhailort) for loading compiled HEF models, configuring network groups, managing virtual streams (vstreams), and executing synchronous/asynchronous inference on Hailo-8/10/15 devices over PCIe, USB, or Ethernet. The repository also contains a command-line tool (hailortcli) for scanning, benchmarking, power measurement, logging, and firmware control; Python bindings (pyHailoRT); a GStreamer plugin suite (hailonet, hailosend/hailorecv, hailodevicestats); and an RPC-based client/server layer (HRPC) that allows a remote hailo service to own devices and GenAI sessions. It is a hybrid repo in the sense that it includes tutorial notebooks and algorithm modules (NMS, tracking, quantization) that resemble research code, but the dominant character is a software application/SDK. The architecture is layered: a common OS abstraction layer (files, sockets, mmap, events, threads), a VDMA/driver layer for hardware DMA and streams, a core-op/resource-manager layer for memory planning and context switching, a net-flow pipeline layer for pre/post-processing (NMS, YOLO, softmax, quantization), and high-level APIs for infer models and GenAI (LLM/VLM/speech2text). The code uses an Expected monad instead of exceptions, pimpl idioms for ABI stability, buffer pools for high-throughput transfers, and protobuf serialization for HRPC. The codebase is large and professional, with strong modularization, but also carries noticeable internal debt: many HRT-TODO markers, duplicated pipeline construction, hardcoded sysfs/tmp paths, deprecated API overloads, and partial Windows/QNX platform stubs. Maintenance is active; the repo is versioned (v5.4.0) and recently committed. Technical drift is medium because core dependencies are current and the project is clearly under active commercial development, yet some legacy/deprecated surfaces and platform-specific ifdefs remain. As a library, meaningful inference requires Hailo hardware; the build is CMake-driven with external dependencies managed via CMake scripts.
This repository is a collection of C++ sample programs for deploying deep learning models on AXERA AI SoCs (NPU chips). It provides ready-to-run inference examples for a wide range of computer vision tasks including object detection (YOLOv5/6/7/8/9/10/11/26, YOLOX, YOLO-World, PPYOLOE+, RT-DETR, SCRFD), instance segmentation, oriented object detection (OBB), pose estimation, depth estimation, semantic segmentation, stereo matching, and more. Each example is a standalone command-line executable that loads a pre-compiled .axmodel, runs inference via the AX_ENGINE API, post-processes the outputs, and saves a result image. The code is organized per target chip (ax650, ax637, ax620e, ax615) with near-identical copies of examples across platforms, differing only in system initialization and minor chip-specific logic. The architecture is not a library or framework but a set of reference implementations demonstrating the integration of AXERA's NPU SDK. The primary data flow is: load image → preprocess (letterbox resize, color conversion) → push to engine → run synchronous inference → decode model outputs (e.g., anchor-based proposals, DFL, NMS) → draw bounding boxes/keypoints/masks → save output. External dependencies include the AX_ENGINE runtime, OpenCV for image processing, and standard C++ libraries. The build system uses CMake with per-chip toolchain files and GitHub Actions CI for cross-compilation verification. While the code is functional and covers a broad model zoo, it exhibits significant duplication across chip directories and hardcoded configuration (thresholds, image sizes, class names). The monolithic detection.hpp (~3500 lines) bundles many model-specific post-processing functions, increasing maintenance burden. There are no automated tests, and manual memory management is prevalent. Despite these quality issues, the repository serves its purpose as a practical reference for developers integrating AXERA hardware, and it stays current with recent model releases (YOLOv13, YOLO26, Depth Anything V2).