$ show hailo-ai/hailort
HailoRT is a production runtime SDK for Hailo AI accelerators. It provides a C/C++ library (libhailort) for loading compiled HEF models, configuring network groups, managing virtual streams (vstreams), and executing synchronous/asynchronous inference on Hailo-8/10/15 devices over PCIe, USB, or Ethernet. The repository also contains a command-line tool (hailortcli) for scanning, benchmarking, power measurement, logging, and firmware control; Python bindings (pyHailoRT); a GStreamer plugin suite (hailonet, hailosend/hailorecv, hailodevicestats); and an RPC-based client/server layer (HRPC) that allows a remote hailo service to own devices and GenAI sessions. It is a hybrid repo in the sense that it includes tutorial notebooks and algorithm modules (NMS, tracking, quantization) that resemble research code, but the dominant character is a software application/SDK. The architecture is layered: a common OS abstraction layer (files, sockets, mmap, events, threads), a VDMA/driver layer for hardware DMA and streams, a core-op/resource-manager layer for memory planning and context switching, a net-flow pipeline layer for pre/post-processing (NMS, YOLO, softmax, quantization), and high-level APIs for infer models and GenAI (LLM/VLM/speech2text). The code uses an Expected monad instead of exceptions, pimpl idioms for ABI stability, buffer pools for high-throughput transfers, and protobuf serialization for HRPC. The codebase is large and professional, with strong modularization, but also carries noticeable internal debt: many HRT-TODO markers, duplicated pipeline construction, hardcoded sysfs/tmp paths, deprecated API overloads, and partial Windows/QNX platform stubs. Maintenance is active; the repo is versioned (v5.4.0) and recently committed. Technical drift is medium because core dependencies are current and the project is clearly under active commercial development, yet some legacy/deprecated surfaces and platform-specific ifdefs remain. As a library, meaningful inference requires Hailo hardware; the build is CMake-driven with external dependencies managed via CMake scripts.
The LLM identified 6 files worth highlighting for reuse:
Self-contained lock-free SPSC ring buffer using power-of-two masking and iterator support. Suitable for any streaming pipeline needing a low-latency bounded queue and easily extractable into a separate project.
Thread-safe SPSC queue with shutdown event and wait-or-shutdown blocking semantics. A clean, reusable building block for producer-consumer work queues in C++ applications.
Generic object pool for efficient reuse of pre-allocated objects, avoiding per-operation allocation in hot paths. Demonstrates a well-contained pool implementation with minimal dependencies.
RAII measurement macros plus accumulator classes for latency, FPS, and numeric values using Welford's online algorithm. Can be lifted into any C++ performance tooling to compute mean/variance without storing full data series.
BufferPool and FastBufferPool abstractions for cyclic buffer reuse, used throughout stream I/O. A self-contained memory management pattern for high-throughput data movement that can be reused in other low-latency systems.
Header-only Expected<T> monad with CHECK/TRY macros for exception-free error propagation. Demonstrates a robust C++ error-handling idiom and is conceptually portable despite its Hailo-specific status codes.