BentoML is a machine-learning model serving framework and deployment platform. In production it lets developers define inference services via a decorator-based SDK (`@bentoml.service`, `@bentoml.api`), package those services into versioned "Bentos" with locked Python dependencies, and build OCI container images or deploy to Kubernetes, EC2, or BentoCloud. The serving layer supports both HTTP (Starlette/Uvicorn) and gRPC, with adaptive batching, model runners, Prometheus metrics, OpenTelemetry tracing, and pluggable monitoring. A comprehensive CLI (`bentoml serve/build/containerize/deploy`) is the primary operational surface, while the `_bentoml_sdk` and `_bentoml_impl` packages provide the user-facing and internal implementation layers respectively. Architecturally, the project separates concern into model registry, service definition, packaging, server workers, and cloud clients. Model artifacts are persisted in a versioned local store via framework adapters for PyTorch, TensorFlow, Transformers, ONNX, scikit-learn, MLflow, XGBoost, LightGBM, and others. Data flows from trained model artifacts into the registry, get referenced by service code, get bundled by `bentoml build` into self-contained Bento packages, and are served by Circus-managed worker processes. The core request path involves serialization, batching, resource allocation, and process supervision, all implemented in internal modules with generally clean abstractions. The repository is mid-migration from a legacy `bentoml.legacy.Service` API to a newer `_bentoml_sdk`, which creates visible duplication: two HTTP client stacks, multiple protobuf versions, and deprecation shims. This is a real maintenance cost but not a functional blocker; the modern SDK is actively developed, documented, and tested. The repo is clearly an application/framework rather than a data-research project; research-like artifacts are limited to test fixtures, training examples, and ML framework integration tests.
This repository is a container build system and runtime for NVIDIA Jetson edge AI devices. It provides a modular, composable framework for building Docker images pre-configured with AI/ML libraries (LLMs, VLMs, robotics, speech, computer vision) optimized for Jetson's ARM64 + CUDA architecture. The system includes a CLI (`jetson-containers`), dependency resolution, multi-stage Docker builds, automated testing, and CI/CD workflow generation. While primarily a build tool, it also ships hundreds of ready-to-use container definitions for popular frameworks like PyTorch, TensorFlow, vLLM, ComfyUI, ROS 2, and Home Assistant. The repository has a hybrid character: the core is an application build system, but many package definitions are research-oriented (e.g., attention kernels, diffusion models, physical AI), reflecting its dual role as both a production deployment tool and an experimentation platform for edge AI.
GraXpert is a desktop application (with CLI support) for astronomical image processing, specifically for removing background gradients from astrophotos. It provides multiple interpolation methods (RBF, Splines, Kriging) that require manual selection of background sample points, as well as an AI-based method that automatically extracts the background using ONNX Runtime models. The application also includes AI-powered denoising and deconvolution (object-only and stars-only) features. The architecture is a monolithic Python application using customtkinter for the GUI, with multiprocessing for parallel interpolation and onnxruntime for AI inference. It supports common astronomical image formats (FITS, XISF, TIFF) and includes a command-line interface for batch processing. The codebase is well-structured with design patterns (command, repository, event bus), localization, and CI/CD pipelines for cross-platform builds.
This repository implements a GitHub Action (`actions/ai-inference`) that runs AI model inference inside GitHub Actions workflows by shelling out to the GitHub Copilot CLI. The action accepts a user prompt (inline or from a file), an optional system prompt, a model identifier, and various configuration options. It builds a combined prompt string, spawns the Copilot CLI with `-p`, `-s`, and `--no-ask-user` flags, captures the model's text response, and exposes it via `response` and `response-file` outputs. The architecture is a single Node.js action (bundled into `dist/index.js`) with modular TypeScript source files: `main.ts` orchestrates input parsing and provider dispatch, `copilot.ts` handles CLI invocation and error handling, `prompt.ts` provides structured `.prompt.yml` file parsing with template variable substitution, and `helpers.ts` contains utility functions. The action is tightly coupled to the Copilot CLI, which must be pre-installed and authenticated on the runner. It supports only the `copilot` provider. The codebase is well-structured, includes unit tests, and follows modern TypeScript practices.
HailoRT is a production runtime SDK for Hailo AI accelerators. It provides a C/C++ library (libhailort) for loading compiled HEF models, configuring network groups, managing virtual streams (vstreams), and executing synchronous/asynchronous inference on Hailo-8/10/15 devices over PCIe, USB, or Ethernet. The repository also contains a command-line tool (hailortcli) for scanning, benchmarking, power measurement, logging, and firmware control; Python bindings (pyHailoRT); a GStreamer plugin suite (hailonet, hailosend/hailorecv, hailodevicestats); and an RPC-based client/server layer (HRPC) that allows a remote hailo service to own devices and GenAI sessions. It is a hybrid repo in the sense that it includes tutorial notebooks and algorithm modules (NMS, tracking, quantization) that resemble research code, but the dominant character is a software application/SDK. The architecture is layered: a common OS abstraction layer (files, sockets, mmap, events, threads), a VDMA/driver layer for hardware DMA and streams, a core-op/resource-manager layer for memory planning and context switching, a net-flow pipeline layer for pre/post-processing (NMS, YOLO, softmax, quantization), and high-level APIs for infer models and GenAI (LLM/VLM/speech2text). The code uses an Expected monad instead of exceptions, pimpl idioms for ABI stability, buffer pools for high-throughput transfers, and protobuf serialization for HRPC. The codebase is large and professional, with strong modularization, but also carries noticeable internal debt: many HRT-TODO markers, duplicated pipeline construction, hardcoded sysfs/tmp paths, deprecated API overloads, and partial Windows/QNX platform stubs. Maintenance is active; the repo is versioned (v5.4.0) and recently committed. Technical drift is medium because core dependencies are current and the project is clearly under active commercial development, yet some legacy/deprecated surfaces and platform-specific ifdefs remain. As a library, meaningful inference requires Hailo hardware; the build is CMake-driven with external dependencies managed via CMake scripts.
This repository serves as the distribution and documentation hub for the Sage application, a proprietary local AI inference engine. It does not contain the source code of the engine itself, which is distributed as compiled binaries via GitHub Releases. Instead, the repository provides platform-specific installation scripts (PowerShell for Windows, shell for macOS), detailed API documentation, release notes, and legal information. The Sage application enables users to run AI models locally on their machines, exposing OpenAI-compatible and Ollama-native APIs through a local server. The documentation outlines how to interact with the server, including endpoints for chat completions, embeddings, and model management. The installation scripts automate the download and setup process, handling platform-specific considerations such as clearing quarantine flags on macOS and ensuring proper TLS settings for Windows PowerShell. While the repository lacks the application's source code, it maintains high-quality documentation and functional installation scripts that facilitate the deployment and use of the Sage desktop application. The project is actively maintained, with recent updates reflecting new features and platform support.
This repository is a curated collection of external resources (articles, papers, tools, videos) focused on AI inference engineering. It covers topics such as LLM serving, GPU kernel programming, attention mechanisms, quantization, distributed inference, and production deployment. The repository contains no application code, data pipelines, or executable scripts; it is purely a reference list organized by topic and difficulty tier. Despite being classified as an 'application' for structural purposes, it functions as a learning guide and bibliography, not a software system. The value lies in the curation and organization of resources, not in any original code or data assets.