$ show google-ai-edge/mediapipe
MediaPipe is a cross-platform framework for building multimodal machine learning pipelines, primarily targeting on-device inference. It provides a graph-based execution model where modular 'calculators' process typed 'packets' through directed acyclic graphs. The framework includes a rich library of pre-built calculators for image processing, tensor manipulation, audio analysis, and ML inference, as well as high-level task APIs for vision (face/hand/pose landmarking, image classification, segmentation), text (classification, language detection), audio (classification), and generative AI (LLM inference). The architecture is layered: a core C++ framework (mediapipe/framework/) defines the graph runtime, stream handlers, and packet infrastructure; a calculator library (mediapipe/calculators/) provides reusable processing nodes; and task-specific APIs (mediapipe/tasks/) offer consistent interfaces across C++, Java, Python, iOS, Web, and C. The system relies on TensorFlow Lite (LiteRT) for model execution, OpenCV for image operations, protobuf for configuration, and Abseil for utilities. Key entry points include the Python package (mediapipe/__init__.py), the C++ task APIs (mediapipe/tasks/cc/), and the web TypeScript modules (mediapipe/web/). The codebase is large and actively developed, with ongoing migration from older API patterns to newer ones (api2/api3), and some deprecated components (legacy Python solutions, older calculator interfaces).
The LLM identified 5 files worth highlighting for reuse:
Implements a generic temporal smoothing calculator using the OneEuroFilter algorithm. The header provides a reusable template function `SmoothLandmarks` that applies low-pass filtering with adaptive cutoff frequency based on velocity. This is a non-trivial signal processing pattern that can be extracted and applied to any time-series landmark data (e.g., hand, face, pose) with minimal adaptation. The implementation is self-contained, uses only standard types and the OneEuroFilter utility, and demonstrates clean separation of filtering logic from calculator boilerplate.
Provides a type-safe, variadic template for merging multiple input streams into a single output stream. The `Merge` function uses compile-time tag dispatching to handle arbitrary numbers of inputs, producing a vector of packets. This is a reusable building block for graph construction that avoids runtime overhead and demonstrates advanced C++ template techniques (parameter packs, constexpr evaluation). It can be extracted and used in any MediaPipe-like pipeline framework.
A concrete calculator that applies the OneEuroFilter to smooth landmark positions over time. It uses a velocity-based adaptive cutoff to reduce jitter while preserving low-latency response. The implementation is self-contained, with clear separation of filter state per landmark, and includes configurable options for min cutoff, beta, and derivate cutoff. This is a production-quality implementation of a common computer vision problem that can be reused in any landmark tracking pipeline.
Defines the `Tensor` class, a unified abstraction for multi-dimensional arrays that supports CPU, GPU (OpenGL, Metal, AHWB), and WebGPU backends. It provides zero-copy views, explicit memory management (pools, shared file descriptors), and type-safe accessors. The design is non-trivial: it uses a pimpl pattern to hide backend-specific implementations, supports on-device tensor operations, and integrates with MediaPipe's memory manager service. This component is highly reusable for any ML inference framework needing cross-platform tensor handling.
Introduces the `Node` class template for defining calculators with compile-time contracts. It uses variadic templates and constexpr strings to specify input/output ports and options, enabling static validation of graph connections. This is a significant improvement over the runtime protobuf-based contract system, demonstrating advanced C++ metaprogramming (type lists, tag dispatching, constexpr evaluation). The pattern is reusable for any framework that needs type-safe, declarative node definitions.