$ show backblaze-labs/genblaze
genblaze is a production-grade Python monorepo SDK for orchestrating generative AI media pipelines — image, video, audio, text-to-speech, and chat — across 16+ provider integrations (OpenAI, Google Gemini/Imagen/Veo, NVIDIA NIM, GMICloud, Runway, Luma, Replicate, FAL, ElevenLabs, AssemblyAI, Hume, LMNT, Stability Audio, Decart, AtlasCloud). The public API is a chainable Pipeline/Step abstraction with batch, streaming, ingest, and agent-loop modes, plus a Click-based CLI (extract/verify/replay/index) for manifest inspection, a schema-first spec package generating TypeScript types from JSON Schemas, and ~30 runnable examples. Architecturally it is a monorepo: libs/core (genblaze-core) contains the framework (pipeline, providers, model registry, storage, media handlers, canonical hashing, signing, observability, webhooks), while each connector under libs/connectors/* is an independently versioned package registering itself via plugin entry points. Key entry points are Pipeline.run()/batch_run()/stream() for programmatic use and the genblaze command line; data flows through provider submit/poll/fetch_output lifecycles, optional caching, S3/B2 object-storage sinks with URL policies, and manifest generation that can be embedded directly into output media files. Two capabilities set this apart. First, provenance: every run produces a canonical, hash-verified Manifest (SHA-256 over deterministically serialized JSON), optionally Ed25519-signed, and byte-preservingly embedded into JPEG/PNG/WebP/MP4/audio files via XMP/atom splicing without re-encoding. Second, security and reliability engineering: SSRF protection with DNS pinning, credential redaction, ReDoS-safe pattern heuristics, ffmpeg injection escaping, configurable retry policies, model liveness probes, and discovery-aware validation (NATIVE/PARTIAL/NONE support levels) that reduces hardcoded slug rot. The project is actively maintained with unusually thorough tooling: golden hash-stability tests, a provider conformance test base class (~15 contract checks per connector), pin-parity and PyPI-metadata release gates, zizmor/dependabot CI, and live probe scripts that fail on API drift. Main risks are the inherent churn of 16+ third-party APIs (mitigated but not eliminated by mock-heavy tests) and a package/versioning layout complex enough that installs rely on documented make targets. Overall this is a strong, well-architected application SDK — suitable both for direct use and as a reference for multi-provider integration frameworks.
The LLM identified 7 files worth highlighting for reuse:
Deterministic JSON serializer producing stable SHA-256 hashes across versions — sorted keys, NFC unicode normalization, normalized floats, depth guards. Backed by golden tests and property-based tests. Extractable into any project needing content-addressed, tamper-evident payloads.
Pattern-keyed ModelFamily registry that resolves provider slugs by regex rather than hardcoded lists, with discovery support levels (NATIVE/PARTIAL/NONE), graded validation outcomes (OK_AUTHORITATIVE, OK_PROVISIONAL, NOT_FOUND, UNKNOWN_PERMISSIVE), single-flight cache, and fallback probes. The anti-slug-rot design is reusable for any API integration layer.
SSRF-safe outbound request helpers: resolve_ssrf and open_pinned_https_connection with DNS pinning, blocklist enforcement, TLS 1.2+; plus credential-sanitization regexes. Drives both pipeline asset validation and the webhook notifier. Extractable into any service that fetches user-supplied URLs.
JPEG XMP segment splicer that embeds/extracts provenance manifests without re-encoding pixel data — preserves EXIF/ICC, handles progressive JPEGs, re-embeds without duplicating XMP packets, uses atomic writes. Demonstrates deep binary-format craft; the approach generalizes to any container format needing non-destructive metadata injection.
Frozen RetryPolicy dataclass with configurable attempts, exponential backoff with jitter, retryable-code sets, and idempotency key strategies; plus a call_with_rate_limit_retry helper honoring Retry-After. Small, dependency-free, and safely reusable across HTTP client layers.
Frozen, hashable SSE-KMS/SSE-C encryption value object that generates per-operation boto3 kwargs (put/head/get/copy) from a single declared policy, eliminating a class of mismatched-parameter bugs. A clean example of a typed configuration value object: extractable into any Python project wrapping boto3.
Pluggable PricingStrategy framework (per_unit, per_input_chars, per_output_second, tiered, bucketed) driven by a declared PricingContext; the SDK ships zero hardcoded prices and enforces that via lint. A clean strategy-based extension point any usage-billed SDK could adopt.