LocalAI is a modular AI inference server providing OpenAI-compatible APIs for running diverse models (LLMs, vision, audio, video) locally. It features a polyglot architecture with Go core services, pluggable backends (C++/Go/Python), and a React-based web UI. The system supports distributed deployment via NATS/PostgreSQL, model gallery management, and multimodal processing. Key capabilities include model loading from Hugging Face, quantization, PII detection, and real-time transcription. The codebase demonstrates strong engineering practices with dependency injection, middleware chains, and clean separation of concerns.
genblaze is a production-grade Python monorepo SDK for orchestrating generative AI media pipelines — image, video, audio, text-to-speech, and chat — across 16+ provider integrations (OpenAI, Google Gemini/Imagen/Veo, NVIDIA NIM, GMICloud, Runway, Luma, Replicate, FAL, ElevenLabs, AssemblyAI, Hume, LMNT, Stability Audio, Decart, AtlasCloud). The public API is a chainable Pipeline/Step abstraction with batch, streaming, ingest, and agent-loop modes, plus a Click-based CLI (extract/verify/replay/index) for manifest inspection, a schema-first spec package generating TypeScript types from JSON Schemas, and ~30 runnable examples. Architecturally it is a monorepo: libs/core (genblaze-core) contains the framework (pipeline, providers, model registry, storage, media handlers, canonical hashing, signing, observability, webhooks), while each connector under libs/connectors/* is an independently versioned package registering itself via plugin entry points. Key entry points are Pipeline.run()/batch_run()/stream() for programmatic use and the genblaze command line; data flows through provider submit/poll/fetch_output lifecycles, optional caching, S3/B2 object-storage sinks with URL policies, and manifest generation that can be embedded directly into output media files. Two capabilities set this apart. First, provenance: every run produces a canonical, hash-verified Manifest (SHA-256 over deterministically serialized JSON), optionally Ed25519-signed, and byte-preservingly embedded into JPEG/PNG/WebP/MP4/audio files via XMP/atom splicing without re-encoding. Second, security and reliability engineering: SSRF protection with DNS pinning, credential redaction, ReDoS-safe pattern heuristics, ffmpeg injection escaping, configurable retry policies, model liveness probes, and discovery-aware validation (NATIVE/PARTIAL/NONE support levels) that reduces hardcoded slug rot. The project is actively maintained with unusually thorough tooling: golden hash-stability tests, a provider conformance test base class (~15 contract checks per connector), pin-parity and PyPI-metadata release gates, zizmor/dependabot CI, and live probe scripts that fail on API drift. Main risks are the inherent churn of 16+ third-party APIs (mitigated but not eliminated by mock-heavy tests) and a package/versioning layout complex enough that installs rely on documented make targets. Overall this is a strong, well-architected application SDK — suitable both for direct use and as a reference for multi-provider integration frameworks.
AudioX-Turbo is a unified framework for efficient anything-to-audio generation (text, video, audio conditions → audio) using Distribution Matching Distillation (DMD) adapted to flow matching and adversarial training. It supports six core tasks: Text-to-Audio, Text-to-Music, Video-to-Audio, Video-to-Music, Text+Video-to-Audio, and Text+Video-to-Music. The system includes training scripts, an interactive Gradio demo, and a modular architecture with interchangeable components (attention, bottlenecks, etc.) via a factory pattern. Key innovations enable high-quality 4-step audio generation, reducing computational cost by up to 25× compared to multi-step diffusion models.
Irodori-TTS is a state-of-the-art Japanese text-to-speech system built on rectified flow diffusion over DACVAE audio latents. It provides a complete training and inference stack: `train.py` supports full fine-tuning, LoRA, speaker inversion, and MeanFlow distillation with distributed training and WSD scheduling; `infer.py` offers a rich CLI with CFG, Sway Sampling, duration scaling, and watermarking; `gradio_app.py` and `gradio_app_voicedesign.py` expose interactive UIs for reference-based cloning and caption-conditioned voice design. The architecture is modular (`irodori_tts` package) with attention dispatch (FA3/SDPA), a hybrid Muon+AdamW optimizer, codec abstraction, and caching-friendly `InferenceRuntime`. While highly engineered for production use (INT4/INT8 quantization, safetensors, HuggingFace Hub integration), it remains a research-oriented codebase with experimental sampling algorithms and many YAML configs for different model sizes/versions. The code is clean and well-factored, though `train.py` is a large monolith and external dependencies (DACVAE, silentcipher) are pinned to specific commits, creating ecosystem fragility.
Audiocraft is a research codebase from Meta (Facebook Research) for state-of-the-art audio generation, including MusicGen, AudioGen, MAGNeT, JASCO, MultiBand Diffusion, and AudioSeal watermarking. It provides training and inference pipelines for text-to-music, text-to-sound, music continuation, style transfer, and symbolic conditioning (chords, drums, melody). The repository is primarily a research library with extensive configuration-driven training (Hydra/Dora), evaluation metrics (FAD, KLD, CLAP, ViSQOL), and interactive Gradio demos. It is not a production application but a collection of models, solvers, and utilities for audio generation research. The codebase is well-structured with modular components (conditioners, codebook patterns, audio effects) and includes unit tests, but relies on internal datasets and cluster paths that limit full reproducibility. Pretrained models are available via HuggingFace Hub for inference.
Amphion is a comprehensive open-source research toolkit for audio, music, and speech generation, developed by OpenMMLab. It covers multiple generative tasks including text-to-speech (TTS), singing voice conversion (SVC), voice conversion (VC), text-to-audio (TTA), neural audio codecs, and vocoders. The codebase is organized around config-driven experiments (JSON + shell scripts) with modular base classes for trainers, datasets, and inference. It ingests a wide variety of public datasets (LJSpeech, LibriTTS, VCTK, M4Singer, Emilia, etc.) and provides extensive preprocessing pipelines for feature extraction (mel-spectrograms, F0, energy, phonetic alignments, self-supervised features from HuBERT/Whisper). The primary research focus is on high-quality, zero-shot, and controllable generation using architectures like diffusion models, flow matching, GANs, autoregressive transformers, and vector-quantized codecs. The repository also includes evaluation metrics (F0, energy, intelligibility, speaker similarity) and visualization tools (SingVisio). While primarily a research codebase, it includes inference scripts and Gradio demos for practical use. External dependencies include PyTorch, HuggingFace Transformers, WeNet, Descript Audio Codec, and various pretrained models.
Higgs Audio v2 is an end-to-end multimodal LLM for expressive text-to-speech and audio generation from Boson AI. This repository contains the full v2 codebase: a Llama-3.2-3B backbone augmented with a DualFFN audio adapter, a unified 25 fps audio tokenizer (semantic + acoustic RVQ codes), ChatML-based data collators, and a serving engine with CUDA graph capture and multi-bucket static KV caches. The model supports zero-shot voice cloning, multi-speaker dialogue generation, and emergent capabilities like humming and background music. The README explicitly states that Higgs Audio v3 is released separately and this repo is only needed for v2/v2.5 reference. The code is organized as a Python package (boson_multimodal) with clear separation between model definition, audio processing, data collation, and serving. The serving engine (HiggsAudioServeEngine) provides both synchronous and async streaming generation. The generation loop is a custom implementation on top of HuggingFace's GenerationMixin that interleaves text token sampling with multi-codebook audio token sampling using a delay pattern. Notable engineering: DualFFN architecture (separate audio FFN paths in selected LLM layers), CUDA graph capture for decode, static KV cache buckets of different sizes, repetition-aware sampling (RAS) for audio tokens, and partially-frozen embedding/linear modules for efficient fine-tuning of new audio tokens. Caveats: the codebase pins transformers to <4.47.0 and relies on internal transformers APIs (LlamaDecoderLayer, LLAMA_ATTENTION_CLASSES), making it fragile against library upgrades. Some files contain broken imports (ac.py references a missing binary module; core_vq.py imports an external xcodec package) but these appear to be dead code paths. The README redirects users to v3, so this repo is primarily useful as v2 reference or for reproducing v2 results.