$ show facebookresearch/audiocraft
Audiocraft is a research codebase from Meta (Facebook Research) for state-of-the-art audio generation, including MusicGen, AudioGen, MAGNeT, JASCO, MultiBand Diffusion, and AudioSeal watermarking. It provides training and inference pipelines for text-to-music, text-to-sound, music continuation, style transfer, and symbolic conditioning (chords, drums, melody). The repository is primarily a research library with extensive configuration-driven training (Hydra/Dora), evaluation metrics (FAD, KLD, CLAP, ViSQOL), and interactive Gradio demos. It is not a production application but a collection of models, solvers, and utilities for audio generation research. The codebase is well-structured with modular components (conditioners, codebook patterns, audio effects) and includes unit tests, but relies on internal datasets and cluster paths that limit full reproducibility. Pretrained models are available via HuggingFace Hub for inference.
The LLM identified 5 files worth highlighting for reuse:
A comprehensive conditioning module implementing text, waveform, and symbolic conditioners (T5, CLAP, chroma, chords, drums, melody). Provides a modular base class hierarchy (BaseConditioner, TextConditioner, WaveformConditioner, JointEmbeddingConditioner) with dropout utilities (AttributeDropout, ClassifierFreeGuidanceDropout) and a ConditionFuser for fusion. This pattern is reusable for any conditional generation task and demonstrates strong software engineering in a research context.
Implements multiple codebook interleaving patterns (Parallel, Delayed, Unrolled, CoarseFirst, MusicLM) for residual vector quantization (RVQ) streams. The CodebooksPatternProvider hierarchy allows flexible sequence arrangement for autoregressive and non-autoregressive models. This is a reusable data engineering pattern for any multi-codebook audio tokenization system.
Provides robust audio I/O utilities (audio_read, audio_write) supporting multiple formats (wav, mp3, ogg, flac) via ffmpeg, with normalization strategies (peak, rms, loudness) and compression (mp3/ogg/flac). Includes convert_audio for sample rate/channel conversion and f32_pcm/i16_pcm for bit depth. This is a standalone, reusable audio processing module that solves common data engineering problems in audio pipelines.
Implements EmbeddingCache for precomputed embeddings (e.g., CLAP, chroma) with disk and memory caching, plus CachedBatchWriter/CachedBatchLoader for mini-batch caching during training. This pattern is reusable for any expensive feature extraction step in data pipelines, improving training efficiency.
A data augmentation pipeline for audio with effects like speed change, reverb, echo, pink noise, filtering, and MP3 compression. Uses a straight-through estimator for differentiable compression. This is a reusable audio augmentation module that can be applied to any audio dataset for robustness training.