VoiceCraft is a token infilling neural codec language model for zero-shot speech editing and text-to-speech (TTS) using only a few seconds of reference audio. The system enables users to generate speech in novel voices or edit existing speech (deletion/insertion/substitution) while preserving speaker identity. It processes audio via EnCodec tokenization and models the token sequence with a transformer architecture featuring adaptive layer normalization and delayed pattern prediction for multi-codebook audio representation. The repository provides a complete pipeline from data preparation to deployment. Training scripts (main.py, z_scripts/*.sh) handle distributed training on datasets like Gigaspeech, while inference is accessible through standalone demos (tts_demo.py, speech_editing_demo.py), batch processing scripts (inference_tts_scale.py, inference_speech_editing_scale.py), a Gradio web interface (gradio_app.py), and a Replicate API wrapper (predict.py). Data processing modules (data/tokenizer.py, data/phonemize_encodec_encode_hf.py) manage phoneme extraction and audio tokenization using external tools like phonemizer and audiocraft's EnCodec. Key architectural strengths include reusable components such as the adaptive layer norm transformer encoder (models/modules/transformer.py), the DelayedPatternProvider for EnCodec's interleaved token structure (models/codebooks_patterns.py), and optimized training utilities like ScaledAdam optimizer and length-based batch sampling. The code demonstrates non-obvious skill in handling audio token generation challenges, though it relies on specific versions of external libraries and includes non-commercial use restrictions. Recent activity (commit 2026-05-30) indicates active maintenance with low technical drift due to dependency pinning.
This repository implements an end-to-end 3D object generation application that takes natural language descriptions and produces 3D mesh models (GLB format). It orchestrates three generative AI models: an LLM (Qwen3 or Llama) for scene planning, a diffusion model (SANA Sprint) for 2D image generation, and Microsoft TRELLIS for 3D generation. The system presents a Gradio web interface for user interaction and includes a Blender addon for direct import of generated assets. It supports two deployment modes: native PyTorch models (for local development) and NIM containers (for scalable inference). Key architectural patterns include a singleton GPU memory manager that coordinates model loading/unloading across VRAM-constrained GPUs, a guardrail service for content safety, and a conditional service abstraction that transparently switches between native and NIM backends based on configuration. The repository is well-structured with modular UI components and comprehensive installation scripts, though it is Windows-only and has complex event chains in the main UI file. The code is actively maintained with recent commits and pinned dependencies, but the heavy reliance on patched versions of TRELLIS and a specific Gradio version introduces some maintenance risk.
SpaTrackerV2 is a 3D point tracking system that processes video input to estimate camera motion, depth, and track user-selected points in 3D space. It integrates multiple depth estimation models (ZoeDepth, DepthAnything, DepthPro) with a transformer-based tracking architecture, offering both offline (full video) and online (sliding window) processing modes. The system provides a Gradio-based web interface for interactive point selection with SAM assistance and outputs 2D tracking videos, 3D visualizations (HTML), and numerical trajectory data. Evaluation modules support benchmarking against datasets like Sintel and CVO. The codebase demonstrates modular design with reusable components for I/O, geometry, alignment, and visualization, though it relies heavily on specific external models and libraries.