$ show kadirnar/whisper-plus
WhisperPlus is a Python library that provides enhanced speech processing capabilities built around Whisper models. It offers a modular pipeline architecture for speech-to-text transcription with optimizations like quantization and flash attention, text summarization (including long-text handling), speaker diarization, text-to-speech conversion, video auto-captioning, and retrieval-augmented generation for chatting with video content. The library integrates with Hugging Face Transformers, Apple MLX for optimized inference on Apple silicon, and various external services like YouTube downloading and vector databases for RAG functionality. The architecture follows a modular design with clearly separated pipelines for each functionality: core speech-to-text (whisper.py), summarization (summarization.py, long_text_summarization.py), text-to-speech (text2speech.py), diarization (whisper_diarize.py), auto-captioning (whisper_autocaption.py), and RAG chatbots (lancedb_chatbot.py, autollm_chatbot.py). Utility modules handle YouTube media downloading (download_utils.py) and text formatting (text_utils.py). The library provides both programmatic API access and a Gradio-based web interface (app.py) for interactive use, along with a CLI entry point (cli.py). Data flows typically begin with audio/video input (from files or YouTube URLs), proceed through specialized processing pipelines (transcription, summarization, etc.), and produce text, audio, or video outputs. For RAG features, transcripts are processed into vector stores enabling question-answering interactions. The library relies on established dependencies including torch, transformers, ffmpeg, pyannote.audio for diarization, and MLX-specific libraries for Apple silicon optimization, demonstrating practical integration of state-of-the-art models into an accessible toolkit.
The LLM identified 2 files worth highlighting for reuse:
Provides robust YouTube video/audio downloading functionality with proper error handling, logging, and pathlib usage. Implements clean separation of concerns for MP3 and MP4 extraction using yt_dlp with FFmpeg post-processing. The component is self-contained, easily extractable to other projects, and demonstrates non-obvious skill in media handling through its thoughtful interface and error management.
Implements intelligent long-text summarization by automatically chunking input based on model token limits while preserving semantic coherence. Features tokenization-aware chunking, proper handling of special tokens, and clean recombination of summaries. The component demonstrates non-obvious skill in addressing transformer sequence length limitations through its chunking strategy and could be adapted to other seq2seq models with minimal changes.