WhisperPlus is a Python library that provides enhanced speech processing capabilities built around Whisper models. It offers a modular pipeline architecture for speech-to-text transcription with optimizations like quantization and flash attention, text summarization (including long-text handling), speaker diarization, text-to-speech conversion, video auto-captioning, and retrieval-augmented generation for chatting with video content. The library integrates with Hugging Face Transformers, Apple MLX for optimized inference on Apple silicon, and various external services like YouTube downloading and vector databases for RAG functionality. The architecture follows a modular design with clearly separated pipelines for each functionality: core speech-to-text (whisper.py), summarization (summarization.py, long_text_summarization.py), text-to-speech (text2speech.py), diarization (whisper_diarize.py), auto-captioning (whisper_autocaption.py), and RAG chatbots (lancedb_chatbot.py, autollm_chatbot.py). Utility modules handle YouTube media downloading (download_utils.py) and text formatting (text_utils.py). The library provides both programmatic API access and a Gradio-based web interface (app.py) for interactive use, along with a CLI entry point (cli.py). Data flows typically begin with audio/video input (from files or YouTube URLs), proceed through specialized processing pipelines (transcription, summarization, etc.), and produce text, audio, or video outputs. For RAG features, transcripts are processed into vector stores enabling question-answering interactions. The library relies on established dependencies including torch, transformers, ffmpeg, pyannote.audio for diarization, and MLX-specific libraries for Apple silicon optimization, demonstrating practical integration of state-of-the-art models into an accessible toolkit.
Promptify is a Python library for performing NLP tasks using large language models (LLMs) with structured, type-safe outputs. It provides a scikit-learn-like interface where users instantiate task classes (e.g., NER, Classify, QA, Summarize) and call them with plain text. The library uses LiteLLM as a universal backend to support 100+ providers (OpenAI, Anthropic, Ollama, etc.), Pydantic v2 for output schemas, Jinja2 for prompt templates, and tenacity for retries. Architecture follows a clean separation: Task classes (promptify/tasks) build prompts via PromptBuilder, send them via LLMEngine (which wraps LiteLLM), and parse responses via a safe Parser that never uses eval(). The package also includes a built-in evaluation framework (promptify/eval) with metrics for precision, recall, F1, accuracy, exact match, and ROUGE, plus dataset loading utilities. The code is well-structured, typed, and includes comprehensive tests. The primary artifact is the "promptify" Python package, installable via pip. The repository is not a data research project; it is a software library for developers.
MMSearch is a framework for evaluating large multimodal models (LMMs) as multi-modal search engines. It provides a pipeline (MMSearch-Engine) that enables any LMM to perform multimodal search by generating queries, retrieving web content, reranking results, and summarizing answers. The repository includes evaluation scripts for four tasks (end2end, requery, rerank, summarization) and a demo for running the engine on custom queries. The system relies on external dependencies like DuckDuckGo search, Playwright for web scraping, and Hugging Face datasets. The architecture is modular with separate components for model loading, prompt templating, web retrieval, content processing, and scoring.
BLURR is a library that integrates Hugging Face transformers with fastai for NLP tasks. It provides high-level, mid-level, and low-level APIs to train and deploy transformer models for text classification, token classification, question answering, summarization, translation, and language modeling. The library leverages fastai's data block API and learner system, wrapping Hugging Face models to work seamlessly within the fastai training loop. Key components include custom data blocks for text processing, model wrappers for Hugging Face integration, specialized loss functions, and callbacks for optimization. The repository includes extensive examples and notebooks demonstrating usage with standard datasets like GLUE, SQuAD, and CNN/DailyMail.