$ show ml-explore/mlx-examples
This repository is a collection of standalone examples demonstrating the MLX framework (Apple's machine learning framework for Apple Silicon). It covers a wide range of deep learning models including LLMs (Mistral, Mixtral, Llama, T5), image generation (Stable Diffusion, FLUX, CLIP), video generation (Wan2.1), audio (Whisper, MusicGen, EnCodec), segmentation (SAM), and more. Each example is self-contained with its own entry point (CLI or Python script), model conversion utilities, and often a training/fine-tuning script. The repository is primarily an application-focused showcase with some research elements (e.g., normalizing flows, GCN). Production entry points include `whisper/mlx_whisper/cli.py` for transcription, `flux/txt2image.py` for image generation, `video/wan2.1/txt2video.py` for video generation, and `lora/lora.py` for fine-tuning. All examples depend on the MLX library and often bundle conversion from PyTorch weights. The architecture is a flat collection of independent projects, not a unified framework, leading to some code duplication (e.g., T5 and EnCodec are copied into multiple directories). However, several components are well-crafted and reusable, notably the Whisper decoding pipeline, the LoRA adapter module, and the flow-matching samplers for diffusion models.
The LLM identified 4 files worth highlighting for reuse:
A modular decoding framework for Whisper ASR. Defines abstract classes TokenDecoder, LogitFilter, SequenceRanker, and Inference, with concrete implementations (GreedyDecoder, SuppressBlank, GreedyRanker, etc.). The DecodingTask orchestrates these components. This design allows easy customization of decoding strategies (e.g., beam search, temperature fallback) and is cleanly separated from the model definition. Could be extracted and reused for any autoregressive sequence model.
Implementation of LoRA (Low-Rank Adaptation) for linear layers. Provides a LoRALinear class that wraps nn.Linear with learnable rank decomposition matrices. Includes class methods `from_linear` and `from_base` for easy insertion into existing models, and `fuse` to merge weights back. This is a clean, reusable pattern for parameter-efficient fine-tuning of any MLX model.
Implements flow-matching samplers for video diffusion: FlowEulerDiscreteScheduler, FlowUniPCMultistepScheduler, and a TeaCache-based caching scheduler. These are self-contained, parameterized, and follow a common scheduler interface. They can be reused in any diffusion/flow-matching pipeline, not just Wan2.1.
High-level transcription pipeline that orchestrates audio loading, model inference, language detection, decoding with fallback, word-level timestamps (via DTW), and hallucination filtering. It is a self-contained function (`transcribe`) that handles all edge cases and can be extracted as a standalone service for speech-to-text.