$ show AIR-hl/llm-interview-code
This repository is a curated collection of Jupyter notebooks implementing core components of modern LLM architectures, intended for interview preparation. It covers attention mechanisms (MHA, GQA, MLA, KV-cache), normalization layers (LayerNorm, RMSNorm), functional components (RoPE, SwiGLU, LoRA, BPE, activation functions, cross-entropy, InfoNCE, sampling, quantization, SFT loss, tool calling parsing), and reinforcement learning losses (DPO, GRPO, DAPO, GSPO, KL divergence). Each notebook is a self-contained, minimal implementation of a specific algorithm, typically using PyTorch. The code is clean, concise, and well-commented, reflecting good engineering practices for educational purposes. While the repository is not a production application and lacks data pipelines, model training scripts, or continuous integration, it serves as a high-quality reference for understanding and reproducing the mathematical foundations of transformer-based language models. The hybrid nature lies in its focus on research-oriented ML components, but its structure is a pure code library rather than a data-driven research project.
The LLM identified 2 files worth highlighting for reuse:
A complete, self-contained implementation of the Byte Pair Encoding (BPE) tokenizer. The class BPETokenizer provides fit() and tokenize() methods, handling vocabulary generation, merge rule learning, and tokenization. It is isolated and can be extracted into a standalone tokenizer module with minimal changes, making it a reusable asset for any NLP project requiring subword tokenization.
Implements a streaming parser for OpenAI-compatible tool call events. The function parse_tool_calls() incrementally assembles function call arguments from delta events, providing a robust pattern for handling real-time API responses. This is a non-trivial, reusable utility for any application that needs to process structured tool outputs from LLMs.