$ account MuLabPKU
MuLabPKU/TransArch is a collection of research projects focused on efficient attention mechanisms for large language models (LLMs), particularly variants of Multi-Head Latent Attention (MLA). The repository provides software artifacts for converting standard attention to hardware-friendly forms (e.g., GQA→MLA via TransMLA, MLA→GQLA via per-group PCA, cross-layer SVD pruning via CLOVER), custom GPU kernels for sparse and FP8 attention (LISA, MISA), and integration patches for production inference frameworks like vLLM and SGLang. Each sub-project is self-contained, with its own entry points, configuration, and evaluation scripts, but they share common calibration utilities and model conversion patterns. The architecture is modular: model conversion pipelines (e.g., `converter.py`, `compression.py`) ingest pre-trained weights and calibration data to produce transformed models, which are then served via patched vLLM or SGLang instances. Custom CUDA/Triton kernels (e.g., LISA's linear and threshold scorers, MISA's tilelang-based sparse attention) are compiled and registered as custom ops. The OpenPWS sub-project provides a rigorous reproduction framework for benchmark experiments, with hash-verified inputs and provenance tracking. While the primary character is application-oriented (delivering reusable software for LLM inference optimization), the repo also contains research evaluation scripts, benchmark results (JSONL files), and calibration datasets, giving it a hybrid nature. The code is research-grade: functional and innovative, but with limited test coverage, hardware lock-in (SM100, Blackwell), and dependency on specific framework versions. Several sub-projects are marked as 'code coming soon' or are preprints, indicating ongoing development.