FlashAttention is a high-performance library implementing IO-aware exact attention for transformer models, with CUDA kernels targeting NVIDIA Ampere (SM80), Hopper (SM90), and Blackwell (SM100) architectures, plus a Python CuTe DSL implementation (FA4), a Composable Kernel backend for AMD, and Triton fallbacks. The library exposes PyTorch-integrated Python APIs (flash_attn_func, flash_attn_varlen_func, flash_attn_with_kvcache) backed by custom CUDA extensions with autograd support. It also ships fused ops (dense layers, layer norm, cross-entropy), model definitions with HuggingFace state-dict remapping (GPT, BERT, LLaMA, Falcon, Baichuan, OPT, GPT-J, GPT-NeoX, ViT), and a PyTorch Lightning + Hydra training pipeline for GPT-style models. The dominant character is a systems library, though it has a hybrid research flavor: experimental kernels for unreleased hardware, benchmark suites, and a training sub-package all coexist in one monorepo. The architecture is a layered monolith: a Python API layer dispatches to multiple kernel backends (CUDA C++ for FA2, CUTLASS/CuTe for FA3, Python CuTe DSL for FA4, CK for AMD, Triton for portability). The core algorithmic contributions are the online softmax with rescaling, tiled attention with shared-memory staging, warp-specialized producer/consumer pipelines using TMA and GMMA, persistent tile schedulers, and deterministic backward passes with semaphore-based reductions. The training sub-package is a secondary component demonstrating the kernels in full GPT training runs with tensor/sequence parallelism. The codebase is actively maintained with extensive CI/CD (including Apptainer-based FA4 CI), a large test suite covering correctness against reference implementations and HuggingFace models, and performance benchmarks. However, it carries significant complexity: multiple overlapping backends with duplicated logic and inconsistent feature support, a complex build system with many environment variables, a monolithic gpt.py model file, and deprecated components (csrc/layer_norm is explicitly deprecated in favor of Triton). The kernel code is world-class and worth studying; the surrounding infrastructure is functional but organically grown.