vLLM is a high-throughput, memory-efficient inference and serving engine for large language models (LLMs). It implements PagedAttention for near-zero waste KV cache management, continuous batching, chunked prefill, prefix caching, CUDA/HIP graphs, speculative decoding, and torch.compile. The system serves OpenAI-compatible APIs (chat completions, completions, embeddings, rerank, speech-to-text), Anthropic Messages API, Cohere API, and custom pooling/classification endpoints. It supports 200+ model architectures including dense, MoE, hybrid SSM/attention, and multimodal models (vision, audio, video). The architecture is a hybrid Python/Rust system: a Python core handles model execution, scheduling, and kernel dispatch, while a Rust frontend provides a high-performance HTTP/gRPC server with middleware for auth, metrics, and load management. Key entry points include the `vllm` CLI (serve, bench, launch), the `LLM` class for offline inference, and the OpenAI-compatible API server. External dependencies include PyTorch, HuggingFace Transformers, FlashAttention, CUTLASS, Triton, NCCL/RCCL, and various quantization libraries. The project is production-grade with extensive CI/CD, Docker images, Helm charts, and deployment integrations for Kubernetes, Ray, SageMaker, and more.
JAX is a high-performance numerical computing library developed by Google Research. It provides a NumPy-compatible API (jax.numpy), automatic differentiation (jax.grad), just-in-time compilation via XLA (jax.jit), automatic vectorization (jax.vmap), and large-scale parallelization (jax.pmap, jax.shard_map). The library is designed for machine learning research and scientific computing, supporting CPU, GPU (NVIDIA, AMD, Intel), and Google TPU hardware. The system is a hybrid Python/C++ library. The Python layer (jax/) handles function tracing into a JAX intermediate representation (JAXpr), applies transformations (autodiff, batching, sharding), and lowers the computation to MLIR/StableHLO. The C++ backend (jaxlib/) compiles this representation via XLA and manages execution on target hardware. It includes a sophisticated plugin system for hardware backends and a custom kernel DSL (Pallas/Mosaic) for writing high-performance fused operations. While primarily an application/library, JAX has strong research roots. It is explicitly described as a research project, and its experimental/ directory contains rapidly evolving features like custom kernel authoring (Pallas), TensorFlow interop (jax2tf), and sparse array support. The codebase is a blend of production-grade engineering (robust CI, deprecation policies, extensive testing) and cutting-edge research (new autodiff algorithms, novel sharding strategies, experimental hardware support). The repository contains several high-quality, isolated components that are reusable in other Python/C++ projects. These include a file-based LRU cache, a weak-reference LRU cache, thread-safe reentrant hash maps, a lazy module loader, a deprecation framework, and a traceback filtering utility. The core autodiff and JIT compilation patterns are the primary value, but these supporting utilities demonstrate excellent software craftsmanship.
Jaxformer is a JAX-based library for training large language models (LLMs) on TPUs with data and model parallelism. It provides a complete training loop supporting distributed training across TPU cores, checkpointing, and resumable data loading from TFRecord files. The codebase includes model definitions for transformer-based architectures (like CodeGen), utilities for learning rate scheduling and optimization, and integration with Hugging Face for model conversion and sampling. The system is designed for scalability, supporting models up to 6B parameters via sharding strategies, and includes tools for preprocessing large datasets into TFRecord format. While the core training functionality is tightly coupled to the TPU environment, several components like the resumable data iterator and checkpointing mechanism demonstrate reusable patterns.