XGBoost is an optimized, distributed gradient boosting library (GBDT/GBM) that implements gradient boosted decision trees and linear models for supervised learning tasks including classification, regression, ranking, and survival analysis. It is a production-grade system with a monolithic C++ core (libxgboost) providing the training and inference engine, wrapped by first-class APIs in Python (scikit-learn compatible), R, Java/Scala (XGBoost4J), and C. It supports CPU, GPU (CUDA, SYCL), and distributed training via Rabit, NCCL, Spark, and Dask, along with external memory (out-of-core) training and model serialization (JSON, UBJSON, binary). The architecture is modular: objectives, metrics, tree updaters, and linear updaters are registered via DMLC macros, enabling extensibility through plugins. Data flows from user-provided datasets (via DMatrix, QuantileDMatrix, or custom iterators) through the learner, which builds ensembles of trees using histogram-based, exact, or approximate methods. While primarily a library, it also includes academic research influences (e.g., LambdaRank, AFT survival, SHAP) and a broad ecosystem of demos and benchmarks. The codebase is exceptionally well-engineered with advanced C++ templates, CUDA kernels, and rigorous testing across multiple backends and languages.
This repository is the official code companion to the book '50 ML projects to understand LLMs' by Mike X Cohen. It contains over 50 Jupyter notebooks organized into chapters (2–7), each exploring a specific concept in large language model interpretability and internals. Topics span tokenization, embeddings, cosine similarity, softmax/logits, loss functions, sampling, hidden state analysis, effective dimensionality, logit lens, attention mechanisms, MLP projections, probing classifiers, and bias measurement. The code is purely educational and experimental—no production services, APIs, or deployable applications. All notebooks are designed to run on Google Colab with minimal setup, using standard libraries (PyTorch, Hugging Face Transformers, scikit-learn, numpy, matplotlib) and pretrained models (GPT-2, BERT, RoBERTa, Pythia). Each project has a 'helper' notebook (incomplete code for students) and a 'solution' notebook (complete reference implementation). The repository is a valuable resource for learning mechanistic interpretability and hands-on LLM analysis, but it lacks modularization, dependency pinning, and test coverage, making it unsuitable for direct reuse in production pipelines.