$ show NannyML/The-Little-Book-of-ML-Metrics
The Little Book of ML Metrics is a comprehensive, collaboratively authored reference book that explains and visualizes evaluation metrics across the full spectrum of machine learning tasks: regression, classification, clustering, ranking, computer vision, NLP, generative AI, probabilistic estimation, bias/fairness, and data observability. The repository contains the LaTeX source code, a collection of Jupyter notebooks (and Python scripts) that generate all figures, and a set of custom tools for book production and review. Its primary function is to produce a high-quality, print-ready PDF that serves as both a pedagogical introduction and a practical decision guide for practitioners. The codebase is dominated by figure-generation scripts that implement each metric with synthetic or small real-world datasets and render Tufte-style, publication-quality plots. While the repository is not a traditional software application, it does include several bespoke tools: an interactive formula tuner (for adjusting TikZ arrow annotations on formulas), a review recorder (a local HTTP server for reviewing the PDF with audio notes), and a formula layout helper that auto-positions arrows. These tools are tightly integrated into the book production pipeline but are not intended for general reuse. The code is well-organized and follows consistent visual patterns (shared color palette, despine helpers, direct annotations), though there is considerable duplication across notebook scripts and a lack of automated tests. The LaTeX build process is robust but depends on a full TeX distribution and a third-party GitHub Action. Dependencies are pinned via pyproject.toml and uv.lock, and the genai figure generation requires a separate heavy environment (torch, transformers) run manually, which reduces full reproducibility out-of-the-box. Overall, this is a well-crafted book project with strong internal consistency, but its code is best viewed as supporting infrastructure rather than a standalone piece of software.
The LLM identified 3 files worth highlighting for reuse:
Shared matplotlib styling module used by all figure scripts. Defines brand colors (NML_CYAN, NML_PURPLE, NML_RED), a custom colormap, rcParams for book-print font sizes, a `despine` helper, and a `save_figure` function that saves figures to the book/figures directory at 300 DPI. This pattern enforces visual consistency across over 40+ figure scripts and could be extracted and reused for any matplotlib-heavy documentation project.
A standalone HTTP server (port 8765) that allows an author to interactively reposition TikZ arrow annotations in LaTeX formulas. It compiles a page snippet with xelatex, renders the page to PNG via pdftoppm, and serves a web UI with drag-and-drop arrow adjustment. The tool writes back modified coordinates to the .tex file and supports undo. Demonstrates a novel pattern for non-trivial LaTeX authoring automation and can be reused for any TikZ-heavy document with arrow overlays.
Automatically computes initial arrow paths for formula annotations by detecting colored symbols, running a simple obstacle-avoidance heuristic, and placing labels outside the text area. It generates LaTeX tikz code that the formula_tuner then adjusts. The algorithm is non-obvious and could be adapted for other layout problems where elements must be connected with minimal overlap.