$ show nlp-with-transformers/notebooks
This repository is the official companion code for the O'Reilly book 'Natural Language Processing with Transformers' by Tunstall, von Werra, and Wolf. It contains 11 Jupyter notebooks, one per chapter, covering the full spectrum of NLP with Transformers: text classification, named entity recognition, text generation, summarization, question answering, model compression, few-shot learning, and training a transformer from scratch. The code is built on the Hugging Face ecosystem (transformers, datasets, accelerate) and is designed to be run interactively on Colab, Kaggle, or local GPU environments. The notebooks are well-structured with clear section headings, explanatory text, and visualizations, making them an excellent educational resource for learning state-of-the-art NLP techniques. Although the repository is primarily educational, it includes a SageMaker adaptation for some chapters, demonstrating deployment of Hugging Face models on AWS. The codebase is not intended as a production-grade library; rather, it serves as a tutorial and reference for practitioners. The overall quality is solid, with clean, reproducible code patterns, but the reliance on pinned, outdated library versions (e.g., transformers 4.16.2, datasets 1.16.1) and the deprecated Chapter 7 (farm-haystack) indicate moderate technical drift. The concepts remain relevant, but the exact code may require adjustments to run with the latest versions.
The LLM identified 4 files worth highlighting for reuse:
Contains the `ConstantLengthDataset` class, a streaming iterable that concatenates tokenized sequences and chunks them into fixed-length blocks. This pattern is essential for training causal language models on code or text corpora significantly larger than memory. It is reusable for any language modeling task that requires efficient, on-the-fly chunking of tokenized data.
Defines the `tokenize_and_align_labels` function, which maps word-level NER tags to subword tokens (e.g., B-PER, I-LOC) by aligning them with the tokenizer's output, setting -100 for special tokens and subtokens. This is a reusable pattern for any token-level classification task (NER, POS, chunking) when using subword tokenizers like BPE or WordPiece.
Implements utility functions `log_probs_from_logits` and `sequence_logprob` to compute the log-probability of a generated sequence given logits. These are reusable for evaluating decoding strategies (greedy, beam search, sampling) and for computing perplexity of generated text without external dependencies.
Contains the `forward_pass_with_label` function that computes per-token cross-entropy loss by hooking into the Trainer's predict method. This enables detailed error analysis by identifying which tokens the model is most uncertain about. The pattern is reusable for any sequence labeling task to gain insights into model weaknesses.