$ show makcedward/nlpaug
nlpaug is a Python library for data augmentation across text (character, word, sentence), audio, and spectrogram modalities. It provides a wide array of augmentation techniques including random insertion/substitution/deletion, synonym replacement via WordNet/PPDB, contextual word embeddings (BERT, GPT-2, RoBERTa), back-translation, TF-IDF-based augmentation, keyboard/OCR error simulation, and audio effects (noise, pitch shift, speed, VTLP). The library follows a consistent pattern: each augmenter inherits from a base class, accepts data via a unified `augment()` method, and returns augmented samples. Composition is supported through `Sequential` and `Sometimes` pipeline classes. It relies heavily on external deep learning models (HuggingFace Transformers, word2vec, GloVe, fastText) and audio processing (librosa). The codebase is well-structured with clear separation of concerns (augmenters, models, utilities, flows), extensive unit tests, and documentation. However, it exhibits significant technical drift due to aggressively pinned dependencies (e.g., `transformers>=5.9.0`, `numpy>=2.4.6`) that may not exist or conflict with current stable releases, and some integration code (conda setup for Python 3.5–3.9) is outdated. Despite this, the core augmentation logic and utility modules are sturdy and reusable.
The LLM identified 5 files worth highlighting for reuse:
Provides pure functions for top-k and nucleus (top-p) filtering of probability distributions, implemented both in NumPy and PyTorch. These are self-contained and can be extracted into any project requiring constrained sampling (e.g., language model outputs, recommendation systems). Demonstrates careful handling of edge cases (zero probabilities, shape consistency) and supports both CPU and GPU tensors.
Contains a `seed()` function that synchronizes random seeds across Python’s `random`, NumPy, and PyTorch with a single call. Also includes a `RandomState` wrapper for deterministic notebook execution. This utility is critical for reproducible experiments and can be reused in any ML or simulation project with minimal dependency.
Implements a thread-safe singleton cache for heavyweight models (e.g., back-translation, language models). Uses a class-level `_cache` dict and lazy initialization via `get_instance()` and `reset_instance()`. The pattern is clean and could be adapted for any expensive resource that should be loaded once and reused across calls.
Provides a `Tokenizer` class with a default regex-based word splitter and a `reverse_tokenizer` function to reassemble token lists into strings. Supports custom tokenizers via injection. The design is simple and decoupled from the rest of the library, making it easy to extract and apply to text preprocessing in other projects.
Implements an `OpenAIChat` language model wrapper that sends prompts to GPT-4o-mini (or configurable model) and parses the response as structured JSON using a predefined schema. Features retry logic, temperature/seed control, and language support. This is a clean, standalone module for generating synthetic data via OpenAI chat completions, reusable in any project requiring LLM-based data augmentation.