$ show hplt-project/sacremoses
Sacremoses is a Python 3 library and CLI tool that reimplements the Moses tokenizer, detokenizer, and truecaser for natural language preprocessing. It provides exact, deterministic text segmentation and normalization for over 30 languages, with specific handling for non‑breaking prefixes, Unicode character properties, and protected patterns. The code is structured as a standard Python package with a `click`‑based CLI entry point, modular tokenization/truecasing modules, and a robust test suite that includes security regression tests. The repository also contains an `audit/` directory documenting a formal security review, with attack scripts and fixed vulnerabilities — a sign of strong engineering discipline for a language‑processing library. A notable architectural decision is the `data‑as‑code` pattern: instead of reading non‑breaking prefix lists and Perl Unicode property files at runtime, the package ships auto‑generated Python modules (`_data_nonbreaking_prefixes.py`, `_data_perluniprops.py`) that contain the same data as dictionary and set literals. This eliminates file‑I/O attack surfaces and simplifies distribution. The generation tool (`tools/generate_data_modules.py`) is itself a clean, reusable script that could be adapted to embed any tabular linguistic resource into a library without runtime file access. The library has undergone a security audit, and several fixes (dependency floors tied to CVEs, bounds on protected patterns, safe temp‑file creation) are baked into the codebase. While the core tokenization logic is a direct port of the Moses Perl script, the implementation shows careful attention to edge cases, concurrency support via `joblib`/`tqdm`, and compatibility with both Python 2 (last supported version) and Python 3. The main technical drift risk comes from static Unicode property snapshots that may lag behind the Unicode standard, but the architecture makes updates straightforward by regenerating the modules. Overall, Sacremoses is a well‑engineered, security‑conscious, production‑ready NLP preprocessing library. It is actively maintained (497 stars, no unresolved Dependabot alerts, recent commits), follows modern packaging conventions, and provides a drop‑in replacement for the Moses tokenizer in Python native code. Its reusable patterns — especially the data‑as‑code embedding and the security audit harness — are valuable for any library that must distribute static linguistic data without introducing file‑I/O vulnerabilities.
The LLM identified 3 files worth highlighting for reuse:
Script that reads non‑breaking prefix text files and Perl Unicode property files from `sacremoses/data/` and generates Python modules (`_data_nonbreaking_prefixes.py`, `_data_perluniprops.py`) containing the same data as dictionaries and sets. This eliminates all runtime file reads, improving security and distribution simplicity. The pattern is fully self‑contained (input directory, output Python modules, no internal imports) and can be extracted and adapted to embed other linguistic resource files into a Python library without runtime file access.
Security attack harness that probes the library for path traversal, regex DoS, unsafe temp‑file creation, and other vulnerabilities documented in `audit/README.md`. Contains isolated attack functions (e.g., path traversal via `Perluniprops.chars()`, numeric‑only prefix injection) with clear preconditions and expected outcomes. This file is a reusable security testing framework for any Python library that processes user‑supplied text data and handles file paths or language resources.
Provides `Perluniprops` and `NonbreakingPrefixes` classes that load Unicode character property sets and language‑specific abbreviation lists from the auto‑generated data modules. Uses a static‑class pattern with lazy loading and caching. The separation of data loading from tokenization logic makes it easy to swap or extend the underlying data sources. The approach of storing data as Python code (rather than runtime file reads) is a clean, security‑conscious pattern that can be replicated in any library needing static linguistic resources.