TOON (Token-Oriented Object Notation) is a human-readable serialization format designed to minimize LLM token usage while preserving lossless round-trip conversion with JSON. This monorepo provides the reference TypeScript implementation (`@toon-format/toon`) and a CLI tool (`@toon-format/cli`) for encoding/decoding TOON. The library features streaming encode/decode, tabular forms for uniform data, normalization of special JavaScript values (BigInt, Date, Set, Map), security hardening against prototype pollution, and a replacer function with path tracking. The CLI supports stdin/stdout conversion, auto-detection by file extension, token statistics, and delimiter options. The repository also includes an extensive benchmark suite comparing TOON against JSON, YAML, XML, and CSV across token efficiency and retrieval accuracy using multiple LLMs (Claude, Gemini, GPT, Grok). Results show TOON matches JSON's accuracy while using ~43% fewer tokens. A documentation site with an interactive playground is provided via VitePress. The codebase is well-structured with clean abstractions, streaming APIs, and thorough testing, though it relies on an external spec repository and has a single author, creating bus factor risk.
Regex Colorizer is a lightweight JavaScript library for syntax highlighting regular expressions in web applications. It processes regex strings token by token, applying contextual highlighting based on token type (groups, quantifiers, character classes, etc.) and active flags (like unicode). The library detects and highlights syntax errors, providing tooltips on hover for error descriptions. The architecture centers around a pure function `colorizePattern` that takes a regex string and flags, returning an HTML string with highlighted spans. This function is used by `colorizeAll` to highlight elements in the DOM. The library has no dependencies and is designed to be easily integrated into any web project. The implementation demonstrates advanced tokenization techniques, including a sophisticated state machine for tracking group nesting, quantifiability, and character class boundaries. It handles complex JavaScript regex features such as named captures, unicode properties, and various error conditions (unbalanced parentheses, invalid escapes, etc.) with precision. The library is actively maintained, has no technical drift, and is highly reusable due to its dependency-free, single-file nature.
JPlag is a mature source-code plagiarism detection engine delivered as a Java library, a picocli-based CLI, and an embedded Vue.js report viewer. In production it parses submissions into language-specific token streams (via ANTLR, JavaCC, or custom parsers), compares token sequences pairwise using greedy string tiling with rolling hashes, merges fragmented matches, applies frequency-based weighting, optionally clusters submissions, and writes a .jplag ZIP report containing JSON models consumed by the viewer. The CLI can also start a local HTTP server to browse results. Architecturally it is a multi-module Maven project: core contains the comparison pipeline and reporting; language-api and language-antlr-utils define a plugin SPI for adding languages; languages/* provides 20+ modules; cli assembles the executable; report-viewer is an npm workspace with model/parser/ui-components packages. Key entry points are de.jplag.cli.CLI and de.jplag.JPlag.run(). External dependencies include ANTLR, picocli, Jackson, JGraphT, Apache Commons Math, and the Vue/TypeScript ecosystem. Although primarily an application, it embeds research-grade components such as spectral and agglomerative clustering with Bayesian optimization, token-sequence normalization, and match-frequency weighting, and it is backed by academic publications. It is not a data pipeline; the inputs are source-code submissions and the output is a similarity report. The repository is actively maintained with extensive CI, end-to-end gold-standard tests, and dependabot, though some language grammars lag language evolution and a few deprecated APIs remain.
Sacremoses is a Python 3 library and CLI tool that reimplements the Moses tokenizer, detokenizer, and truecaser for natural language preprocessing. It provides exact, deterministic text segmentation and normalization for over 30 languages, with specific handling for non‑breaking prefixes, Unicode character properties, and protected patterns. The code is structured as a standard Python package with a `click`‑based CLI entry point, modular tokenization/truecasing modules, and a robust test suite that includes security regression tests. The repository also contains an `audit/` directory documenting a formal security review, with attack scripts and fixed vulnerabilities — a sign of strong engineering discipline for a language‑processing library. A notable architectural decision is the `data‑as‑code` pattern: instead of reading non‑breaking prefix lists and Perl Unicode property files at runtime, the package ships auto‑generated Python modules (`_data_nonbreaking_prefixes.py`, `_data_perluniprops.py`) that contain the same data as dictionary and set literals. This eliminates file‑I/O attack surfaces and simplifies distribution. The generation tool (`tools/generate_data_modules.py`) is itself a clean, reusable script that could be adapted to embed any tabular linguistic resource into a library without runtime file access. The library has undergone a security audit, and several fixes (dependency floors tied to CVEs, bounds on protected patterns, safe temp‑file creation) are baked into the codebase. While the core tokenization logic is a direct port of the Moses Perl script, the implementation shows careful attention to edge cases, concurrency support via `joblib`/`tqdm`, and compatibility with both Python 2 (last supported version) and Python 3. The main technical drift risk comes from static Unicode property snapshots that may lag behind the Unicode standard, but the architecture makes updates straightforward by regenerating the modules. Overall, Sacremoses is a well‑engineered, security‑conscious, production‑ready NLP preprocessing library. It is actively maintained (497 stars, no unresolved Dependabot alerts, recent commits), follows modern packaging conventions, and provides a drop‑in replacement for the Moses tokenizer in Python native code. Its reusable patterns — especially the data‑as‑code embedding and the security audit harness — are valuable for any library that must distribute static linguistic data without introducing file‑I/O vulnerabilities.
rustbpe is a Rust library with Python bindings that implements a BPE (Byte Pair Encoding) tokenizer for training and inference. The system provides efficient training of BPE models using parallel processing (Rayon), GPT-4 style regex pre-tokenization, and incremental update algorithms to avoid recomputing pair counts from scratch. It exports trained tokenizers to TikTok format for fast inference. The architecture centers around a Tokenizer struct that maintains merge rules and patterns, with methods for training, encoding, decoding, and batch processing. The Python interface mirrors the core Rust implementation, allowing seamless integration into Python NLP workflows while leveraging Rust's performance for computationally intensive operations.
This repository is a curated collection of Jupyter notebooks implementing core components of modern LLM architectures, intended for interview preparation. It covers attention mechanisms (MHA, GQA, MLA, KV-cache), normalization layers (LayerNorm, RMSNorm), functional components (RoPE, SwiGLU, LoRA, BPE, activation functions, cross-entropy, InfoNCE, sampling, quantization, SFT loss, tool calling parsing), and reinforcement learning losses (DPO, GRPO, DAPO, GSPO, KL divergence). Each notebook is a self-contained, minimal implementation of a specific algorithm, typically using PyTorch. The code is clean, concise, and well-commented, reflecting good engineering practices for educational purposes. While the repository is not a production application and lacks data pipelines, model training scripts, or continuous integration, it serves as a high-quality reference for understanding and reproducing the mathematical foundations of transformer-based language models. The hybrid nature lies in its focus on research-oriented ML components, but its structure is a pure code library rather than a data-driven research project.
soynlp is a pure-Python library for unsupervised Korean natural language processing, providing statistical methods for tokenization, noun extraction, POS tagging, lemmatization, and text normalization. The architecture is modular, with subpackages for hangle (Hangul decomposition/composition), lemmatizer, normalizer, noun extraction (v1, v2, news), POS tagging (dictionary-based and LR-template-based), tokenization (LTokenizer, MaxScoreTokenizer, RegexTokenizer), vectorization (TF-IDF-like), word extraction (cohesion, branching entropy), and utility data structures (LRGraph, EojeolCounter). Key entry points include WordExtractor, LRNounExtractor_v2, LTokenizer, MaxScoreTokenizer, BaseVectorizer, and DoublespaceLineCorpus. Data flows typically involve reading a corpus of double-space-separated sentences, building LR substring frequency graphs, extracting word/noun scores, and then using those scores for tokenization or tagging. External dependencies are limited to numpy and scikit-learn (older versions). The repository also includes research-oriented tutorials (Jupyter notebooks) and pre-built linguistic resources (frequency dictionaries, trained noun predictors), making it a hybrid of a production library and a research codebase. The unsupervised approach is well-suited for homogeneous document collections where domain-specific adaptation is needed.