htmlparser2 is a fast, forgiving HTML/XML parser for Node.js and browsers, written in TypeScript. It provides a low-level streaming tokenizer (Tokenizer) that feeds events to a higher-level Parser, which handles tag matching, implicit closing, foreign content (SVG/MathML), and entity decoding. The library also includes Node.js WritableStream and Web Streams API WritableStream adapters for piping data directly into the parser. It is the foundation for cheerio and other DOM manipulation libraries. The codebase is well-structured, thoroughly tested, and actively maintained, with a focus on performance and spec compliance for real-world HTML.
Kagome is a self-contained Japanese morphological analyzer written in pure Go. It provides tokenization with part-of-speech tagging, multiple segmentation modes (normal, search, extended), and dictionaries embedded in the binary for easy deployment. The system includes a CLI tool with subcommands (tokenize, server, lattice, sentence), a RESTful API server mode, WebAssembly support for browser use, and C library API for FFI integration with languages like Python and PHP. The architecture centers around a lattice-based Viterbi algorithm for tokenization, with clean separation between tokenizer logic, dictionary handling, filtering utilities, and interface layers.
Sudachi is a professional-grade Japanese morphological analyzer implemented as a Java library and CLI tool. It performs tokenization, part-of-speech tagging, and text normalization using a plugin-based architecture with configurable input rewriting, out-of-vocabulary (OOV) handling, and lattice path rewriting. The system relies on precompiled binary dictionaries (system and user) built from CSV lexicons, and supports three segmentation modes (A, B, C) to balance between shortest and longest tokenization for different use cases. Internally, Sudachi uses a double-array trie for fast common-prefix search, memory-mapped file I/O for efficient dictionary loading, and a Viterbi-style lattice search for optimal path selection. The project includes a command-line interface, a comprehensive test suite, JMH benchmarks, and CI/CD pipelines. It is designed for embedding in larger applications (e.g., Elasticsearch via a separate plugin) and has been used in production at Works Applications.
This is the canonical implementation of OpenAI's Harmony response format, the structured prompt format used by the gpt-oss open-weight model series. The repository provides a high-performance Rust core for rendering conversations into token sequences and parsing model outputs back into structured messages. It supports multi-channel reasoning (analysis, commentary, final), tool/function calling with JSON Schema-defined parameters, structured outputs, and streaming incremental parsing. The library is packaged with first-class Python bindings via PyO3 and a JavaScript/WASM build for browser use. A Next.js demo visualizes the token-level rendering. The codebase demonstrates strong engineering: a custom BPE tokenizer (tiktoken) implementation in Rust, a streaming parser that gracefully handles partial UTF-8 sequences, and a JSON Schema-to-TypeScript converter for tool definitions. Tests are comprehensive with parity between Rust and Python test suites.
This repository is the official companion code for the book *Build a Large Language Model (From Scratch)* by Sebastian Raschka (105k+ stars). It provides a step-by-step, educational implementation of GPT-2 and its modern variants (Llama 3, Qwen3, Gemma 3/4, Olmo3) entirely in PyTorch, organized by book chapters (2–7 plus appendices). The codebase is a mix of Jupyter notebooks for interactive learning and Python modules that form a reusable pip package (`llms-from-scratch`), accompanied by extensive CI/CD pipelines, unit tests, and performance analysis scripts. Architecturally, the core is a modular transformer library that includes tokenization (BPE, tiktoken, custom), multi-head attention with multiple variants (GQA, MLA, SWA, DSA, cross-layer KV sharing), feedforward blocks, normalization (LayerNorm, RMSNorm), and LoRA. The code supports pretraining from scratch, classification finetuning (spam, IMDB), instruction finetuning (with/without LoRA), and preference tuning (DPO). It can load pretrained weights from OpenAI, Hugging Face, and local checkpoints. The package exposes clean interfaces (`GPTModel`, `generate`, `train_model_simple`) and includes bonus materials for KV cache implementations, mixture-of-experts, memory estimators, and user interfaces (Chainlit). While primarily educational, the code exhibits production-quality patterns: isolated attention modules, configurable model architecture, well-structured training loops, and robust weight loading with shape validation. The repository is actively maintained with recent commits, cross-platform support (Linux, macOS, Windows), and a strong emphasis on reproducibility (fixed seeds, deterministic settings, environment definitions via pixi/uv/conda). Duplication exists across bonus notebooks and a few hardcoded paths, but the core package and test suite are well-coordinated.
Sacremoses is a Python 3 library and CLI tool that reimplements the Moses tokenizer, detokenizer, and truecaser for natural language preprocessing. It provides exact, deterministic text segmentation and normalization for over 30 languages, with specific handling for non‑breaking prefixes, Unicode character properties, and protected patterns. The code is structured as a standard Python package with a `click`‑based CLI entry point, modular tokenization/truecasing modules, and a robust test suite that includes security regression tests. The repository also contains an `audit/` directory documenting a formal security review, with attack scripts and fixed vulnerabilities — a sign of strong engineering discipline for a language‑processing library. A notable architectural decision is the `data‑as‑code` pattern: instead of reading non‑breaking prefix lists and Perl Unicode property files at runtime, the package ships auto‑generated Python modules (`_data_nonbreaking_prefixes.py`, `_data_perluniprops.py`) that contain the same data as dictionary and set literals. This eliminates file‑I/O attack surfaces and simplifies distribution. The generation tool (`tools/generate_data_modules.py`) is itself a clean, reusable script that could be adapted to embed any tabular linguistic resource into a library without runtime file access. The library has undergone a security audit, and several fixes (dependency floors tied to CVEs, bounds on protected patterns, safe temp‑file creation) are baked into the codebase. While the core tokenization logic is a direct port of the Moses Perl script, the implementation shows careful attention to edge cases, concurrency support via `joblib`/`tqdm`, and compatibility with both Python 2 (last supported version) and Python 3. The main technical drift risk comes from static Unicode property snapshots that may lag behind the Unicode standard, but the architecture makes updates straightforward by regenerating the modules. Overall, Sacremoses is a well‑engineered, security‑conscious, production‑ready NLP preprocessing library. It is actively maintained (497 stars, no unresolved Dependabot alerts, recent commits), follows modern packaging conventions, and provides a drop‑in replacement for the Moses tokenizer in Python native code. Its reusable patterns — especially the data‑as‑code embedding and the security audit harness — are valuable for any library that must distribute static linguistic data without introducing file‑I/O vulnerabilities.
ChatRWKV is a collection of inference tools, demos, and a pip package for the RWKV language model family (v4-v7), a 100% RNN architecture that rivals transformers in performance. The repository provides interactive chat scripts (chat.py, v2/chat.py), API demos (API_DEMO_CHAT.py), model conversion utilities, state merging, music generation (MIDI/ABC), and benchmarking scripts. The core RWKV model class (rwkv_pip_package/src/rwkv/model.py) supports flexible deployment strategies (CPU/GPU, fp16/fp32/bf16, int8 quantization, streaming) and includes custom CUDA kernels for efficient WKV operations. The tokenizer module (tokenizer/rwkv_tokenizer.py) offers a fast trie-based byte-level tokenizer alongside a reference implementation. While the repository is primarily a research/demo project with many ad-hoc scripts and hardcoded paths, the pip package and core components are well-crafted and demonstrate significant engineering skill in GPU optimization and model inference. The codebase is actively maintained and supports the latest RWKV-7 architecture.
This repository provides a cross-platform C++ tokenizer binding library that wraps HuggingFace tokenizers (via Rust), SentencePiece, and a custom RWKV world tokenizer. It exposes a common abstract interface (`Tokenizer`) with factory methods to create instances from in-memory blobs (JSON, SentencePiece model, BPE vocab/merges, RWKV model). The architecture is a static library with a C FFI layer (`tokenizers_c.h`) and a C++ wrapper (`tokenizers_cpp.h`). The Rust component (`rust/src/lib.rs`) implements the C bindings for the HuggingFace tokenizers crate, while the C++ source files (`src/`) provide implementations for SentencePiece and RWKV. The project also includes a JavaScript/WebAssembly binding via Emscripten (`web/`). Key entry points are the factory functions in `tokenizers_cpp.h` and the `example/example.cc` demonstrating usage. External dependencies include the `tokenizers` Rust crate (v0.21.2), `sentencepiece`, `msgpack-c`, and for the web build, Emscripten. The codebase is well-structured, with clean abstractions and support for multiple platforms (iOS, Android, Windows, Linux, Web).
Kronos is a foundation model for financial K-line (OHLCV) time series data, accepted at AAAI 2026. It introduces a novel two-stage architecture: a BinarySphericalQuantizer tokenizer that converts continuous multi-dimensional market data into hierarchical discrete tokens, followed by an autoregressive decoder-only Transformer (with RoPE, RMSNorm, hierarchical embedding, and dependency-aware layers) pre-trained on data from 45+ global exchanges. The repository provides a complete research pipeline including model definition, tokenizer training, predictor fine-tuning (via Qlib or custom CSV), backtesting, and inference through multiple interfaces (Python API, CLI examples, a Tkinter GUI, and a Flask web UI). While the core model and tokenizer demonstrate strong research craftsmanship, the codebase exhibits signs of rapid prototyping: many example scripts with overlapping functionality, hardcoded paths, AI-generated comments, and prediction outputs that sometimes degenerate to constant values or numerical instability, indicating the model is still in an experimental stage. The repository is hybrid in nature—primarily a research project with application wrappers for demonstration and evaluation.
This repository provides an Emacs package that implements bash completion within Emacs shell buffers and command prompts. It works by spawning or reusing a bash process to generate completions via `compgen`, respecting the user's bash configuration including aliases, functions, and programmable completion. The package handles edge cases like quoted strings, redirections, and colon-separated variables, and works with TRAMP for remote shells. The architecture is centered around a single Emacs Lisp file that defines functions for parsing shell commands, communicating with bash processes, and formatting completion results. Key entry points include `bash-completion-setup` for initialization and `bash-completion-dynamic-complete` for integration with Emacs' completion system.
soynlp is a pure-Python library for unsupervised Korean natural language processing, providing statistical methods for tokenization, noun extraction, POS tagging, lemmatization, and text normalization. The architecture is modular, with subpackages for hangle (Hangul decomposition/composition), lemmatizer, normalizer, noun extraction (v1, v2, news), POS tagging (dictionary-based and LR-template-based), tokenization (LTokenizer, MaxScoreTokenizer, RegexTokenizer), vectorization (TF-IDF-like), word extraction (cohesion, branching entropy), and utility data structures (LRGraph, EojeolCounter). Key entry points include WordExtractor, LRNounExtractor_v2, LTokenizer, MaxScoreTokenizer, BaseVectorizer, and DoublespaceLineCorpus. Data flows typically involve reading a corpus of double-space-separated sentences, building LR substring frequency graphs, extracting word/noun scores, and then using those scores for tokenization or tagging. External dependencies are limited to numpy and scikit-learn (older versions). The repository also includes research-oriented tutorials (Jupyter notebooks) and pre-built linguistic resources (frequency dictionaries, trained noun predictors), making it a hybrid of a production library and a research codebase. The unsupervised approach is well-suited for homogeneous document collections where domain-specific adaptation is needed.