$ show mlc-ai/tokenizers-cpp
This repository provides a cross-platform C++ tokenizer binding library that wraps HuggingFace tokenizers (via Rust), SentencePiece, and a custom RWKV world tokenizer. It exposes a common abstract interface (`Tokenizer`) with factory methods to create instances from in-memory blobs (JSON, SentencePiece model, BPE vocab/merges, RWKV model). The architecture is a static library with a C FFI layer (`tokenizers_c.h`) and a C++ wrapper (`tokenizers_cpp.h`). The Rust component (`rust/src/lib.rs`) implements the C bindings for the HuggingFace tokenizers crate, while the C++ source files (`src/`) provide implementations for SentencePiece and RWKV. The project also includes a JavaScript/WebAssembly binding via Emscripten (`web/`). Key entry points are the factory functions in `tokenizers_cpp.h` and the `example/example.cc` demonstrating usage. External dependencies include the `tokenizers` Rust crate (v0.21.2), `sentencepiece`, `msgpack-c`, and for the web build, Emscripten. The codebase is well-structured, with clean abstractions and support for multiple platforms (iOS, Android, Windows, Linux, Web).
The LLM identified 3 files worth highlighting for reuse:
Implements the RWKV world tokenizer using a Trie tree for efficient longest-prefix matching. The TrieTree class is self-contained, demonstrating a clean, recursive data structure for tokenization. It deserializes a msgpack-encoded word-to-id mapping and builds a trie. This component is reusable because the trie-based tokenization pattern is dataset-agnostic and can be extracted for other subword tokenization tasks.
Defines the abstract `Tokenizer` base class with a clean interface (`Encode`, `Decode`, `GetVocabSize`, `IdToToken`, `TokenToId`) and static factory methods (`FromBlobJSON`, `FromBlobSentencePiece`, `FromBlobByteLevelBPE`, `FromBlobRWKVWorld`). This interface is well-designed, decoupling users from backend implementations, and follows the factory pattern. It is highly reusable as a template for any tokenizer abstraction.
Provides C FFI bindings to the HuggingFace tokenizers Rust library. Wraps the `Tokenizer` struct with methods like `encode`, `decode`, `id_to_token`, etc. Uses `Box::into_raw` for ownership and `#[no_mangle]` functions. This is a clean example of exposing a Rust library to C/C++ code, and the pattern can be reused for other Rust-to-C++ integrations.