This library provides encoding and decoding for the Concise Binary Object Representation (CBOR) serialization format (RFC 8949). It features a hybrid architecture with core implementation in Rust for performance and safety, exposed through Python bindings. The library supports advanced CBOR features including value sharing for cyclic references, string referencing for deduplication of repeated strings, canonical encoding, and extensive semantic tag support for types like dates, UUIDs, IP addresses, and more. It provides both a programmatic API and a command-line diagnostic tool.
rustbpe is a Rust library with Python bindings that implements a BPE (Byte Pair Encoding) tokenizer for training and inference. The system provides efficient training of BPE models using parallel processing (Rayon), GPT-4 style regex pre-tokenization, and incremental update algorithms to avoid recomputing pair counts from scratch. It exports trained tokenizers to TikTok format for fast inference. The architecture centers around a Tokenizer struct that maintains merge rules and patterns, with methods for training, encoding, decoding, and batch processing. The Python interface mirrors the core Rust implementation, allowing seamless integration into Python NLP workflows while leveraging Rust's performance for computationally intensive operations.