$ show WorksApplications/Sudachi
Sudachi is a professional-grade Japanese morphological analyzer implemented as a Java library and CLI tool. It performs tokenization, part-of-speech tagging, and text normalization using a plugin-based architecture with configurable input rewriting, out-of-vocabulary (OOV) handling, and lattice path rewriting. The system relies on precompiled binary dictionaries (system and user) built from CSV lexicons, and supports three segmentation modes (A, B, C) to balance between shortest and longest tokenization for different use cases. Internally, Sudachi uses a double-array trie for fast common-prefix search, memory-mapped file I/O for efficient dictionary loading, and a Viterbi-style lattice search for optimal path selection. The project includes a command-line interface, a comprehensive test suite, JMH benchmarks, and CI/CD pipelines. It is designed for embedding in larger applications (e.g., Elasticsearch via a separate plugin) and has been used in production at Works Applications.
The LLM identified 4 files worth highlighting for reuse:
Implements common-prefix search using a double-array trie over UTF-8 byte arrays. The trie is stored in compact arrays (base, check) and lookup traverses byte-by-byte. The code is self-contained, uses only primitive arrays, and can be extracted as a standalone trie implementation for any application requiring fast prefix matching with minimal memory overhead.
Provides a thin wrapper around java.nio.MappedByteBuffer with a static `map()` method that loads a file into memory, and an `unmap()` method that uses JDK internals (Unsafe in JDK 9+, Cleaner in JDK 8) to eagerly release the mapping. This pattern is reusable for any application that needs efficient large-file loading and deterministic cleanup without relying on garbage collection.
Implements POS (part-of-speech) matching with wildcard support using a BitSet-based approach. Given a list of POS patterns (each can be a full string or null as wildcard), it compiles them into an array of BitSets indexed by POS part position. The match operation computes a BitSet for the target POS and checks membership with a threshold count. This pattern is reusable for any hierarchical category matching system (not just NLP) where efficient wildcard matching against a multi-part code is needed.
Implements streaming tokenization by reading input characters in chunks via a SurrogateAwareReadable buffer and yielding tokenized sentence results lazily. This avoids loading the entire text into memory, making it suitable for processing large documents. The pattern demonstrates how to combine a pull-based character reader with a sentence detector and tokenizer to produce results on demand, reusable for any NLP pipeline that must handle unbounded input.