thehardwarestore.ai

Kagome is a self-contained Japanese morphological analyzer written in pure Go. It provides tokenization with part-of-speech tagging, multiple segmentation modes (normal, search, extended), and dictionaries embedded in the binary for easy deployment. The system includes a CLI tool with subcommands (tokenize, server, lattice, sentence), a RESTful API server mode, WebAssembly support for browser use, and C library API for FFI integration with languages like Python and PHP. The architecture centers around a lattice-based Viterbi algorithm for tokenization, with clean separation between tokenizer logic, dictionary handling, filtering utilities, and interface layers.

Sudachi is a professional-grade Japanese morphological analyzer implemented as a Java library and CLI tool. It performs tokenization, part-of-speech tagging, and text normalization using a plugin-based architecture with configurable input rewriting, out-of-vocabulary (OOV) handling, and lattice path rewriting. The system relies on precompiled binary dictionaries (system and user) built from CSV lexicons, and supports three segmentation modes (A, B, C) to balance between shortest and longest tokenization for different use cases. Internally, Sudachi uses a double-array trie for fast common-prefix search, memory-mapped file I/O for efficient dictionary loading, and a Viterbi-style lattice search for optimal path selection. The project includes a command-line interface, a comprehensive test suite, JMH benchmarks, and CI/CD pipelines. It is designed for embedding in larger applications (e.g., Elasticsearch via a separate plugin) and has been used in production at Works Applications.