$ show ikawaha/kagome
Kagome is a self-contained Japanese morphological analyzer written in pure Go. It provides tokenization with part-of-speech tagging, multiple segmentation modes (normal, search, extended), and dictionaries embedded in the binary for easy deployment. The system includes a CLI tool with subcommands (tokenize, server, lattice, sentence), a RESTful API server mode, WebAssembly support for browser use, and C library API for FFI integration with languages like Python and PHP. The architecture centers around a lattice-based Viterbi algorithm for tokenization, with clean separation between tokenizer logic, dictionary handling, filtering utilities, and interface layers.
The LLM identified 2 files worth highlighting for reuse:
Implements a robust sentence splitter for Japanese text handling various punctuation marks (。, ．, ！, !, ？, ?), follower characters, whitespace options, and double line feed splitting. Provides a bufio.SplitFunc interface for easy integration with Go's scanner. Demonstrates skilled Unicode text boundary detection with configurable parameters for different use cases.
A generic, type-safe object pool implementation wrapping sync.Pool for efficient resource reuse. Uses Go generics to provide type safety while eliminating allocations and ensuring concurrent safety. Demonstrates skilled application of pooling patterns to minimize GC pressure in high-throughput tokenization scenarios.