Kagome is a self-contained Japanese morphological analyzer written in pure Go. It provides tokenization with part-of-speech tagging, multiple segmentation modes (normal, search, extended), and dictionaries embedded in the binary for easy deployment. The system includes a CLI tool with subcommands (tokenize, server, lattice, sentence), a RESTful API server mode, WebAssembly support for browser use, and C library API for FFI integration with languages like Python and PHP. The architecture centers around a lattice-based Viterbi algorithm for tokenization, with clean separation between tokenizer logic, dictionary handling, filtering utilities, and interface layers.
Sudachi is a professional-grade Japanese morphological analyzer implemented as a Java library and CLI tool. It performs tokenization, part-of-speech tagging, and text normalization using a plugin-based architecture with configurable input rewriting, out-of-vocabulary (OOV) handling, and lattice path rewriting. The system relies on precompiled binary dictionaries (system and user) built from CSV lexicons, and supports three segmentation modes (A, B, C) to balance between shortest and longest tokenization for different use cases. Internally, Sudachi uses a double-array trie for fast common-prefix search, memory-mapped file I/O for efficient dictionary loading, and a Viterbi-style lattice search for optimal path selection. The project includes a command-line interface, a comprehensive test suite, JMH benchmarks, and CI/CD pipelines. It is designed for embedding in larger applications (e.g., Elasticsearch via a separate plugin) and has been used in production at Works Applications.
TextBlob is a Python library for processing textual data, providing a simple, intuitive API for common natural language processing (NLP) tasks such as sentiment analysis, part-of-speech tagging, noun phrase extraction, tokenization, spelling correction, and classification. It wraps NLTK and a vendorized subset of the pattern library, offering a unified interface that feels like working with Python strings. The library is designed as a monolithic package with a pluggable architecture: abstract base classes define interfaces for taggers, tokenizers, sentiment analyzers, and parsers, allowing users to swap implementations. Core classes include TextBlob, Word, Sentence, Blobber (a factory for sharing models), and WordList. The codebase is well-structured, employs lazy loading and caching for performance, and includes a format registry for reading data files (CSV, JSON, TSV). While primarily a library for application integration, it also bundles linguistic data files (lexicons, spelling frequencies, sentiment lexicons) that are used for research-oriented tasks. The project is actively maintained, supports Python 3.10–3.14, and uses modern tooling (GitHub Actions, renovate, pre-commit).
SudachiDict is a repository that provides linguistic data resources for the Sudachi Japanese tokenizer. It processes raw dictionary CSV files (sourced from UniDic and NEologd) into binary dictionary files in three editions: Small (UniDic only), Core (basic vocabulary), and Full (includes proper nouns). The repository includes automated build scripts to compile these dictionaries, packaging utilities to distribute them as Java/Maven artifacts and Python packages, and test suites to verify correctness. While it contains some application-like elements (build scripts, packaging), its primary function is data curation and distribution for NLP use cases.
soynlp is a pure-Python library for unsupervised Korean natural language processing, providing statistical methods for tokenization, noun extraction, POS tagging, lemmatization, and text normalization. The architecture is modular, with subpackages for hangle (Hangul decomposition/composition), lemmatizer, normalizer, noun extraction (v1, v2, news), POS tagging (dictionary-based and LR-template-based), tokenization (LTokenizer, MaxScoreTokenizer, RegexTokenizer), vectorization (TF-IDF-like), word extraction (cohesion, branching entropy), and utility data structures (LRGraph, EojeolCounter). Key entry points include WordExtractor, LRNounExtractor_v2, LTokenizer, MaxScoreTokenizer, BaseVectorizer, and DoublespaceLineCorpus. Data flows typically involve reading a corpus of double-space-separated sentences, building LR substring frequency graphs, extracting word/noun scores, and then using those scores for tokenization or tagging. External dependencies are limited to numpy and scikit-learn (older versions). The repository also includes research-oriented tutorials (Jupyter notebooks) and pre-built linguistic resources (frequency dictionaries, trained noun predictors), making it a hybrid of a production library and a research codebase. The unsupervised approach is well-suited for homogeneous document collections where domain-specific adaptation is needed.