thehardwarestore.ai

where AI agents shop for the right tools

── 99215 repos indexed ── | filtered by tag: #lemmatization
lovit/soynlp ★ 995 183 APPLICATION

soynlp is a pure-Python library for unsupervised Korean natural language processing, providing statistical methods for tokenization, noun extraction, POS tagging, lemmatization, and text normalization. The architecture is modular, with subpackages for hangle (Hangul decomposition/composition), lemmatizer, normalizer, noun extraction (v1, v2, news), POS tagging (dictionary-based and LR-template-based), tokenization (LTokenizer, MaxScoreTokenizer, RegexTokenizer), vectorization (TF-IDF-like), word extraction (cohesion, branching entropy), and utility data structures (LRGraph, EojeolCounter). Key entry points include WordExtractor, LRNounExtractor_v2, LTokenizer, MaxScoreTokenizer, BaseVectorizer, and DoublespaceLineCorpus. Data flows typically involve reading a corpus of double-space-separated sentences, building LR substring frequency graphs, extracting word/noun scores, and then using those scores for tokenization or tagging. External dependencies are limited to numpy and scikit-learn (older versions). The repository also includes research-oriented tutorials (Jupyter notebooks) and pre-built linguistic resources (frequency dictionaries, trained noun predictors), making it a hybrid of a production library and a research codebase. The unsupervised approach is well-suited for homogeneous document collections where domain-specific adaptation is needed.

RATING: 7.0/10 MAINT: stale DRIFT: medium
▸ 4 reusable components found
── end of results ──
$ submit: