$ show BYVoid/OpenCC
OpenCC (Open Chinese Convert) is a deterministic, dictionary-based Chinese text conversion library and CLI tool that converts between Simplified Chinese, Traditional Chinese, and regional variants (Taiwan, Hong Kong, Japan), including experimental Small Seal Script (Unicode 18.0). It is used in production by input method engines (RIME, fcitx), dictionary tools (GoldenDict), and NLP pipelines for text normalization and preprocessing. The core is a C++ library with a plugin-based segmentation system (built-in mmseg, optional jieba). Conversion is config-driven via JSON files that define normalization, segmentation, and a chain of dictionary groups with match policies (short_circuit, union). Dictionaries are compiled from text files into binary formats (marisa-trie .ocd2, legacy .ocd). The library exposes C, C++, Python (pybind11), and Node.js (N-API) APIs. A CLI tool provides conversion, inspection, ambiguity detection, and dictionary management. Data flows: text input is segmented, then each segment is looked up in a chain of dictionaries (character, phrase, regional variants). The first match is used (short_circuit) or all matches are unioned. The result is reassembled. The system supports streaming conversion with bounded memory via chunked processing. External dependencies include marisa-trie, darts-clone, rapidjson, pybind11, Google Test, and Google Benchmark; optional jieba (C++ fork) for segmentation plugin.
The LLM identified 4 files worth highlighting for reuse:
Implements a fast prefix matching algorithm with UTF-8 skip-scan optimization. Uses a character-level skip table to quickly bypass non-matching positions. Supports single-dict fast path and multi-dict table path with caching. Self-contained component that could be extracted for other text processing tasks requiring high-performance prefix lookup.
Defines an abstract ResourceProvider interface with filesystem and zip implementations. Decouples resource loading from conversion logic, enabling flexible addition of new resource sources (network, encrypted, etc.). The pattern is reusable for any project needing pluggable resource resolution.
Provides a template interface for dictionary serialization/deserialization with TryLoadFromFile and NewFromFile methods. Implementations include BinaryDict, MarisaDict, DartsDict, and TextDict. This pattern allows adding new dictionary formats without modifying the conversion engine, demonstrating clean separation of concerns.
Provides base classes Dict and RichDict for reading, writing, and manipulating tab-separated dictionary files. Used by all data maintenance scripts (sort, reverse, extract). This is a reusable utility for any project working with similar key-value dictionary formats, with support for comments, multiple values, and reverse-preference annotations.