$ show wy-luke/All-Chinese-Character-Set
This repository provides curated Chinese character sets (3500 common, 7000 common, and a comprehensive set) combined with symbols (Latin letters, digits, punctuation, full-width variants). The data is sourced from official Chinese character frequency lists and community-maintained character tables. The repository includes simple Node.js scripts for building the final text files from source files, merging new characters, and removing duplicates. The primary use case is generating character sets for Unity TextMeshPro font creation, but the data can be used in any application requiring a comprehensive Chinese character inventory. The codebase is minimal and well-documented in AGENTS.md, with clear instructions for editing source files and regenerating outputs. The repository is actively maintained with no open issues and a recent commit.
The LLM identified 2 files worth highlighting for reuse:
The build script reads source character files (3500.txt, 7000.txt, all.txt, symbols.txt) from the source/ directory, applies full-width conversion to symbols, deduplicates using Set, and writes the combined output files to the repository root. It is a self-contained ETL pipeline that can be adapted for any character set aggregation task by modifying the input file paths and transformation logic.
A generic deduplication utility that reads a text file, identifies duplicate characters with their positions, logs them, and writes back the unique characters. It operates on the source files and can be reused for any text-based dataset where duplicate removal is needed. The logging of duplicate positions is a good practice for data quality auditing.