$ show openvenues/libpostal
libpostal is a production-grade C library for parsing, normalizing, and expanding international street addresses. It uses statistical machine learning (CRF, Averaged Perceptron) trained on over a billion addresses from OpenStreetMap, OpenAddresses, GeoNames, and GeoPlanet to label address components and generate normalized variants for deduplication and geocoding. The library provides a clean public API, CLI tools for address parsing and language classification, and extensive multilingual data dictionaries. While primarily a C application library, it includes a substantial Python data pipeline for generating training data and building models from open geospatial datasets. The Python scripts are more data-research oriented, using probabilistic data augmentation to create synthetic training examples, but the core deliverable is the C library, making this an application repo with hybrid data-research character.
The LLM identified 4 files worth highlighting for reuse:
Provides generic dynamic string containers (char_array, cstring_array, string_tree) and UTF-8-aware string operations (comparison, case conversion, tokenization, trimming, replacement). These are self-contained, header-based templates widely used across the codebase. An engineer could extract this file into any C project needing efficient, portable string handling with minimal modification.
Implements a double-array trie for fast prefix and suffix lookup, with functions for adding keys, searching, and serializing/deserializing to binary file. The design is compact and memory-efficient. Could be reused in any C application requiring a dictionary for autocomplete, spell-check, or phrase matching. The serialization format is simple and portable.
Implements multiple string distance algorithms (affine gap, Damerau-Levenshtein, Jaro-Winkler) with clean, dependency-free C functions. Each algorithm is isolated and could be extracted into a standalone fuzzy matching library. The implementations demonstrate careful memory management and algorithmic optimization.
Provides a configurable string normalization pipeline that applies transliteration, Unicode case folding, hyphen stripping, and numeric expression expansion (numex) based on a bitmask of options. The design allows composing transformations without tight coupling. Could be reused in any multilingual text processing system.