$ show tobymao/sqlglot
SQLGlot is a pure-Python SQL parser, transpiler, optimizer, lineage analyzer, diff engine, and in-process executor with support for 30+ SQL dialects including Snowflake, BigQuery, DuckDB, Spark, Trino, Postgres, and MySQL. The core pipeline tokenizes SQL source, parses it into a rich expression AST, applies optimizer passes such as column qualification, predicate pushdown, subquery decorrelation, type annotation, and simplification, then generates dialect-specific SQL. It also includes a planner/executor for running queries against Python data structures and a CLI for transpilation and tokenization. Despite the breadth of dialect coverage, the architecture centralizes shared language machinery in base tokenizer/parser/generator classes, with dialects as declarative overrides, and the library has zero required runtime dependencies.
The LLM identified 6 files worth highlighting for reuse:
A highly configurable SQL tokenizer engine driven by pluggable keyword, quote, comment, string, and single-token maps. It handles all scanning details for every dialect while remaining independent of dialect-specific grammar, making it a reusable foundation for building tokenizers for new SQL-like languages.
Implements a Change Distiller-style semantic diff over SQL expression trees, producing Insert/Remove/Move/Update/Keep edits with similarity scoring via Dice coefficients. The algorithm is reusable for comparing any tree-structured data and is especially valuable for SQL query change analysis and migration tooling.
Provides nested catalog/database/table/column schema modeling with trie-based lookup, dialect-aware normalization, visibility filtering, and caching. The MappingSchema abstraction is directly reusable in any SQL analysis tool that needs schema-aware column resolution or type inference.
A compact generic trie implementation used to build efficient longest-match keyword and format-pattern matchers for tokenizers. It is fully self-contained and can be extracted for any prefix-matching or keyword-tokenization problem without modification.
Normalizes SQL boolean expressions into CNF or DNF using distributive-law rewriting with distance heuristics to control expression growth. This demonstrates a non-trivial optimizer pass that can be reused as a standalone expression normalization utility or adapted for other logic-synthesis tasks.
Rewrites SQL identifiers, strings, and comments into anonymized forms while preserving token positions and string lengths. This is a reusable privacy-preserving transform for query logging, debugging, and sharing SQL workloads without leaking sensitive literals or column names.