Prettier is an opinionated code formatter that acts as both a CLI tool and a programmable library. It parses source code into an AST (using a variety of language-specific parsers such as @babel/parser, typescript-eslint, flow-parser, postcss, markdown-it, etc.), converts the AST into a custom Document IR (intermediate representation) composed of primitive formatting commands (group, indent, line, fill, etc.), and then prints the Document IR into a formatted string. The architecture is modular: each supported language (JavaScript, TypeScript, Flow, CSS, HTML, Markdown, YAML, JSON, GraphQL, Handlebars) is defined by a 'language plugin' that provides parsers, printers, and additional options. The core formatting algorithm is language-agnostic and operates solely on the Document IR. Key entry points are the CLI (bin/prettier.cjs) and the JavaScript API (src/index.js). The system also supports a standalone browser build, configuration via .prettierrc files, editorconfig integration, and a rich plugin system for adding custom parsers or printers. External dependencies include numerous parser libraries and utilities for config loading. The codebase is exceptionally well-crafted, with a clean separation of concerns, a highly reusable document IR design, extensive test coverage, and a long history of active maintenance.
Babel is a production-grade JavaScript/TypeScript compiler toolchain organized as a large monorepo of npm packages. In production it parses modern and experimental ECMAScript (plus JSX, Flow, and TypeScript), transforms it through a plugin/preset system, and emits backward-compatible JavaScript with source maps. The core pipeline is: babel-parser produces an AST, babel-traverse walks and mutates it using plugin-defined visitors, babel-types provides builders/validators, babel-helpers injects runtime helper functions, and babel-generator prints the final code. Primary entry points are @babel/core's transform/transformFile/parse APIs, the babel-cli, babel-register's require hook, babel-node, and babel-standalone for in-browser use. Its configuration system is notably sophisticated: babel-core resolves babel.config.*, .babelrc, and package.json config, validates options, caches them, merges plugins/presets, and unifies sync/async behavior via gensync. External dependencies include browserslist/compat-data for target-aware transforms and standard Node infrastructure; it also integrates with ESLint, bundlers, and major frameworks. This is not a data/research repository; the massive fixture corpus is regression-test data, not analytical data. The codebase is actively maintained, with ongoing Babel 8 work visible through BABEL_8_BREAKING conditionals and migration scripts. Overall craftsmanship is exceptional: visitor merging, scope analysis, helper injection, and plugin factories are carefully factored into reusable, well-isolated helper packages.
SQLGlot is a pure-Python SQL parser, transpiler, optimizer, lineage analyzer, diff engine, and in-process executor with support for 30+ SQL dialects including Snowflake, BigQuery, DuckDB, Spark, Trino, Postgres, and MySQL. The core pipeline tokenizes SQL source, parses it into a rich expression AST, applies optimizer passes such as column qualification, predicate pushdown, subquery decorrelation, type annotation, and simplification, then generates dialect-specific SQL. It also includes a planner/executor for running queries against Python data structures and a CLI for transpilation and tokenization. Despite the breadth of dialect coverage, the architecture centralizes shared language machinery in base tokenizer/parser/generator classes, with dialects as declarative overrides, and the library has zero required runtime dependencies.
ESLint is a mature, widely-used static analysis tool for JavaScript and TypeScript. It operates as both a CLI tool and a programmatic library, parsing source code into an AST and applying a configurable set of lint rules to detect and fix code quality, style, and correctness issues. The architecture is modular: a core linter (`lib/linter/linter.js`) orchestrates parsing (via espree or custom parsers), rule execution, and fix application; a configuration system (`lib/config/`) resolves flat config files; and a CLI engine (`lib/cli.js`) handles command-line invocation, file globbing, and result formatting. Key entry points include `bin/eslint.js` (CLI), `lib/api.js` (public API exposing `ESLint`, `Linter`, `RuleTester`, `SourceCode`), and `lib/universal.js` (browser-compatible Linter). Data flows: source files are read, parsed into ASTs, traversed by rule visitors, and lint messages are collected, filtered, and formatted. External dependencies include espree (parser), eslint-scope (scope analysis), eslint-visitor-keys, and various community utilities. The codebase is self-hosting, using its own rules and config for development, and includes extensive test suites, fuzzing tools, and ecosystem regression testing.
This repository is the remark markdown ecosystem monorepo. It ships four npm packages: remark-parse (markdown to mdast AST), remark-stringify (mdast AST to markdown), remark (a preconfigured unified processor with both), and remark-cli (a terminal interface). In practice, these packages are thin adapter plugins for unified: they attach parser/compiler functions to a unified processor, merge user settings from processor data, and expose extension hooks that other plugins use to add syntax support. The heavy CommonMark parsing and serialization is delegated to mdast-util-from-markdown and mdast-util-to-markdown, so the code here is intentionally small and focused on clean integration rather than reimplementation. The architecture is a workspace-based monorepo of ESM-only packages. Entry points are `remark()`, `unified().use(remarkParse)`, `unified().use(remarkStringify)`, and the `remark` CLI binary. The primary data flow is: markdown text -> unified parse -> mdast tree -> plugin transformations -> unified compile -> markdown text. Extension mechanisms use processor data fields (`micromarkExtensions`, `fromMarkdownExtensions`, `toMarkdownExtensions`), which keeps core packages stable while allowing the plugin ecosystem to grow without coupling. Maintenance is active: CI runs against multiple Node versions, tests downstream packages as canaries, and the codebase enforces strict TypeScript coverage. Technical drift is low: the packages target maintained Node versions, use current unified/micromark-based dependencies, and follow modern ESM conventions. The design is a strong example of building a stable, plugin-driven toolchain around an AST-processing core.
Bandit is a well-established Python security linter that performs static analysis by parsing source code into ASTs and running a configurable set of plugins against AST nodes. It is primarily a CLI tool, but also exposes a plugin architecture for custom checks and formatters via setuptools entry points. Key entry points include the main CLI (bandit.cli.main), baseline comparison utility (bandit.cli.baseline), and config generator (bandit.cli.config_generator). The core engine (bandit/core/manager.py, node_visitor.py, context.py) orchestrates file discovery, AST traversal, test execution, and result aggregation. It supports multiple output formats (JSON, SARIF, CSV, HTML, XML, YAML, text, screen) and integrates with CI/CD pipelines, pre-commit hooks, and Git via .bandit files. The project is actively maintained, supports Python 3.10+, and has extensive test coverage, though some legacy config conversion code and removed test IDs (e.g., B309, B320) indicate minor drift. Bandit is a reference implementation for a plugin-based security linter with a clean separation between core, plugins, and formatters.
Graphify is a CLI tool and library that extracts structural and semantic knowledge graphs from source code across 36+ programming languages. It parses files using tree-sitter grammars, builds a graph of nodes (functions, classes, modules, etc.) and edges (calls, imports, inheritance, etc.), performs community detection via Leiden/Louvain, and generates reports, wikis, and exports to formats like Obsidian, Neo4j, and FalkorDB. The system integrates with AI coding assistants through an MCP server (stdio/HTTP), skill files (SKILL.md, CLAUDE.md), and git hooks for automatic graph updates. Key entry points include `graphify extract`, `graphify query`, `graphify serve`, and `graphify install`. External dependencies include tree-sitter grammars, LLM APIs (OpenAI, Anthropic, Gemini, Ollama), and optional databases. The architecture is modular: per-language extractors in `graphify/extractors/`, graph assembly in `build.py`, clustering in `cluster.py`, and export in `export.py`. The codebase demonstrates strong engineering practices with extensive tests, atomic file writes, security validation (SSRF, path traversal), and cross-platform support.
PsySH is a production-grade interactive PHP shell, REPL, and runtime debugger. It accepts PHP input from a terminal, validates and rewrites it through a pipeline of PHP-Parser AST passes, evaluates it inside an isolated execution context, and renders results using a custom VarDumper presenter, formatters, and support for paging, themes, clipboard copy, and OSC 8 hyperlinks. It is distributed both as a Composer library and a self-updating PHAR binary, with primary entry points in src/functions.php (sh(), debug(), info(), bin()) and the bin/psysh launcher. The architecture is modular and interface-driven: a central Shell/Configuration core coordinates commands built on Symfony Console, multiple readline backends (GNU, libedit, userland, and a new experimental interactive readline), a completion engine with typed sources and refiners, legacy tab-completion matchers, execution-loop listeners, manual/documentation loaders, and updater strategies for both the PHP manual and PsySH itself. The main data flow is: terminal input -> readline -> code cleaning/validation -> eval -> output; completion and history are layered on top via AST analysis, fuzzy matching, suggestions, and JSONL-backed history persistence. The codebase is unusually thorough for a developer tool: nearly every subsystem has unit and integration tests, CI spans PHP 7.4 through 8.6 plus 32-bit and downstream consumer tests, and static analysis baselines manage compatibility across php-parser 4-5 and Symfony 3.4-7.x. The main weaknesses are the vendored Hoa terminal library, a wide compatibility surface requiring deprecation baselines and version shims, and the complexity of the experimental interactive readline, which creates maintenance risk. Overall this is a high-craft, actively maintained project with several genuinely reusable components.
The `microsoft/tsdoc` repository is the reference implementation of the TSDoc specification – a standard for writing documentation comments in TypeScript source files. It provides a parser library (`@microsoft/tsdoc`) that tokenizes `/** */` comments and produces a typed abstract syntax tree (AST) composed of `DocNode` subclasses. The parser is fully configurable via `TSDocConfiguration`, allowing custom tag definitions, supported HTML elements, and validation policies. A companion package (`@microsoft/tsdoc-config`) loads and validates `tsdoc.json` configuration files, supporting inheritance via `extends`. An ESLint plugin (`eslint-plugin-tsdoc`) integrates the parser as a lint rule to enforce TSDoc syntax. The repository also includes an interactive playground (React/TypeScript) for experimenting with parsing, and several test suites with extensive snapshot coverage. Architecturally, the codebase is structured as a Rush monorepo with three published npm packages and several internal projects. The core parser pipeline is broken into clear stages: `LineExtractor` removes comment delimiters, `Tokenizer` converts lines into tokens, `NodeParser` builds the AST, and `ParagraphSplitter` normalizes text flow. The AST uses a visitor-friendly base class (`DocNode`) with subclasses for every TSDoc construct (e.g., `DocCodeSpan`, `DocLinkTag`, `DocParamBlock`). Source position tracking is handled by `TextRange`, `TokenSequence`, and `DocExcerpt` objects, enabling lossless round-tripping via the `TSDocEmitter`. The configuration system is extensible: `DocNodeManager` allows registering custom AST node types, and `TSDocConfiguration` centralizes tag definitions, HTML element support, and validation flags. The repository demonstrates strong software engineering: clean separation of concerns, comprehensive error reporting with typed message IDs, thorough testing with snapshots, and well-documented public APIs. While some components are tightly coupled to the parser pipeline (e.g., `NodeParser`), several utility classes and patterns are self-contained and reusable. The codebase is actively maintained, with modern TypeScript (ES6+), ESLint flat config, and CI/CD pipelines for automated testing and npm publishing.
This repository contains the core PostCSS library, a programmatic CSS transformation tool. It functions as a parser, an AST manipulator, and a stringifier. Its primary production use is as a library dependency in web development build pipelines (e.g., integrated via webpack, Gulp, or used directly by tools like Autoprefixer and Stylelint) to automate CSS transformations such as adding vendor prefixes, linting, or transpiling future CSS syntax. The architecture is a classic pipeline: CSS text is tokenized and parsed into a tree of Node objects (Root, Rule, Declaration, etc.). A Processor runs an ordered list of plugins, each of which can traverse and modify the tree using a visitor pattern. The modified tree is then stringified back to CSS, with optional source map generation. The core processing engine in `lib/lazy-result.js` is a sophisticated implementation of the visitor pattern, handling synchronous and asynchronous plugins, event filtering, and safe tree mutation. The codebase is well-structured with clear separation of concerns. Several components, such as the input handler (`lib/input.js`), error reporter (`lib/css-syntax-error.js`), value splitter (`lib/list.js`), and source map handler (`lib/previous-map.js`), are isolated, well-crafted, and reusable in other projects. The library is actively maintained, has a comprehensive test suite with fuzzing, and a formal security model. The primary technical debt is the tight coupling between the parser and the specific AST node classes, which makes extracting the parser alone difficult, and the transitional state of supporting both CommonJS and ES modules.
The php-ast repository is a PHP extension that exposes the internal Abstract Syntax Tree (AST) used by PHP 7 and 8. It provides functions to parse PHP code into an inspectable AST structure that mirrors PHP's internal compiler representation, enabling static analysis, code transformation, refactoring tools, and other metaprogramming tasks. Unlike pure-PHP AST parsers, this extension leverages PHP's own parser for maximum performance and accuracy. The extension implements a sophisticated versioning system to maintain compatibility across PHP releases. Rather than breaking when PHP introduces new language features, it defines AST versions (currently 50 through 120) and can represent newer constructs using older node types when necessary for backward compatibility. This allows developers to write analysis tools that work across multiple PHP versions without constantly updating for each new release. Key architectural components include the C-based extension core that interfaces with Zend's parser, a comprehensive metadata system describing node kinds and their associated flags, and utility functions for AST inspection. The extension handles the full PHP grammar including recent additions like attributes, match expressions, nullsafe operators, constructor property promotion, and enum support. In production use, this extension serves as the foundation for many PHP static analysis tools, enabling them to operate with near-native performance by avoiding the overhead of pure-PHP parsing while providing a stable, versioned interface to PHP's AST structure.
ppxlib is the standard library for writing PPX rewriters in OCaml—compile-time preprocessors that transform the OCaml abstract syntax tree (AST). It provides a unified framework for defining extension points (`[%...]`), deriving generators (`[@@deriving ...]`), context‑free rewrite rules, and global transformations. The library abstracts over multiple OCaml compiler versions (4.08 through 5.06) via its own frozen AST definitions and bidirectional migration modules, insulating PPX authors from version-specific parsetree changes. The core architecture consists of a driver that orchestrates transformation phases (lint, preprocess, instrument, context‑free, global, last instrument) and can be compiled into standalone executables or used as a Dune build plugin. The codebase is actively maintained with extensive documentation, benchmark infrastructure (including a vendored ppx_sexp_conv for comparison), and a comprehensive test suite covering error embedding, code path tracking, and AST round‑tripping. Despite the complexity of cross‑version AST support, the internal APIs are cleanly separated into reusable components such as typed attribute handlers, composable deriving arguments, and a GADT‑based DSL for context‑free rules.
This repository is a pnpm monorepo for @stylistic/eslint-plugin, a community-maintained ESLint plugin that provides formatting and stylistic rules for JavaScript, TypeScript, JSX, and related syntaxes. It ports formatting rules that were deprecated or removed from ESLint core and typescript-eslint, giving users a drop-in path to continue enforcing style conventions. The main deliverable is an npm library consumed through ESLint flat config; there is no server, database, or runtime application. The plugin exposes roughly a hundred rules, each with typed options, autofix support, parser-specific test coverage, and generated metadata and type declarations.
This repository contains 'officeParser', a robust Node.js and browser library for parsing 12 office document formats (DOCX, PPTX, XLSX, ODT, ODP, ODS, PDF, RTF, CSV, MD, HTML, EPUB) into a unified abstract syntax tree (AST) and generating high-fidelity output in 9 formats (JSON, text, Markdown, HTML, CSV, RTF, PDF, EPUB, and RAG-focused chunks). It provides a first-class CLI (`npx officeparser`) and a programmatic API (`parseOffice()` → AST → `.to(format, config?)`), with extensive configuration for OCR, chunking strategies, security limits, and format-specific dialects. The architecture follows a strict pipeline: format-specific parsers normalize documents into `OfficeParserAST`, then format-agnostic generators produce the desired output, enabling cross-format conversion and round-trip fidelity. The codebase demonstrates high craftsmanship with reusable utility modules (sanitization, typed error/warning systems, lazy ZIP handling, MathML-to-LaTeX conversion) and a comprehensive test suite covering parser baselines, generator baselines, security injection tests, and exhaustively tested edge cases. It supports browser and Node runtimes via dual CJS/ESM packages and bundled builds, including slim variants that minimize dependencies for browser environments. Key data flows include document metadata extraction, attachment handling (images, charts), OCR via Tesseract.js, PDF generation via Puppeteer, and semantic chunking for LLM ingestion. External dependencies include `pdfjs-dist`, `tesseract.js`, `fflate`, `chart.js`, and `puppeteer` (optional). While the repository is actively maintained with recent commits and CI across multiple Node versions, it shows some technical drift: deprecated API fields remain documented alongside replacements, and large files (e.g., OpenOfficeParser.ts at ~1600 lines) increase maintenance complexity. The builder has addressed past packaging issues (webpack context-module) and maintains sophisticated browser shims. Overall, it is a production-grade document conversion toolkit with exceptional reusability in its core utilities and a clean separation between parsing and generation.
C++ Insights is a source-to-source transformation tool built on LLVM/Clang that reveals compiler-generated code by making implicit C++ mechanisms explicit. It processes C++ source through Clang's AST, applies transformations to show template expansions, lambda desugaring, coroutine state machines, lifetime tracking, and other low-level details, outputting valid C++ that illustrates what the compiler does internally. The architecture centers on AST traversal via custom visitors and code generators, with modular components for different transformation modes (standard, Cfront-style, coroutine-specific). Key entry points include Insights.cpp (main driver) and specialized generators like CoroutinesCodeGenerator.cpp. The tool serves an educational purpose, helping developers understand C++ language semantics through visualizing compiler transformations.
Spoon is a comprehensive, open-source Java library for source code analysis, transformation, and generation. Developed at INRIA, it provides a complete AST metamodel covering all modern Java constructs (up to Java 25), along with powerful APIs for pattern matching, template-based code generation, static dataflow analysis (with Z3 integration), control flow graph construction, Javadoc parsing, and decompilation. The architecture is modular with submodules for specific tasks (e.g., spoon-dataflow, spoon-smpl for semantic patches, spoon-visualisation). Spoon is used both as a research platform for program analysis and as a practical tool for building custom code transformations, linters, and refactoring tools. The codebase demonstrates high engineering quality with clean separation of interfaces and implementations, extensive use of visitor and factory patterns, and a sophisticated query API.
Excellent Migrations is an Elixir library that analyzes Ecto migration files to detect potentially dangerous database operations that could cause downtime or data loss. It provides a Credo check, Mix tasks, and direct API usage to identify issues such as adding columns with default values, removing columns, running raw SQL, and more. The tool allows marking operations as safe via special comments or configuration to avoid false positives. The architecture is centered around AST parsing of migration files, with separate modules for detecting dangers, parsing safety comments, and filtering results. Key components like the AST parser and comment parser are well-crafted and reusable in other contexts.
Goldmark is a CommonMark-compliant Markdown parser and HTML renderer written in Go, designed for high performance and extensibility. The v2 branch introduces a complete redesign with Go generics, a cleaner separation of parser and renderer, and more semantic AST nodes. The core library exposes a `parser` package with `New()` and `Parse()` to produce an AST, and a `renderer/html` package with `New()` and `Render()` to output HTML. Extensions (tables, footnotes, strikethrough, task lists, typographer, linkify, definition lists) are implemented via a well-defined extension pattern using `parser.Extension` and `html.Extension` interfaces, making it straightforward to add custom syntax. The architecture is a library rather than a standalone application; it has no CLI or server, but is used by many static site generators and other tools. The codebase demonstrates strong engineering practices: extensive test coverage (including CommonMark spec tests and fuzzing), benchmarking against the C reference implementation, and a modular design that abstracts parsing, rendering, and text handling. The v2 version is still maturing, with some breaking changes from v1 that may affect third-party extensions, but the core is robust and well-documented.
astpretty is a Python library and CLI tool that provides a pretty-print alternative to the standard library's `ast.dump`. It formats Python AST nodes (from `ast.parse`) into a human-readable, indented representation with optional line/column offsets. The library is self-contained, relying only on the Python standard library, and exposes two main functions: `pformat` (returns a string) and `pprint` (prints to stdout). The CLI entry point reads a Python file, parses it, and prints the AST. The code is well-typed, uses modern Python features (3.11+), and includes comprehensive tests covering edge cases like type comments and version-specific AST changes. It is a polished, focused utility for developers working with Python's AST.
GritQL is a declarative query language and CLI tool for searching, linting, and transforming source code across many programming languages (JS/TS, Python, Java, Go, Rust, Ruby, PHP, SQL, etc.). The repository implements a Rust workspace comprising a pattern-matching engine (grit-pattern-matcher) that compiles GritQL patterns into executable AST-based matchers, a CLI client (grit) with commands like `apply`, `check`, `list`, `patterns`, and `lsp`, an LSP server for IDE integration, WebAssembly bindings for browser use (playground), and Node.js NAPI bindings for programmatic access. Patterns are authored in `.grit` files or embedded in Markdown with sample-based testing. The system uses tree-sitter parsers (vendored grammars) to parse source files, supports pattern caching for performance, and includes authentication, analytics, and remote module resolution. The architecture is modular: core pattern matching is isolated in `crates/grit-pattern-matcher`, pattern compilation in `crates/core`, language support in `crates/language`, and UX in `crates/cli`. The documentation site (Next.js with Markdoc) provides an interactive playground. External dependencies include tree-sitter parsers, NAPI-RS, WebAssembly tooling, and optional LLM APIs for AI-assisted patterns.