$ show johnkerl/miller
Miller (mlr) is a command-line data processing tool for structured text formats including CSV, TSV, JSON, JSON Lines, YAML, DKVP, and others. It operates as a streaming pipeline: input records are read from files or stdin, processed through a chain of transformer verbs (e.g., cut, sort, join, stats1, put, filter), and written to output in any supported format. The tool includes a domain-specific language (DSL) for record-level transformations, with support for user-defined functions, subroutines, loops, and higher-order functions. Miller is designed for performance, using Go channels for concurrent streaming, batch processing, and a slab allocator for efficient memory use. The architecture is a monolithic CLI application written in Go with no external runtime dependencies. Key entry points include the main binary (cmd/mlr/main.go), the entrypoint dispatcher (pkg/entrypoint), and the terminal subcommands (help, repl, script, regtest, mcp). The codebase is organized into packages for input readers, output writers, transformers (verbs), DSL parsing and execution, CLI flag handling, and a core value type system (Mlrval). Miller also provides an MCP (Model Context Protocol) server for AI agent integration, shell completion scripts, and a REPL for interactive use. Miller is not a research project; it is a production-grade data munging tool comparable to awk, sed, cut, join, and sort but for name-indexed data. Its primary data flows involve reading heterogeneous records, applying transformations via verbs or DSL expressions, and writing results. The tool handles compressed files, in-place editing, and output redirection to files or pipes.
The LLM identified 5 files worth highlighting for reuse:
A generic ordered map implementation that maintains insertion order while providing O(1) key lookups via a lazy hash index. The hash index is only built when first needed (after a configurable threshold of fields), optimizing for small maps. This data structure is used throughout Miller for record field ordering and deterministic iteration in aggregations. It is self-contained and could be extracted into any Go project requiring ordered key-value storage.
A self-contained ANSI colorization module that handles terminal color output with TTY detection, environment variable overrides (MLR_NO_COLOR, MLR_ALWAYS_COLOR, etc.), and configurable color schemes. It provides a clean interface for colorizing output without coupling to the rest of the application. Could be extracted as a standalone terminal color utility.
A TSV (tab-separated values) encoder/decoder that handles field escaping, embedded newlines, and round-trip safety. It provides encode and decode functions that are self-contained and could be reused in any project needing TSV processing. The codec is designed for correctness and performance.
A utility for interpreting C-style escape sequences in string literals (e.g., \n, \t, \x00, \uXXXX). It handles octal, hex, and Unicode escapes with proper error handling. This is a common need in DSL interpreters and configuration parsers, and the implementation is self-contained and easily extractable.
A circular buffer implementation for sliding window operations over record streams. It maintains a fixed-size window of values and supports operations like delta, ratio, shift, and sliding statistics. While currently tied to Miller's value types, the circular buffer pattern is generic and could be extracted for any streaming window computation.