$ show datafusion-contrib/datafusion-dft
datafusion-dft (dft) is a batteries-included Rust application that wraps Apache DataFusion into a multi-front-end SQL query and analysis tool. It provides four interfaces sharing one execution core: a scriptable CLI (file/command queries, benchmarking, FlightSQL client mode, CSV/JSON/Parquet/Vortex output), a ratatui terminal UI with SQL editor, paginated results, and history tabs, an Arrow FlightSQL gRPC server with full metadata and prepared-statement support, and an HTTP REST server. The core library (crates/datafusion-app) constructs a DataFusion SessionContext with a trait-based extension plugin system, dedicated CPU/IO executors, statistics collection, and observability hooks.
The LLM identified 6 files worth highlighting for reuse:
Implements tcp_conversations, a DataFusion table function that transforms raw pcap/pcapng packets into aggregate TCP connection flows (5-tuple keyed with byte/packet counts and timing). The packet-to-flow aggregation is self-contained domain logic — portable to any DataFusion-based network monitoring, security, or performance tooling.
Provides capture/capture_wide live packet-capture table functions, bridging blocking libpcap C calls into an async DataFusion source via a background thread and channel. Planning deliberately avoids opening the network device during EXPLAIN. This native-to-async bridge pattern is reusable for integrating any blocking C library with DataFusion streams.
A DedicatedExecutor (InfluxDB-inspired) that offloads CPU-bound query work to a dedicated tokio runtime while retaining the caller's IO runtime, preventing query execution from blocking server I/O. Self-contained concurrency primitive, directly reusable in any async service embedding DataFusion.
ExecutionPlanVisitor implementations that walk DataFusion physical plans to collect fine-grained IO and compute statistics (bytes read, rows scanned, per-operator timing). A portable visitor-based observability pattern for any project needing query-level metrics over DataFusion execution plans.
Exposes Parquet bloom-filter metadata as a queryable table, decoding low-level parquet crate structures (filter type, algorithm, hash function, bitset) into Arrow batches via the standard UDTF pattern. The decoding logic is isolated within a dedicated crate and reusable for any Parquet inspection or data-profiling tool.
Bridges WASM linear memory and Arrow IPC: the guest allocates memory via an exported alloc function, and the host constructs Arrow IPC buffers (record batches) directly from that memory region. A non-obvious FFI design for running cross-language UDFs, extractable as a template for WASM-based extension systems in Arrow-native projects.