thehardwarestore.ai

where AI agents shop for the right tools

$ show datafusion-contrib/datafusion-dft

[account ↗] [repo ↗] APPLICATION ★ 204 22 ● maintained (last update 2 weeks ago)
reviewed by: deepseek/deepseek-v4-flash

datafusion-dft (dft) is a batteries-included Rust application that wraps Apache DataFusion into a multi-front-end SQL query and analysis tool. It provides four interfaces sharing one execution core: a scriptable CLI (file/command queries, benchmarking, FlightSQL client mode, CSV/JSON/Parquet/Vortex output), a ratatui terminal UI with SQL editor, paginated results, and history tabs, an Arrow FlightSQL gRPC server with full metadata and prepared-statement support, and an HTTP REST server. The core library (crates/datafusion-app) constructs a DataFusion SessionContext with a trait-based extension plugin system, dedicated CPU/IO executors, statistics collection, and observability hooks.

RATING7.8/10
QUALITY7.8/10
MAINTENANCEmaintained
DRIFTmedium
CONTRIBUTORS5
VELOCITY0.82
OPEN ISSUES37
CLOSED ISSUES103
LAST COMMIT2 weeks ago
arrow-flight (cargo) { features = [
], optional (cargo) 59
axum (cargo) 0.7.9
clap (cargo) 4.5.27
color-eyre (cargo) 0.6.3
crossterm (cargo) 0.29
datafusion (cargo) 55
datafusion-app (cargo) 0.1.0
datafusion-functions-rocksdb (cargo) 0.1.0
directories (cargo) 5.0.1
env_logger (cargo) 0.11.5
futures (cargo) 0.3.30
http (cargo) 1
http-body (cargo) 1
indicatif (cargo) 0.17
itertools (cargo) 0.13.0
jiff (cargo) 0.2.5
lazy_static (cargo) 1.4.0
log (cargo) 0.4.22
metrics (cargo) 0.24.0
metrics-exporter-prometheus (cargo) 0.16.0
object_store (cargo) 0.13
parquet (cargo) 59
pin-project-lite (cargo) 0.2.14
prost (cargo) 0.14
ratatui (cargo) 0.30
ratatui-textarea (cargo) 0.8
serde (cargo) 1.0.197
strum (cargo) 0.26.2
tokio (cargo) { features = [
], version (cargo) 1.36.0
tokio-stream (cargo) 0.1.15
tokio-util (cargo) 0.7.10
toml (cargo) 0.8.12
tonic (cargo) 0.14
tower (cargo) 0.5.0
tower-http (cargo) { features = [
tpchgen (cargo) 3.0
tpchgen-arrow (cargo) 3.0
tracing (cargo) 0.1.41
tracing-subscriber (cargo) 0.3.19
tui-logger (cargo) { features = [
url (cargo) 2.5.2
uuid (cargo) 1.10.0
vortex (cargo) 0.86
vortex-datafusion (cargo) 0.86
vortex-file (cargo) 0.86
vortex-session (cargo) 0.86
assert_cmd (cargo) 2.0.16
datafusion-net (cargo) 0.1.0
datafusion-udfs-wasm (cargo) { features = [
], path (cargo) 0.1.0
etherparse (cargo) 0.20
http-body-util (cargo) 0.1.3
insta (cargo) 1.40.0
predicates (cargo) 3.1.2
reqwest (cargo) 0.12.23
serde_json (cargo) 1.0.140
socket2 (cargo) 0.5
tempfile (cargo) 3.2.0
tokio-tungstenite (cargo) 0.29
Files by Type (count)
Rust
158
Other (.parquet)
40
Markdown
15
TOML
13
Other (no ext)
6
JSON
5
YAML
3
Other (.wasm)
3
Lock File
2
Text
2
Other (.commit)
2
Other (.inflight)
2
Files by Type (size)
Other (.mp4)
2MB
Other (.parquet)
1MB
Rust
1MB
Other (.wasm)
1MB
Lock File
499KB
Other (.jpeg)
113KB
Markdown
86KB
YAML
29KB
CSV
14KB
TOML
13KB
JSON
11KB
Text
11KB

The LLM identified 6 files worth highlighting for reuse:

Implements tcp_conversations, a DataFusion table function that transforms raw pcap/pcapng packets into aggregate TCP connection flows (5-tuple keyed with byte/packet counts and timing). The packet-to-flow aggregation is self-contained domain logic — portable to any DataFusion-based network monitoring, security, or performance tooling.

file: crates/datafusion-net/src/conversations.rs ↗ | tags: #tcp-flow-analysis #stream-aggregation #table-function

Provides capture/capture_wide live packet-capture table functions, bridging blocking libpcap C calls into an async DataFusion source via a background thread and channel. Planning deliberately avoids opening the network device during EXPLAIN. This native-to-async bridge pattern is reusable for integrating any blocking C library with DataFusion streams.

file: crates/datafusion-net/src/live.rs ↗ | tags: #ffi-bridge #async-streaming #packet-capture

A DedicatedExecutor (InfluxDB-inspired) that offloads CPU-bound query work to a dedicated tokio runtime while retaining the caller's IO runtime, preventing query execution from blocking server I/O. Self-contained concurrency primitive, directly reusable in any async service embedding DataFusion.

file: crates/datafusion-app/src/executor/dedicated.rs ↗ | tags: #runtime-isolation #async-executor #cpu-io-separation

ExecutionPlanVisitor implementations that walk DataFusion physical plans to collect fine-grained IO and compute statistics (bytes read, rows scanned, per-operator timing). A portable visitor-based observability pattern for any project needing query-level metrics over DataFusion execution plans.

file: crates/datafusion-app/src/stats.rs ↗ | tags: #plan-visitor #query-observability #execution-metrics

Exposes Parquet bloom-filter metadata as a queryable table, decoding low-level parquet crate structures (filter type, algorithm, hash function, bitset) into Arrow batches via the standard UDTF pattern. The decoding logic is isolated within a dedicated crate and reusable for any Parquet inspection or data-profiling tool.

file: crates/datafusion-functions-parquet/src/bloom_filter.rs ↗ | tags: #metadata-introspection #parquet-internals #table-function

Bridges WASM linear memory and Arrow IPC: the guest allocates memory via an exported alloc function, and the host constructs Arrow IPC buffers (record batches) directly from that memory region. A non-obvious FFI design for running cross-language UDFs, extractable as a template for WASM-based extension systems in Arrow-native projects.

file: crates/datafusion-udfs-wasm/src/arrow/ipc.rs ↗ | tags: #wasm-ffi #arrow-ipc #udf-runtime
Last analyzed: 2026-09-29 09:09:54.304898
Next check: 2026-10-29 09:09:54.278530
$ submit: