$ show databendlabs/databend
Databend is an open-source, cloud-native distributed SQL query engine and data warehouse, built in Rust. It provides a full SQL interface for analytical workloads, supporting operations such as SELECT, INSERT, COPY INTO/OUT, MERGE INTO, and complex aggregations with window functions, subqueries, and joins. The architecture follows a storage-compute separation model: a metadata service (Meta) manages schemas, users, and catalogs via a Raft-based consensus layer, while query nodes handle SQL parsing, planning, optimization (Cascades optimizer with rule-based and cost-based passes), and distributed execution over a pipelined framework. Multiple storage engines are supported, including a native columnar Fuse engine optimized for cloud object stores (S3, MinIO, Azure Blob), as well as external formats like Iceberg, Delta Lake, Hive, Paimon, Parquet, and ORC. The system exposes access via MySQL wire protocol, HTTP API, Flight SQL, and gRPC, and includes enterprise features such as data masking, row access policies, materialized views, streams, network policies, and role-based access control.
The LLM identified 4 files worth highlighting for reuse:
A custom metrics framework providing Counter, Gauge, Histogram, and registry with Prometheus-compatible output. It is self-contained, uses generic family types for label support, and includes a process collector. This module can be extracted into a separate crate for use in other Rust applications requiring lightweight, high-performance metrics collection without external dependencies.
A suite of custom hash table implementations (linear probing, Swiss-style, short string specialized) designed for low-level, cache-efficient aggregation and join operations in a data engine. The code is isolated within the `hashtable` crate, with well-defined traits and no dependency on the rest of Databend's internals. This can be reused in any Rust project needing high-throughput, memory-controlled hash tables.
A transactional key-value API over a distributed metadata store, implementing optimistic concurrency control with conditions, read sets, write sets, and idempotent operations. The design is generic and decoupled from the specific metadata domain, making it reusable for any project building a linearizable, conditional KV store on top of Raft or similar consensus.
An LRU cache implementation with configurable byte capacity and item count limits, supporting insertion, eviction, and update callbacks. It uses a doubly-linked list and hash map internally, with no external dependencies beyond std. This is a solid, reusable building block for any application needing a bounded, evictable cache.