DuckDB is a high-performance, embeddable, in-process analytical SQL database engine written in C++. It provides a full SQL dialect with vectorized execution, columnar storage, advanced compression (ALP, Chimp, dictionary, FSST), adaptive indexing (ART), and a rich optimizer with rule-based and cost-based transformations. The system is designed for OLAP workloads, supporting complex queries with window functions, recursive CTEs, pivot operations, and lambda expressions. It exposes multiple client APIs: a C API (v1 and v2), C++ library, Swift bindings, and a command-line shell. Extensibility is a first-class concern via a plugin system for table/scalar/aggregate functions, custom storage backends, and data format readers (Parquet, CSV, JSON, Arrow). The repository includes extensive benchmark suites (TPC-H, TPC-DS, ClickBench, LDBC, IMDB, recursive CTE) and a comprehensive test infrastructure. While primarily an application (database engine), it also contains research-grade components in query optimization, compression algorithms, and parallel execution, giving it a hybrid character.
ClickHouse is a high-performance, column-oriented OLAP database management system written primarily in C++. It is designed for real-time analytics on large datasets, supporting SQL queries, distributed execution, and various storage engines (MergeTree, ObjectStorage, etc.). The architecture is monolithic but modular, with distinct layers for parsing (Parsers), planning (Analyzer/Planner), execution (Processors), and storage (Storages). Key entry points include the server binary (`clickhouse-server`) and client tools (`clickhouse-client`). It relies heavily on C++ templates, RAII, and a plugin-like registration system for functions and storage engines. It supports distributed queries via ZooKeeper/Keeper and integrates with various cloud storage providers (S3, Azure, GCS).
ParadeDB is a PostgreSQL extension that provides full-text search, vector search, and analytics directly within the database, eliminating the need for a separate search service. The core engine is `pg_search`, which implements a custom index access method (`paradedb`/`bm25`) using Tantivy for search indexing and DataFusion for analytic aggregations. It supports BM25 scoring, phrase/term/boolean queries, fuzzy matching, proximity search, hybrid search (combining full-text and vector), and a rich set of tokenizers (including jieba, ICU, Lindera, ngram). The extension integrates deeply with PostgreSQL's planner through custom scan nodes (BaseScan, AggregateScan, JoinScan, MPP) to push down search, aggregation, and join operations into the index. It also offers columnar storage via fast fields, parallel query execution, and WAL-level integration for crash safety. Architecturally, the system is a Rust library compiled as a shared library loaded by PostgreSQL, using the `pgrx` framework for FFI. Entry points include SQL functions for search (e.g., `paradedb.query`, `pdb.score`), DDL for index creation, and custom plan/exec hooks. Data flows: user SQL queries are intercepted by the planner, which decides whether to use a custom scan path; if so, Tantivy or DataFusion executes the search/aggregation/join, producing tuples that are optionally post-processed by PostgreSQL (e.g., heap fetches, visibility checks). The extension relies on external dependencies: Tantivy (forked for block storage), DataFusion (for aggregate execution), ICU (for Unicode tokenization), Lindera (for Japanese tokenization), and Apache Arrow (for columnar data exchange).
pco/pcodec is a high-performance lossless compression library specializing in numerical sequences (integers and floats). It implements a novel three-stage pipeline: mode selection (classic, dictionary, integer/float multiplier, float quantizer), delta encoding (consecutive, lookback, conv1), and binning with tANS entropy coding. The library achieves near-Shannon-entropy compression ratios, particularly for structured numeric data like time series, columnar store blocks, and scientific datasets. It provides multiple APIs: a core Rust crate (`pco`), Python bindings via PyO3 (`pcodec` on PyPI), Java bindings via JNI (`pco-jni` on Maven Central), a C API (`pco_c`), and a feature-rich CLI for compression, decompression, inspection, and benchmarking against many other codecs (zstd, snappy, parquet, blosc, etc.). The codebase includes a legacy predecessor (`quantile-compression/q_compress`) also maintained in the same workspace. The architecture is modular, with separate crates for bit I/O (`better_io`), type dispatch macros (`dtype_dispatch`), and extensive testing including format backward compatibility and corruption recovery. It is actively developed with CI pipelines for Rust, Python, and Java.
VictoriaLogs is a high-performance, lightweight log management system designed for both small and large-scale deployments. It consists of multiple Go-based services for log ingestion (vlagent, vlinsert), storage (vlstorage), and querying (vlselect), complemented by a Preact/TypeScript web UI (vmui). The system uses a columnar storage approach with bloom filters, custom encoding, and a pipeline-based query engine (LogsQL) for efficient log processing. Key architectural patterns include composable data transformation pipes, extensible filter systems, mergeable statistical aggregations, and object pooling for performance optimization. The frontend features a modular component architecture with custom hooks for state management, contextual controls, and interactive visualizations.
Apache Kudu is a distributed columnar storage system designed for fast analytics on fast data. It provides a horizontally scalable, fault-tolerant storage layer that integrates tightly with the Hadoop ecosystem (Spark, Impala, Hive, Flink). The architecture consists of master servers managing metadata and tablet servers storing data in columnar files (CFile) with Raft-based replication for consistency. The codebase includes production-grade C++ core, multi-language client libraries (Java, C++, Python, Spark), a comprehensive RPC framework, security features (Kerberos, TLS, Ranger), and extensive testing infrastructure. While primarily an application, it also contains benchmarking and experimental code (TPC-H, Jepsen, performance experiments) that give it a minor hybrid character. The repository is actively maintained with a large community and demonstrates strong engineering practices, though some vendored code and platform-specific workarounds indicate moderate technical drift.
Parca is a continuous profiling platform that collects, stores, queries, and visualizes pprof profiles in real time. The Go backend ingests profiles via gRPC (both pprof and OpenTelemetry OTLP), normalizes them into Apache Arrow columnar format, and stores them in either FrostDB (default) or ClickHouse. It provides a rich gRPC+JSON API for querying profiles (flamegraph, table, source view, callgraph, pprof download), scraping Prometheus-style targets, managing debug information for symbolization, and sharing profiles via links. The TypeScript/React frontend delivers an interactive profiling UI with flamegraphs, tables, source views, metrics graphs, sandwich views, and comparison mode, all backed by efficient Arrow-based querying. The codebase demonstrates strong engineering with reusable patterns (generic caching, signed request abstraction, Arrow compaction), but carries some drift risk from dual storage backends, generated protobuf code, and a complex UI state layer built with multiple state management strategies.