pgGraph is a PostgreSQL extension written in Rust (via pgrx) that adds high-performance graph query capabilities directly into PostgreSQL. It allows users to model graphs by registering existing relational tables as nodes and edges, building a compressed sparse row (CSR) adjacency index for sub-millisecond traversals, shortest paths, and connected components. The extension exposes SQL functions (`graph.traverse`, `graph.shortest_path`, `graph.search`, `graph.gql`, `graph.cypher`) and supports both openCypher and a subset of GQL for querying property graphs. It integrates deeply with PostgreSQL security (row-level security, multi-tenancy, ACLs), provides resource governance (memory, disk, work unit budgets), and includes a durable projection layer with atomic manifests, delta segments, compaction, and crash recovery. The codebase is structured as a pgrx extension with a Rust core, SQL bootstrap scripts, background workers for build/maintenance jobs, and an extensive test suite including fuzz targets, property-based tests, and heavy integration tests for concurrency, crash recovery, and performance gates.
ParadeDB is a PostgreSQL extension that provides full-text search, vector search, and analytics directly within the database, eliminating the need for a separate search service. The core engine is `pg_search`, which implements a custom index access method (`paradedb`/`bm25`) using Tantivy for search indexing and DataFusion for analytic aggregations. It supports BM25 scoring, phrase/term/boolean queries, fuzzy matching, proximity search, hybrid search (combining full-text and vector), and a rich set of tokenizers (including jieba, ICU, Lindera, ngram). The extension integrates deeply with PostgreSQL's planner through custom scan nodes (BaseScan, AggregateScan, JoinScan, MPP) to push down search, aggregation, and join operations into the index. It also offers columnar storage via fast fields, parallel query execution, and WAL-level integration for crash safety. Architecturally, the system is a Rust library compiled as a shared library loaded by PostgreSQL, using the `pgrx` framework for FFI. Entry points include SQL functions for search (e.g., `paradedb.query`, `pdb.score`), DDL for index creation, and custom plan/exec hooks. Data flows: user SQL queries are intercepted by the planner, which decides whether to use a custom scan path; if so, Tantivy or DataFusion executes the search/aggregation/join, producing tuples that are optionally post-processed by PostgreSQL (e.g., heap fetches, visibility checks). The extension relies on external dependencies: Tantivy (forked for block storage), DataFusion (for aggregate execution), ICU (for Unicode tokenization), Lindera (for Japanese tokenization), and Apache Arrow (for columnar data exchange).
Apache AGE is a PostgreSQL extension that adds graph database capabilities, implementing the openCypher query language. It allows users to create, query, and manipulate property graphs (vertices and edges) directly within PostgreSQL, leveraging the database's transactional guarantees, security, and indexing. The extension provides a custom data type (agtype) for graph entities, a Cypher-to-SQL parser, and custom scan executors for efficient graph operations. It includes a comprehensive test suite, multi-language client drivers (Go, Java, Node.js, Python), and integration with NetworkX for graph analysis. The architecture is tightly integrated with PostgreSQL internals through hooks, custom nodes, and memory management, making it a production-grade graph database solution for PostgreSQL users.
This repository provides **pgrx**, a framework for writing PostgreSQL extensions entirely in Rust. It includes a CLI tool (`cargo-pgrx`) for project scaffolding, building, testing, packaging, and schema generation. The core library (`pgrx`) offers safe Rust wrappers for PostgreSQL internals: memory contexts, SPI (Server Programming Interface), arrays, triggers, aggregates, GUCs, datetime types, and more. Procedural macros (`#[pg_extern]`, `#[pg_trigger]`, `#[pg_aggregate]`, `#[derive(PostgresType)]`, etc.) automatically generate SQL DDL and Rust boilerplate, enabling a pure-Rust extension development experience. The framework supports PostgreSQL versions 13 through 19 via feature-gated FFI bindings in `pgrx-pg-sys`, which uses `bindgen` to produce per-version Rust declarations. SQL schema generation is handled at compile time by `pgrx-sql-entity-graph`, which embeds metadata in the shared library's binary sections. The architecture is modular: the `pgrx` crate re-exports public API, `pgrx-macros` provides proc macros, `pgrx-pg-sys` contains raw FFI bindings, `pgrx-sql-entity-graph` handles SQL generation, and `cargo-pgrx` is the CLI frontend. Example extensions in `pgrx-examples/` demonstrate features like custom types, background workers, logical decoding, and SPI usage. A comprehensive test suite in `pgrx-unit-tests/` includes compile-fail tests, property-based tests (via `proptest`), and integration tests that run against real PostgreSQL instances. Key external dependencies include the PostgreSQL development headers (for `pg_config` and C API), `libclang` for bindgen, and Rust ecosystem crates such as `serde`, `syn`, `quote`, `clap`, `ureq`, and `criterion` (for benchmarking). The framework is designed for extension authors; it is not a data analysis or research project.
pg_bulkload is a high-performance bulk data loading tool for PostgreSQL designed to bypass shared buffers for faster loads. It provides a command-line utility and PostgreSQL extension supporting multiple input formats (CSV, binary, SQL functions), parallel processing, configurable error handling, and crash recovery. The architecture follows a modular pipeline: reader → parser → checker → tuple former → writer, with deep integration into PostgreSQL internals for memory management, error reporting, and index building. Key features include direct path writing (bypassing WAL), multi-process loading via shared memory IPC, and B-tree index construction from sorted data.
Activeloop's DeepLake is a production-grade, multi-vector database engine for AI data. At its core, it is a columnar, chunked, compressed storage system for tensors (multi-dimensional arrays) with native support for embeddings, images, video, text, and other complex AI data types. It exposes a Python library for dataset management, ingestion (COCO, Labelbox, CSV, Parquet), version control, and integration with PyTorch/TensorFlow/MMDetection. A high-performance C++ backend provides lazy NumPy-like array operations (nd::array), a composable async framework (async::promise), and a Tensor Query Language (TQL). The most distinctive component is a PostgreSQL extension (pg_deeplake) that allows SQL access to DeepLake datasets via custom table and index access methods, supporting vector similarity search (cosine, BM25) and hybrid search. Query execution is delegated to DuckDB through a custom scan node, enabling efficient analytical queries. The extension also includes WAL-based DDL replication across instances. This is primarily an application (a database system), with substantial research-oriented features such as medical image formats, compression codecs, and TPC-H benchmarking.
This repository implements the Adaptive Query Optimization (AQO) extension for PostgreSQL, a machine learning-based system that improves query planning by learning from historical query execution statistics. The extension integrates with PostgreSQL's planner and executor via hook mechanisms to override cardinality estimation with predictions from per-query-class k-nearest neighbors regression models. It features a multi-layer architecture: preprocessing (query classification and mode handling), cardinality estimation (ML-based prediction), postprocessing (statistics collection and model updates), and shared memory persistence for query statistics and ML models. Key operational modes include disabled, learn, intelligent, controlled, and forced, controlled via GUCs. The system automatically tunes learning/usage flags based on prediction stability and includes background workers for maintenance. The extension requires patching PostgreSQL source and rebuilding, with tight coupling to internal planner structures (e.g., RelOptInfo, Plan nodes) and executor functions. Deployment involves setting shared_preload_libraries, configuring GUCs like AQO.mode, and managing memory limits for query classes and feature subspaces. It provides SQL interfaces for managing query classes (enable/disable, status checks) and EXPLAIN integration for runtime visibility. The test suite includes pgbench integration, SQL scripts for feature validation, and Perl scripts for pg_stat_statements compatibility. While the core AQO logic is highly specialized to PostgreSQL, two components demonstrate extractable excellence: the hash function for query normalization (ignoring constants/whitespace) and the standalone k-NN regression implementation with linear smoothing. These show non-obvious skill in handling SQL variability and efficient online learning. However, the deep integration with PostgreSQL internals, version-specific patching requirements, and complex state management introduce maintenance challenges and technical debt risks, particularly during PostgreSQL version upgrades.