H2 is a full-featured, embeddable SQL database engine written in pure Java. It operates as a JDBC driver supporting both embedded (in-process) and client-server (TCP, PostgreSQL wire protocol, web console) modes. The codebase implements a complete SQL parser, query optimizer, transaction engine with MVCC, multiple storage engines (MVStore, legacy PageStore), and a rich set of data types, functions, and compatibility modes for databases like MySQL, PostgreSQL, Oracle, and MSSQLServer. It also includes tools for backup, recovery, CSV import/export, and a web-based administration console. The architecture is modular: the `org.h2.mvstore` package provides a standalone B-tree based transactional key-value store, `org.h2.value` defines a comprehensive type system, and `org.h2.expression` offers an extensible expression and function framework. The repository is exhaustively tested with thousands of SQL scripts and unit tests covering concurrency, crash recovery, and cross-database compatibility. It is actively maintained, with recent commits and support for Java 21 virtual threads and GraalVM native-image.
The POCO C++ Libraries (pocoproject/poco) is a mature, modular collection of reusable C++ class libraries for building network-centric, cross-platform applications. It does not contain a single application; rather, it provides a SDK with libraries for networking (HTTP, TCP, UDP, SSL/TLS), data access (SQLite, MySQL, PostgreSQL, ODBC via unified Data layer), XML/JSON parsing, cryptography (OpenSSL wrappers, JWT), logging (including a FastLogger based on Quill), threading, caching, configuration management, and more. The architecture is layered: Foundation provides core abstractions (streams, threading, events), and higher-level modules (Net, Data, XML, Crypto) depend on it. External dependencies include OpenSSL, MySQL/PostgreSQL client libraries, and ODBC drivers, while many third-party libraries (expat, zlib, quill, cpptrace, 7zip) are bundled in the repository. The codebase is actively maintained with CI, fuzzing, and extensive test suites, and uses modern C++17/20 idioms alongside traditional C++ patterns.
The Data Engineering Wiki is a community-maintained knowledge base dedicated to documenting data engineering concepts, tools, and best practices. Rather than being an executable application or data processing pipeline, it serves as a comprehensive reference resource for individuals learning and working in the data engineering field. The repository is organized into thematic sections covering foundational concepts (data architecture, modeling, processing, storage), practical tools and technologies, implementation guides, frequently asked questions, and curated learning resources. Structurally, the wiki follows a logical progression from theoretical foundations to practical application, with concepts explained through clear definitions, diagrams, and real-world examples. Key strengths include its comprehensive coverage of both established and emerging data engineering topics, practical guides on data pipeline best practices and testing methodologies, and active maintenance evidenced by recent commits. The content is presented in an accessible markdown format with consistent frontmatter, making it easy to navigate and consume. While not containing executable code or data pipelines itself, the wiki provides reusable knowledge assets in the form of methodological guides and reference materials that can be directly applied to data engineering projects. Notably, the Data Pipeline Best Practices and Testing Your Data Pipeline guides offer actionable frameworks for building reliable, scalable data systems. The repository's value lies in its role as a living document that captures and organizes community knowledge to support learning and professional practice in data engineering.
TablePro is a native macOS and iOS database client built with SwiftUI and AppKit, designed as an open-source alternative to commercial tools like TablePlus. It supports a vast array of database engines (PostgreSQL, MySQL, MSSQL, Oracle, MongoDB, Redis, Snowflake, ClickHouse, DuckDB, Teradata, Trino, etc.) through a plugin-based driver architecture defined in TableProPluginKit. The application features a custom code editor (CodeEditSourceEditor) with tree-sitter syntax highlighting, line folding, minimap, and Vim mode; a powerful data grid for inline editing; SSH tunneling with multi-factor authentication; CloudKit-based sync for connections and settings; and an integrated AI assistant with MCP (Model Context Protocol) server support for agentic database interactions. The codebase is actively maintained, demonstrates high engineering craftsmanship with clean protocols, extensive unit and UI tests, and many reusable components that could be extracted into independent libraries.
This repository is a distribution of the classic 'employees' sample database, a realistic relational dataset of ~300,000 employees, ~2.8 million salary records, and associated department/title/manager data. It is not an application but a data asset and benchmark fixture used to test database servers, application integrations, and migration tooling. The repo contains MySQL and PostgreSQL schemas, raw data dumps, partitioned table variants, stored procedures/views, and an integrated integrity-test suite that verifies row counts and incremental MD5/SHA-1/SHA-256 checksums. It also bundles a version of the Sakila sample database. CI workflows exercise loading and integrity checks across MySQL 5.6-9.6, Percona Server, MariaDB, and PostgreSQL 16/17, and the SHA-256 checksums are intentionally identical across all supported engines. The primary value is a non-trivial, self-contained dataset with strong cross-database compatibility and a clever, reusable data-integrity verification pattern.
This repository is a comprehensive French-language training course for PostgreSQL 18, structured as 22 progressive chapters and 7 reference annexes. It serves as educational material covering PostgreSQL fundamentals, SQL querying, advanced features, administration, performance tuning, extensions, and modern architectures. The content combines theoretical explanations with practical examples, best practices, and version-specific features (like asynchronous I/O, UUIDv7, and native OAuth). While not containing executable code or data pipelines, it provides reusable reference materials such as SQL query templates, shell scripts for backup/monitoring, configuration guides, and checklists that can be adapted across projects. The repository is actively maintained with recent updates and demonstrates low technical drift due to its focus on enduring database principles alongside version-specific annotations.
Medoo is a lightweight PHP database framework that provides a fluent, array-based API for building SQL queries and mapping results. It supports multiple database engines (MySQL, PostgreSQL, SQLite, MSSQL, Oracle, Sybase) via the PDO extension. The library is contained in a single source file (src/Medoo.php) and offers CRUD operations, joins, aggregates, raw SQL, transactions, and column type mapping. It is designed to be simple and intuitive, fitting naturally into various PHP frameworks. The codebase includes a comprehensive test suite that validates SQL generation across different database dialects, and the project is actively maintained with modern PHP version support.
fselect is a Rust command-line tool that enables searching the filesystem using SQL-like queries. It parses a SELECT-FROM-WHERE-GROUP BY-HAVING-ORDER BY-LIMIT syntax, traverses directories, and extracts rich metadata from files including size, timestamps, permissions, media information (EXIF, audio/video dimensions, duration), cryptographic hashes, MIME types, extended attributes, ACLs, git status, and more. The tool supports aggregation functions, subqueries (both FROM-subselects and correlated IN/EXISTS), and multiple output formats (tabular, CSV, JSON, HTML). It can accelerate searches via external indexes (Everything on Windows, plocate on Linux) and includes an interactive REPL mode. The architecture is a single binary with clean separation of concerns: lexer/parser for query processing, a searcher module for filesystem traversal and expression evaluation, utilities for metadata extraction, and pluggable output formatters.
Authorizer is a self-hosted, production-grade authentication and authorization server written in Go. It implements OAuth 2.0 / OIDC, SAML IdP/SP, SCIM 2.0, WebAuthn/passkeys, TOTP, email/SMS OTP, and fine-grained authorization via an embedded OpenFGA engine. The architecture is transport-agnostic: a shared service layer (internal/service) is called by GraphQL resolvers, gRPC handlers, and HTTP handlers (Gin). It supports multiple storage backends (SQL, MongoDB, Cassandra, DynamoDB, Couchbase, ArangoDB) via a unified storage.Provider interface, plus Redis and in-memory stores for sessions and caching. The system includes a React admin dashboard, an end-user login SPA, and an MCP (Model Context Protocol) server for AI tool exposure. Security is a primary focus, with extensive regression tests for CVEs, SSRF protection, constant-time comparisons, and audit logging.
This is a production-grade Kubernetes operator written in Go that manages the full lifecycle of MariaDB databases (including Galera clusters, asynchronous replication, and MaxScale proxies) on Kubernetes. It implements a custom resource definition (CRD) pattern with controllers, validating webhooks, and sidecar agents to handle deployment, scaling, TLS, backups (logical and physical), point-in-time recovery, and SQL resource management (users, grants, databases). The operator is built with controller-runtime and kubebuilder, and includes extensive integration tests using Ginkgo/Gomega and envtest. Its architecture is modular, with separate controllers for each CRD, reusable builder packages for Kubernetes objects, and well-factored utility packages for data structures, PKI, condition management, and statefulset helpers. The codebase is actively maintained, with a rich set of examples, Helm charts, and multi-cluster support. It does not include any data research or analytics components.
Sqlvet is a static analysis CLI tool for Go projects that scans raw SQL queries embedded in Go source code to detect potential runtime errors at build time. It uses Go's SSA (Static Single Assignment) representation and call graph analysis to trace SQL query strings through function calls, handling common patterns like string concatenation, functions returning query strings, and pass-through functions like sqlx.Rebind. Extracted queries are parsed with a PostgreSQL grammar (via pg_query_go) to validate syntax, table/column names against a schema, and detect unsafe dynamic SQL that could lead to injection. The tool supports custom SQL function matchers via TOML configuration, allowing users to extend analysis to their own wrapper libraries. The architecture is a single Go binary with modular packages for configuration, schema parsing, query extraction, and SQL validation, with comprehensive test coverage including integration tests using real Go packages.
Blazer is a Ruby on Rails engine that provides a self-service business intelligence web application. It allows users to run raw SQL queries against multiple databases (PostgreSQL, MySQL, Redshift, BigQuery, Snowflake, 30+ others via pluggable adapters), visualize results as charts and maps, build interactive dashboards, and schedule recurring data quality checks with email or Slack notifications. The architecture follows a monolith Rails engine pattern with clear separation of concerns: adapters for database connectivity, a statement parser for variable substitution, a result caching layer, and controllers/views for the web UI. It also supports anomaly detection and forecasting as optional enhancements. The codebase is well-structured, extensively tested, and actively maintained with frequent releases. Notable patterns include the adapter interface in `lib/blazer/adapters/base_adapter.rb`, the SQL statement parsing in `lib/blazer/statement.rb`, and the execution pipeline in `lib/blazer/run_statement.rb`.
Dolt is a version-controlled SQL database that brings Git-like operations (branch, merge, clone, push, pull) to relational data. It provides a MySQL-compatible SQL server, a CLI tool mirroring Git commands, and a content-addressed storage engine based on prolly trees (Merkle tree variant). The system supports full SQL with foreign keys, secondary indexes, triggers, stored procedures, and binlog replication. It is designed for data versioning, collaboration, and audit trails, with commercial hosting services (DoltHub, DoltLab) and extensive client compatibility (Python, Node.js, Java, Go, etc.). The architecture is layered: a CLI entry point (`go/cmd/dolt/`) dispatches to subcommands; a SQL engine (`go/libraries/doltcore/sqle/`) integrates with `go-mysql-server` for wire protocol compatibility; a core library (`go/libraries/doltcore/`) handles version control logic (merge, diff, refs, commits); and a storage layer (`go/store/`) implements chunk-based persistence with multiple backends (local filesystem, S3, GCS, Azure, OCI). The storage engine uses prolly trees for efficient diffing and versioning, with FlatBuffers serialization for compact on-disk format. Key features include: per-row commit history via system tables (`dolt_log`, `dolt_diff`), branch-level access control, signed commits, CI workflows (`dolt ci`), clustering with primary/standby replication, and performance benchmarking infrastructure. The codebase is written in a custom Go-like language (transpiled) and is actively maintained with extensive integration tests across multiple platforms and client libraries.
Exposed is a Kotlin SQL ORM library by JetBrains that provides a type-safe DSL for database operations, an optional DAO/entity layer, schema migration tooling, and multi-database support. The architecture is modular: a core module (exposed-core) defines the SQL DSL, column types, expression trees, and vendor dialects; separate modules implement synchronous JDBC (exposed-jdbc) and reactive R2DBC (exposed-r2dbc) backends; the DAO layer (exposed-dao, exposed-dao-r2dbc) adds entity management with caching, relationships, and lifecycle hooks; additional modules provide encryption, JSON, money types, Spring Boot auto-configuration, and Gradle/Maven plugins for migration generation. The library supports H2, MySQL, MariaDB, PostgreSQL, Oracle, SQL Server, SQLite, and Amazon Redshift, with extensive dialect-specific SQL generation and type mapping. Key entry points include Database.connect() for JDBC and R2dbcDatabase.connect() for R2DBC, transaction scopes (transaction {} / suspendTransaction {}), table/column definitions via DSL functions, and entity classes (IntEntity, UuidEntity, etc.) for the DAO pattern. Data flows through a query builder that constructs parameterized SQL statements, which are then executed by backend-specific executables (BlockingExecutable for JDBC, SuspendExecutable for R2DBC). The library also provides statement interceptors for logging and monitoring, and a migration utility that can generate Flyway-compatible scripts from table definitions. External dependencies include JDBC drivers for each supported database, R2DBC drivers, Spring Framework (for the Spring Boot starters and transaction integration), and optional libraries for encryption (AES, BCrypt, Argon2). The build system is Gradle with Kotlin DSL, and the project publishes multiple artifacts to Maven Central. The codebase demonstrates strong engineering practices with extensive cross-database test suites, parameterized tests, and continuous integration across multiple database containers.
This repository implements a Model Context Protocol (MCP) server that exposes DuckDB and MotherDuck databases to AI assistants and IDEs. The server provides five MCP tools: execute_query for running SQL queries, list_databases/list_tables/list_columns for catalog exploration, and switch_database_connection for changing the active database at runtime. It supports multiple database sources including in-memory DuckDB, local .duckdb files, S3-hosted databases, and MotherDuck cloud databases. The server can run in read-only or read-write mode, with additional security features like SaaS mode for MotherDuck that restricts local filesystem access. The codebase is well-architected with clear separation of concerns: a DatabaseClient class handles all connection management and query execution, tool implementations are pure functions living in a dedicated subpackage, and server configuration is centralized in configs.py. The use of FastMCP for the MCP protocol provides a solid foundation. The repository includes a comprehensive test suite of over 30 E2E tests covering local files, in-memory databases, MotherDuck, S3, read-only mode, concurrent access, timeouts, result limits, type serialization, catalog tools, and HTTP transport modes. CI/CD pipelines automatically run tests, publish to PyPI and the MCP registry, and create GitHub releases with MCPB bundles. The server is production-ready with features like lazy connection initialization, query timeout via threading, automatic result truncation based on row and character limits, and support for ephemeral connections that avoid file locks. The code handles edge cases such as MotherDuck read-scaling token validation, S3 credential chain support, and proper redaction of sensitive tokens in switch database responses. While the architecture is solid, most components are tightly coupled to the DatabaseClient interface, limiting direct reusability outside this project. The instructions.py module is the most easily extractable component, providing a template for generating dynamic MCP server instructions based on configuration.
This repository is a comprehensive educational resource for data engineering, structured as a handbook and bootcamp. It contains curated learning paths (beginner, intermediate, Databricks AI bootcamp) with lecture materials, homework, and sample datasets. The codebase includes teaching examples of core data engineering patterns: dimensional data modeling (cumulative tables, SCD type 2), fact data modeling (array metrics, datelist int), Apache Spark jobs with unit tests, Apache Flink streaming jobs, and analytical SQL patterns (retention, funnel, grouping sets, growth accounting). It also includes a small Flask server for A/B experimentation and Docker Compose setups for local development. While primarily educational, the SQL scripts and job templates are production-quality and reusable across projects. The repository is actively maintained with pinned dependencies and clear documentation, making it highly reproducible for learners.
drawDB is a free, open-source, client-side web application for designing relational database schemas and generating SQL. It provides an interactive ER diagram editor with support for multiple database dialects (MySQL, PostgreSQL, SQLite, MariaDB, MSSQL, Oracle), import/export in SQL, DBML, JSON, Mermaid, and Markdown, and features like auto-arrange, auto-connect foreign keys, schema validation, migration generation, and versioning via GitHub Gists. The architecture is a React single-page application (Vite + Tailwind CSS + Semi UI) with state managed through custom React Contexts and hooks, persistence via IndexedDB (Dexie), and an optional cloud backend through an extension system. It is primarily an application, with no data research or ML components.
Drizzle ORM is a headless, type-safe SQL query builder and ORM for TypeScript/JavaScript, supporting PostgreSQL, MySQL, SQLite, and serverless databases (Neon, PlanetScale, Cloudflare D1, Turso, etc.). The repository is a monorepo containing the core ORM library (`drizzle-orm`), a CLI tool (`drizzle-kit`) for schema introspection, migration generation, and database push/pull, a data seeding library (`drizzle-seed`), validation schema generators (`drizzle-zod`, `drizzle-valibot`, `drizzle-typebox`, `drizzle-arktype`), and an ESLint plugin (`eslint-plugin-drizzle`). The architecture follows a dialect abstraction pattern: each database has its own column builders, query builders (select, insert, update, delete), and session implementations that wrap native client libraries. The core provides composable SQL building via template literals, runtime type identification via the `entityKind` pattern, and a caching layer with tag-based invalidation. The CLI uses Zod-based configuration validation and generates migration SQL by diffing serialized schema snapshots. Integration tests run against real databases in Docker containers, and extensive type tests ensure compile-time correctness. External dependencies include various database client libraries (pg, mysql2, better-sqlite3, libsql, @neondatabase/serverless, etc.) and validation libraries (zod, valibot, typebox, arktype).
DBeaver is a universal database management tool and SQL client, built as a desktop application on the Eclipse RCP (OSGi plugin) framework. It provides a rich graphical interface for connecting to over 100 database systems via JDBC/ODBC drivers, including relational databases (MySQL, PostgreSQL, Oracle, SQL Server, SQLite, etc.), cloud databases (BigQuery, Athena, Snowflake), and NoSQL systems (MongoDB, Cassandra) in commercial editions. The application offers a full suite of database development features: a multi-tab SQL editor with syntax highlighting, code completion, and execution plan visualization; a data editor with grid, form, and GIS geometry views; ER diagram designer; data export/import wizards supporting CSV, JSON, XML, SQL, Markdown, and Excel; dashboard widgets for query-based monitoring; and an integrated AI assistant (OpenAI, Copilot) for natural language query generation and schema analysis. The architecture follows a strict model/UI separation enforced by OSGi plugin boundaries. The core model plugin (`org.jkiss.dbeaver.model`) defines interfaces for database objects, data values, execution contexts, and security, which are implemented by database-specific extension plugins (`org.jkiss.dbeaver.ext.*`). UI plugins (`org.jkiss.dbeaver.ui.*`) provide editors, navigators, and dialogs, while data transfer, debugging, and AI features are isolated in their own plugins. This design allows the model layer to be reused in the web-based CloudBeaver product. Key entry points include the `DBeaverApplication` class for startup, the `SQLEditor` for query editing, and the `DataTransferWizard` for import/export. External dependencies include the Eclipse RCP framework, JDBC drivers, ANTLR4 for SQL grammar parsing, JSQLParser for SQL analysis, and various third-party libraries for SSH tunneling, AWS SDK, and AI API communication. Data flows center around database connections: users configure data sources via connection wizards, execute SQL queries through the editor, and view results in data grids or export pipelines. The data transfer subsystem implements a producer/consumer pattern with pluggable transformers and serializers, supporting streaming for large datasets. The AI integration uses a provider abstraction to interface with different LLM APIs, generating SQL from natural language and providing schema-aware context. The repository also includes a CLI tool (`dbvr`) for CI/CD automation and a headless test platform. While the community edition is fully open-source, commercial versions add NoSQL support, additional AI providers, and premium features, with some code paths guarded by feature flags.
Liam is a full-stack web application and CLI toolchain for database schema design and visualization. It parses schemas from multiple formats (PostgreSQL, Prisma, Drizzle, tbls, schema.rb), renders interactive Entity-Relationship Diagrams (ERDs) using React Flow, and provides an AI-assisted workflow powered by LangGraph agents. Users can create design sessions, chat with a team of AI agents (PM, DB, QA) that analyze requirements, generate schema changes, create migration SQL, and validate via pgTAP tests. The frontend is built with Next.js and Supabase for authentication, real-time updates, and data persistence. A CLI tool allows local ERD generation from schema files. The monorepo includes numerous internal packages for schema parsing, UI components, error handling (neverthrow), and in-browser PostgreSQL execution (PGlite).