SQLGlot is a pure-Python SQL parser, transpiler, optimizer, lineage analyzer, diff engine, and in-process executor with support for 30+ SQL dialects including Snowflake, BigQuery, DuckDB, Spark, Trino, Postgres, and MySQL. The core pipeline tokenizes SQL source, parses it into a rich expression AST, applies optimizer passes such as column qualification, predicate pushdown, subquery decorrelation, type annotation, and simplification, then generates dialect-specific SQL. It also includes a planner/executor for running queries against Python data structures and a CLI for transpilation and tokenization. Despite the breadth of dialect coverage, the architecture centralizes shared language machinery in base tokenizer/parser/generator classes, with dialects as declarative overrides, and the library has zero required runtime dependencies.
Ockam is a framework for building secure, interconnected systems that provides identity-based end-to-end encrypted communication, secure channel establishment, and policy-based access control. The architecture features clear separation of concerns with modules for core types, identity management, secure channels, transport abstractions, vault cryptography, API orchestration, and a command-line interface. Implemented primarily in Rust with Elixir components, it uses async/await and OTP patterns for concurrency. Key entry points include the ockam CLI, node builder/context, and orchestrator API. The system enables secure multi-hop routing, message forwarding, and service integration with external systems like Kafka and InfluxDB.
This is a production-grade Customer Data Platform (CDP) backend written in Go. It ingests events via HTTP (gateway), transforms and enriches them (processor), and routes them to over 90 destinations including streaming services (e.g., Kafka, Kinesis) and data warehouses (e.g., Snowflake, BigQuery, ClickHouse). The core of the system is a PostgreSQL-backed job queue (JobsDB) that provides partitioned, multi-consumer, durable processing. The architecture is a monolith with multiple internal services that can be deployed in different modes (gateway-only, processor-only, embedded). It includes enterprise features like OAuth token management, user suppression, regulatory deletion (GDPR), and reporting with HyperLogLog aggregation. The codebase is well-structured with clear separation of concerns, extensive unit and integration tests, and many reusable patterns (worker pool, circuit breaker, transaction listeners, config reloading). It is actively maintained with regular commits and security patching.
DBeaver is a universal database management tool and SQL client, built as a desktop application on the Eclipse RCP (OSGi plugin) framework. It provides a rich graphical interface for connecting to over 100 database systems via JDBC/ODBC drivers, including relational databases (MySQL, PostgreSQL, Oracle, SQL Server, SQLite, etc.), cloud databases (BigQuery, Athena, Snowflake), and NoSQL systems (MongoDB, Cassandra) in commercial editions. The application offers a full suite of database development features: a multi-tab SQL editor with syntax highlighting, code completion, and execution plan visualization; a data editor with grid, form, and GIS geometry views; ER diagram designer; data export/import wizards supporting CSV, JSON, XML, SQL, Markdown, and Excel; dashboard widgets for query-based monitoring; and an integrated AI assistant (OpenAI, Copilot) for natural language query generation and schema analysis. The architecture follows a strict model/UI separation enforced by OSGi plugin boundaries. The core model plugin (`org.jkiss.dbeaver.model`) defines interfaces for database objects, data values, execution contexts, and security, which are implemented by database-specific extension plugins (`org.jkiss.dbeaver.ext.*`). UI plugins (`org.jkiss.dbeaver.ui.*`) provide editors, navigators, and dialogs, while data transfer, debugging, and AI features are isolated in their own plugins. This design allows the model layer to be reused in the web-based CloudBeaver product. Key entry points include the `DBeaverApplication` class for startup, the `SQLEditor` for query editing, and the `DataTransferWizard` for import/export. External dependencies include the Eclipse RCP framework, JDBC drivers, ANTLR4 for SQL grammar parsing, JSQLParser for SQL analysis, and various third-party libraries for SSH tunneling, AWS SDK, and AI API communication. Data flows center around database connections: users configure data sources via connection wizards, execute SQL queries through the editor, and view results in data grids or export pipelines. The data transfer subsystem implements a producer/consumer pattern with pluggable transformers and serializers, supporting streaming for large datasets. The AI integration uses a provider abstraction to interface with different LLM APIs, generating SQL from natural language and providing schema-aware context. The repository also includes a CLI tool (`dbvr`) for CI/CD automation and a headless test platform. While the community edition is fully open-source, commercial versions add NoSQL support, additional AI providers, and premium features, with some code paths guarded by feature flags.
Databend is an open-source, cloud-native distributed SQL query engine and data warehouse, built in Rust. It provides a full SQL interface for analytical workloads, supporting operations such as SELECT, INSERT, COPY INTO/OUT, MERGE INTO, and complex aggregations with window functions, subqueries, and joins. The architecture follows a storage-compute separation model: a metadata service (Meta) manages schemas, users, and catalogs via a Raft-based consensus layer, while query nodes handle SQL parsing, planning, optimization (Cascades optimizer with rule-based and cost-based passes), and distributed execution over a pipelined framework. Multiple storage engines are supported, including a native columnar Fuse engine optimized for cloud object stores (S3, MinIO, Azure Blob), as well as external formats like Iceberg, Delta Lake, Hive, Paimon, Parquet, and ORC. The system exposes access via MySQL wire protocol, HTTP API, Flight SQL, and gRPC, and includes enterprise features such as data masking, row access policies, materialized views, streams, network policies, and role-based access control.
Jitsu is an open-source event data platform (alternative to Segment) that ingests events from websites, apps, and servers, transforms them via user-defined functions (UDFs) and built-in destination functions, and delivers them to data warehouses (ClickHouse, BigQuery, Snowflake, Redshift, Postgres, S3, GCS) and SaaS tools (Amplitude, PostHog, HubSpot, etc.). The architecture is multi-service: a Go-based bulker handles ingestion and warehouse delivery, a TypeScript-based rotor executes event transformations and profile building, a Next.js console provides the management UI and API, and a CLI tool supports project management and function deployment. The data flow is: HTTP ingest → Kafka → rotor (functions) → bulker (SQL/file destinations) → ClickHouse/PostgreSQL for metrics and logs. External dependencies include Kafka, PostgreSQL, ClickHouse, Redis, MongoDB, and Google Cloud Storage. The console supports multi-tenant workspaces, authentication (Firebase, OIDC, NextAuth), billing, audit logging, rate limiting, and an MCP server for AI tool integration. The codebase is a monorepo with pnpm workspaces and Go modules, featuring extensive test suites (unit, integration with testcontainers, e2e) and CI/CD pipelines. It demonstrates strong engineering practices with reusable libraries (juava, jitsubase), custom high-performance JSON parsers, and well-abstracted repository and configuration patterns. However, there is some technical drift: duplicate JSON libraries (jsoniter and jsonorder), commented-out code, and ESLint disable comments in admin routes. Overall, it is a mature, actively maintained platform suitable for production use.
Redash is a full-stack business intelligence platform that enables users to connect to multiple data sources, write and schedule queries, build interactive visualizations, and create dashboards with alerting and sharing capabilities. The backend is a Python Flask application with SQLAlchemy for database access, Redis for caching and task queuing, and RQ for background job execution. The frontend is a React single-page application using Ant Design, Plotly, D3, and Leaflet, organized into pages for queries, dashboards, alerts, data sources, and administration. The system exposes a RESTful API for all operations and supports authentication via password, Google OAuth, SAML, LDAP, JWT, and API keys. The architecture follows a modular plugin pattern for query runners and visualizations. Over 35 query runners (e.g., PostgreSQL, BigQuery, Snowflake, MongoDB, Prometheus) are registered via a common base class, each implementing run_query, get_schema, and configuration_schema. The visualization library (viz-lib) is a separate package that registers chart types (line, bar, pie, map, table, cohort, sankey, etc.) with editors and renderers. Dashboards use a grid layout with widgets that can display visualizations, text, or embedded content. Alerts evaluate query results against conditions and send notifications to destinations like Slack, email, PagerDuty, and webhooks. Key data flows: users write SQL or use query runners to fetch data from connected sources; results are cached and can be visualized; dashboards aggregate multiple visualizations with parameter bindings; scheduled queries refresh results and trigger alerts. The system also supports embedding visualizations in external sites and public sharing of dashboards. While primarily an application, the extensive query runner and visualization plugin systems make it a flexible platform for data exploration and operational reporting.
This repository is a curated collection of over 30 independent Streamlit in Snowflake (SiS) demo applications, each self-contained in its own folder. The demos span a wide range of use cases including business intelligence dashboards, LLM-powered chatbots (Cortex, DeepSeek-R1, RAG), geospatial H3 visualizations, custom UI components (AgGrid, ECharts, Slickgrid), CI/CD pipelines with Snowflake CLI, and embedding SiS into external portals via Next.js. Each app includes a `streamlit_app.py` entry point, a `data/creation_script.sql` for Snowflake object setup, and sample data. The code is educational, heavily commented, and demonstrates patterns like caching, dynamic page navigation, and local/cloud-aware connections. While the apps are not production-grade (some hardcoded values, duplication), they serve as excellent reference implementations for building SiS applications.