This repository contains the complete source code for SQLite, a self-contained, high-reliability, embedded SQL database engine. The codebase implements a full SQL database system including a SQL parser (generated by the included Lemon parser generator), a bytecode virtual machine (VDBE) for executing prepared statements, a B-tree storage engine, a transaction manager with WAL and journaling, and a pluggable VFS (Virtual File System) layer for OS abstraction. The architecture is a monolithic C library designed for embedding into applications, with optional loadable extensions for full-text search (FTS3/4/5), R-tree spatial indexing, JSON processing, vector similarity search (vec1), and dozens of utility functions. Key entry points include the public C API defined in sqlite3.h, the command-line shell (shell.c), and the Tcl test harness. Data flows through the SQL parser to the query optimizer (where.c), which generates VDBE bytecode executed by the virtual machine, ultimately reading/writing pages through the pager and B-tree layers. External dependencies are minimal: only standard C library and optional ICU for Unicode support. The repository also includes extensive test suites, fuzz testing harnesses, and build systems for Unix, Windows, and WebAssembly.
This repository contains the SQLite embedded SQL database engine source code, mirrored from the actively maintained SQLite project. The core is a C library that parses SQL, compiles it into a custom bytecode virtual machine (VDBE), and executes it against a B-tree storage layer backed by a transactional pager and a pluggable OS abstraction (VFS). Primary entry points are the public C API in src/sqlite.h.in, the sqlite3 command-line shell built from src/shell.c.in, and the init functions of many loadable extensions.
GRDB.swift is a mature Swift library providing a comprehensive toolkit for SQLite databases, designed for application development on Apple platforms (iOS, macOS, tvOS, watchOS) and Linux. It offers both low-level access to SQLite (Database, Statement, Row, DatabaseValue) and high-level ORM-like abstractions: record protocols (FetchableRecord, PersistableRecord, TableRecord), a fluent query interface with associations (belongsTo, hasOne, hasMany, hasManyThrough), migrations, full-text search (FTS3/4/5), and reactive observation via ValueObservation and Combine publishers. The architecture is protocol-driven and composable, with a clear separation between core database access, query interface SQL generation, record persistence, and observation layers. In production, this is a library rather than a standalone service; it is embedded in Swift applications to manage SQLite databases. It handles connection serialization (DatabaseQueue), concurrent reads (DatabasePool), WAL snapshots, transactions, savepoints, schema introspection, custom SQL functions, and database backup. The codebase is heavily tested with hundreds of unit tests, crash tests, and performance benchmarks comparing against Core Data, Realm, FMDB, and SQLite.swift. The library is actively maintained (v7.11.1, Swift 6.1+, Xcode 16.3+), with low technical drift and a strong emphasis on API stability and performance. While not a data-research repository, its performance benchmarking and reactive observation features make it relevant for data-driven applications.
Paperless-ngx is a production-grade, self-hosted document management system. It ingests physical and digital documents through a consume folder, REST API uploads, and IMAP email fetching, then runs OCR via Tesseract/ocrmypdf, extracts text through a pluggable parser registry, generates archive PDFs and thumbnails, and stores structured metadata (tags, correspondents, document types, custom fields). Documents are indexed in Tantivy and exposed through a Django REST API consumed by an Angular SPA that provides search, editing, bulk operations, permissions, workflows, sharing, and an admin interface. Background processing is handled by Celery/Redis with real-time status pushed over WebSockets. The system also includes optional AI/LLM classification, RAG chat over the document index, and a workflow engine that reacts to consumption and document lifecycle events with actions like assignment, emailing, webhooks, and trash moves. Though primarily an application, it contains a substantial search-engine subsystem and an optional LLM/embedding pipeline, giving it a mild hybrid character.
ParadeDB is a PostgreSQL extension that provides full-text search, vector search, and analytics directly within the database, eliminating the need for a separate search service. The core engine is `pg_search`, which implements a custom index access method (`paradedb`/`bm25`) using Tantivy for search indexing and DataFusion for analytic aggregations. It supports BM25 scoring, phrase/term/boolean queries, fuzzy matching, proximity search, hybrid search (combining full-text and vector), and a rich set of tokenizers (including jieba, ICU, Lindera, ngram). The extension integrates deeply with PostgreSQL's planner through custom scan nodes (BaseScan, AggregateScan, JoinScan, MPP) to push down search, aggregation, and join operations into the index. It also offers columnar storage via fast fields, parallel query execution, and WAL-level integration for crash safety. Architecturally, the system is a Rust library compiled as a shared library loaded by PostgreSQL, using the `pgrx` framework for FFI. Entry points include SQL functions for search (e.g., `paradedb.query`, `pdb.score`), DDL for index creation, and custom plan/exec hooks. Data flows: user SQL queries are intercepted by the planner, which decides whether to use a custom scan path; if so, Tantivy or DataFusion executes the search/aggregation/join, producing tuples that are optionally post-processed by PostgreSQL (e.g., heap fetches, visibility checks). The extension relies on external dependencies: Tantivy (forked for block storage), DataFusion (for aggregate execution), ICU (for Unicode tokenization), Lindera (for Japanese tokenization), and Apache Arrow (for columnar data exchange).
Meilisearch is a high-performance search engine written in Rust, organized as a workspace of crates. The core `milli` crate implements indexing, ranking, faceting, filtering, and vector search; the `meilisearch` crate exposes an HTTP API via actix-web. It supports typo-tolerant full-text search, facet filters, geo search, hybrid keyword/vector search, and federated search across remote instances. The system persists data in LMDB and provides dump/restore with backward-compatible readers for multiple dump versions. Beyond search, the repository contains data-intensive components: an indexing pipeline with incremental updates, facet tree structures, embedding generation with multiple providers (OpenAI, HuggingFace, Ollama, REST), and a task scheduler for asynchronous batch operations. Experimental features include chat completions, personalization, and dynamic search rules. The codebase is primarily an application, but includes research-grade algorithms for tokenization, typo tolerance, and vector similarity. Maintenance risks include dual indexing pipelines (old and new update paths), many experimental feature flags, and vendored external crates. The code is generally well-structured with reusable crates for dump handling, filter parsing, and HTTP client utilities.
Typesense is a high-performance, open-source search engine written in C++17, designed as a drop-in alternative to Algolia and Elasticsearch. It provides a RESTful HTTP API for full-text search, faceted navigation, geospatial filtering, vector search (with HNSW), and hybrid (semantic + keyword) search. The architecture is a distributed, Raft-based clustered system with leader election, log replication, and snapshotting, ensuring high availability and consistency. The server integrates with external AI providers (OpenAI, Google, Azure, Cloudflare) for embedding generation and natural language query processing, and supports on-device ONNX models for text, image, and voice search. The codebase is modular, with singleton managers for collections, embeddings, analytics, rate limiting, and authentication, all built on top of a RocksDB-backed key-value store. Extensive test coverage includes C++ unit tests (gtest), TypeScript integration tests (Bun), and benchmark infrastructure (k6, InfluxDB). While primarily an application, it incorporates data-intensive features like statistical sketches (TDigest, CountMinSketch), custom indexing data structures (ART, numeric trie, hat-trie), and analytics pipelines, giving it a hybrid character.
slacrawl is a local-first Slack archiving CLI tool written in Go. It ingests messages, files, users, and channels from multiple sources—Slack API (bot/user tokens), Slack Desktop local cache (LevelDB + IndexedDB), MCP (Model Context Protocol) adapters, and external provider subprocesses—and stores them in a local SQLite database with full-text search (FTS5). The tool provides a rich command surface including sync, search, TUI browsing, analytics reports (digest, trends, quiet channels), retention purging, and Git-based archive sharing (publish/subscribe). It is designed for privacy-conscious users who want offline access to their Slack history without cloud dependency. The architecture follows a clean internal package separation: `cmd/slacrawl` is the entry point, `internal/cli` handles command routing via Kong, `internal/store` provides the SQLite persistence layer with schema migrations and batch writes, `internal/syncer` orchestrates multi-source synchronization, and `internal/slackapi`, `internal/slackdesktop`, `internal/slackmcp`, and `internal/provider` implement source-specific adapters. The codebase is well-tested with extensive Go tests, CI/CD via GitHub Actions, and cross-platform releases (Linux, macOS, Docker). While primarily an application, slacrawl exhibits data pipeline characteristics: it ingests raw Slack data, transforms it into a normalized relational schema with FTS indexing, and exposes analytical outputs (reports, digests). The embedding_jobs table hints at future ML integration. The Git archive sharing module is a novel pattern for distributed data replication. Overall, the codebase demonstrates strong engineering practices with reusable components like the generic MCP client, platform-specific file locking, and the external provider protocol.
Papra is a self-hostable, open-source document management platform that provides a full-stack solution for archiving, organizing, and searching documents. The system consists of a Hono-based API server with SQLite (via Drizzle ORM and libsql/Turso), a SolidJS web client, a React Native mobile app (Expo), a CLI tool, and several standalone packages. It supports multi-tenant organizations, role-based access, document upload with content extraction (OCR via Tesseract, Mistral OCR, Azure Document Intelligence, Docling, and a custom HTTP strategy), full-text search via SQLite FTS5 with a custom query parser, tagging with automatic tagging rules, document sharing with password-protected links, intake emails, webhooks, and subscriptions with Stripe integration. The architecture is modular, using repository and usecase patterns, an in-process event system, and a driver-based approach for storage (filesystem, S3, Azure Blob), email delivery, and KV stores. The codebase is heavily tested with unit, integration, and e2e tests, and includes a comprehensive demo mode.
SeekStorm is a high-performance search engine library and multi-tenancy server implemented in Rust. It provides both lexical (BM25F with proximity) and vector (ANN with IVF clustering and scalar/Turbo quantization) search capabilities, supporting real-time indexing, faceted search, spelling correction, query auto-completion, highlighting, and hybrid result fusion via RRF. The architecture consists of a core library (`seekstorm`), a RESTful server (`seekstorm_server`) with embedded web UI, and a Rust client SDK (`seekstorm_client`). The server uses memory-mapped files for persistence, supports per-index configuration of tokenizers, stemmers, stop words, and compression, and enforces multi-tenant isolation via API keys with resource quotas. Search performance is enhanced with SIMD acceleration (AVX2/NEON) for posting list intersection, vector quantization, and distance computations. The codebase is well-tested with extensive integration and consistency tests, and follows an explicit commit model for durability.
Infinity is a high-performance AI-native vector database designed for LLM applications, offering hybrid search over dense embeddings, sparse embeddings, tensors, and full-text. Its architecture is a distributed C++ core with a modular planner/executor, multiple index types (HNSW, IVF, BMP, DiskANN, EMVB, PLAID, full-text), and a transaction system with WAL and checkpointing. The database exposes a SQL interface, a Thrift RPC API, an HTTP API, and native SDKs for Python, Go, and C++. It also includes a web GUI (Next.js) for administration. The codebase is heavily optimized with SIMD instructions, custom analyzers for multiple languages, and advanced query optimization (fast rough filters, result caching). While the repository is actively maintained with a comprehensive test suite and CI/CD pipeline, there is some technical drift from deprecated KNN implementations and disabled unit tests, and the reliance on C++20 modules may introduce compiler compatibility risks.
Min is a production desktop web browser built on Electron, focused on speed, minimal UI, and privacy. The repository contains the complete application: main-process window and view management, tab/task state, a plugin-driven search/address bar, session restore, download handling, reader mode, PDF viewing, content blocking, translation, and password-manager integrations. The dominant architecture is an Electron main/renderer split with IPC-driven communication, several hidden service pages (places, translation), and a custom module bundling pipeline for main, browser, and preload scripts. Although it embeds data-oriented components such as a history full-text search engine and a bookmark tag index, this is an application, not a data/research project; those components serve browser features rather than analytical workflows. The primary entry points are `main/main.js` for the Electron main process, `js/default.js` for browser UI initialization, and `index.html` for the renderer shell. Core data flows are local: browsing history and bookmarks are persisted through Dexie/IndexedDB in a hidden places service, full-text history search uses a custom BM25-style ranking implementation, and network requests are filtered by a modified ABP filter parser. External dependencies include Electron, Dexie, pdfjs-dist, Readability, drag-and-drop libraries, external CLI password managers, EasyList-style filter lists, the Public Suffix List, and Bergamot translation models. The searchbar plugin system and password-manager abstraction are the most notable architectural patterns. The codebase is actively maintained but shows medium technical drift: it relies on a custom concatenation-based build instead of a standard bundler, mixes legacy callbacks with modern async/await, vendors modified forks of third-party code, and contains Electron-version-specific workarounds. No automated test suite is present in the repository, and some platform-specific code is fragile across Electron upgrades. Overall it is a solid, working desktop application with several clean abstractions and a small number of genuinely reusable components, but it would benefit from test coverage and modernization.
This repository is the official archive and static site for '老司机 iOS 周报' (Senior Driver iOS Weekly), a long-running curated newsletter for iOS developers. The project spans from 2018 to the present (2026) and contains over 380 weekly issues stored as structured Markdown files. While the core content is a curated list of articles, tools, and news, the repository also includes a build pipeline and a single-page application (SPA) that generates a searchable, full-text archive deployed to GitHub Pages. The architecture is a hybrid of content repository and static site generator: the `scripts/build-index.js` script parses the Markdown files, extracts metadata (sections, recommenders, reading time), and builds a search index using MiniSearch with CJK tokenization. The `docs/` directory contains the resulting SPA (`app.js`, `index.html`) and vendored libraries. The CI workflow (`.github/workflows/pages.yml`) automates building and deploying. The project is community-driven, with contributions submitted via GitHub Issues. It is actively maintained with recent updates, and the codebase is focused on reliable content delivery rather than complex application logic.
Blackbird is a lightweight, asynchronous SQLite wrapper and model layer for Swift applications. It provides a type-safe persistence layer using Swift concurrency and Codable, with automatic schema migrations, key-path-based querying, caching, change notifications, and SwiftUI integration. The library avoids external dependencies and focuses on developer experience over raw performance, using Swift's type system for compile-time safety. While primarily designed as a reusable framework for local data storage in Apple platform apps, it includes advanced features like full-text search with highlighting and transactional support. The codebase shows evidence of active maintenance but relies on some Swift internals that may pose future compatibility risks.