$ show activeloopai/deeplake
Activeloop's DeepLake is a production-grade, multi-vector database engine for AI data. At its core, it is a columnar, chunked, compressed storage system for tensors (multi-dimensional arrays) with native support for embeddings, images, video, text, and other complex AI data types. It exposes a Python library for dataset management, ingestion (COCO, Labelbox, CSV, Parquet), version control, and integration with PyTorch/TensorFlow/MMDetection. A high-performance C++ backend provides lazy NumPy-like array operations (nd::array), a composable async framework (async::promise), and a Tensor Query Language (TQL). The most distinctive component is a PostgreSQL extension (pg_deeplake) that allows SQL access to DeepLake datasets via custom table and index access methods, supporting vector similarity search (cosine, BM25) and hybrid search. Query execution is delegated to DuckDB through a custom scan node, enabling efficient analytical queries. The extension also includes WAL-based DDL replication across instances. This is primarily an application (a database system), with substantial research-oriented features such as medical image formats, compression codecs, and TPC-H benchmarking.
The LLM identified 4 files worth highlighting for reuse:
A fully featured C++ promise/future implementation with chaining (then, then_async), combine (all/any), cancellation, error propagation, progress callbacks, and priority scheduling. It uses a spin-lock protected shared state and supports both immediate and background execution. This is a self-contained, reusable async primitive that could be extracted for any C++ project needing composable asynchronous operations.
Implements lazy, compile-time composition of array transformations using expression templates. Supports unary and binary kernel expressions (map, zip) over nd::arrays with type erasure through virtual dispatch. Demonstrates advanced C++ template metaprogramming and is reusable as a standalone array expression engine.
Injects precomputed column statistics (null fraction, distinct values, most common values, histogram) from DeepLake datasets directly into PostgreSQL's pg_statistic table, bypassing expensive ANALYZE sampling. This is a clever integration pattern that improves query planning for large-scale data and could be adapted for other external storage engines.
Provides robust conversions between bounding box formats (COCO, Pascal VOC, YOLO) in both pixel and fractional coordinate spaces. Includes polygon-to-mask conversion, evaluation metric wrapping, and seamless integration with MMDetection. This is a self-contained utility module that any computer vision pipeline could reuse independently of DeepLake.