$ show scallop-lang/scallop
Scallop is a differentiable probabilistic Datalog engine implemented in Rust with Python bindings (scallopy). It compiles Scallop programs (a Datalog variant with provenance annotations) through a three-stage compiler (front AST → back normalized Datalog → RAM relational algebra) to a runtime that executes with configurable provenance semirings (unit, boolean, tropical, probabilistic, differentiable, etc.). The system supports neuro-symbolic AI by integrating with PyTorch for gradient-based learning, and provides plugins for LLMs (GPT, Gemini) and vision models (CLIP, OWL-ViT, SAM). The architecture is a monolithic Rust library (core) with multiple sub-crates for CLI tools (scli, sclc, sclrepl), Python bindings via PyO3, and a plugin system for extending with foreign functions, predicates, and attributes. Key data flows: user-provided Scallop source or facts are parsed, analyzed (type inference, boundness), transformed, and compiled into a relational algebra plan that is executed with provenance-aware dataflow operators, producing output collections. The codebase is well-structured with clear separation between compiler, runtime, and provenance layers, and includes several reusable components such as copy-on-write containers, galloping search, dual numbers for AD, batch streaming abstractions, and a self-contained SDD library for weighted model counting. While the code demonstrates strong engineering and research-grade sophistication, there is some code duplication across provenance variants and experimental/legacy directories that lower the overall quality score. The repository is actively maintained with recent commits, CI workflows, and a growing plugin ecosystem.
The LLM identified 7 files worth highlighting for reuse:
Implements a generic CopyOnWrite container using Rc or Arc for cheap clone and deferred mutation. The container wraps a value and a flag; clone is O(1) and only deep-copies on write. This is a reusable utility for any Rust project requiring shared immutable state with occasional mutations, demonstrating a clean implementation of a classic pattern.
Provides galloping (exponential) search functions for iterating over sorted batches efficiently. Used in join and difference dataflow operators to skip ahead when keys diverge. This is a standalone, reusable implementation of a well-known algorithm for sorted list operations, extractable for any project needing efficient sorted batch processing.
Defines a DualNumber type for automatic differentiation within semiring computations. Supports standard arithmetic operations (add, multiply) and gradient accumulation. This component is self-contained (depends only on basic numeric traits) and can be extracted as a lightweight dual-number library for differentiable semirings in Rust.
A standalone library for constructing and evaluating Sentential Decision Diagrams (SDDs), a form of knowledge compilation used for weighted model counting. Includes vtree construction, bottom-up SDD compilation, semiring evaluation, and disjunction handling. This is a fully self-contained, reusable library for probabilistic inference and compilation.
Defines the AttributeProcessor trait and registry for user-defined attributes (e.g., @gpt, @demand). Processors can inspect and modify the AST, register foreign functions/predicates, and affect compilation. This clean extension point allows third-party plugins to add language features without modifying the core compiler, demonstrating a well-designed plugin architecture.
Defines the Batch trait and implementations for stable, recent, empty, single, chain, and join batches. These abstractions enable efficient, composable streaming dataflow operations (e.g., join, antijoin) over sorted collections. The pattern is reusable for any incremental or streaming computation system needing batch-oriented processing.
Provides a Plugin base class with hooks for setup_argparse, configure, and load_into_context. Plugins can add CLI arguments, initialize resources, and register foreign attributes/functions into the Scallop context. This clean, extensible design enables the plugin ecosystem (GPT, Gemini, CodeQL, etc.) and can be reused as a pattern for any Python framework that needs plugin-based extensibility.