$ show VectifyAI/PageIndex
PageIndex is a production-ready RAG framework that replaces vector databases with a reasoning-based, hierarchical tree index over documents. Its core innovation is the 'flash' pipeline: a sophisticated, LLM-free PDF structure extractor that uses character-level layout analysis, trie-based keyword matching, and heuristic classification to build a table of contents tree from PDFs in seconds. The system then enables agentic retrieval by feeding this tree to an LLM, which reasons over the hierarchy to find relevant pages without embedding or chunking. The repository provides a full Python SDK (PageIndexClient) with local and cloud modes, CLI tool, integrations with OpenAI Agents SDK, Anthropic SDK, and Claude Agent SDK, plus an MCP bridge for remote tool execution. While primarily an application, it contains a deep research-grade data processing pipeline (flash) with extensive multilingual dictionaries and layout analysis heuristics, giving it a hybrid character. The codebase is well-structured with clear module boundaries, extensive test coverage, and reusable patterns like the tool contract system and trie-based matching. However, it carries medium technical drift risk due to hardcoded thresholds, dependency on private APIs of LLM frameworks, and pdfium version sensitivity.
The LLM identified 5 files worth highlighting for reuse:
The entire flash module is a self-contained, highly engineered PDF structure extraction system that can be extracted as a standalone library. It performs character-level parsing via pdfium, column detection, line clustering, block building, heading detection, and outline assembly—all without LLM calls. Demonstrates non-obvious skill in handling complex PDF layouts, multilingual text, and academic document structures. The module is loosely coupled internally but could be packaged independently with minimal changes.
Implements a clean tool contract pattern for agentic document retrieval. Defines tools (browse, get, structure, page content, delete) with JSON Schema definitions, response size limits, and cloud-local parity enforcement. The pattern is reusable for any agent tool system and includes framework adapters for OpenAI, Anthropic, and Claude. The contract is tested for byte-identical schema between local and cloud modes.
A generic MCP (Model Context Protocol) client over HTTP with SSE streaming. Handles session management, re-initialization on 404, blob stubbing, and proper error propagation. Reusable for any MCP server without modification. Demonstrates careful handling of protocol nuances like tool call ID matching and isError propagation.
Implements a tree optimization algorithm for refining hierarchical document indices. Uses a cost model (scan pages + routing cost) to decide merge/expand operations. The frontier-based cost analysis is a generic pattern applicable to any hierarchical structure optimization problem. Demonstrates thoughtful engineering of trade-offs between tree depth and node size.
Implements a trie-based keyword matching system (similar to Aho-Corasick) for efficient lookup of boilerplate phrases, headings, and section keywords. Includes tokenizer, hashing, and token type classification. The trie is used throughout the flash pipeline for classification. Reusable for any text pattern matching task.