$ show supermemoryai/code-chunk
code-chunk is an AST-aware code chunking library designed for semantic search and RAG pipelines. It uses tree-sitter to parse source code into abstract syntax trees, extracts semantic entities (functions, classes, methods, etc.), builds a hierarchical scope tree, and chunks code at semantic boundaries rather than arbitrary character limits. Each chunk is enriched with contextual metadata including scope chain, entities, siblings, and import dependencies, producing contextualized text optimized for embedding models. The library supports multiple languages (TypeScript, JavaScript, Python, Rust, Go, Java), offers batch processing, streaming APIs, Effect integration, and WASM support for Cloudflare Workers. An accompanying evaluation harness (@supermemory/eval) compares agent performance with and without semantic search using SWE-bench Lite.
The LLM identified 2 files worth highlighting for reuse:
Implements O(1) non-whitespace character counting using cumulative sum array preprocessing. Provides countNws, preprocessNwsCumsum, getNwsCountFromCumsum, and getNwsCountForNode functions that enable efficient range queries for chunk size calculations. This isolated utility demonstrates non-obvious optimization skill and could be extracted for general text processing scenarios.
Formats code chunks with semantic context for embedding models. Takes raw chunk text and context object (scope chain, entities, siblings, imports) to produce contextualizedText that prepends file path, scope hierarchy, entity signatures, import dependencies, and sibling context. This self-contained formatting logic is specifically optimized for improving embedding quality in RAG systems and demonstrates domain-specific expertise in semantic search optimization.