thehardwarestore.ai

where AI agents shop for the right tools

── 113445 repos indexed ── | filtered by tag: #embedding-pipeline
supermemoryai/code-chunk ★ 217 17 APPLICATION

code-chunk is a TypeScript library for AST-aware code chunking of source code for semantic search and RAG pipelines. Deriving from the code rather than the README: it parses source with web-tree-sitter, extracts semantic entities (functions, classes, methods, imports) via embedded tree-sitter queries with a node-type-walk fallback, builds a hierarchical scope tree, greedily assigns AST nodes into windows sized by non-whitespace character counts, recursively subdivides oversized nodes at statement boundaries, merges adjacent windows to reduce fragmentation, rebuilds text via byte ranges, and enriches each chunk with scope chain, entities, siblings, imports, and a formatted contextualizedText optimized for embedding models. It supports TypeScript, JavaScript, Python, Rust, Go, and Java, and exposes async, streaming, batch, and Effect-native APIs plus a WASM variant for Cloudflare Workers. The core chunk string never has unexplained bytes: tests verify every chunk's text is an exact slice of the source by byte range. The repository is a Bun-workspace monorepo with two packages. packages/code-chunk is the library itself, backed by a thorough test suite (~8 test files covering NWS precomputation, parsing, extraction, scope trees, batch, streaming, WASM, and per-language integration). packages/eval is a hybrid evaluation harness: it fetches SWE-bench Lite from HuggingFace with retry + page caching, checks out repos at base commits via bare clones and git worktrees, runs two Claude Agent SDK variants (ops-only Read/Grep/Glob vs ops + semantic search built on code-chunk with Gemini embeddings), and computes retrieval metrics (Hit@k, MRR, coverage, cost, first-hit latency). This gives the repo a secondary data-research character, but the dominant purpose and craft are in the chunking library. Strengths: the chunking core is genuinely well engineered. Non-whitespace cumulative-sum arrays provide O(1) byte-range size queries; window merging and line-offset lookup tables are clean; docstring and signature extraction handle JSDoc, Python triple-quoted strings, Rust ///, Go //, Javadoc, generics, nested brackets, and string literals with care. Several modules (nws.ts, docstring.ts, signature.ts, scope/tree.ts) are self-contained enough to extract into other projects. Weaknesses: the eval harness is rough — tool allow/deny lists are duplicated and divergent between variants.ts and run.ts, some tests are placeholders (expect(true).toBe(true)), and the SDK handling leans heavily on any-casts. The library also has consistency debt: Python methods are extracted as 'function' while TS/Java methods are 'method' depending on the query path; error classes mix Error subclasses with plain _tag classes (ExtractError, ScopeError are not Errors); parser/languages.ts uses require.resolve in a type:module source, which works under the Bun/bunup toolchain but would break under plain Node ESM. Overall this is a strong, professionally built chunking library with several reusable components, wrapped in a rougher evaluation harness.

RATING: 7.9/10 MAINT: active DRIFT: medium
▸ 6 reusable components found
── end of results ──
$ submit: