$ show going-doer/Paper2Code
Paper2Code is a research system that automatically generates runnable code repositories from machine learning research papers. The core pipeline, PaperCoder, uses a multi-agent LLM architecture (planning→analysis→coding) implemented as a sequence of Python scripts orchestrated via shell commands. It accepts papers in PDF (converted to JSON via s2orc-doc2json), JSON, or LaTeX format, and produces a complete Python project with config, model definitions, training loops, and evaluation scripts. The codebase supports both OpenAI API (o3-mini by default) and open-source models via vLLM (e.g., DeepSeek-Coder-V2-Lite-Instruct). It includes a debug stage that iteratively fixes runtime errors using LLM-generated SEARCH/REPLACE patches, and an evaluation module that scores generated repositories (reference-free or reference-based) using LLM judgment. The repository also bundles the Paper2Code benchmark dataset (90 papers from top ML conferences with gold repositories) and prompts for evaluation. Although primarily an application, it has research artifacts (evaluation scripts, prompt templates, benchmark data) that give it a hybrid character.
The LLM identified 3 files worth highlighting for reuse:
A collection of utility functions for parsing structured LLM outputs (JSON with optional [CONTENT] wrappers), extracting code from markdown responses, computing API costs, reading repository files, and formatting planning artifacts. The content_to_json function chain (4 fallback levels) demonstrates a robust pattern for handling inconsistent model output formatting. The cost calculator and token counter are reusable across any OpenAI API project.
Implements an LLM-driven debugging loop: given a code repository and an error log, it asks the model to suggest SEARCH/REPLACE diffs, then applies them to create a fixed version (with backup). This pattern is self-contained and can be extracted as a generic debugger for any codebase. It handles multiple file modifications, backup management, and error handling. The diff format (SEARCH/REPLACE) is a non-obvious design choice that works well with LLMs.
Refines a YAML configuration file by detecting model and dataset names (via OpenAI) and replacing them with concrete Hugging Face repository IDs (queried via huggingface_hub). This solves a common reproducibility problem: ambiguous references to 'Llama-3' or 'TriviaQA' become specific downloadable identifiers. The pattern is entirely reusable for any configuration-driven ML project that depends on external model repositories.