$ show huangkiki/dailypaper-skills
This repository is a collection of Claude Code skills and Python scripts that automate daily paper recommendation and note-taking for researchers. It fetches papers from HuggingFace Daily Papers, HuggingFace Trending, and arXiv API, scores them against user-defined keywords, enriches metadata (authors, affiliations, figures, method names), and generates a tiered daily recommendation markdown file (must-read, worth reading, skip). For must-read papers, it invokes a paper-reader skill that produces detailed structured notes with figures, formulas, tables, and concept wiki entries, all saved into an Obsidian vault. It also fetches GitHub Trending repositories and writes notes. The system includes a web viewer (FastAPI + vanilla JS) for browsing the vault content, with full-text search, wikilink resolution, and a Claude chat interface. The architecture is a three-step pipeline (fetch, review, notes) orchestrated by Claude Code skills, with Python scripts handling data fetching, scoring, enrichment, and file management. The codebase demonstrates solid engineering: async HTTP with fallback chains, robust HTML parsing, heuristic text extraction, and clean modular design. It is primarily an application (a set of tools) rather than a data research project, though it involves significant data processing.
The LLM identified 6 files worth highlighting for reuse:
Batch-enriches arXiv papers with metadata (authors, affiliations, figures, method names) using asyncio + subprocess curl for concurrent HTTP requests, pure regex HTML parsing, and multiple fallback strategies (HTML -> abs page -> PDF extraction). Demonstrates robust error handling, semaphore-limited concurrency, and intelligent merging of data from multiple sources. Could be extracted and reused for any arXiv paper enrichment pipeline.
Fetches papers from HuggingFace Daily, HuggingFace Trending, and arXiv API, scores them based on user-defined keywords/negative/domain-boost lists, merges by arXiv ID, deduplicates against history (with weekend relaxation), and selects top N. Supports multi-day mode, backfill from history when pool is thin, and outputs JSON. A self-contained, well-structured data pipeline that could be reused for any paper recommendation system.
Extracts institution affiliations from PDF text (pdftotext output) using heuristic rules: institution keyword matching, numbered footnote patterns, copyright lines, and positional extraction (lines after author block before abstract). Handles noise filtering, sentence detection, and deduplication. A standalone script that reads stdin and outputs JSON, demonstrating clever text processing techniques applicable to any PDF metadata extraction task.
Generates Map of Content (MOC) index files recursively for a directory tree. Creates or updates a markdown file per directory with links to subdirectories (with note counts) and notes. Idempotent: only rewrites if content changed. Uses dataclass for summary. A clean, self-contained module that could be reused for any file tree indexing or navigation page generation.
FastAPI backend that serves an Obsidian vault viewer. Provides endpoints for daily papers, GitHub trending, paper notes, concept wiki, full-text search, and wikilink index. Includes a Claude chat endpoint that streams SSE responses from the local `claude` CLI. Demonstrates clean API design, frontmatter parsing, tier table parsing, and config loading with fallback. Could be reused as a template for any markdown vault web viewer.
Frontend JavaScript for the web viewer. Implements a hash-based router, custom marked extensions for LaTeX blocks/inline and wikilinks (with resolution against a server-built index), callout processing, and a Claude chat client that consumes SSE streams. Demonstrates non-trivial frontend engineering without a framework, including responsive design and dark theme. The wikilink resolution and tier card rendering are particularly well-done.