$ show Doragd/Algorithm-Practice-in-Industry
This repository is a comprehensive data pipeline and curation system for academic papers in the fields of search, recommendation, and advertising (known as 搜广推 in Chinese). It consists of three primary automated workflows: daily arXiv paper fetching with LLM-based relevance scoring (rough and fine ranking) and output to HTML/Feishu notifications; conference paper aggregation where metadata from top venues (KDD, SIGIR, RecSys, etc.) is crawled from DBLP/DBLP and stored as markdown tables; and a curated industry practice article index with its own HTML page. All are driven by GitHub Actions and deploy to GitHub Pages. The codebase is functional but exhibits typical research-project messiness: legacy scripts coexist with a newer paperBotV2 refactor, many conference paper tables have incomplete fields (empty abstracts, zero recommendation scores), and the environment relies on unpinned Python package versions and external APIs (DeepSeek, arXiv, DBLP). The arXiv daily pipeline is the most polished component, featuring retry logic, status tracking, and two-stage LLM ranking. Overall, the repo is a pragmatic automation for researchers but not production-grade software.
The LLM identified 3 files worth highlighting for reuse:
Core daily arXiv pipeline: fetches papers from arXiv API across multiple categories, applies a two-stage LLM-based relevance ranking (rough title-only, fine title+abstract), and saves filtered results as JSON. Features aggressive retry logic with exponential backoff and jitter, per-category rate limiting, and full status tracking via the ArxivDailyStatus class. Could be extracted for any academic paper curation task by swapping the API source and relevance prompts.
Centralized repository of LLM prompt templates for the arXiv ranking pipeline. Contains prompts for rough ranking (topic relevance based on title) and fine ranking (detailed relevance based on title and abstract), plus a translation prompt. Each prompt includes a system message, structured JSON output instructions, and an adjustable scoring rubric. Self-contained and immediately reusable in any LLM-based content filtering or scoring system.
Defines the ArxivDailyStatus dataclass used to track the end-to-end state of each daily pipeline run. Records workflow trigger, fetch stats per arXiv category, LLM ranking success/failure counts, average scores, output flags (JSON written, HTML generated, Feishu sent), and errors. Provides a `to_csv()` method for logging to a persistent CSV file. Implements a clean observability pattern that could be reused for any scheduled data pipeline.