$ show Rimagination/scansci-pdf
This is a comprehensive academic paper downloader application that retrieves PDFs from over 20 sources, including open access repositories (Unpaywall, OpenAlex, arXiv, EuropePMC), publisher APIs (Elsevier, Springer Nature TDM, Wiley), and institutional access channels (WebVPN, CARSI, EZProxy). It is distributed as a Python CLI tool (Typer), an MCP server (FastMCP) for AI agent integration, and a web UI, with support for batch processing, browser automation (Playwright/CloakBrowser), Tor, and FlareSolverr for anti-bot bypass. The architecture is centered on a multi-lane scheduling pipeline (publisher-specific, OA, grey sources, institutional) with tiered parallel racing, adaptive scoring, and negative caching. It includes publisher-specific PDF routing strategies for 19+ publishers, a registry pattern for extensibility, and a CookieStore for persistent session management. The application also provides citation export (BibTeX/RIS/Endnote), Zotero integration, and markdown conversion. While the primary character is a software application, it exhibits hybrid data-research traits through its extensive data source integration, metadata enrichment (Crossref, Semantic Scholar), and caching layer. The codebase is well-structured with clear modules but suffers from some code duplication (notably an institutional subpackage that mirrors core modules) and hardcoded paths, reducing maintainability. Overall, it is a sophisticated, production-ready tool that demonstrates advanced patterns in browser automation and anti-crawl techniques.
The LLM identified 3 files worth highlighting for reuse:
Defines an abstract BasePublisherStrategy class with methods for article URL, PDF URL, SSO configuration, and browser automation selectors. Uses a registry pattern (StrategyRegistry) with decorator registration, enabling dynamic dispatch by publisher. This is a clean, isolated pattern that can be reused for any multi-provider download system beyond academic publishing.
Implements a CookieStore class that serializes/deserializes browser cookies to JSON with expiry filtering and import/export for requests sessions. Supports multiple cookie formats and integrates with browser contexts for persistent login state. This is a self-contained utility usable in any Python project requiring persistent cookie storage across sessions.
Provides a thread-safe per-domain rate limiter with configurable request-per-second limits and host-level concurrency semaphores. Uses minimal locking and is used by all HTTP fetches. This class is a general-purpose rate limiting component that can be extracted and applied to any project needing domain-aware throttling.