$ show jimmc414/onefilellm
OneFileLLM is a versatile CLI tool and web application designed to process diverse inputs (local files, GitHub repositories, YouTube transcripts, arXiv papers, web pages, etc.) into a single structured XML-like output optimized for LLM consumption. The system features format-specific processors that extract and convert content (e.g., Excel to markdown, PDF text extraction, Jupyter notebook to code), token counting via tiktoken with optional NLTK-based compression, and metadata reporting. It includes a Rich-powered terminal interface with colored output and progress bars, a Flask-based web app with Server-Sent Events for real-time progress updates, and an alias system for JSON-stored shortcuts with placeholder substitution. Security is prioritized through SSRF protection, local path validation, and offline mode support. The architecture separates core processing logic (onefilellm.py), CLI handling (cli.py), utilities (utils.py), and web components (extras/web_app.py), enabling modular reuse.
The LLM identified 2 files worth highlighting for reuse:
Provides isolated utility functions for safe file reading (encoding fallback), clipboard/stdin handling, network utilities with streaming downloads and GitHub auth forwarding, SSRF-protected URL validation (_is_safe_url), file type filtering, and format detection/parsing dispatch. These functions demonstrate non-obvious skills in security hardening and robustness, have minimal internal dependencies, and can be extracted for reuse in other projects requiring reliable input handling.
Contains the AliasManager class implementing a JSON-based alias system with user-defined aliases overriding core defaults, placeholder substitution ({}) for dynamic command templates, and validation (alphanumeric plus -_/ characters). This self-contained component depends only on standard libraries, isolates alias logic from core processing, and offers a reusable pattern for CLI tools needing configurable shortcuts with safe variable interpolation.