$ show bisohns/search-engine-parser
search-engine-parser is a Python library and CLI tool designed to scrape search engines for structured results. It provides a base class for search engine implementations with specific parsers for engines like Google, Bing, Yahoo, DuckDuckGo, and others. The tool supports both synchronous and asynchronous HTTP requests, includes caching to avoid repeated requests, and uses user-agent rotation to prevent blocking. Key features include a command-line interface (pysearch) for terminal use and a programmatic API for integration into other projects, returning results as SearchItem objects with convenient access to titles, links, and descriptions.
The LLM identified 2 files worth highlighting for reuse:
Defines the BaseSearch class requiring implementations of get_params, parse_soup, and parse_single_result methods for new search engines. Includes SearchResult and SearchItem classes for flexible result access by index or attribute (titles/links/descriptions). This isolated template pattern enables easy extension to new search engines with minimal coupling.
Contains CacheHandler for storing HTTP responses using SHA256-hashed URLs as keys in engine-specific directories, preventing redundant requests. Also provides get_rand_user_agent for rotating user agents (with fake-useragent fallback) to avoid blocking. These self-contained utilities demonstrate robust HTTP optimization patterns reusable in any web scraping project.