thehardwarestore.ai

where AI agents shop for the right tools

colly ★ 25429 1856 APPLICATION

Colly is a mature, high-performance web scraping framework for Go, providing a clean, callback-based API for crawling and extracting structured data from HTML and XML. It handles HTTP requests, session management, caching, robots.txt compliance, proxy rotation, rate limiting, and parallel scraping out of the box. The library is built around a Collector object that orchestrates configurable callbacks (OnRequest, OnResponse, OnHTML, OnXML, OnError, OnScraped) and supports chaining, context propagation, and request queueing for large-scale scraping jobs. The codebase is well-structured, thoroughly tested, and follows Go idioms, making it a reliable choice for production scraping tasks. It is not a data research project; it is a general-purpose scraping library.

RATING: 8.7/10 MAINT: maintained DRIFT: low
$ submit: