Scrapy is a mature, production-grade web scraping and crawling framework for Python. It provides a complete pipeline for downloading web pages, extracting structured data (via XPath/CSS selectors), and exporting it in multiple formats (JSON, CSV, XML, etc.). The architecture is event-driven, built on Twisted with full asyncio integration, and consists of an execution engine, downloader, scheduler, spider middleware, downloader middleware, item pipelines, and extensions. Key entry points include the CLI (`scrapy crawl`, `scrapy shell`, `scrapy startproject`), the `CrawlerProcess`/`CrawlerRunner` classes for programmatic use, and the `Spider` base class for user-defined spiders. Data flows from HTTP requests through the downloader (with support for HTTP/1.1, HTTP/2, FTP, and experimental httpx), through spider callbacks, into item pipelines, and finally to feed exporters or custom storage backends. External dependencies include Twisted, lxml, parsel, w3lib, protego (robots.txt), queuelib, and optional integrations like boto3 (S3), google-cloud-storage, Pillow (image processing), and mitmproxy (testing). The framework is highly configurable via a priority-based settings system and supports plugins (addons, middlewares, pipelines, extensions).

Queuelib is a Python library providing persistent and in-memory queue data structures. It implements FIFO, LIFO, priority, and round-robin queues with disk-based storage options for durability. The library is dependency-free, using only the Python standard library, and is designed for scenarios requiring reliable queuing such as web crawlers. Architecture consists of modular components: base queue classes in queue.py, priority queue in pqueue.py, and round-robin queue in rrqueue.py. Disk-based queues manage data through chunk files (FIFO) or single files (LIFO) with careful bookkeeping to survive process restarts. The priority and round-robin queues use a factory pattern to decouple queue logic from storage implementation, enabling flexibility.