Implements a composable middleware chain using async generators for initialization and cleanup. Each middleware step yields control to the next, with proper resource management via try/finally. This pattern is reusable for any request-processing pipeline and demonstrates clean separation of concerns.

A generic class that persists any Pydantic model to a key-value store with event-driven saving, explicit-only mode, and deep-copy defaults. Used by SessionPool, RequestList, and RenderingTypePredictor. Can be extracted for any project needing crash-recoverable state without coupling to the rest of Crawlee.

Wraps any RequestManager with per-domain delay tracking, HTTP 429 backoff, crawl-delay respect, and domain stall detection. Routes requests to sub-managers based on domain. Demonstrates robust rate-limit handling that can be reused in any web crawling or API client project.

Streaming SAX-based sitemap parser that handles XML, TXT, gzipped, and nested sitemaps with size caps, robots.txt discovery, and cross-origin filtering. Handles real-world quirks like CDATA, malformed XML, and gzip bombs. The streaming approach avoids loading entire files into memory, making it suitable for large sitemaps.

Dynamically adjusts concurrency based on CPU, memory, event loop, and client load snapshots. Uses a weighted overload ratio to decide whether to scale up or down. Reusable for any async system needing adaptive concurrency without hardcoding limits.