$ show scrapy/queuelib
Queuelib is a Python library providing persistent and in-memory queue data structures. It implements FIFO, LIFO, priority, and round-robin queues with disk-based storage options for durability. The library is dependency-free, using only the Python standard library, and is designed for scenarios requiring reliable queuing such as web crawlers. Architecture consists of modular components: base queue classes in queue.py, priority queue in pqueue.py, and round-robin queue in rrqueue.py. Disk-based queues manage data through chunk files (FIFO) or single files (LIFO) with careful bookkeeping to survive process restarts. The priority and round-robin queues use a factory pattern to decouple queue logic from storage implementation, enabling flexibility.
The LLM identified 3 files worth highlighting for reuse:
Contains FifoDiskQueue and LifoDiskQueue classes implementing durable FIFO/LIFO queues. FifoDiskQueue uses chunk-based file storage with automatic cleanup and efficient space management. LifoDiskQueue uses a single file with size tracking. Both are self-contained, relying only on standard library, and demonstrate advanced techniques for disk persistence and resource management.
Implements PriorityQueue using a factory pattern to create underlying queues per priority level. Decouples priority logic from storage mechanism, allowing any queue-like object (FIFO/LIFO, in-memory/disk) to be used. Demonstrates clean separation of concerns and high reusability through dependency injection via the queue factory.
Implements RoundRobinQueue with factory-based underlying queues per key and a key queue for round-robin distribution. Similar to PriorityQueue, it decouples queue logic from storage, enabling flexible backend usage. Shows effective use of the strategy pattern for cyclic processing scenarios.