$ show soxoj/maigret
Maigret is a username-based OSINT reconnaissance tool that checks across thousands of websites for accounts associated with a given username, extracts profile information via the socid_extractor library, and generates multi-format reports (HTML, PDF, CSV, JSON, Markdown, XMind, Neo4j, etc.). It operates as a CLI tool, an embeddable Python library, a Flask web application with live scan streaming, and a Docker image. The architecture is a monolith with a clean separation between site database management (MaigretDatabase/MaigretSite models), asynchronous checking engine (AsyncioQueueGeneratorExecutor, checkers for aiohttp, curl_cffi, Cloudflare bypass), extraction enrichment, and report generation. External dependencies include aiohttp, socid_extractor, curl_cffi, aiodns, and potentially FlareSolverr and OpenAI-compatible APIs (optional). The tool supports proxy/Tor/I2P routing, self-validation of site configurations, and auto-update of its site database. It is actively maintained, with frequent releases and a large community, and is used by professionals in OSINT, academic research, cryptocurrency investigations, and privacy analysis.
The LLM identified 4 files worth highlighting for reuse:
The AsyncioQueueGeneratorExecutor class provides a generic asynchronous parallel task executor with configurable concurrency, per-task timeout, and graceful cancellation on exceptions or timeout. It is fully self-contained (no internal imports beyond standard library) and can be extracted into any async Python project that needs to run many I/O-bound tasks concurrently with fine-grained error handling. This is a high-quality reusable pattern for task scheduling.
The MaigretDatabase and MaigretSite classes model a large, versioned, JSON-serializable site database with engines, tags, check types, activation methods, and mirror relationships. They include methods for filtering by rank, category, status, and name, and handle field overrides from base engines. This is a well-crafted data access layer that could be reused for any project requiring a flexible, hierarchical configuration with JSON persistence and querying capabilities.
The module provides a CheckError class with typed error categories (DNS, timeout, captcha, etc.) and a mapping of common error strings to these types, along with human-readable solution messages. It also tracks error frequency per site to determine if a site should be disabled. This pattern is highly reusable for any web scraping or checking tool that needs to classify transient vs. permanent failures and provide actionable advice to users.
The functions parse_usernames, run_url_mutations, and _canonical_url / _domain_tail implement a clean extraction pipeline for parsing additional usernames and IDs from profile pages, filtering out non-plausible identifiers (URLs, emails), and deduplicating URLs to avoid redundant fetches. The URL deduplication logic is a robust helper that could be extracted into any project needing normalized URL comparison and filtering.