$ show sherlock-project/sherlock
Sherlock is a monolithic Python CLI for username enumeration across social networks. For a given username it fans out concurrent HTTP probes to roughly 400 external sites (defined declaratively in sherlock_project/resources/data.json), then classifies each response as Claimed / Available / Unknown / Illegal / WAF using three detection modes: HTTP status code, presence/absence of a site-specific error message in the response body, or redirect behavior (redirects disabled, 2xx/3xx means claimed). Requests run in parallel via a requests-futures session capped at 20 workers; an optional per-site regex short-circuits illegal usernames without sending a request, and hardcoded fingerprint strings detect Cloudflare / PerimeterX / AWS WAF blocks. The CLI accepts one or more usernames, supports {?} wildcard expansion into `_`, `-`, `.` variants, and reports results to the terminal via a QueryNotify observer, with optional .txt, .csv, and .xlsx export. The system is fully data-driven: no per-site code exists. Each manifest entry declares the probe URL, error detection method, error signatures, optional headers/request method/JSON payload, and NSFW flag; a JSON Schema (data.schema.json) enforces conditional consistency (e.g., errorMsg required when errorType includes 'message'). By default the manifest is fetched live from https://data.sherlockproject.xyz, with an ephemeral false-positive exclusion list pulled from a dedicated GitHub branch; --local, --json, and --site flags override this behavior. Entry points are the `sherlock` console script, `python -m sherlock_project`, a Docker image entrypoint, and an Apify Actor (.actor/) that wraps the Docker image and pushes JSON results into an Apify dataset. Engineering strengths include the declarative manifest/schema design, a genuinely useful test suite (schema validation, live-target probe tests, randomized false-positive fuzzing, a Unicode regression test for issue #2730), and CI that runs a tox matrix across Python 3.10-3.14t on three OSes, validates only manifest entries modified by a PR, comments the validation summary back on the PR, and updates the exclusions list daily. Weaknesses: nearly all orchestration and output-format logic lives in a ~1,200-line sherlock.py; notify.py uses a module-global result counter; sites.py has mutable default arguments and a dead username_unclaimed parameter; the --print-found flag is always True (store_true combined with default=True); stem is an unused declared dependency; and the request layer relies on requests-futures, an effectively unmaintained library. The project is actively maintained and extremely popular (93k+ stars), and its health depends on hundreds of external sites remaining stable, hence the extensive removed-sites documentation and dedicated false-positive/false-negative issue templates. Technical drift is moderate rather than severe: the Python ^3.9 floor is now EOL, requests-futures is unmaintained, the WAF fingerprint catalog has not been extended since late 2024, and the PyPI readme still documents removed --tor flags.
The LLM identified 3 files worth highlighting for reuse:
Self-contained CLI that parses JUnit XML test output with defusedxml (XXE-safe) and renders a per-target pass/fail Markdown table for GitHub PR comments. It distinguishes simple failures from hard errors, reports aggregate counts, and depends only on stdlib plus defusedxml, so it can be dropped into any CI pipeline that emits JUnit XML.
Contains set_pattern_upper_bound, a stdlib-only helper that rewrites unbounded regex quantifiers (+ , * , {n,}) into bounded forms before feeding them to the rstr fuzz generator - a clever way to keep generated usernames short enough for fast probing. The surrounding module demonstrates a reusable strategy (randomized false-positive probes and known-claimed-username false-negative probes) for validating any account-existence checker driven by a declarative site manifest.
SherlockFuturesSession subclasses requests-futures' FuturesSession to inject a monotonic response-time measurement hook that is always placed first in the hook chain, correctly handling list, tuple, single-callable, and missing-hook cases. Extracting this class yields a ready-made pattern for per-request latency tracking in any concurrent HTTP workload.