Maigret is a username-based OSINT reconnaissance tool that checks across thousands of websites for accounts associated with a given username, extracts profile information via the socid_extractor library, and generates multi-format reports (HTML, PDF, CSV, JSON, Markdown, XMind, Neo4j, etc.). It operates as a CLI tool, an embeddable Python library, a Flask web application with live scan streaming, and a Docker image. The architecture is a monolith with a clean separation between site database management (MaigretDatabase/MaigretSite models), asynchronous checking engine (AsyncioQueueGeneratorExecutor, checkers for aiohttp, curl_cffi, Cloudflare bypass), extraction enrichment, and report generation. External dependencies include aiohttp, socid_extractor, curl_cffi, aiodns, and potentially FlareSolverr and OpenAI-compatible APIs (optional). The tool supports proxy/Tor/I2P routing, self-validation of site configurations, and auto-update of its site database. It is actively maintained, with frequent releases and a large community, and is used by professionals in OSINT, academic research, cryptocurrency investigations, and privacy analysis.
CodExt is a Python library and command-line tool that extends the standard library's codecs with over 120 custom encodings for data transformation. It provides encoding/decoding functionalities across domains including base encodings (baseN, base45, base85, etc.), cryptography (Vigenère, RSA, XOR), compression (gzip, LZ77/LZ78), steganography, and language-based transformations (Leetspeak, Morse, Braille). The system features a heuristic-based guess mode for automated multi-layered decoding and a macro system for chaining codecs. Architected as a plugin-based library, it allows dynamic registration of new codecs and is designed for CTF, security, and data obfuscation tasks. The codebase demonstrates strong modularity with clear separation of concerns between core functionality, codec implementations, and utility modules.
HAITI (HAsh IdenTIfIer) is a Ruby library and CLI tool for identifying hash types from input strings. It matches against a comprehensive database of over 675 hash types (including modern algorithms like SHA3, Blake2, and VeraCrypt) using regex patterns. The library provides concise output with hashcat mode numbers, John the Ripper format names, and example hashes. Multiple CLI entry points exist: the main `haiti` command, `haiti-parsable` for script-friendly output, `haiti-fzf` for interactive selection, and `hashcat-haiti`/`john-haiti` for converting hashes into hashcat/JtR formats. The codebase is well-structured, with clean separation between the identification engine, data definitions, and CLI wrappers. It includes comprehensive documentation (man pages, YARD API docs, a docsify site) and is packaged for numerous Linux distributions and Homebrew. The identification logic is straightforward but effective: iterate over a JSON array of prototypes, apply each regex to the input, and return matching types. The data file (`prototypes.json`) is manually maintained and extended with each release.
Yeti is a threat intelligence platform that ingests, enriches, correlates, and stores cyber threat observables, entities, indicators (YARA, Sigma, Suricata, regex, forensic artifacts), and DFIQ investigative questions. It provides a FastAPI-based REST API for CRUD operations, graph traversal, semantic search (via ChromaDB), and RBAC. The system uses ArangoDB as its primary graph/document database, Redis for message brokering, and Celery for background task scheduling. It includes a plugin architecture for feeds (30+ public sources like MITRE ATT&CK, CISA KEV, MISP, AbuseIPDB) and analytics (Shodan, Censys, VirusTotal, etc.). The codebase is well-structured with clean abstractions (ArangoYetiConnector, event system, polymorphic schemas, migration framework) but contains deprecated plugins and some configuration drift. It is primarily an application, not a data research project.
openSquat is a CLI tool for detecting domain squatting threats including phishing, typosquatting, IDN homograph attacks, doppelgänger domains, and bitsquatting. It operates in three modes: Community (free NRD feed), Premium Feed (paid NRD feed with local Levenshtein detection), and Premium API (server-side lookalike API queries). The tool processes keywords against domain feeds using optimized Levenshtein distance with early exit, integrates with VirusTotal, Quad9 DNS, Certificate Transparency, and phishing databases, and outputs results in TXT, JSON, or CSV formats. Architecture features modular components for feed management, DNS validation, squatting detection, API client, and output handling, with parallel processing for performance.
Sherlock is a monolithic Python CLI for username enumeration across social networks. For a given username it fans out concurrent HTTP probes to roughly 400 external sites (defined declaratively in sherlock_project/resources/data.json), then classifies each response as Claimed / Available / Unknown / Illegal / WAF using three detection modes: HTTP status code, presence/absence of a site-specific error message in the response body, or redirect behavior (redirects disabled, 2xx/3xx means claimed). Requests run in parallel via a requests-futures session capped at 20 workers; an optional per-site regex short-circuits illegal usernames without sending a request, and hardcoded fingerprint strings detect Cloudflare / PerimeterX / AWS WAF blocks. The CLI accepts one or more usernames, supports {?} wildcard expansion into `_`, `-`, `.` variants, and reports results to the terminal via a QueryNotify observer, with optional .txt, .csv, and .xlsx export. The system is fully data-driven: no per-site code exists. Each manifest entry declares the probe URL, error detection method, error signatures, optional headers/request method/JSON payload, and NSFW flag; a JSON Schema (data.schema.json) enforces conditional consistency (e.g., errorMsg required when errorType includes 'message'). By default the manifest is fetched live from https://data.sherlockproject.xyz, with an ephemeral false-positive exclusion list pulled from a dedicated GitHub branch; --local, --json, and --site flags override this behavior. Entry points are the `sherlock` console script, `python -m sherlock_project`, a Docker image entrypoint, and an Apify Actor (.actor/) that wraps the Docker image and pushes JSON results into an Apify dataset. Engineering strengths include the declarative manifest/schema design, a genuinely useful test suite (schema validation, live-target probe tests, randomized false-positive fuzzing, a Unicode regression test for issue #2730), and CI that runs a tox matrix across Python 3.10-3.14t on three OSes, validates only manifest entries modified by a PR, comments the validation summary back on the PR, and updates the exclusions list daily. Weaknesses: nearly all orchestration and output-format logic lives in a ~1,200-line sherlock.py; notify.py uses a module-global result counter; sites.py has mutable default arguments and a dead username_unclaimed parameter; the --print-found flag is always True (store_true combined with default=True); stem is an unused declared dependency; and the request layer relies on requests-futures, an effectively unmaintained library. The project is actively maintained and extremely popular (93k+ stars), and its health depends on hundreds of external sites remaining stable, hence the extensive removed-sites documentation and dedicated false-positive/false-negative issue templates. Technical drift is moderate rather than severe: the Python ^3.9 floor is now EOL, requests-futures is unmaintained, the WAF fingerprint catalog has not been extended since late 2024, and the PyPI readme still documents removed --tor flags.
Hawk-Eye is a command-line application designed to scan various data sources for Personally Identifiable Information (PII) and secrets. It supports cloud storage (S3, GCS, Firebase), databases (MySQL, PostgreSQL, MongoDB, Redis), file systems, and collaboration platforms (Slack, Google Drive). The tool uses regex pattern matching and OCR to detect sensitive data in multiple file formats including images, PDFs, Office documents, and videos. The architecture is modular with a central dispatcher in main.py that routes commands to specific implementations in the commands/ directory. Shared functionality like file reading, OCR, and pattern matching is encapsulated in internals/system.py. This design allows easy extension to new data sources by adding new command modules. The tool is intended for security auditing and can be run standalone or integrated into scripts/CI/CD pipelines. Features include alert deduplication, severity-based reporting, and Slack/Jira integrations. Despite its comprehensive feature set, the tool maintains separation of concerns, making it maintainable and extensible. The use of third-party libraries for specific data sources keeps core logic focused on scanning and reporting.
This repository provides a collection of 52 specialized AI agent definitions for penetration testing and red team operations, designed to be used with Claude Code. It is distributed as a Claude Code plugin, with an interactive installer (install.sh), a SQLite findings database (db/findings.sh) for persisting engagement data, and a versioned migration system. The agents are defined as Markdown files with YAML frontmatter, each containing structured system prompts that enforce scope validation, OPSEC tagging, evidence handling, and MITRE ATT&CK mapping. The system includes a swarm orchestrator for multi-agent coordination, a scope guard for safety, and a two-tier safety model (advisory vs. Bash-capable agents). While primarily a configuration/instruction set rather than a traditional software application, it demonstrates strong engineering practices in prompt architecture, database schema design, and distribution automation. The repository is actively maintained with CI validation, shellcheck, and unit tests for the database scripts.
iKy (I Know You) is a modular OSINT (Open Source Intelligence) aggregation platform built with FastAPI, Celery, Redis, and an Angular frontend. The system ingests a username or email and concurrently executes 26+ parallel tasks across social media, data breaches, search engines, and dark web sources, returning a structured JSON contract (module, param, validation, raw, graphic, profile, timeline, tasks) consumed by a visual graph/timeline UI. Key architectural patterns include a dynamic module registry that maps route names to Celery task paths, a decorator-based task wrapper (@iky_task) that centralizes dev-mode golden-file bypass, error classification, and performance logging, and a pluggable search provider orchestration layer. The codebase demonstrates a strong separation of concerns between API routing, task execution, and data enrichment, but suffers from fragile scraping logic (LinkedIn RSC parsing, TikTok SSR regex), heavy reliance on browser session cookies, and several deprecated external APIs (Twitter v1.1, FullContact). The project is actively maintained with an extensive test suite (~4K lines) and Docker Compose deployment, though many tests are heavily mocked and may not detect real API drift.
reNgine is a Django-based automated security reconnaissance and attack-surface management platform. It orchestrates 20+ external security tools (Subfinder, Amass, Naabu, Nuclei, FFUF, GoSpider, Dalfox, WAFW00f, EyeWitness, theHarvester, h8mail) via Celery task chains to perform subdomain discovery, port scanning, HTTP crawling, directory fuzzing, OSINT gathering, technology fingerprinting, WAF detection, and vulnerability scanning. Results are parsed from tool outputs (nmap XML, nuclei JSON, dalfox JSON, ffuf JSON), persisted in PostgreSQL, and exposed through a Django REST Framework API plus a jQuery/Bootstrap frontend with server-side DataTables, dashboards, and WeasyPrint PDF reports. The system supports multi-project tenancy, role-based access control (sys_admin, penetration_tester, auditor), scheduled scans via Celery Beat, and deep HackerOne bug-bounty integration for program import, bookmarking, and vulnerability report submission. It also integrates LLMs (OpenAI and local Ollama) for vulnerability description enrichment and attack-surface suggestions. Architecturally it is a monolithic Django application with four feature apps (dashboard, scanEngine, startScan, targetApp, recon_note) and a Docker deployment composed of web, celery, celery-beat, redis, postgres, nginx, and ollama services. The orchestration core, web/reNgine/tasks.py (~3000 lines), composes Celery groups/chords/chains, executes shell commands against external tools, parses their structured output, and persists scan results. The frontend is server-rendered Django templates with heavy inline JavaScript and a supplementary DRF API layer in web/api/views.py. Although it collects and aggregates large volumes of scan-derived data, this is fundamentally an operational security application, not a data-research project. The codebase is actively patched (recent CVE fixes for command injection and XSS) and feature-rich, but carries notable technical debt: an end-of-life Django 3.2 base, old vendored JS plugins (DataTables 1.10.16, Lightbox from 2018), hardcoded container filesystem paths, a monolithic task module mixing orchestration/parsing/persistence, and dead or commented code. The strongest reusable pieces are the RengineTask Celery base class (standardized task logging, caching, and notifications), the dual-provider LLM abstraction with OpenAI/Ollama backends, and a custom server-side datatable filtering language applying operators (=, >, <, !, &, |) across subdomain, endpoint, and vulnerability views. It is best characterized as effective, pragmatic, security-tool engineering rather than clean, academically elegant architecture.
This repository is a distribution bundle of third-party reverse engineering and malware analysis tools for Windows, packaged with an installer and updater. The tools are organized into categories such as Analysis, Decompilers, Disassemblers, Hex Editors, Network Monitors, Unpackers, and more. The project provides Inno Setup scripts for building installers, a custom updater executable, and CI/CD workflows (GitHub Actions) for automated builds and signing. The custom code (Python scripts for installer generation, batch files for tool updates) is functional but tightly coupled to this specific packaging project. The repository serves as a convenient, curated collection of tools rather than an original software application, making it valuable for analysts who need a ready-to-use toolkit.
This repository is a collection of blog posts, presentations, Jupyter notebooks, and scripts by John Lambert (@JohnLaTwC), a renowned security engineer from Microsoft. It serves as a knowledge base for security defenders, covering topics such as attack graph theory, the MS08-067 vulnerability story, and the 'Githubification of InfoSec' concept, which advocates for open, community-driven security practices using MITRE ATT&CK, Sigma rules, and Jupyter notebooks. The notebooks focus on malware analysis, shellcode reverse engineering, and threat hunting, often leveraging VirusTotal APIs and custom disassembly tools. The Yara rules provide detection for document-based attacks. The PowerShell scripts are utilities to restore Windows Defender to a secure configuration. While the content is highly educational and demonstrates deep security expertise, the repository does not constitute a structured application or a reusable data pipeline; it is a curated set of resources for learning and reference.
This repository is a comprehensive, curated collection of cybersecurity study plans covering 20+ specialized domains such as application security, cloud security (AWS/Azure/GCP), DevSecOps, threat modeling, GRC, GenAI security, and more. It is not a software application or data research project; rather, it is a knowledge base of structured markdown roadmaps designed to guide learners and professionals through role-specific cybersecurity skill development. The plans reference up-to-date frameworks (OWASP Top 10:2025, NIST CSF 2.0, EU AI Act), include curated resources (books, videos, courses, certifications), and are consistently maintained. The repository has high community engagement (5k+ stars) and remains actively updated, though it contains no executable code, data pipelines, or notebooks.
This repository is a curated directory of cybersecurity tools and resources, organized into categories such as AI security, API pentesting, C2 frameworks, forensics, OSINT, and more. It is built as a static documentation site using Retype, with markdown pages listing external links and brief descriptions. The repository contains no executable application code, data pipelines, or analytical models. Its primary function is to serve as a reference index for security professionals, maintained through periodic updates to links and categories. The codebase is essentially a collection of markdown files and configuration for the static site generator, with a large set of default password text files included as a resource. There is no software architecture, entry points, or data flows beyond the static site generation. The repo is well-organized and regularly updated, but it offers no reusable software components or novel engineering patterns.
This repository is an archived collection of interview preparation notes for penetration testing and cybersecurity positions. It contains three markdown files: HR interview tips, a comprehensive technical question bank covering topics like SQL injection, XSS, network security, cryptography, and system administration, and a log of actual interview questions from specific companies. The content is well-organized and detailed, providing both strategic advice and technical depth. The repository is not a software application but a knowledge resource. It is explicitly archived by the author, who has moved on from the security field, but remains a valuable reference for aspiring security professionals.
SecLists is a massive, curated collection of security testing wordlists and payloads, not a software application. It serves as a companion dataset for penetration testers and security researchers, aggregating lists of usernames, passwords, URLs, fuzzing payloads, web shells, and other sensitive data patterns into a single repository. The 'architecture' is purely organizational: a directory tree of text files grouped by category (e.g., Passwords, Fuzzing, Discovery, Payloads), supplemented by a set of maintenance scripts in `.bin/` for updating lists from remote sources, validating file integrity, and automating contributions via GitHub Actions. The primary 'data flow' is curation: lists are sourced from external projects (e.g., FuzzDB, RAFT, Probable-Wordlists, sqlmap), committed to the repo, and periodically refreshed by automated scripts. There are no models, analytics, or dashboards produced. The repository's value lies entirely in the breadth and quality of its aggregated lists. Maintenance is active, with commits as recent as June 2026 and automated hourly/daily update workflows for select lists (JWT secrets, API endpoints, etc.). However, from a data engineering perspective, the repo has significant limitations. There is no dependency management (no requirements.txt, Dockerfile, or environment spec), no data validation beyond basic newline/empty-line checkers, and no documentation of data lineage or update frequency for the vast majority of lists. Many READMEs are outdated or reference defunct sources. The collection is highly heterogeneous: some lists are meticulously curated and documented, while others are raw dumps with no metadata. There are no tests, no schema definitions, and no quality metrics. The `.bin/` scripts are functional but minimal—straightforward Python and shell scripts for fetching remote files and checking formatting, none of which demonstrate non-obvious engineering craftsmanship. In summary, SecLists is an invaluable reference dataset for the security community, but it is not a data pipeline or analytical project. It is a file repository with light automation, judged by data engineering standards: low reproducibility, minimal documentation, no dependency management, and no data quality framework beyond basic file hygiene.