thehardwarestore.ai

This is a full-stack web application for performing open-source intelligence (OSINT) analysis on any website. The system runs 40+ checks encompassing DNS, SSL/TLS, HTTP security headers, mail configuration (SPF/DKIM/DMARC), social media presence, trackers, blocklists, threat feeds, carbon footprint, tech stack, and more. The frontend is built with Astro and React, featuring a reactive results view with a masonry grid, advisory panel, and progress bar. The backend is a Node.js Express server (also serverless-friendly) with individual API handlers per check, wrapped in a common middleware layer for error handling, rate limiting, and skip logic. The application is designed for self-hosting and supports deployment to Vercel, Netlify, Docker, Render, and Fly.io. It relies on many external sources (DNS, WHOIS, SSL Labs, HIBP, Google Safe Browsing, Shodan, etc.) and uses Puppeteer/Chromium for headless browser tasks. The codebase demonstrates a clean modular architecture, reusable UI components, and a novel analysis rule system that transforms raw data into structured findings with severity levels.

Maigret is a username-based OSINT reconnaissance tool that checks across thousands of websites for accounts associated with a given username, extracts profile information via the socid_extractor library, and generates multi-format reports (HTML, PDF, CSV, JSON, Markdown, XMind, Neo4j, etc.). It operates as a CLI tool, an embeddable Python library, a Flask web application with live scan streaming, and a Docker image. The architecture is a monolith with a clean separation between site database management (MaigretDatabase/MaigretSite models), asynchronous checking engine (AsyncioQueueGeneratorExecutor, checkers for aiohttp, curl_cffi, Cloudflare bypass), extraction enrichment, and report generation. External dependencies include aiohttp, socid_extractor, curl_cffi, aiodns, and potentially FlareSolverr and OpenAI-compatible APIs (optional). The tool supports proxy/Tor/I2P routing, self-validation of site configurations, and auto-update of its site database. It is actively maintained, with frequent releases and a large community, and is used by professionals in OSINT, academic research, cryptocurrency investigations, and privacy analysis.

IntelOwl is a production-grade, open-source threat intelligence platform that automates the analysis of observables (IPs, domains, URLs, hashes) and files (PE, PDF, APK, etc.) through a pluggable plugin system. The backend is built with Django REST Framework, Celery for asynchronous task execution, and Django Channels for real-time WebSocket updates. The frontend is a React single-page application using react-router, react-flow for pipeline visualizations, and Zustand for state management. The system supports multi-tenancy, organizations, role-based access, and integrates with dozens of external threat intelligence APIs and open-source analysis tools via containerized microservices. The architecture is centered around a plugin framework with five plugin types: analyzers (file and observable), connectors (push results to external systems), visualizers (render reports), pivots (create child jobs from analysis results), and ingestors (pull threat feeds). Playbooks bundle these plugins into repeatable analysis workflows. A data model manager normalizes analyzer outputs into structured evaluations with reliability scores, and a user event system allows manual verdict overrides with decay logic. The platform also includes an AI chatbot assistant powered by a local Ollama LLM and LangGraph, enabling natural-language queries and one-click analysis launches. Key operational features include a comprehensive REST API, token-based authentication with optional OAuth/LDAP/RADIUS, Elasticsearch for BI reporting, and Docker-based deployment with separate containers for specialized analyzers. The codebase is actively maintained with extensive unit and integration tests, CI/CD pipelines, and a large community (4695 stars). While the migration system is somewhat repetitive, the overall architecture is clean, modular, and demonstrates strong software engineering practices.

deanonymizer is a command-line tool designed for privacy self-audits. It collects public data from platforms like Reddit, Hacker News, GitHub, and Stack Overflow for specified usernames, then analyzes this data to identify personally identifiable information (PII) and other linkable information. The analysis combines a configurable LLM backend (Anthropic, OpenAI, or Claude Code) for inferring soft signals (location, employer, etc.) with a deterministic regex pass for hard identifiers (emails, social handles). The tool outputs a detailed report highlighting findings with confidence levels and remediation suggestions, enabling users to reduce their online footprint. The architecture is modular, separating concerns into CLI handling, platform-specific data fetching, analysis orchestration, LLM abstraction, and report generation.

WorldMonitor is a full-stack, open-source global intelligence platform providing real-time monitoring of geopolitical, economic, military, supply chain, and natural events. The codebase is overwhelmingly an application (single-page web dashboard, Vercel Edge API, Convex backend, Tauri desktop app, CLI, and SDKs) with a secondary data pipeline character (consumer price scraping, external API seeding). It powers a commercial SaaS with free and Pro tiers, featuring a map-based dashboard with 100+ configurable panels, an MCP (Model Context Protocol) server for AI agents, and extensive multi-language support (28 locales). The architecture follows a seed-cache pattern where Railway cron jobs fetch external data into Redis, and serverless Edge functions read cached JSON for low-latency responses. The frontend uses vanilla TypeScript with Vite, deck.gl/MapLibre for mapping, and a panel system for modular UI composition. The system is actively maintained, with daily commits, comprehensive test coverage (unit, integration, E2E with Playwright), and a well-documented API surface with 298 protobuf definitions across 36 domain services. While the codebase is large and occasionally duplicates shared constants across modules, it demonstrates strong software engineering practices including circuit breakers, idempotency keys, rate limiting, and a consistent handler pattern.

OpenCTI is an open-source cyber threat intelligence platform that ingests, manages, and analyzes threat data using the STIX 2.0/2.1 standard. The system is a monorepo containing a Node.js/TypeScript GraphQL backend (opencti-graphql), a React/Relay frontend (opencti-front), a Python client library (client-python), and a Python worker service (opencti-worker). The backend uses Elasticsearch/OpenSearch as its primary data store, Redis for caching and pub/sub, RabbitMQ for connector queues, and PostgreSQL for migrations. The frontend provides full CRUD UIs, dashboards, workspaces, playbook automation, and workflow management. Key entry points include the GraphQL API (resolvers per entity type), the frontend React app with Relay data fetching, and the Python client for programmatic access. Data flows involve ingestion via CSV/JSON/TAXII/RSS feeds, manual creation, connector enrichment, and inference rules that automatically derive relationships. External dependencies include Elasticsearch, Redis, RabbitMQ, MinIO/S3 for file storage, and optional AI services. The platform supports enterprise features (playbooks, PIR, XTM Hub) gated by license checks.

`gau` (getallurls) is a high-performance Go CLI tool that aggregates known URLs for a given domain from multiple public web archives: Wayback Machine, CommonCrawl, AlienVault OTX, and URLScan. It is designed primarily for security researchers and penetration testers to enumerate URL endpoints, supporting concurrent fetches, filtering by status code/mime-type/date, blacklisting extensions, and deduplication of parameters. The architecture follows a clean provider interface pattern, with each data source implemented as a separate package satisfying a common `Provider` interface. The runner orchestrates a worker pool for concurrent processing, and output is handled via configurable JSON or plain text writers. The codebase is well-structured, testable in principle (though no tests are present in the visible tree), and the dependency graph is minimal and modern, relying on `fasthttp` for efficient HTTP requests and `viper` for configuration management.

Sightline is a Next.js web application for discovering and analyzing real-world infrastructure using OpenStreetMap data. It provides a search interface that accepts natural language queries (e.g., 'telecom towers in karnataka') or structured syntax (e.g., 'type:telecom operator:airtel region:karnataka'). The application uses Nominatim for geocoding to convert location queries to bounding boxes, then queries the Overpass API for matching OSM assets. Results are displayed on an interactive Leaflet map with marker clustering and in a filterable list. The architecture features a clean separation between frontend React components, backend API routes for search and geocoding proxying, and utility libraries for query parsing, caching, and API interactions. The application implements proper rate limiting, timeout handling, fallback API endpoints, and efficient caching to respect external service limits while providing a responsive user experience.

Linkook is a Python CLI tool for OSINT-driven discovery of linked social media accounts and associated emails across multiple platforms. It takes a single username as input, scans a configurable set of websites (defined in a provider.json file), and recursively extracts linked accounts, usernames, and emails from discovered profiles. The tool supports breach checking via HudsonRock's database or Have I Been Pwned, and can export results to a Neo4j-compatible JSON format for graph visualization. The architecture is modular: a Provider class handles URL templates and regex extraction, a SiteScanner performs HTTP requests and keyword-based existence checks, and a ScannerManager manages concurrent scanning via a thread pool with a queue. The code is well-structured, includes unit tests, and demonstrates solid engineering practices with clean separation of concerns, error handling, and logging.

Favicorn is a command-line OSINT tool that searches for websites by their favicon hashes. It accepts input via favicon URL, file, or domain, computes multiple cryptographic and perceptual hashes (MMH3, MD5, SHA256, etc.), and generates search URLs for 11 security/search platforms (Shodan, ZoomEye, Censys, etc.). Optionally, it fetches and displays preview results from Shodan, ZoomEye, and Netlas using API keys. The architecture features a modular design: the Favicon class handles hash computation and platform-specific URL generation, while specialized Fetcher classes (Shodan, ZoomEye, Netlas) encapsulate API interactions, caching, and response parsing. The CLI orchestrates workflow, parallel execution, and output formatting. Key dependencies include requests, mmh3, dnspython, and platform-specific SDKs.

This is a comprehensive command-line tool and self-hosted web server for network reconnaissance, written entirely in Bash. It performs ASN lookups, BGP statistics, RPKI validation, IP geolocation, IP reputation checks (via StopForumSpam, IPQualityScore, GreyNoise), Shodan InternetDB scanning for open ports and vulnerabilities, AS path tracing with detailed hop information, and more. The architecture is a monolithic Bash script (~7000 lines) that integrates numerous public APIs (RIPEStat, Team Cymru, pWhois, PeeringDB, CAIDA, ipinfo.io, ip-api, Cloudflare Radar, etc.) and command-line tools (mtr, whois, jq, curl, ncat). It can run in interactive mode, JSON output mode (as an API), or as an HTTP server (using ncat) that generates HTML reports with clickable links and a bookmarklet for browser integration. The code is well-structured for a shell script, with extensive error handling, caching of API responses, and support for API tokens to enhance functionality. While it is not modular in a traditional sense (everything is in one file), it demonstrates high craftsmanship in Bash and is actively maintained.

openSquat is a CLI tool for detecting domain squatting threats including phishing, typosquatting, IDN homograph attacks, doppelgänger domains, and bitsquatting. It operates in three modes: Community (free NRD feed), Premium Feed (paid NRD feed with local Levenshtein detection), and Premium API (server-side lookalike API queries). The tool processes keywords against domain feeds using optimized Levenshtein distance with early exit, integrates with VirusTotal, Quad9 DNS, Certificate Transparency, and phishing databases, and outputs results in TXT, JSON, or CSV formats. Architecture features modular components for feed management, DNS validation, squatting detection, API client, and output handling, with parallel processing for performance.

Sherlock is a monolithic Python CLI for username enumeration across social networks. For a given username it fans out concurrent HTTP probes to roughly 400 external sites (defined declaratively in sherlock_project/resources/data.json), then classifies each response as Claimed / Available / Unknown / Illegal / WAF using three detection modes: HTTP status code, presence/absence of a site-specific error message in the response body, or redirect behavior (redirects disabled, 2xx/3xx means claimed). Requests run in parallel via a requests-futures session capped at 20 workers; an optional per-site regex short-circuits illegal usernames without sending a request, and hardcoded fingerprint strings detect Cloudflare / PerimeterX / AWS WAF blocks. The CLI accepts one or more usernames, supports {?} wildcard expansion into `_`, `-`, `.` variants, and reports results to the terminal via a QueryNotify observer, with optional .txt, .csv, and .xlsx export. The system is fully data-driven: no per-site code exists. Each manifest entry declares the probe URL, error detection method, error signatures, optional headers/request method/JSON payload, and NSFW flag; a JSON Schema (data.schema.json) enforces conditional consistency (e.g., errorMsg required when errorType includes 'message'). By default the manifest is fetched live from https://data.sherlockproject.xyz, with an ephemeral false-positive exclusion list pulled from a dedicated GitHub branch; --local, --json, and --site flags override this behavior. Entry points are the `sherlock` console script, `python -m sherlock_project`, a Docker image entrypoint, and an Apify Actor (.actor/) that wraps the Docker image and pushes JSON results into an Apify dataset. Engineering strengths include the declarative manifest/schema design, a genuinely useful test suite (schema validation, live-target probe tests, randomized false-positive fuzzing, a Unicode regression test for issue #2730), and CI that runs a tox matrix across Python 3.10-3.14t on three OSes, validates only manifest entries modified by a PR, comments the validation summary back on the PR, and updates the exclusions list daily. Weaknesses: nearly all orchestration and output-format logic lives in a ~1,200-line sherlock.py; notify.py uses a module-global result counter; sites.py has mutable default arguments and a dead username_unclaimed parameter; the --print-found flag is always True (store_true combined with default=True); stem is an unused declared dependency; and the request layer relies on requests-futures, an effectively unmaintained library. The project is actively maintained and extremely popular (93k+ stars), and its health depends on hundreds of external sites remaining stable, hence the extensive removed-sites documentation and dedicated false-positive/false-negative issue templates. Technical drift is moderate rather than severe: the Python ^3.9 floor is now EOL, requests-futures is unmaintained, the WAF fingerprint catalog has not been extended since late 2024, and the PyPI readme still documents removed --tor flags.

Shadowbroker is a full-stack intelligence platform that aggregates real-time OSINT data from 60+ public sources into a geospatial dashboard with secure mesh networking capabilities. The architecture consists of a Next.js/React frontend with MapLibre GL visualization, a Python/FastAPI backend handling data fetching and mesh protocols, and a Tauri-based desktop application for local operation. Key features include aircraft/ship/satellite tracking, encrypted Wormhole messaging, InfoNet reputation/governance system, Agentic AI channel for automated analysis, OSINT reconnaissance tools, and time machine snapshot playback. The platform emphasizes security through SSRF protection, HMAC authentication, and privacy-preserving mesh communications while providing extensibility via toggleable data layers and plugin-like agent skills.

octosuite is a Python command-line and interactive terminal (TUI) application for querying and analyzing public GitHub data. It provides a library interface for programmatic access and a CLI/TUI for interactive use. It wraps the GitHub REST API v3 with an in-memory cache and data sanitization. It supports users, organizations, repositories, and search. The codebase is well-structured with clear separation between API layer, models, CLI argument parsing, and TUI using questionary and rich. It is published on PyPI and uses modern Python packaging (uv). However, it lacks automated tests and does not handle GitHub API authentication or rate limiting, which limits its production readiness. The code is clean and follows good practices, making it a solid foundation for extension.

OSINT Brazuca Regex is a CLI tool and curated dataset of regular expressions for detecting Brazilian and global PII/OSINT-relevant patterns. The scanner (`osint_regex_scan.py`) applies the dataset to files, directories, or stdin, supporting multi-threading, text extraction from documents (PDF, DOCX, XLSX, RTF, EPUB) via stdlib or optional `pypdf`/`pdftotext`, and noise filtering (blob masking, line length limits). Output formats include terminal, CSV, JSON, or raw values. The dataset is split into two JSON files (Brazil-specific and general), merged at load time, and includes patterns for CPF, CNPJ, credit cards, crypto wallets, API credentials, and more. The repository also includes a build script (`build.py`) for managing the split/merged dataset. Core functionality requires only Python 3.12+ stdlib, with optional dependencies for extended PDF support and encoding detection.

OneForAll is a comprehensive subdomain enumeration and reconnaissance tool designed for security researchers and penetration testers. It aggregates subdomains from over 60 passive sources (certificate transparency logs, DNS datasets, search engines, threat intelligence feeds, and web crawlers), performs active brute-forcing via massdns, and enriches results with DNS resolution, HTTP probing, CDN detection, wildcard filtering, and subdomain takeover risk assessment. The tool is built as a modular Python CLI application, with each data source implemented as an independent plugin inheriting from base classes in `common/`. Output is stored in a SQLite database and can be exported to CSV, JSON, XLSX, and other formats. While primarily an application, the repository also includes curated data files (subdomain wordlists, CDN CIDR ranges, cloud service fingerprints) that are valuable for security research.

iKy (I Know You) is a modular OSINT (Open Source Intelligence) aggregation platform built with FastAPI, Celery, Redis, and an Angular frontend. The system ingests a username or email and concurrently executes 26+ parallel tasks across social media, data breaches, search engines, and dark web sources, returning a structured JSON contract (module, param, validation, raw, graphic, profile, timeline, tasks) consumed by a visual graph/timeline UI. Key architectural patterns include a dynamic module registry that maps route names to Celery task paths, a decorator-based task wrapper (@iky_task) that centralizes dev-mode golden-file bypass, error classification, and performance logging, and a pluggable search provider orchestration layer. The codebase demonstrates a strong separation of concerns between API routing, task execution, and data enrichment, but suffers from fragile scraping logic (LinkedIn RSC parsing, TikTok SSR regex), heavy reliance on browser session cookies, and several deprecated external APIs (Twitter v1.1, FullContact). The project is actively maintained with an extensive test suite (~4K lines) and Docker Compose deployment, though many tests are heavily mocked and may not detect real API drift.

reNgine is a Django-based automated security reconnaissance and attack-surface management platform. It orchestrates 20+ external security tools (Subfinder, Amass, Naabu, Nuclei, FFUF, GoSpider, Dalfox, WAFW00f, EyeWitness, theHarvester, h8mail) via Celery task chains to perform subdomain discovery, port scanning, HTTP crawling, directory fuzzing, OSINT gathering, technology fingerprinting, WAF detection, and vulnerability scanning. Results are parsed from tool outputs (nmap XML, nuclei JSON, dalfox JSON, ffuf JSON), persisted in PostgreSQL, and exposed through a Django REST Framework API plus a jQuery/Bootstrap frontend with server-side DataTables, dashboards, and WeasyPrint PDF reports. The system supports multi-project tenancy, role-based access control (sys_admin, penetration_tester, auditor), scheduled scans via Celery Beat, and deep HackerOne bug-bounty integration for program import, bookmarking, and vulnerability report submission. It also integrates LLMs (OpenAI and local Ollama) for vulnerability description enrichment and attack-surface suggestions. Architecturally it is a monolithic Django application with four feature apps (dashboard, scanEngine, startScan, targetApp, recon_note) and a Docker deployment composed of web, celery, celery-beat, redis, postgres, nginx, and ollama services. The orchestration core, web/reNgine/tasks.py (~3000 lines), composes Celery groups/chords/chains, executes shell commands against external tools, parses their structured output, and persists scan results. The frontend is server-rendered Django templates with heavy inline JavaScript and a supplementary DRF API layer in web/api/views.py. Although it collects and aggregates large volumes of scan-derived data, this is fundamentally an operational security application, not a data-research project. The codebase is actively patched (recent CVE fixes for command injection and XSS) and feature-rich, but carries notable technical debt: an end-of-life Django 3.2 base, old vendored JS plugins (DataTables 1.10.16, Lightbox from 2018), hardcoded container filesystem paths, a monolithic task module mixing orchestration/parsing/persistence, and dead or commented code. The strongest reusable pieces are the RengineTask Celery base class (standardized task logging, caching, and notifications), the dual-provider LLM abstraction with OpenAI/Ollama backends, and a custom server-side datatable filtering language applying operators (=, >, <, !, &, |) across subdomain, endpoint, and vulnerability views. It is best characterized as effective, pragmatic, security-tool engineering rather than clean, academically elegant architecture.