$ show ShielderSec/webtech
WebTech is a Python-based tool for detecting technologies used on websites. It analyzes HTTP responses by examining headers, HTML content, meta tags, cookies, JavaScript, and URL patterns to identify web servers, frameworks, CMS, and other technologies. The tool provides both a command-line interface and a library API, with optional integration as a Burp Suite extension for active/passive scanning. Technology detection relies on a JSON-based signature database that is regularly updated from a Wappalyzer-compatible source. The architecture is modular, separating concerns into distinct modules: database management (downloading and merging signature files), HTML parsing (extracting meta tags and scripts), core detection logic (checking each technology signature across multiple vectors), and utility functions. The command-line tool processes single URLs or files containing multiple URLs, outputting results in human-readable, grep-friendly, or JSON formats. The Burp extension functions as a passive scanner, analyzing traffic intercepted by Burp Suite. While the core detection logic is well-structured and demonstrates reusable patterns, the codebase shows signs of technical debt. It maintains Python 2/3 compatibility (increasing burden as Python 2 is EOL), disables SSL verification by default (creating security risks), and the Burp extension depends on outdated Jython 2.7.0. The dependency on an external signature database introduces potential fragility if the upstream format changes. Despite these issues, the modular design allows for extraction of reusable components such as the Tech namedtuple, custom JSON encoder for sets, and HTML parser for meta/script extraction.
The LLM identified 3 files worth highlighting for reuse:
Contains the Tech namedtuple for representing detected technology and version, plus utility functions for platform detection (user data directory) and domain validation. The Tech namedtuple provides a lightweight, immutable data structure ideal for representing key-value pairs with optional version information. The platform detection logic abstracts OS-specific user data directory handling, demonstrating a reusable cross-platform utility pattern.
Implements SetEncoder, a custom JSON encoder that extends json.JSONEncoder to handle Python sets by converting them to lists. This isolated component solves a specific serialization problem when using sets in data structures intended for JSON output. With no dependencies beyond the standard library, it can be trivially extracted and reused in any project requiring JSON serialization of sets.
Defines WTParser, a subclass of HTMLParser, that extracts meta tags and script sources from HTML content. It provides a clean interface: feed HTML and retrieve a dictionary of meta attributes and a list of script URLs. This self-contained component (only uses html.parser and re) demonstrates a focused, reusable approach to extracting specific information from HTML without the overhead of a full parsing library, suitable for any HTML scraping or analysis task.