$ show Charcoal-SE/SmokeDetector
SmokeDetector is a headless chatbot that monitors Stack Exchange for spam and abusive content in real time. It connects to the Stack Exchange WebSocket to receive new and edited posts, then runs them through a multi-layered detection engine that checks against extensive blacklists (keywords, websites, usernames, phone numbers, IP ranges, nameservers, ASNs) and applies heuristic rules including DNS lookups, homoglyph detection, and similarity analysis. Detected spam is reported to designated chat rooms on Stack Exchange Chat, where moderators can provide feedback that feeds back into the system via the Metasmoke statistics service. The architecture is event-driven: the main loop in `ws.py` listens for WebSocket events, enqueues posts for scanning via `bodyfetcher.py`, and dispatches results to `spamhandling.py` for chat reporting. A command system in `chatcommands.py` allows operators to manage blacklists, inspect posts, and control the bot through chat messages. Persistent state is stored in pickle files, and the bot integrates with GitHub for automated blacklist pull requests. The detection logic in `findspam.py` is rule-based with over 100 rules registered via a decorator pattern, each with configurable filters for site, reputation, and score. Key reusable components include a generic TTL cache (`metasmoke_cache.py`), an extensible blacklist parser hierarchy (`blacklists.py`), a phone number matching module with homoglyph support (`phone_numbers.py`), and a command registration decorator. The codebase is well-tested with extensive parametrized tests and has CI/CD pipelines. While the core detection file is large and tightly coupled, many supporting modules are cleanly separated and could be extracted for other projects.
The LLM identified 4 files worth highlighting for reuse:
Implements a hierarchy of blacklist parsers (BlacklistParser base class, BasicListParser, TSVDictParser, YAMLParserCIDR, YAMLParserNS, YAMLParserASN) that load and validate blacklist data from plain text, TSV, and YAML files. Each parser handles a specific format with validation (e.g., CIDR, nameserver pairs, ASN numbers). The design is clean, extensible, and could be reused in any project requiring structured blacklist loading with format-specific validation.
Encapsulates a feedback event (type, flags like blacklist/silent, associated post) and provides methods to send it to the Metasmoke API. The class is self-contained, with clear validation and serialization logic. It could be extracted as a reusable feedback/flagging component for any moderation system that needs to record and transmit user feedback.
Provides a generic TTL cache with generator-based population, expiry, and optional pickle persistence. The cache supports lazy loading via generators, automatic expiry, and dump/load from disk. It is thread-safe and designed for reuse in any Python application needing a simple, persistent cache with time-based invalidation.
Contains functions for matching phone numbers in text, including support for homoglyph characters (e.g., replacing 'O' with '0') and North American number handling. The module is self-contained with its own test data and could be reused in any application that needs to detect obfuscated phone numbers in user-generated content.