SmokeDetector is a headless chatbot that monitors Stack Exchange for spam and abusive content in real time. It connects to the Stack Exchange WebSocket to receive new and edited posts, then runs them through a multi-layered detection engine that checks against extensive blacklists (keywords, websites, usernames, phone numbers, IP ranges, nameservers, ASNs) and applies heuristic rules including DNS lookups, homoglyph detection, and similarity analysis. Detected spam is reported to designated chat rooms on Stack Exchange Chat, where moderators can provide feedback that feeds back into the system via the Metasmoke statistics service. The architecture is event-driven: the main loop in `ws.py` listens for WebSocket events, enqueues posts for scanning via `bodyfetcher.py`, and dispatches results to `spamhandling.py` for chat reporting. A command system in `chatcommands.py` allows operators to manage blacklists, inspect posts, and control the bot through chat messages. Persistent state is stored in pickle files, and the bot integrates with GitHub for automated blacklist pull requests. The detection logic in `findspam.py` is rule-based with over 100 rules registered via a decorator pattern, each with configurable filters for site, reputation, and score. Key reusable components include a generic TTL cache (`metasmoke_cache.py`), an extensible blacklist parser hierarchy (`blacklists.py`), a phone number matching module with homoglyph support (`phone_numbers.py`), and a command registration decorator. The codebase is well-tested with extensive parametrized tests and has CI/CD pipelines. While the core detection file is large and tightly coupled, many supporting modules are cleanly separated and could be extracted for other projects.
Prosopo is a privacy-focused, self-hosted CAPTCHA alternative that provides multiple challenge types—image selection, puzzle (image jigsaw), proof-of-work, and frictionless (behavioral bot detection). The system comprises an Express-based provider API (with MongoDB and Redis for persistence and caching), a CLI for administrative tasks, client-side embeddable widgets (vanilla JS bundle, React/Angular/Vue/Svelte wrappers), and a suite of supporting libraries for cryptography, locale, datasets, and native Rust modules (JA4 TLS fingerprinting, Merkle tree acceleration, puzzle rendering). Bot detection is multi-layered: client-side behavioral data (mouse/keyboard/touch events, SIMD readings) is collected and server-side scored, combined with JA4 TLS fingerprint analysis, IP reputation (ipapi.is + MaxMind), DNS asymmetry detection, spam email heuristics, and custom decision machines per site key. The architecture is a monorepo with ~70 workspaces, leveraging Vite for bundling, Vitest for unit/integration tests, and Cypress for E2E visual regression testing. The provider enforces access policies (allow/block/restrict) via a Redis-backed rules engine and maintains usage counters for rate limiting. Asset sealing (AES-256-GCM) protects image CDN URLs, and puzzle difficulty is sampled with a CSPRNG-stratified ladder to frustrate solver calibration.
Mosparo is a self-hosted, open-source spam protection system that replaces CAPTCHA on web forms. It is built as a monolithic Symfony 6.4 application (PHP 8.1+) with Doctrine ORM supporting MySQL, PostgreSQL, and SQLite. The system provides a multi-tenant architecture where each project (website) can define custom spam detection rules (field rules for words, emails, domains, IPs, user agents, etc. and submission rules like random value detection) and security measures (honeypot, minimum time, proof-of-work, request delaying, IP lockout, equal submission blocking). Integration is done via a frontend JavaScript widget that communicates with a verification API. The codebase also includes an administration panel for system settings, user management, GeoIP2 integration, rule package imports, statistics, and self-update capabilities. The architecture follows standard Symfony patterns with controllers, services (helpers), repositories, and Twig templates. The rule engine is extensible via interfaces and is reasonably well-tested at the unit level. The project is actively maintained with a robust CI/CD pipeline and Docker support.
This is a large, multi-language monorepo from X (formerly Twitter) containing the core code for the For You feed recommendation system. It includes real-time Rust services (home-mixer, thunder, vm-ranker), a Scala rule engine (botmaker) for spam/abuse detection, Python ML training infrastructure (phoenix/xrex), content safety pipelines (grox), and batch feature extraction jobs (simclusters, agatha). The system is composed of loosely coupled services communicating via thrift/gRPC and Kafka, with a strong emphasis on modular pipeline architectures (sources, hydrators, filters, scorers, selectors). While the code demonstrates high craftsmanship with clean abstractions and extensive use of traits/interfaces, it is deeply integrated with internal infrastructure (Manhattan, Strato, Finagle) and requires significant adaptation for external reuse. The repo is hybrid, containing both production application services and substantial ML research/training code.