thehardwarestore.ai

Strix is an open-source, AI-powered penetration testing tool that orchestrates multiple LLM-driven agents to autonomously discover and validate security vulnerabilities in web applications, APIs, source code, and infrastructure. The system operates as a CLI application with three interface surfaces: a headless/non-interactive mode for CI/CD pipelines, a feature-rich Go Bubble Tea TUI for interactive scanning, and a local React web viewer for reviewing results. Scans run inside Docker sandboxes with a Caido interception proxy sidecar, allowing agents to execute shell commands, browse applications, modify requests, and apply patches. The architecture follows a multi-agent coordination pattern where a root agent delegates tasks to child agents, tracks coverage, manages budgets, and produces comprehensive reports including CVSS-scored vulnerabilities, proof-of-concept exploits, and fix recommendations. The codebase demonstrates strong engineering practices with extensive test coverage, atomic file operations, output bounding, MCP tool integration, and a sophisticated IPC protocol between the Python backend and Go TUI. Key features include: multi-agent orchestration with parent/child messaging and budget enforcement; Docker sandboxed execution with Caido proxy integration; support for multiple LLM providers (OpenAI, Anthropic, LiteLLM, Bedrock, etc.); MCP (Model Context Protocol) tool integration; SARIF and PDF report generation; cloud platform integration for managed scans; and a local web viewer with live transcript and vulnerability triage. The repository is primarily an application but includes data-oriented components like LLM cost tracking, coverage ledgers, and vulnerability deduplication using LLM-based comparison.

garak is a CLI tool and Python library for automated red-teaming and vulnerability scanning of large language models (LLMs). It systematically probes models with adversarial inputs (jailbreaks, prompt injections, encoding attacks, etc.) and uses detectors to classify model responses for safety failures. It supports multiple model backends (OpenAI, Anthropic, HuggingFace, etc.) and generates structured reports with severity scoring. The system follows a modular plugin architecture with base classes for probes, detectors, generators, harnesses, evaluators, and buffs. Plugins are dynamically loaded and configured via a hierarchical config system. The default harness runs probes with their associated detectors, while a cross-product harness allows all probes against all detectors. Results are logged as JSONL and can be visualized via a React-based web UI (garak-report) that displays DEFCON-level severity, calibration Z-scores, and technique-intent matrices. While primarily an application, garak incorporates significant research elements: it implements state-of-the-art attack techniques (GCG, AutoDAN, BEAST, TAP) and includes curated datasets of adversarial prompts and calibration data. The codebase serves both as a practical security tool and as a research platform for LLM vulnerability assessment.

Mythic is a cross-platform, post-exploitation, red teaming command-and-control (C2) framework. It enables operators to manage remote agents (callbacks), generate payloads, issue tasks, exfiltrate files, harvest credentials, and automate workflows through an event-driven system. The architecture is a multi-container Docker deployment: a Go backend (using Gin, RabbitMQ, PostgreSQL, gRPC) handles agent communication, task orchestration, and data persistence; a React frontend (Material-UI, Apollo Client) provides a rich operator interface with real-time updates via GraphQL subscriptions (backed by Hasura); and a CLI tool (Mythic_CLI) manages container lifecycle, configuration, and service installation. Key entry points include the agent message webhook (HTTP POST/GET), web controllers for CRUD operations, and RabbitMQ RPC handlers for internal service communication. External dependencies include RabbitMQ, PostgreSQL, Hasura GraphQL engine, Docker, and gRPC. The system is designed for extensibility through payload type containers, C2 profiles, translation containers, and custom browser scripts.

Agentic Security is an open-source LLM vulnerability scanner that probes AI endpoints for security weaknesses such as jailbreaks, prompt injection, data leakage, and sandbox escapes. It provides both a FastAPI web server (port 8718) with a Vue.js frontend and a Fire-based CLI (server, ci, init, ls, scan). The core scan engine sends adversarial prompts (text, image, audio) to target LLMs via configurable HTTP specs, classifies responses using a multi-layered refusal detection system (phrase-based, OneClassSVM, PII detector, LLM-as-judge, hybrid voting), and generates detailed reports with failure rates and cost metrics. The architecture is modular: LLM provider abstraction (OpenAI, Anthropic, LiteLLM), resilience primitives (rate limiter, circuit breaker, concurrent executor), plugin-based refusal classifiers, and a dataset registry supporting HuggingFace, CSV, Google Sheets, and dynamic transformations (stenography, encoding). It also integrates external red-teaming tools (Garak, Inspect AI) and includes reinforcement learning and Bayesian optimization for adaptive prompt selection. While primarily an application, it contains significant data/research components (ML models, datasets, refusal classification research) that support its scanning mission.

DeepTeam is a production-oriented Python library and CLI for red-teaming LLM applications. It does not merely document vulnerabilities; it executes automated adversarial campaigns against a user-supplied model callback, generating attack prompts through LLM-driven templates (single-turn, multi-turn, tree/crescendo jailbreaks), scoring target responses with LLM-as-judge metrics, and aggregating outcomes into risk assessments with CVSS scoring. It also ships a code scanner for static AI-security review, a trace scanner for agent executions, and deployable input/output guardrails. Main entry points are the red_team() Python API and the deepteam CLI (login, run, scan), with YAML configuration driving model selection, vulnerability types, and attack budgets. Architecturally, the repo is a modular Python package built on deepeval: vulnerabilities inherit from BaseVulnerability, all metrics inherit from BaseRedTeamingMetric, attacks implement configurable templates and schemas, and frameworks (OWASP, MITRE ATLAS, NIST, EU AI Act, BeaverTails, Aegis) map risks to pre-built categories. Results flow through RTTestCase objects and can be saved locally as JSON or pushed to the Confident AI platform. The library is dual sync/async and includes progress reporting, cost accumulation, and retry logic. Although it embeds research-style synthetic data generation and dataset-driven framework integrations, the dominant character is an application library/CLI rather than a data research project. Strengths include broad vulnerability and attack coverage, clear base-class contracts, a substantial test suite, and active CI. Weaknesses include heavy copy-paste duplication across 30+ vulnerability, metric, and template modules, tight coupling to deepeval internals, reliance on hardcoded model defaults such as gpt-3.5-turbo-0125, and a known logic bug where is_successful in the metric base class compares instead of assigns, always returning False unless an error is set. Telemetry code also contains hardcoded service keys. Overall, it is a functional, well-documented framework with real reuse value, but it would benefit from deduplication and dependency hardening.

LLMMap is a CLI-based automated prompt injection testing framework for LLM-integrated applications, modeled after sqlmap. It systematically discovers injection points (query parameters, headers, body, cookies, path), generates goal-aware adversarial prompts using a dual-LLM architecture (Generator + Judge), and evaluates responses through pattern matching, semantic similarity, and out-of-band callbacks. The tool supports 227 injection techniques across 18 attack families, multiple LLM providers (Ollama, OpenAI, Anthropic, Google), and provides reliability confirmation via Wilson confidence intervals. It includes comprehensive reporting, data-flow tracing, and Burp Suite integration. The codebase is well-structured with clean abstractions, extensive tests, and reusable components, though some modules are reserved for future development and external dependencies like interactsh-client introduce fragility.

This repository is a comprehensive collection of red teaming tactics, techniques, and procedures (TTPs) presented as markdown cheatsheets and guides. It is not a software application or data pipeline; rather, it is a knowledge base containing hundreds of embedded code snippets (bash, PowerShell, Python, Go, Rust, C, etc.) organized by platform (Linux, Windows, macOS, Cloud, ICS, Web). The repo includes a few standalone tools such as an Azure subdomain enumeration Metasploit module, a Flask-based C2 redirector container, a PDF generation script, and a PAM module for credential harvesting. The primary value is as a reference for offensive security operators, not as a runnable application. Architecture is document-based with no API, CLI entry point, or library exports. External dependencies are minimal (mostly system tools).