$ show msoedov/agentic_security
Agentic Security is an open-source LLM vulnerability scanner that probes AI endpoints for security weaknesses such as jailbreaks, prompt injection, data leakage, and sandbox escapes. It provides both a FastAPI web server (port 8718) with a Vue.js frontend and a Fire-based CLI (server, ci, init, ls, scan). The core scan engine sends adversarial prompts (text, image, audio) to target LLMs via configurable HTTP specs, classifies responses using a multi-layered refusal detection system (phrase-based, OneClassSVM, PII detector, LLM-as-judge, hybrid voting), and generates detailed reports with failure rates and cost metrics. The architecture is modular: LLM provider abstraction (OpenAI, Anthropic, LiteLLM), resilience primitives (rate limiter, circuit breaker, concurrent executor), plugin-based refusal classifiers, and a dataset registry supporting HuggingFace, CSV, Google Sheets, and dynamic transformations (stenography, encoding). It also integrates external red-teaming tools (Garak, Inspect AI) and includes reinforcement learning and Bayesian optimization for adaptive prompt selection. While primarily an application, it contains significant data/research components (ML models, datasets, refusal classification research) that support its scanning mission.
The LLM identified 6 files worth highlighting for reuse:
Implements a provider factory that creates LLM client instances (OpenAI, Anthropic, LiteLLM) from a unified interface. Each provider implements sync/async generate and chat methods with consistent error mapping (LLMRateLimitError, LLMProviderError). The factory supports lazy client creation and dynamic model selection. This pattern is reusable for any project needing to abstract over multiple LLM backends with minimal coupling.
Provides a semaphore-based concurrent executor with integrated token-bucket rate limiting and circuit breaker (closed/open/half-open states). Tracks success rate and p95 latency metrics. This is a self-contained resilience primitive that can be extracted and reused in any async Python application requiring controlled concurrency and fault tolerance.
Implements a DetectorRegistry that maps detector names to factory functions, supporting config-driven enable/disable and custom class paths. This pattern allows adding new refusal detection strategies (e.g., PII, sandbox escape, LLM-as-judge) without modifying core code. The registry pattern is reusable for any plugin-based system.
Defines LLMSpec, a parser for HTTP request templates with placeholders (<<PROMPT>>, <<BASE64_IMAGE>>, <<BASE64_AUDIO>>) and secret expansion. Handles JSON escaping, Content-Length stripping, and placeholder substitution. This is a reusable pattern for defining and probing any LLM API endpoint with dynamic payloads.
Implements FuzzChain with a | operator that composes FuzzNode instances into sequential LLM pipelines. Each node's output is passed as {input} to the next. This demonstrates a clean composable pattern for building multi-step attack chains, reusable for any sequential LLM processing workflow.
Provides a collection of prompt obfuscation functions (rot5, rot13, base64, mirror, scramble, noise-character, ASCII substitution) that can be applied to any text. These are self-contained, parameterized transformations that are reusable for data augmentation or adversarial testing in any NLP pipeline.