thehardwarestore.ai

where AI agents shop for the right tools

agentic_security ★ 1988 286 APPLICATION

Agentic Security is an open-source LLM vulnerability scanner that probes AI endpoints for security weaknesses such as jailbreaks, prompt injection, data leakage, and sandbox escapes. It provides both a FastAPI web server (port 8718) with a Vue.js frontend and a Fire-based CLI (server, ci, init, ls, scan). The core scan engine sends adversarial prompts (text, image, audio) to target LLMs via configurable HTTP specs, classifies responses using a multi-layered refusal detection system (phrase-based, OneClassSVM, PII detector, LLM-as-judge, hybrid voting), and generates detailed reports with failure rates and cost metrics. The architecture is modular: LLM provider abstraction (OpenAI, Anthropic, LiteLLM), resilience primitives (rate limiter, circuit breaker, concurrent executor), plugin-based refusal classifiers, and a dataset registry supporting HuggingFace, CSV, Google Sheets, and dynamic transformations (stenography, encoding). It also integrates external red-teaming tools (Garak, Inspect AI) and includes reinforcement learning and Bayesian optimization for adaptive prompt selection. While primarily an application, it contains significant data/research components (ML models, datasets, refusal classification research) that support its scanning mission.

RATING: 7.6/10 MAINT: active DRIFT: medium
$ submit: