Presidio is a production-grade PII detection and anonymization framework. It operates as both a Python library and a set of microservices (Flask REST APIs) for detecting and redacting personally identifiable information (PII) in text, images (including DICOM), and structured/tabular data. The system is built around three core engines: the Analyzer (entity recognition via regex patterns, NLP models, and LLMs), the Anonymizer (applies operators like replace, redact, mask, hash, encrypt), and the Image Redactor (OCR-based text extraction and bounding-box redaction). A CLI tool and a structured data engine complement the main services. The architecture is modular and extensible: recognizers are pluggable (predefined regex-based, NLP-based, or custom), anonymization operators follow a consistent interface, and NLP engines (spaCy, Stanza, Transformers, GLiNER, HuggingFace) are swappable via YAML configuration. The system supports context-aware detection, score thresholds, allow/deny lists, and batch processing. External dependencies include NLP libraries, OCR (Tesseract), cloud services (Azure AI Language, Azure Health Data Services), and LLM providers (Azure OpenAI, Ollama). Key entry points include the AnalyzerEngine.analyze() method, AnonymizerEngine.anonymize(), the REST endpoints (/analyze, /anonymize, /redact), and the presidio-cli command-line tool. The primary data flow is: input text/image → Analyzer (detects entities with scores) → Anonymizer (applies operators) → output. For images, OCR extracts text before analysis. The system is designed for compliance with data privacy regulations (GDPR, HIPAA, etc.) and is used in production deployments with Kubernetes, Azure, and Spark.
This is a document redaction and anonymization platform designed for processing unstructured data (documents, images, PDFs) and structured data (CSV, databases) to detect and redact sensitive information like PII, PHI, and financial data. The system features a React/TypeScript frontend with a Python/FastAPI backend, utilizing AI models for NER (HaS Text), OCR (PaddleOCR-VL), and visual detection (LocateAnything). It provides both single-file processing via a playground interface and batch processing workflows with review/approval steps. The platform includes role-based access control, audit logging, configurable presets for different industries, and structured data handling capabilities. Deployment is containerized with Docker and supports GPU acceleration for ML models.
OSINT Brazuca Regex is a CLI tool and curated dataset of regular expressions for detecting Brazilian and global PII/OSINT-relevant patterns. The scanner (`osint_regex_scan.py`) applies the dataset to files, directories, or stdin, supporting multi-threading, text extraction from documents (PDF, DOCX, XLSX, RTF, EPUB) via stdlib or optional `pypdf`/`pdftotext`, and noise filtering (blob masking, line length limits). Output formats include terminal, CSV, JSON, or raw values. The dataset is split into two JSON files (Brazil-specific and general), merged at load time, and includes patterns for CPF, CNPJ, credit cards, crypto wallets, API credentials, and more. The repository also includes a build script (`build.py`) for managing the split/merged dataset. Core functionality requires only Python 3.12+ stdlib, with optional dependencies for extended PDF support and encoding detection.