$ show data-privacy-stack/presidio
Presidio is a production-grade PII detection and anonymization framework. It operates as both a Python library and a set of microservices (Flask REST APIs) for detecting and redacting personally identifiable information (PII) in text, images (including DICOM), and structured/tabular data. The system is built around three core engines: the Analyzer (entity recognition via regex patterns, NLP models, and LLMs), the Anonymizer (applies operators like replace, redact, mask, hash, encrypt), and the Image Redactor (OCR-based text extraction and bounding-box redaction). A CLI tool and a structured data engine complement the main services. The architecture is modular and extensible: recognizers are pluggable (predefined regex-based, NLP-based, or custom), anonymization operators follow a consistent interface, and NLP engines (spaCy, Stanza, Transformers, GLiNER, HuggingFace) are swappable via YAML configuration. The system supports context-aware detection, score thresholds, allow/deny lists, and batch processing. External dependencies include NLP libraries, OCR (Tesseract), cloud services (Azure AI Language, Azure Health Data Services), and LLM providers (Azure OpenAI, Ollama). Key entry points include the AnalyzerEngine.analyze() method, AnonymizerEngine.anonymize(), the REST endpoints (/analyze, /anonymize, /redact), and the presidio-cli command-line tool. The primary data flow is: input text/image → Analyzer (detects entities with scores) → Anonymizer (applies operators) → output. For images, OCR extracts text before analysis. The system is designed for compliance with data privacy regulations (GDPR, HIPAA, etc.) and is used in production deployments with Kubernetes, Azure, and Spark.
The LLM identified 5 files worth highlighting for reuse:
Abstract base class for anonymization operators with operate(), validate(), operator_name(), and operator_type() methods. Combined with OperatorsFactory, this enables dynamic registration and invocation of anonymization/deanonymization operations (hash, mask, redact, replace, encrypt, decrypt, keep, custom). The pattern is cleanly separated and could be extracted as a generic operator framework for any transformation pipeline.
CharacterBasedTextChunker and TokenizerBasedTextChunker provide robust text splitting with word-boundary preservation, offset adjustment, and overlap deduplication. These are essential for processing large texts within NLP engine limits. The design is isolated and reusable for any text processing pipeline requiring chunking with accurate character offsets.
Singleton utility that auto-detects GPU/CPU availability for NLP engines (spaCy, Transformers, GLiNER, ONNX) with environment variable override (PRESIDIO_DEVICE). Provides a consistent device selection mechanism across the codebase. Simple, well-tested, and easily extractable for any ML project needing hardware-aware configuration.
Boosts recognizer scores based on surrounding context words using lemma matching. Supports configurable matching modes (substring, whole_word) and context similarity factors. Demonstrates a clean pattern for context-aware confidence adjustment that could be applied to other entity recognition systems requiring contextual disambiguation.
Provides AES-CBC encryption/decryption with key derivation from a password using PBKDF2. Used by the Encrypt and Decrypt operators for reversible anonymization. A self-contained cryptographic utility that could be reused in any Python project requiring symmetric encryption with password-based key derivation.