$ show microsoft/msticpy
MSTICpy is a Python library for cybersecurity threat hunting and investigation, primarily designed for use in Jupyter Notebooks. It provides a unified interface to query multiple data sources (Azure Sentinel, Splunk, Kusto, MDE, etc.), enrich data with threat intelligence and geolocation, perform anomaly detection and process tree analysis, and visualize results with interactive timelines, maps, and graphs. The architecture follows a modular, plugin-based design: data drivers abstract away backend-specific protocols, threat intelligence providers are registered via a consistent base class, and a pivot framework dynamically attaches investigation functions to entity types. The library also includes configuration editors, Azure authentication helpers, and notebook widgets to streamline analyst workflows. While the core purpose is security analysis, the repository also contains substantial research-oriented components, such as anomalous sequence detection (Markov models on command sessions), time-series anomaly detection, and event clustering. These are packaged alongside the production-grade query and enrichment infrastructure, making the repo a hybrid of an application library and a data analysis toolkit. The codebase is well-tested with extensive unit and integration tests, though many require mocked or optional external services. Key entry points include the `QueryProvider` class for data acquisition, `TILookup` for threat intelligence, `Pivot` for entity-driven investigations, and the `mp` pandas accessor for inline transformations. External dependencies span Azure SDK, Bokeh, ipywidgets, and multiple third-party APIs, which introduces some maintenance overhead. Overall, MSTICpy is a mature, actively maintained tool that demonstrates strong engineering practices with reusable patterns, but some modules show signs of version drift and compatibility workarounds.
The LLM identified 5 files worth highlighting for reuse:
Declarative pipeline framework that reads YAML definitions and executes a sequence of steps (filter, enrich, transform) on DataFrames. Each step is a registered function with parameter mappings. The pipeline can generate executable Python code from the YAML, enabling reproducible investigation workflows. Demonstrates a clean separation of configuration and execution, and is reusable for any DataFrame-oriented processing chain.
Collection of foundational decorators and utility classes: `@export` to automatically populate `__all__`, `check_kwargs` for function parameter validation, `singleton` metaclass, `enum_parse` for flexible enum conversion, and `ImportPlaceholder` for graceful missing-dependency handling. These are self-contained, reusable across any Python project, and demonstrate careful API design.
Query templating engine that loads YAML query definitions with parameterized placeholders, type validation, datetime conversion, and macro substitution (e.g., `{{time_range}}`). Supports aliasing parameter names and provides error reporting. This pattern is directly reusable for any system that needs to define and execute parameterized queries from configuration files.
Abstract base class for all threat intelligence providers, defining a consistent interface (`lookup_ioc`, `lookup_ip`, etc.) with common preprocessing (observable type validation, normalization) and result formatting. Subclasses implement only the HTTP request logic. The pattern is a textbook example of the Template Method design pattern, and the base class can be extracted to build a new TI provider framework.
Shared helper functions for Bokeh timeline plots, including tooltip generation, range selectors, tick formatters, and reference lines. Contains a `bokeh_figure` wrapper that normalizes API differences between Bokeh v2 and v3 (e.g., `Circle` radius vs size). This module is relatively isolated and demonstrates how to build a reusable visualization utility layer that absorbs dependency changes.