Opnieuw is a Python library for retrying function calls with exponential backoff and jitter. It provides a clean, opinionated decorator API that requires only two parameters: maximum total calls and a retry window. The library handles both synchronous and asynchronous functions, supports exception chaining, and offers context managers to override retry behavior (e.g., disable retries or skip waits during testing). The core design uses a `BackoffCalculator` class that computes backoff intervals using a geometric series, and a `Clock` abstraction to allow time manipulation in tests. The library is small, fully typed, and demonstrates strong software craftsmanship with proper use of context variables for thread/async safety.
ManimGL is a mature, well-architected Python library for programmatic mathematical animation, using OpenGL (via ModernGL) for GPU-accelerated rendering. Originally created by Grant Sanderson (3Blue1Brown), it is the upstream predecessor to the community edition. The codebase is structured as a library with a CLI entry point (`manimgl`), and it cleanly separates concerns into animation logic, scene management, camera/frame transforms, mobject (math object) types, shader pipelines, and utility modules. The architecture centers on a scene-graph model: `Scene` objects orchestrate `Animation` instances that interpolate between `Mobject` states, while a `Camera` with a `CameraFrame` handles view/projection and feeds into a shader-based rendering pipeline. The system supports real-time interactive preview via an embedded IPython shell, video export via FFmpeg piping, and LaTeX-to-SVG conversion for text. The GLSL shader system is notably sophisticated, with custom `#INSERT` preprocessing to share code between vertex/fragment/geometry shaders for quadratic Bézier rendering, fractals, and lit surfaces. Key reusable components include: a robust event-driven interactive system (event dispatcher + listeners), a disk-caching decorator for expensive computations, a deeply capable Bézier/geometry math library, a flexible color system with colormaps, and a module-reload system that enables live coding. The codebase demonstrates high craftsmanship in its shader architecture, the `CameraFrame` as a fully animatable mobject, and the `TransformMatchingParts` algorithm for morphing between shapes.
Scrapy is a mature, production-grade web scraping and crawling framework for Python. It provides a complete pipeline for downloading web pages, extracting structured data (via XPath/CSS selectors), and exporting it in multiple formats (JSON, CSV, XML, etc.). The architecture is event-driven, built on Twisted with full asyncio integration, and consists of an execution engine, downloader, scheduler, spider middleware, downloader middleware, item pipelines, and extensions. Key entry points include the CLI (`scrapy crawl`, `scrapy shell`, `scrapy startproject`), the `CrawlerProcess`/`CrawlerRunner` classes for programmatic use, and the `Spider` base class for user-defined spiders. Data flows from HTTP requests through the downloader (with support for HTTP/1.1, HTTP/2, FTP, and experimental httpx), through spider callbacks, into item pipelines, and finally to feed exporters or custom storage backends. External dependencies include Twisted, lxml, parsel, w3lib, protego (robots.txt), queuelib, and optional integrations like boto3 (S3), google-cloud-storage, Pillow (image processing), and mitmproxy (testing). The framework is highly configurable via a priority-based settings system and supports plugins (addons, middlewares, pipelines, extensions).
Freqtrade is a production-grade cryptocurrency trading bot written in Python. It provides a full-fledged platform for designing, backtesting, and executing automated trading strategies across major exchanges (Binance, Kraken, Bybit, OKX, etc.) via the CCXT library. The bot supports both spot and futures trading, dry-run simulation, and live deployment with docker and systemd integration. At its core, it exposes a plugin architecture for strategy definitions, pairlist filters, and risk protection modules, all configured through a unified YAML/JSON configuration system. It also includes an embedded REST API and WebSocket server for remote monitoring and control, with optional Telegram and Discord notification channels. Beyond live trading, freqtrade offers a comprehensive research subsystem: a deterministic backtesting engine with realistic fee/slippage modeling, hyperparameter optimization (hyperopt) using Optuna with over a dozen loss functions, and a machine learning module (FreqAI) that integrates sklearn, XGBoost, LightGBM, PyTorch, and reinforcement learning (SB3). The codebase includes tools for lookahead and recursive analysis to detect strategy biases, plus data download/convert/clean utilities supporting multiple storage formats (feather, parquet, JSON). Architecturally, the project follows a modular design with clear separation of concerns: exchange abstraction layer, strategy resolver, data provider, persistence (SQLAlchemy), and RPC multiplexing. It uses dependency injection through resolvers, a logging mixin for debounced messages, and a custom scheduler for periodic tasks. The test suite is extensive with over 800 test files covering all major components. While the codebase is large and some modules (e.g., exchange-specific subclasses) are tightly coupled to CCXT, many internal patterns such as the plugin system, resolver, and data handler interfaces are well-abstracted and could be extracted for reuse in other Python trading or data pipeline projects.
pyenv is a mature, widely-used CLI tool for managing multiple Python installations per-user or per-project. It operates by intercepting Python-related commands via shims (small wrapper scripts) that redirect execution to the correct Python version based on a `.python-version` file, the `PYENV_VERSION` environment variable, or a global default. The core is implemented in Bash with a plugin architecture: the `python-build` plugin handles compiling Python from source or downloading pre-built binaries (CPython, PyPy, Anaconda, Miniconda, Miniforge, GraalPy, etc.), while `pyenv-binary` (experimental) packages installed versions into relocatable archives. The codebase includes shell completions for Bash, Zsh, Fish, and PowerShell, a hook system for extensibility, and an optional C extension for fast `realpath` resolution. It is actively maintained with frequent releases, extensive CI testing across multiple OS/arch combinations, and a large community (45k+ stars). The repository is primarily an application (CLI tool) but also contains a substantial data-like collection of build definitions and patches for hundreds of Python versions.
Ray is a distributed compute framework that unifies task/actor execution, data processing, model training, hyperparameter tuning, RL, and model serving under a single runtime. The core is written in C++ (raylet, GCS, object store, scheduling) with a Python frontend, communicating via gRPC and Arrow-compatible plasma object storage. Production entry points include `ray.init()` for cluster bootstrap, `@ray.remote` for task/actor offload, Ray Serve for model deployment (FastAPI/DAG/gang-scheduling patterns), Ray Data for streaming ETL, Ray Train/Tune for distributed training and hyperparameter search, and RLlib for reinforcement learning. The architecture is a hybrid: a distributed systems platform (scheduler, object store, autoscaler) layered with high-level AI libraries, all built via Bazel with extensive CI (Docker images, build matrices, release tests). It is not a data research repo, though Ray Data and RLlib contain research-grade data pipeline and RL primitives; the dominant character is a production-grade application framework.
Gradio is a full-stack Python library and web framework for rapidly creating interactive web UIs for machine learning models, APIs, or any Python function. It provides a high-level Interface class for simple demos, a flexible Blocks API for custom layouts and event-driven apps, and a ChatInterface for chatbot applications. The backend is built on FastAPI and Uvicorn, serving both a REST API and WebSocket-based streaming. The frontend is a Svelte 5 / TypeScript application with per-component packages, a SvelteKit SSR app, and a standalone SPA that can be embedded via a custom HTML element. The library includes a rich component library (30+ input/output widgets), layout primitives (Row, Column, Tabs, Accordion, Sidebar), theming support, internationalization, and a CLI for development, deployment, and custom component creation. It also provides official Python and JavaScript client libraries for programmatic access to Gradio apps, and supports MCP (Model Context Protocol) for tool/resource integration. The repository contains over 100 runnable demos covering a wide range of ML use cases (image classification, text generation, speech recognition, object detection, chatbots, etc.), many of which integrate with Hugging Face models and datasets. While primarily an application framework, the extensive demo collection and focus on ML model visualization give it a secondary data/research character.
SearXNG is a privacy-respecting metasearch engine that aggregates results from over 100 search engines (Google, Bing, DuckDuckGo, Wikipedia, academic sources, etc.) via HTTP requests. It runs as a Flask web application, supports multiple output formats (HTML, JSON, RSS), and includes a plugin system, bot detection, rate limiting, and user preferences stored in cookies. The architecture is a monolithic Python application with a separate TypeScript/Less frontend built with Vite. It uses Valkey/Redis for caching and rate limiting, and SQLite for persistent caches. The codebase is well-structured with a clear separation of concerns: engine modules implement a common request/response interface, result types are typed with msgspec, and plugins are registered via a simple hook system. It is actively maintained with CI/CD, Docker support, and comprehensive documentation.
Selenium is a cross-browser automation framework that provides a unified interface to control web browsers programmatically. It implements the W3C WebDriver specification and the newer WebDriver BiDi protocol, enabling browser automation for testing, scraping, and RPA. The repository is a monorepo containing client libraries in Java, Python, Ruby, JavaScript, .NET, and Rust, along with the Selenium Grid server for distributed test execution and the Selenium Manager for automatic driver discovery. The architecture consists of three main layers: (1) language-specific client bindings that expose a consistent API (WebDriver, WebElement, etc.) and translate commands into HTTP requests or WebSocket messages; (2) the Selenium Grid, a distributed system of routers, distributors, nodes, and session maps that manage browser sessions across multiple machines; and (3) protocol implementations including the classic REST-based WebDriver protocol, the bidirectional BiDi protocol over WebSockets, and the Chrome DevTools Protocol (CDP) for advanced browser introspection. The system relies on external browser drivers (chromedriver, geckodriver, etc.) and communicates with browsers via HTTP and WebSocket. Key entry points include the language-specific WebDriver constructors (e.g., webdriver.Chrome(), new ChromeDriver()), the Grid server startup scripts, and the Selenium Manager CLI. Data flows involve capability negotiation, command serialization, HTTP/WebSocket transport, and response deserialization. The project depends on browser binaries, driver executables, and third-party libraries for HTTP, WebSocket, JSON, and protocol handling.
MLX is an array framework for machine learning on Apple Silicon and NVIDIA GPUs, developed by Apple's machine learning research team. It provides a NumPy-like API for Python and C++ with composable function transformations (autograd, vectorization, graph optimization), lazy computation, dynamic graph construction, and multi-backend support (CPU, Apple Metal, NVIDIA CUDA). The framework includes high-level neural network layers, optimizers, distributed training primitives, quantization support, and model I/O (safetensors, GGUF). It is designed for efficient on-device ML research and inference, with an emphasis on performance via custom GPU kernels, JIT compilation, and unified memory.
Cookiecutter is a command-line utility for creating projects from templates (cookiecutters). It uses Jinja2 to templatize file and directory names, prompts users for input, and generates a customized project structure. The tool supports local and remote templates (via Git/Hg), includes a hook system for customizing the generation process, and provides a Python API for programmatic use. Architecture-wise, it is a modular Python package with distinct components for CLI, configuration, environment setup, template discovery, generation, hooks, prompting, replay, VCS operations, and utilities. Key entry points are the `cookiecutter` CLI command and the `cookiecutter.main.cookiecutter()` function. The system follows a data flow where template location is resolved, context is gathered (from defaults, user config, and prompts), templates are rendered with Jinja2, and output is written to a directory, optionally executing user-defined hooks at various stages.
python-dotenv is a production-grade Python library that reads key-value pairs from .env files and sets them as environment variables, following the 12-factor configuration principles. It is widely used in development and deployment workflows to separate configuration from code. The library provides both programmatic APIs and a command-line interface for managing environment variables without manual file editing. The architecture centers on the `DotEnv` class, which orchestrates parsing, variable interpolation, and environment loading. Parsing is handled by a custom streaming parser (`parser.py`) that uses a `Reader` with position tracking and regex-based tokenization to handle single/double quoted values, comments, escape sequences, and UTF-8 BOM stripping. Variable expansion (`variables.py`) implements POSIX-style `${VAR:-default}` interpolation using an `Atom` abstract base class with `Literal` and `Variable` strategies. The core library has zero external dependencies; the CLI (built on `click`) and IPython integration are optional extras. Key entry points include `load_dotenv()` (loads .env into `os.environ`), `dotenv_values()` (returns a dict without touching the environment), `set_key()`/`unset_key()` for programmatic file manipulation, and `find_dotenv()` for locating .env files. The CLI offers `list`, `get`, `set`, `unset`, and `run` subcommands, with support for JSON/shell/export output formats. The IPython `%dotenv` magic provides interactive loading. Data flows from file or stream through the parser into an ordered dict, with optional interpolation and override control. External dependencies are minimal: `click` for the CLI, `IPython` for the magic extension, and standard library modules (`os`, `io`, `re`, `tempfile`, `pathlib`, etc.). The library supports Python 3.10–3.14 (including free-threaded 3.14t) and PyPy 3.11, with comprehensive CI testing across Linux and Windows.
This repository produces and maintains the official Jupyter Docker Stacks: a family of layered Docker images that ship JupyterLab, Jupyter Notebook, JupyterHub single-user servers, and a broad set of scientific computing kernels and libraries. The actual deliverable is infrastructure-as-code: Dockerfiles, entrypoint/startup scripts, a Python tagging system, CI workflows, and a container-based test suite. Although the images serve data-science workflows (pandas, Spark, PyTorch, R, Julia, etc.), the repository itself is an application/devops project for building, versioning, and publishing reproducible computing environments, not an analytics or ETL codebase. This hybrid nature shows in bundled research-oriented packages and notebook-based tests, but the core engineering artifact is container image definitions and release automation.
GEF (GDB Enhanced Features) is a production-grade GDB plugin for exploit development, reverse engineering, and binary analysis. It is a single Python script (gef.py) that loads into GDB (>=10.0) and provides over 90 commands covering memory inspection, heap analysis, security mitigation checks (checksec), pattern generation, format string detection, and remote debugging. The architecture is monolithic but well-organized, with a clear separation of concerns via class hierarchies for architectures, memory regions, heap structures, and command registration. The test suite is extensive, using a Docker-based CI matrix that runs on multiple architectures (x86, ARM, AARCH64, etc.) and validates behaviour through RPyC-driven remote GDB sessions. While primarily an application, it also serves as a research tool for heap allocator internals and glibc version-specific behaviour, blending practical utility with deep system understanding.
Bandit is a well-established Python security linter that performs static analysis by parsing source code into ASTs and running a configurable set of plugins against AST nodes. It is primarily a CLI tool, but also exposes a plugin architecture for custom checks and formatters via setuptools entry points. Key entry points include the main CLI (bandit.cli.main), baseline comparison utility (bandit.cli.baseline), and config generator (bandit.cli.config_generator). The core engine (bandit/core/manager.py, node_visitor.py, context.py) orchestrates file discovery, AST traversal, test execution, and result aggregation. It supports multiple output formats (JSON, SARIF, CSV, HTML, XML, YAML, text, screen) and integrates with CI/CD pipelines, pre-commit hooks, and Git via .bandit files. The project is actively maintained, supports Python 3.10+, and has extensive test coverage, though some legacy config conversion code and removed test IDs (e.g., B309, B320) indicate minor drift. Bandit is a reference implementation for a plugin-based security linter with a clean separation between core, plugins, and formatters.
MockServer is a highly sophisticated, multi-protocol HTTP(S) mock server and proxy for testing, written in Java with Netty. It supports HTTP/1.1, HTTP/2, HTTP/3 (QUIC), gRPC, WebSockets, DNS, TCP, and binary protocols on a single port. The core engine provides expectation-based request matching, response generation, forwarding, proxying, recording, verification, and chaos engineering (latency, connection drops, error injection). It also includes built-in LLM mocking (OpenAI, Anthropic, Gemini, etc.), OIDC/OAuth2/SAML identity providers, SCIM provisioning, WASM custom rules, and drift detection. The repository is a monorepo containing the Java server, multi-language client libraries (Java, Node, Python, Ruby, Go, .NET, Rust, PHP), a React dashboard UI, IDE plugins (JetBrains, VS Code), Helm charts, Docker images, and extensive CI/CD pipelines via Buildkite. The architecture is modular with clear separation of concerns: core model/matcher/serialization in `mockserver-core`, Netty transport in `mockserver-netty`, language-specific clients in `mockserver-client-*`, and a React UI in `mockserver-ui`. The codebase demonstrates strong engineering practices including DTO patterns with Jackson serializers, a matcher hierarchy with compiled-once validators, bounded caches, constant-time equality checks, and extensive test coverage with integration tests across multiple environments.
StatsForecast is a production-grade Python library for large-scale statistical time series forecasting. It provides a scikit-learn-like API (fit/predict) with over 30 models including AutoARIMA, AutoETS, AutoCES, AutoTheta, TBATS, MSTL, GARCH, and intermittent demand models. The library achieves high performance through a hybrid architecture: core algorithms are implemented in C++ using Eigen and pybind11, while Python handles orchestration, model selection, and validation. It supports distributed computing via Fugue (Spark, Dask, Ray) and multiprocessing, enabling forecasting on millions of time series. The library also includes conformal prediction intervals, cross-validation, simulation, and feature engineering utilities. While primarily an application library, the repository has a strong research component with extensive benchmarking experiments against R's forecast package, pmdarima, Prophet, NeuralProphet, and Amazon Forecast. These experiments are well-documented with reproducible environments (Docker, conda, Makefiles) and use public datasets (M3, M4, M5, Tourism, ERCOT). The codebase is actively maintained with CI/CD across multiple OS/Python versions, automated wheel building, and documentation pipelines using Quarto and Mintlify. The architecture is clean and modular: models inherit from a common base class (_TS), distributed backends implement a ParallelBackend interface, and C++ code is separated into submodules. Notable reusable components include a generic Nelder-Mead optimizer, a unified distribution framework, and a stepwise ARIMA search algorithm ported from R. The library is well-tested with comprehensive unit and integration tests.
Piccolo is an async-first Python ORM and query builder targeting PostgreSQL, SQLite, and CockroachDB. It provides a declarative table definition system, automatic schema migrations, a CLI for project scaffolding, user authentication, fixture dump/load, and Pydantic model generation. The codebase is well-structured with clean abstractions (column types, query delegates, engine abstraction, migration auto-generation) and an extensive test suite covering multiple database backends. It is actively maintained with frequent releases and supports recent Python and database versions.
django-anymail is a Django library that provides a unified interface for sending transactional email through multiple Email Service Providers (ESPs) such as Amazon SES, Mailgun, SendGrid, Postmark, SparkPost, Mailjet, Brevo, Resend, Mailtrap, Postal, Scaleway, Unisender Go, and MailerSend. It extends Django's built-in email framework with ESP-specific features like tags, metadata, inline images, merge data, templates, and click/open tracking. The library also handles inbound email and tracking webhooks, normalizing ESP-specific events into Django signals. Architecture follows a backend pattern: each ESP has a concrete EmailBackend class that implements `build_message_payload` and `post_to_esp`, with common logic in base classes (`AnymailRequestsBackend`, `BasePayload`). Webhook views are similarly structured with base classes for signature validation and event dispatch. The codebase is well-tested with both mock unit tests and live integration tests run weekly against real ESP APIs. It is a mature, actively maintained library with a clean separation of concerns and extensive documentation.
hypertools is a well‑engineered Python library for high‑dimensional data analysis and visualization, with a strong focus on animated 3D trajectories, multi‑subject alignment (Haxby hyperalignment, SRM, Procrustes), cluster analysis, time‑series forecasting, and missing‑data imputation. The package exposes a scikit‑learn–compatible Pipeline: 'manip → normalize → reduce → align → plot', plus standalone dispatchers ('hyp.plot', 'hyp.align', 'hyp.cluster', 'hyp.predict', 'hyp.impute') that share a consistent model‑spec grammar (string, class, instance, dict) and support `return_model` reuse. Two plotting backends (matplotlib and plotly) produce parity for kaleidoscopic animations (serialitary, chemo, bullet, morph) and static charts. The codebase is fortified by an audit‑driven test suite of >160 test files, many using real instead of mocked data, and delivery‑orchestrated through a coherent CI pipeline. Vendored code (PPCA, brainiak SRM, Noto Sans fonts) and a lazy‑import mechanism that auto‑installs optional extras on demand keep the core lean while enabling rich functionality.