Awkward Array is a Python library for manipulating nested, ragged, and mixed-type data structures that do not fit naturally into NumPy's rectangular array model. It is primarily a software library, not a data-research pipeline, though it contains research-oriented studies and benchmarks. The architecture is a Python front-end backed by compiled C++ kernels, with optional integrations for PyArrow, JAX, CuPy, Numba, CUDA, and ROOT. The core data model is built around composable layouts, forms, and types, and operations are dispatched through a high-level function decorator that separates public API from backend-specific implementations.
Arroyo is a distributed stream processing platform written in Rust, inspired by Apache Flink. It allows users to define real-time data pipelines using SQL, which are compiled into physical plans executed across a cluster of workers. The system supports stateful operations (windows, joins, aggregations), exactly-once semantics via checkpointing, and a wide variety of connectors (Kafka, Kinesis, MQTT, Redis, filesystems, Iceberg, etc.). It includes a web UI for managing pipelines, connections, and UDFs, as well as a REST API and Kubernetes-native deployment. The architecture separates concerns into an API server, controller, worker, and compiler service, with gRPC communication. The codebase demonstrates high engineering effort with custom extensions to Apache DataFusion for streaming SQL planning, a rigorous checkpoint protocol, and a pluggable connector framework. Notable craftsmanship includes the state protocol for generation-based checkpointing, the ArrowOperator abstraction, and the two-phase commit sink pattern.