$ show ray-project/ray
Ray is a distributed compute framework that unifies task/actor execution, data processing, model training, hyperparameter tuning, RL, and model serving under a single runtime. The core is written in C++ (raylet, GCS, object store, scheduling) with a Python frontend, communicating via gRPC and Arrow-compatible plasma object storage. Production entry points include `ray.init()` for cluster bootstrap, `@ray.remote` for task/actor offload, Ray Serve for model deployment (FastAPI/DAG/gang-scheduling patterns), Ray Data for streaming ETL, Ray Train/Tune for distributed training and hyperparameter search, and RLlib for reinforcement learning. The architecture is a hybrid: a distributed systems platform (scheduler, object store, autoscaler) layered with high-level AI libraries, all built via Bazel with extensive CI (Docker images, build matrices, release tests). It is not a data research repo, though Ray Data and RLlib contain research-grade data pipeline and RL primitives; the dominant character is a production-grade application framework.
The LLM identified 4 files worth highlighting for reuse:
PartitionPredicatePruner implements pushed-down predicate filtering for Ray Data datasources. It must exactly mirror reader-side pruning semantics, a correctness constraint typical of query engines. The pattern is reusable for any columnar/parquet datasource that needs to prune file fragments before scan, and demonstrates non-obvious skill in aligning optimizer-level pruning with storage-level filtering.
Converts Jupyter notebooks to plain Python scripts by translating `%%bash` and `!` magic cells into `subprocess.run` calls. This is a clean, self-contained CI utility that enables running notebook-based examples in headless test pipelines. It is directly extractable for any project that needs to execute notebooks in CI without a Jupyter kernel, and demonstrates a pragmatic pattern for bridging research-style notebooks with production test infrastructure.
PlatformEventBuilder constructs OpenTelemetry-compatible platform events for Ray Serve, capturing deployment lifecycle and autoscaling decisions. It isolates telemetry schema construction from business logic, making it a reusable pattern for any distributed system that needs structured, vendor-neutral operational events. The builder pattern keeps event fields consistent and testable without coupling to a specific tracing backend.
Implements the datasource v2 architecture for Parquet with row-group coalescing, footer type extraction, and file listing strategies. It demonstrates a clean separation between file discovery, chunking, and read path — a pattern reusable for any columnar storage adapter. The row-group coalescing logic is non-trivial and shows how to batch small row groups into efficient block reads, a common performance pitfall in data engineering.