$ show scality/cloudserver
This repository implements Zenko CloudServer, an open-source, S3-compatible object storage server written in Node.js. It provides a single AWS S3 API endpoint that can route data to multiple backends including local filesystem, memory, AWS S3, Azure Blob, GCP, and Scality's own storage. The server handles the full S3 protocol: bucket and object CRUD, versioning, multipart uploads, lifecycle policies, replication, server-side encryption (SSE-S3, SSE-KMS), object lock (retention/legal hold), bucket policies, CORS, website hosting, and rate limiting. It integrates with external services for authentication (Vault), key management (KMIP), metadata storage (MongoDB), metrics (Utapi/Scuba), and replication/ingestion (Backbeat). The architecture is a multi-process Node.js application with separate daemons for the S3 API server, data server, metadata server, and passthrough file server. Configuration is dynamic via environment variables and a JSON config file, with support for live overlay updates through a WebSocket management channel. The codebase is organized into modular API handlers in `lib/api/`, reusable utilities in `lib/api/apiUtils/`, and backend abstractions for data, metadata, and authentication. Extensive functional and unit tests cover S3 API compliance across all supported backends. Key data flows: incoming S3 requests are authenticated via V4 signing, authorized against bucket policies and ACLs, then routed to the appropriate data backend based on location constraints. Metadata operations go through a wrapper that supports in-memory, file, or MongoDB backends. Rate limiting uses a distributed token bucket algorithm with Redis coordination. Metrics are pushed to Utapi and exposed via Prometheus. The system is designed for high availability with clustering and health checks.
The LLM identified 4 files worth highlighting for reuse:
Implements the Generic Cell Rate Algorithm (GCRA) for rate limiting. This is a self-contained module that calculates whether a request should be allowed based on a token bucket model. It can be extracted and reused in any Node.js application requiring precise, distributed rate limiting without external dependencies beyond basic math.
A Node.js Transform stream that verifies AWS Signature V4 for chunked uploads in real-time. It reconstructs the canonical request from streaming chunks, validates the signature, and emits the verified payload. This is a non-trivial streaming authentication pattern that could be reused in any service needing to verify AWS SigV4 chunked uploads without buffering the entire request.
A Redis Lua script that atomically grants tokens for the rate limiter. It implements the GCRA state management in Redis, handling token bucket refill and consumption in a single atomic operation. This script is self-contained and can be reused in any Redis-backed rate limiting system, demonstrating skill in distributed systems and Lua scripting for Redis.
Provides a dynamic configuration overlay system that applies patches to the running configuration via WebSocket push. It caches overlays, validates version ordering, and persists them to the metadata store. This pattern is reusable for any application needing live configuration updates with rollback safety, demonstrating a clean separation between base config and overlays.