ProjectSend is a self-hosted secure file sharing platform built as a modular Laravel application with an Inertia.js + React frontend. The codebase is organized into domain modules — Files, Clients, Groups, Identity, Comments, Notifications, Audit, Platform, and Api — each containing controllers, services, policies, models, and event hooks. Staff members upload and share files with clients or groups, manage roles and permissions, enforce download limits, expire or version files, and monitor activity through a detailed audit log. Clients access their own portal with four switchable themes (default, compact, drive, gallery), and public share links support additional presentation themes with pagination, previews, and controlled downloads. The application is a monolith with strong modular boundaries rather than microservices. Authentication supports local accounts, LDAP, social login/SSO via OIDC, two-factor authentication, CAPTCHA, and password policies. A REST API protected by Sanctum tokens offers scoped abilities, RFC 7807 problem details, polling-based pagination, and an OpenAPI contract that is drift-tested against the live route surface. Background jobs handle zip downloads, email digests, thumbnails, file cleanup, and scheduled maintenance. The system persists settings, activity logs, notification preferences, and upload metadata in a relational database, while file payloads can be stored on local disks, S3-compatible storage, or Google Cloud Storage. The code is notably security-conscious: secrets are encrypted at rest, old input flashing excludes credentials, CSRF cookie naming is customized to avoid collisions, download paths are never exposed in API responses, and regression tests cover disclosure boundaries, permission scoping, token revocation, and account erasure. The test suite is extensive, using Pest feature tests to exercise permissions, file lifecycle edge cases, version chains, comment visibility, API contracts, and authentication flows. The architecture favors event-based extension points and registry patterns, making modules independently testable and partially reusable, though the project remains an application rather than a framework or library. Overall this is a well-crafted, actively maintained application with clean engineering practices, clear separation of concerns, and unusually thorough security regression coverage. Its main complexity is the large surface area of interchangeable themes, editions, authentication backends, and storage drivers, which creates some ongoing drift risk, but current CI, recent commits, and dependency management show the project is keeping pace with its ecosystem.
Onedump is a Go-based CLI tool for database administration, specializing in backup, restore, and migration workflows. It supports MySQL (with a zero-dependency native dumper and mysqldump) and PostgreSQL (via pg_dump), and can persist backups to local files, AWS S3, Google Drive, Dropbox, or SFTP endpoints. The architecture follows a clean pipeline pattern: YAML configuration drives concurrent job execution, each job uses a dumper interface to stream database content through gzip compression and fan-out to multiple storage backends. Beyond core backup, it includes advanced features: MySQL binlog streaming to S3 with checksum-based deduplication, point-in-time restore via binlog replay, a MySQL slow query log parser with query masking and deduplication, and a resumable/concurrent SFTP file transfer utility with exponential backoff. The codebase is well-structured, heavily tested, and uses modern Go idioms, making it suitable as a reference for building database-adjacent infrastructure tooling.
This is a standalone Rust web server that implements Turborepo's remote caching API. It acts as a sidecar process, typically started inside CI pipelines (GitHub Actions, GitLab, Kubernetes), that forwards artifact storage and retrieval requests to any S3-compatible object store (AWS S3, Cloudflare R2, RustFS). The server exposes a small set of REST endpoints: health check, artifact upload/download/head, event logging, and a team artifact listing stub. Artifacts are streamed directly from the HTTP body to S3 (and vice versa) without buffering into memory. Optional Bearer-token authentication uses constant-time comparison, and the server is instrumented with OpenTelemetry for distributed traces and system-level metrics (CPU, memory) that are exportable via OTLP. The repository also includes a GitHub Action wrapper (Node.js) that manages the binary lifecycle—startup health polling, graceful shutdown, and log collection—and a multi-stage Docker build that produces statically linked Linux binaries and a universal macOS binary. The architecture is a monolith (single binary) built with actix-web, with clear separation into modules for settings, storage, authentication, routes, and telemetry. All artifacts are scoped under a team prefix derived from the Turborepo client's query parameters.
This repository implements Zenko CloudServer, an open-source, S3-compatible object storage server written in Node.js. It provides a single AWS S3 API endpoint that can route data to multiple backends including local filesystem, memory, AWS S3, Azure Blob, GCP, and Scality's own storage. The server handles the full S3 protocol: bucket and object CRUD, versioning, multipart uploads, lifecycle policies, replication, server-side encryption (SSE-S3, SSE-KMS), object lock (retention/legal hold), bucket policies, CORS, website hosting, and rate limiting. It integrates with external services for authentication (Vault), key management (KMIP), metadata storage (MongoDB), metrics (Utapi/Scuba), and replication/ingestion (Backbeat). The architecture is a multi-process Node.js application with separate daemons for the S3 API server, data server, metadata server, and passthrough file server. Configuration is dynamic via environment variables and a JSON config file, with support for live overlay updates through a WebSocket management channel. The codebase is organized into modular API handlers in `lib/api/`, reusable utilities in `lib/api/apiUtils/`, and backend abstractions for data, metadata, and authentication. Extensive functional and unit tests cover S3 API compliance across all supported backends. Key data flows: incoming S3 requests are authenticated via V4 signing, authorized against bucket policies and ACLs, then routed to the appropriate data backend based on location constraints. Metadata operations go through a wrapper that supports in-memory, file, or MongoDB backends. Rate limiting uses a distributed token bucket algorithm with Redis coordination. Metrics are pushed to Utapi and exposed via Prometheus. The system is designed for high availability with clustering and health checks.
This repository is a storage provider plugin for Synapse, the Matrix homeserver, that stores media in S3-compatible object storage. It is a Python package installed as a module and loaded by Synapse at runtime. The core is a single module, s3_storage_provider.py, which implements Synapse's storage provider API and translates local media operations into S3 GET/PUT/HEAD/DELETE calls via boto3. It supports SSE-C encryption, custom S3 endpoints, and a local cache. A companion CLI script, s3_media_upload, migrates existing local media to S3. Architecturally, the system is a library/plugin rather than a standalone service. Synapse invokes the provider's methods to store and retrieve media; the provider uses background threads (_S3DownloadThread, _S3UploadThread) to perform asynchronous operations with backpressure. The code uses threading locks and events to coordinate concurrent access, and includes tests, CI workflows, and release automation. The primary entry point is the S3StorageProvider class instantiated by Synapse; the module can also be run as a script for maintenance, and scripts/s3_media_upload provides a CLI. External dependencies include boto3 for S3 access, Synapse as the host application, and Twisted for tests. The repository is actively maintained with recent commits, dependabot updates, and GitHub Actions for CI and releases.
Attic is a self-hostable Nix binary cache server written in Rust. It provides a multi-tenant, globally deduplicated cache for Nix store paths, supporting both local filesystem and S3-compatible storage backends. The system consists of a server binary (`atticd`) that implements the Nix binary cache HTTP API and a client CLI (`attic`) for authentication, pushing store paths, and managing caches. Authentication is stateless via signed JWTs (RS256/HS256) with fine-grained permissions per cache. The server uses content-defined chunking (FastCDC) to split NAR archives into variable-sized chunks, enabling deduplication at both the NAR and chunk levels. Compression (brotli, zstd, xz) is applied during upload with simultaneous hash computation for integrity verification. Garbage collection supports per-cache retention periods and cascading deletion from local caches to global chunk storage. The codebase is organized as a Cargo workspace with four crates: `attic` (core library with chunking, hashing, Nix store FFI), `server` (HTTP server, database, storage, garbage collection), `client` (CLI with push, watch-store, cache management), and `token` (JWT creation and verification). The architecture is modular, with clear separation between API handlers, database entities, storage backends, and authentication. While the project is still in early prototype stage with many TODOs and some fragile FFI bindings to Nix's C++ store library, it demonstrates solid engineering practices with reusable components like streaming hash readers, compression streams, and a custom serde serializer for Nix manifest format.
HydroShare is a Django-based web application for sharing, managing, and publishing hydrologic data and models. It provides a rich resource management system with support for various file types (NetCDF, GeoTIFF, timeseries, etc.), fine-grained access control (groups, communities, privileges), metadata extraction, and discovery via a Vue.js frontend. The system uses PostgreSQL, Redis, Celery, MinIO (S3-compatible storage), and MongoDB for indexing. It exposes REST APIs for programmatic access and includes background tasks for bag creation, metadata extraction, and DOI registration. While primarily an application, it incorporates data engineering patterns for scientific metadata handling and file type aggregation.