$ show SeldonIO/seldon-core
Seldon Core v2 is a production-grade, Kubernetes-native MLOps platform for deploying, scaling, and managing machine learning inference at scale. The system is architected as a set of microservices: a Kubernetes operator manages CRDs (Model, Pipeline, Server, Experiment), a centralized scheduler coordinates model placement and scaling across server replicas, an agent sidecar runs on each inference server to handle model loading/unloading, caching, and reverse proxying, and a Kafka Streams-based dataflow engine executes multi-step inference pipelines with joins, batching, and triggers. Inference requests enter via an Envoy proxy that routes to model servers or pipeline gateways, with Kafka as the underlying data plane for pipeline flows. The platform supports the V2 inference protocol, A/B experiments, explainability, drift detection, and OpenAI API translation. It integrates with MLServer, Triton Inference Server, and custom runtimes, and provides a CLI for lifecycle management.
The LLM identified 6 files worth highlighting for reuse:
Implements per-item locking around an LRU cache, allowing concurrent access with eviction and reload transactions. The `CacheTransactionManager` provides a `DoWithLock` method that acquires a lock on a cache key, performs the operation, and retries if the key was evicted. This is a clean, reusable concurrency pattern for any cache that needs safe eviction and reload semantics.
Provides a generic vector-clock-based conflict resolution framework for aggregating status updates from distributed components (e.g., pipeline and model status from multiple agents). Includes a `ConflictResolver` that merges status vectors and detects conflicts, with specialized implementations for pipelines and models. This pattern is reusable in any distributed system requiring eventual consistency with conflict detection.
A lightweight health check framework that allows registering multiple checks (startup, liveness, readiness) and exposes them via an HTTP server. The `Manager` aggregates results and supports custom check implementations. This is a self-contained, reusable component for any Go service needing Kubernetes health probes.
Implements a `CertificateStore` that manages TLS certificates with both file-based and Kubernetes secret-based backends, supporting hot reload via file watchers or informers. The store provides a uniform interface for getting current certificates, and the TLS configuration utility (`util.go`) selects the appropriate security protocol (PLAINTEXT, TLS, mTLS) based on environment variables. This is a reusable component for any Go service needing dynamic TLS configuration.
Translates OpenAI Chat Completions API requests to the Seldon OIP (Open Inference Protocol) format and back, including SSE streaming support, gzip handling, and structured output extraction. The translator pattern (`translator/base.go`) defines a generic interface for bidirectional conversion, making it easy to add new API translators. This is a clean, reusable pattern for building API gateways that translate between external and internal protocols.
A wrapper around `sync.Map` that adds a fast `Len()` method using an atomic counter incremented on Store and decremented on Delete. This solves the common problem of not having a built-in length function for `sync.Map` while maintaining concurrent safety. The implementation is minimal, well-tested, and directly reusable in any Go project requiring a concurrent map with length tracking.