$ show replicate/cog
Cog is an open-source CLI and runtime for packaging machine learning models into production-ready Docker containers. It automatically generates HTTP inference servers with OpenAPI schema, handles GPU/CUDA/cuDNN compatibility, and provides a full weight management pipeline for large model artifacts. The project consists of a Go CLI (build, push, serve, predict, doctor, weights management), a Rust/PyO3 prediction server (coglet) with slot-based concurrency, a Python SDK for defining model interfaces (BaseRunner, Input, streaming), and a TypeScript playground UI. Key workflows include building OCI images from cog.yaml, pushing to registries, serving models via HTTP with SSE streaming, and managing model weights via content-addressed storage and lockfiles. The architecture is multi-language but cleanly separated, with well-defined protocols between components.
The LLM identified 4 files worth highlighting for reuse:
Implements a typestate-based permit slot for prediction concurrency. The PermitInUse, PermitIdle, and PermitPoisoned states are enforced at compile time via Rust's type system, ensuring safe state transitions and preventing slot leaks. This pattern could be extracted for any resource pool with strict lifecycle requirements.
A content-addressed file store with atomic writes via temp-file+rename, digest verification, context-cancellable reads, and concurrency-safe operations. The PutFile method drains readers and verifies digest/size. This is a self-contained storage primitive reusable for any idempotent caching or asset management system.
Defines a JSON-line IPC protocol between the Rust orchestrator and Python subprocess using serde-tagged enums for control and slot messages. Features inline vs spill input handling (MAX_INLINE_IPC_SIZE), streaming output chunks, metrics, and file outputs. The protocol is versioned and has comprehensive snapshot tests, making it a robust pattern for parent-child communication in multi-language runtimes.
A phased tree-sitter-based Python source code parser that extracts predictor class signatures, type annotations, imports, and module-scope constants for OpenAPI schema generation. It handles forward references, unions, choices, and decorators without executing user code. The multi-phase approach and AST utilities are reusable for any Python static analysis tool.