$ show bentoml/BentoML
BentoML is a machine-learning model serving framework and deployment platform. In production it lets developers define inference services via a decorator-based SDK (`@bentoml.service`, `@bentoml.api`), package those services into versioned "Bentos" with locked Python dependencies, and build OCI container images or deploy to Kubernetes, EC2, or BentoCloud. The serving layer supports both HTTP (Starlette/Uvicorn) and gRPC, with adaptive batching, model runners, Prometheus metrics, OpenTelemetry tracing, and pluggable monitoring. A comprehensive CLI (`bentoml serve/build/containerize/deploy`) is the primary operational surface, while the `_bentoml_sdk` and `_bentoml_impl` packages provide the user-facing and internal implementation layers respectively. Architecturally, the project separates concern into model registry, service definition, packaging, server workers, and cloud clients. Model artifacts are persisted in a versioned local store via framework adapters for PyTorch, TensorFlow, Transformers, ONNX, scikit-learn, MLflow, XGBoost, LightGBM, and others. Data flows from trained model artifacts into the registry, get referenced by service code, get bundled by `bentoml build` into self-contained Bento packages, and are served by Circus-managed worker processes. The core request path involves serialization, batching, resource allocation, and process supervision, all implemented in internal modules with generally clean abstractions. The repository is mid-migration from a legacy `bentoml.legacy.Service` API to a newer `_bentoml_sdk`, which creates visible duplication: two HTTP client stacks, multiple protobuf versions, and deprecation shims. This is a real maintenance cost but not a functional blocker; the modern SDK is actively developed, documented, and tested. The repo is clearly an application/framework rather than a data-research project; research-like artifacts are limited to test fixtures, training examples, and ML framework integration tests.
The LLM identified 5 files worth highlighting for reuse:
Implements the CORK dynamic batching dispatcher, which aggregates concurrent requests into batches under configurable max batch size and max latency constraints. It is self-contained and can be reused in any async Python service that needs latency-aware request batching.
Provides a secure import/export abstraction for model and Bento artifacts using fsspec, with explicit validation of archive member paths, symlink safety, and directory boundary checks. This is a reusable security pattern for any code that extracts untrusted tar/zip archives.
A small utility that defers heavy third-party imports (numpy, torch, tensorflow, pandas) until first use while preserving type hints and raising friendly errors for missing optional dependencies. This pattern is directly extractable for any Python library with optional extras.
Defines the DataContainer abstraction for converting model payloads between batched and unbatched forms, with a typed registry (AutoContainer) and PEP 574 pickle-based serialization. It cleanly separates runner input/output handling from serving logic and can be adapted for other inference or batch-processing frameworks.
Generates Dockerfiles from a small set of base-image Jinja2 templates, parameterizing CUDA, Python, conda, and OS variants. The template-driven approach is reusable for any project that needs to produce multi-variant container images from metadata.