$ show yomorun/yomo
This Rust codebase is an LLM inference proxy and tool-routing gateway. It exposes an OpenAI-compatible HTTP API (`/v1/chat/completions`, `/v1/models`) and forwards requests to multiple upstream LLM providers (OpenAI, Anthropic via direct/Bedrock, Vertex AI, vLLM DeepSeek, TokenHub) through a unified Provider trait. A QUIC server (s2n-quic, TLS 1.3) accepts connections from serverless tools written in Go or Node.js; the agent loop performs multi-round tool calling, invokes registered tools, and streams results back in SSE format. A separate model API proxy layer handles non-chat endpoint shapes (messages, responses, generateContent) with usage extraction and injection. Architecturally, the system is a Rust CLI binary (`yomo`) with three subcommands: `serve` starts the HTTP API and QUIC zipper, `init` scaffolds serverless tool projects, and `run` launches a tool subprocess and connects it to the zipper. Tool registration is done at runtime over QUIC and stored in-memory, so state is lost on restart. The HTTP layer is Axum, auth is bearer-token, and observability uses OpenTelemetry. The serverless execution model is subprocess-based: it creates temporary directories, runs `npm install`/`go mod tidy`, and exchanges metadata over stdout. Reusable engineering exists in the framing protocol (`io`), streaming SSE mappers, usage aggregation logic, TLS configuration, and provider error separation. However, the provider implementations are duplicated and brittle to upstream API drift; hard-coded model limits, ephemeral tool state, and near-empty serverless stubs reduce long-term maintainability. No integration tests against real upstream APIs were found in the reviewed surface.
The LLM identified 4 files worth highlighting for reuse:
Implements a length-prefixed frame protocol for reliable message exchange over any async byte stream. It is self-contained and can be lifted into any Rust network service to delimit messages over QUIC or TCP without stream fragmentation or partial-read bugs.
Pluggable mapper that translates provider-specific streaming chunks into OpenAI-compatible SSE events. The abstraction is reusable for any LLM backend that needs to speak the OpenAI chat-completions streaming dialect while keeping upstream differences isolated.
Defines EndpointUsage with conversions and aggregation logic from provider payloads to the OpenAI usage shape, plus an async handler trait for external billing or observability systems. The conversion patterns can be extracted into a generic usage-normalization layer for LLM proxies.
Builds server and client TLS configurations with optional mutual TLS, custom CA support, dev certificates, and the `yomo-v2` ALPN protocol. The setup logic is portable to other Rust QUIC/TLS applications needing mTLS without reimplementing certificate loading and validation.