$ show OpenNMT/CTranslate2
CTranslate2 is a high-performance inference engine for Transformer models, optimized for CPU and GPU deployment. It supports a wide range of architectures including encoder-decoder (BART, T5, NLLB, M2M-100, Whisper), decoder-only (LLaMA, GPT-2, Mistral, Gemma, Qwen), and encoder-only (BERT, DistilBERT, XLM-RoBERTa) models. The system provides a C++ library with Python bindings, a command-line tool (ct2-translator), and converters from popular frameworks like HuggingFace Transformers, Fairseq, Marian, OpenNMT-py/tf, and OPUS-MT. Key features include quantized inference (INT8/INT16/FP16/BF16/AWQ), flash attention, multi-GPU tensor parallelism, streaming generation, and token-level penalties. The architecture follows a modular design with device-agnostic operations, layer abstractions, and a model registry that allows plugging in new architectures without modifying the core engine. The codebase is actively maintained with frequent releases, extensive test coverage (C++ and Python), and CI pipelines for multiple platforms and backends.
The LLM identified 2 files worth highlighting for reuse:
Implements a configurable sequence decoding framework with strategy pattern (BeamSearch, GreedySearch) and a chain of responsibility for logits processors (repetition penalty, n-gram blocking, biased decoding, top-k/top-p sampling). The design cleanly separates search strategies from logit modification, enabling easy extension with custom processors or search variants. This module is reusable for any sequence generation project requiring advanced decoding algorithms with modular logic.
Thread-safe model replica pool that manages multiple model copies across devices for parallel inference. Provides queue-based job dispatch, worker affinity, and automatic cache clearing. The design is agnostic to model type and can be extracted for any model-serving scenario that requires high-throughput asynchronous inference with multiple replicas.