$ show deepseek-ai/DeepSeek-OCR
DeepSeek-OCR is an inference repository for a multimodal OCR model that combines vision encoders (SAM and CLIP) with a large language model (DeepSeek-V2/V3) to perform optical character recognition with layout awareness, supporting native and dynamic resolution processing. The system provides two inference paths: a Hugging Face Transformers integration for simple usage and a custom vLLM integration for high-throughput batched inference, including PDF processing and evaluation benchmarks. The architecture is a monolithic model definition with separate modules for vision encoding (deepencoder), image preprocessing (process), and vLLM glue code, all tightly coupled to the specific model architecture. The codebase is functional and demonstrates competent engineering for research-grade inference, but lacks production hardening, documentation, and consistent coding standards.
The LLM identified 3 files worth highlighting for reuse:
The DeepseekOCRForCausalLM class implements a vLLM-compatible multimodal model that fuses features from a SAM ViT and a CLIP vision encoder via a projector, then passes them into a DeepSeek language model. It demonstrates a clean pattern for registering custom multimodal processors with vLLM's registry, handling pixel-to-embedding conversion with dynamic cropping, and loading weights with prefix remapping. This pattern is reusable for integrating any vision encoder + LLM combination into vLLM.
A custom transformers LogitsProcessor that suppresses n-gram repetition within a sliding window, with optional whitelist token IDs. It is self-contained, well-structured, and can be reused in any text generation pipeline to reduce repetition without modifying the model. The window-based search is efficient for long sequences.
The MlpProjector class supports multiple projection types (linear, MLP, downsample, hybrid split) for mapping vision features to LLM embedding space. It is modular and configurable via an easydict config, making it reusable as a generic feature projector in multimodal architectures. The code handles various fusion strategies cleanly.