$ show microsoft/OmniParser
The OmniParser repository provides a system for parsing GUI screenshots into structured elements (text and icons) to enable vision-based GUI agents. It combines object detection (YOLO), OCR (PaddleOCR), and vision-language models (Florence2) to detect and describe screen elements, outputting bounding boxes and structured data. The system includes a FastAPI server for parsing as a service, Gradio and Streamlit demos for interactive use, and agent frameworks that integrate with LLMs (like GPT-4o, Claude) to perform automated computer tasks within a Windows VM environment (via the OmniBox component). The architecture is modular, with core parsing utilities in the `util` directory, agent tools and loops in `omnitool/gradio`, and VM management in `omnitool/omnibox`. Key reusable components include the model loading and processing utilities (`util/utils.py`), a tool abstraction pattern for defining agent actions (`omnitool/gradio/tools/`), and a visualization utility for annotating screenshots (`util/box_annotator.py`). The system demonstrates non-obvious skill in integrating multiple ML models for comprehensive screen understanding and providing a standardized interface for agent-LLM-tool interaction. While actively maintained (last commit July 2026), the repository shows signs of technical drift, including dependency version pinning that may become outdated and deprecation warnings in the demo notebook (e.g., timm layers, Florence2 GenerationMixin). The reliance on external services (Hugging Face models, LLM APIs) and the complexity of the Windows VM setup also introduce maintenance considerations. Nonetheless, the codebase delivers on its promise of enabling vision-based GUI agents and offers several patterns reusable in similar projects.
The LLM identified 4 files worth highlighting for reuse:
Provides self-contained functions for loading YOLO, Florence2, and PaddleOCR models, performing OCR, and generating screen parsing outputs. This module abstracts complex model inference into an easy-to-use interface, making it reusable in any project requiring GUI element detection and understanding.
Defines a ToolCollection class for managing agent tools, along with BaseAnthropicTool and standardized ToolResult/ToolFailure classes. This pattern enables consistent tool definition and execution in LLM-driven agents, and can be extracted for use in other agent frameworks.
Implements a FastAPI server that loads the OmniParser model and exposes a /parse/ endpoint for screen parsing. This serves as a clean, reusable pattern for serving any ML model via a REST API, with model loading handled in a startup event.
Contains functions to annotate images with bounding boxes and labels, tailored for screen parsing output. This utility is generally useful for any computer vision project requiring visualization of detection results, and is decoupled from the core parsing logic.