$ show tencentmusic/page-eyes-agent
PageEyes Agent is a production-grade Python framework for natural-language-driven UI automation, published on PyPI as `page-eyes` (v1.3.9). It provides LLM agents that perceive screens and execute actions (click, input, swipe, assert) across Web (Playwright), Android (ADB), iOS (WebDriverAgent), HarmonyOS (hdc), and Electron (CDP). The core architecture separates a PlanningAgent that decomposes user instructions into atomic steps from per-platform execution agents that call tools through Pydantic AI's tool-calling framework. Screen perception is delegated either to a vision-language model in lightweight mode or to the bundled OmniParser2 service in full mode. Each run produces a self-contained HTML report with step logs, screenshots, element bounding-box overlays, and video, which can be uploaded to COS or MinIO. The repo is hybrid: it is primarily an application framework but embeds a significant ML-serving sub-project, OmniParser2, which is a FastAPI microservice running PaddleOCR for text detection, a fine-tuned YOLOv11n for icon detection, a LoRA-fine-tuned Florence-2 model for icon captioning, and a YOLO overlay detector for popups/loading states, with optional Milvus-based image caching. Model weights and tokenizer artifacts (including a ~96k-token BPE merges file) are committed to the repository, and the docs include detailed articles on Florence-2 LoRA fine-tuning and YOLO overlay detection, giving the repo a data-research flavor alongside its application character. The codebase shows genuine engineering skill: a generic Device abstraction with factory methods, a storage strategy pattern (COS/MinIO/Base64), a reusable skills system (e.g., gui-popup-handler), and an agent loop with planning/execution/reflection separation. CI/CD, tests, a changelog, and mkdocs documentation are present. Weaknesses include heavy ML dependencies with override conflicts, a committed binary blob (yadb), code duplication between the main package and OmniParser2, and coupling to Tencent-internal infrastructure (mirror URLs, email addresses, platform names), which limits external portability.
The LLM identified 4 files worth highlighting for reuse:
BoxesHandler performs post-processing on detected UI element boxes: overlap removal, spatial sorting into reading order, and label annotation that avoids text overlap. It is an isolated, self-contained CV utility that could be extracted into any OCR or object-detection pipeline without modification, demonstrating non-obvious skill in deterministic box deduplication and layout ordering.
StorageClient with interchangeable CosStrategy, MinioStrategy, and Base64Strategy backends for uploading screenshots, videos, and reports. The strategy pattern keeps upload logic decoupled from the agent framework, making it directly reusable in any Python project needing pluggable artifact storage with different providers.
A generic Device[ClientT, DeviceT] dataclass with a create() async factory method that abstracts client construction and lifecycle per platform (Web, Android, iOS, HarmonyOS, Electron). It demonstrates a clean pattern for adding new platform backends with minimal ripple effects and could be adapted to other multi-platform driver or device-control projects.
AsyncImageVectorStorage computes CLIP embeddings for screenshots and performs Milvus-based similarity lookup to cache repeated UI parsing results. This solves a recurring problem — avoiding redundant model inference on visually identical screenshots — as a self-contained, async-friendly component that could be reused in any vision pipeline needing perceptual deduplication or retrieval.