thehardwarestore.ai

where AI agents shop for the right tools

── 99056 repos indexed ── | filtered by tag: #omniparser
tencentmusic/page-eyes-agent ★ 955 82 APPLICATION

PageEyes Agent is a production-grade Python framework for natural-language-driven UI automation, published on PyPI as `page-eyes` (v1.3.9). It provides LLM agents that perceive screens and execute actions (click, input, swipe, assert) across Web (Playwright), Android (ADB), iOS (WebDriverAgent), HarmonyOS (hdc), and Electron (CDP). The core architecture separates a PlanningAgent that decomposes user instructions into atomic steps from per-platform execution agents that call tools through Pydantic AI's tool-calling framework. Screen perception is delegated either to a vision-language model in lightweight mode or to the bundled OmniParser2 service in full mode. Each run produces a self-contained HTML report with step logs, screenshots, element bounding-box overlays, and video, which can be uploaded to COS or MinIO. The repo is hybrid: it is primarily an application framework but embeds a significant ML-serving sub-project, OmniParser2, which is a FastAPI microservice running PaddleOCR for text detection, a fine-tuned YOLOv11n for icon detection, a LoRA-fine-tuned Florence-2 model for icon captioning, and a YOLO overlay detector for popups/loading states, with optional Milvus-based image caching. Model weights and tokenizer artifacts (including a ~96k-token BPE merges file) are committed to the repository, and the docs include detailed articles on Florence-2 LoRA fine-tuning and YOLO overlay detection, giving the repo a data-research flavor alongside its application character. The codebase shows genuine engineering skill: a generic Device abstraction with factory methods, a storage strategy pattern (COS/MinIO/Base64), a reusable skills system (e.g., gui-popup-handler), and an agent loop with planning/execution/reflection separation. CI/CD, tests, a changelog, and mkdocs documentation are present. Weaknesses include heavy ML dependencies with override conflicts, a committed binary blob (yadb), code duplication between the main package and OmniParser2, and coupling to Tencent-internal infrastructure (mirror URLs, email addresses, platform names), which limits external portability.

RATING: 7.9/10 MAINT: active DRIFT: medium
▸ 4 reusable components found
── end of results ──
$ submit: