$ show X-PLUG/MobileAgent
This repository is a comprehensive family of GUI automation agents (Mobile-Agent series) developed by Tongyi Lab, Alibaba Group. It provides end-to-end solutions for automating interactions across mobile (Android), desktop (Windows/macOS/Linux), and web platforms using vision-language models. The system perceives GUI elements through screenshots, grounds natural language instructions to actionable commands (click, type, scroll), and executes tasks via platform-specific controllers. The architecture includes multiple agent versions (v1-v3.5, E, UI-S1, PC-Agent) with shared core modules for perception, action parsing, and controller interfaces, alongside training frameworks (VERL) and evaluation suites (Android World, MiniWoB, Mind2Web). Key entry points are version-specific run scripts (e.g., run_ma35.py) and API services for remote agent control. The system relies on external dependencies like ADB for Android, PyAutoGUI for desktop, Playwright for web, and VLMs (Qwen series) for reasoning.
The LLM identified 4 files worth highlighting for reuse:
Defines an abstract Controller class with platform-specific implementations (AndroidController, DesktopController, WebController) for device interaction (screenshot, tap, type, slide, back, home). Demonstrates clean abstraction for GUI automation, isolating platform details while providing a uniform interface for agent logic. Easily reusable in other automation frameworks.
Standardizes the action space across agent versions (click, input_text, scroll, etc.) and provides robust JSON parsing/validation utilities for LLM outputs. Eliminates ad-hoc action formatting, ensuring consistency between perception and execution layers. The pattern is self-contained with minimal dependencies, making it portable to other agent systems.
Calculates effective sequence lengths for multimodal inputs (text + image) in Qwen-VL models, accounting for vision token encoding. Solves the non-obvious problem of aligning text and vision sequences in transformer models, preventing truncation errors. Encapsulated utility with clear inputs/outputs, reusable across similar multimodal architectures.
Implements a modular reward manager for RLHF training that processes rule-based or model-based rewards. Demonstrates clean separation of reward computation from training loop, supporting various reward functions through a common interface. The abstract base pattern allows easy extension for custom rewards while maintaining training stability.