$ show zai-org/Open-AutoGLM
Open-AutoGLM (Phone Agent) is an AI-powered mobile automation framework that uses vision-language models to understand phone screens and execute user tasks through automated touch, swipe, and text input operations. The system works by repeatedly capturing device screenshots, sending them to a vision model (AutoGLM-Phone-9B or compatible API), parsing the model's decision into actionable commands, and executing those commands via platform-specific debugging tools (ADB for Android, HDC for HarmonyOS, WebDriverAgent for iOS). The architecture employs a device factory pattern to abstract platform differences behind a unified interface, an OpenAI-compatible model client with streaming and performance metrics, and a command-based action handler that supports tap, type, swipe, launch, back, home, long press, and takeover operations. The repository is structured as both a CLI application and a Python library, with comprehensive configuration for 50+ Android apps, 60+ HarmonyOS apps, and 100+ iOS apps, along with bilingual system prompts (Chinese/English), configurable timing parameters, and callback hooks for sensitive operation confirmation and manual intervention. Notable features include remote debugging support, verbose step-by-step execution with model thinking output, and integration with third-party model hosting services.
The LLM identified 5 files worth highlighting for reuse:
Abstract factory that transparently dispatches all device operations (screenshot, tap, swipe, type, launch, etc.) to the correct platform module (ADB, HDC, or iOS) based on a DeviceType enum. Each platform module provides the same function signatures, allowing the agent code to remain platform-agnostic. This component is directly reusable in any multi-platform phone automation project.
ModelClient wraps the OpenAI Python SDK with streaming support, real-time thinking/action parsing via marker detection (looking for 'do(action=' or 'finish(message=' in the stream), and automatic performance metrics generation (TTFT, thinking end time, total inference time). MessageBuilder provides structured system/user/assistant message creation with image embedding and context space optimization by removing images after use. The performance tracking pattern is directly reusable for any model-based agent pipeline.
ActionHandler implements a clean command dispatch pattern: it maps action names (Launch, Tap, Type, Swipe, Back, Home, etc.) to handler methods, converts relative coordinates (0-1000) to absolute pixels using screen dimensions, handles sensitive operation confirmation via callback, and manages keyboard switching for text input. The parse_action function uses AST-based parsing (not eval) for safe extraction of action arguments. The handler structure with ActionResult dataclass and callback injection is a reusable pattern for any command-execution system.
ADBConnection provides robust connection management for Android devices, supporting USB, WiFi, and remote TCP/IP connections. It includes device discovery via 'adb devices -l', connection/disconnection with timeout handling, TCP/IP mode enabling for wireless debugging, IP address extraction from 'ip route' output, server restart management, and structured DeviceInfo dataclass. The pattern of wrapping shell command invocations with structured result types and retry logic is directly reusable for any device tool abstraction.
Three-tier timing configuration system using nested dataclasses (ActionTimingConfig, DeviceTimingConfig, ConnectionTimingConfig) with automatic environment variable overrides in __post_init__. Provides a global TIMING_CONFIG singleton with get_timing_config()/update_timing_config() accessors. This pattern cleanly separates operational timing from business logic and allows runtime customization without code changes. Directly reusable in any project needing configurable delays.