mai-ui-agents
Scale GUI agents to real-world complexity via extended action space (user interaction, tool calls) and device-cloud collaboration. Online RL supports 500+ parallel environments with asynchronous handl
SKILL.mdskills/skillxiv-v0.0.2-claude-opus-4.6/mai-ui-agents/SKILL.md
GitHubOverview
MAI-UI addresses critical limitations in existing GUI agents through pragmatic design choices: extended actions enable richer interactions, device-cloud collaboration preserves privacy, and online RL at scale improves agentic reasoning.
Core Technique
Extended Action Space: Beyond pure UI operations, agents can request clarification and invoke tools.
class ExtendedActionSpace:
# Actions: click, type, scroll, + new ones
user_ask = "ask_user" # Request clarification
mcp_call = "mcp_call" # Use external tools
Device-Cloud Collaboration: Local agent monitors alignment; cloud only handles complex cases.
class HybridAgent:
def should_handoff_to_cloud(self, trajectory, instruction):
deviation = self.alignment_monitor.evaluate(trajectory)
if deviation > threshold:
return True # Handoff to cloud
return False # Continue locally
Key Performance
- 73.5% grounding (ScreenSpot-Pro)
- 76.7% mobile navigation (AndroidWorld)
- 41.7% real-world tasks (MobileWorld)
- 500+ parallel environments
References
- Extended action space design
- Device-cloud collaboration architecture
- Online RL with asynchronous parallelism