mai-ui-agents

Scale GUI agents to real-world complexity via extended action space (user interaction, tool calls) and device-cloud collaboration. Online RL supports 500+ parallel environments with asynchronous handl

SKILL.md

GitHub

Overview

MAI-UI addresses critical limitations in existing GUI agents through pragmatic design choices: extended actions enable richer interactions, device-cloud collaboration preserves privacy, and online RL at scale improves agentic reasoning.

Core Technique

Extended Action Space: Beyond pure UI operations, agents can request clarification and invoke tools.

class ExtendedActionSpace:
    # Actions: click, type, scroll, + new ones
    user_ask = "ask_user"      # Request clarification
    mcp_call = "mcp_call"      # Use external tools

Device-Cloud Collaboration: Local agent monitors alignment; cloud only handles complex cases.

class HybridAgent:
    def should_handoff_to_cloud(self, trajectory, instruction):
        deviation = self.alignment_monitor.evaluate(trajectory)
        if deviation > threshold:
            return True  # Handoff to cloud
        return False  # Continue locally

Key Performance

  • 73.5% grounding (ScreenSpot-Pro)
  • 76.7% mobile navigation (AndroidWorld)
  • 41.7% real-world tasks (MobileWorld)
  • 500+ parallel environments

References

  • Extended action space design
  • Device-cloud collaboration architecture
  • Online RL with asynchronous parallelism