m19
Kontrol mesin lokal & dunia fisik/simulasi: desktop macOS tanpa nyolong kursor, robotika NVIDIA (Isaac Sim/Omniverse), scene prep, dan mobility. Opsional, gated. Aksi yang nyentuh hardware/desktop
SKILL.mdskills/superagent/m19/SKILL.md
GitHubm19 — Desktop & Physical Control (NEW in v4.1)
Kontrol mesin lokal & dunia fisik/simulasi: desktop macOS tanpa nyolong kursor, robotika NVIDIA (Isaac Sim/Omniverse), scene prep, dan mobility. Opsional, gated. Aksi yang nyentuh hardware/desktop operator = punya risk gate sendiri (R9), terpisah dari governor dana.
| Capability | Surface | Risk |
|---|---|---|
| macOS native control (background) | AppleScript / osascript / Accessibility API |
desktop — R9 gate sekali |
| Robotics sim | Isaac Sim / Omniverse (USD + Python API) | sim only — aman |
| Scene preparation | USD scene assembly | aman |
| Mobility / movement | Isaac Lab RL / ROS2 bridge | sim default; real-robot = R9 |
1. macOS desktop control — TANPA nyolong kursor
Prinsip inti: background-first. Jangan pakai cliclick/PyAutoGUI yang ngerebut kursor fisik operator. Pakai jalur yang nge-drive app langsung:
# kirim aksi ke app spesifik via AppleScript — gak ganggu kursor/fokus operator
osascript -e 'tell application "Notes" to make new note with properties {body:"dari agent"}'
# UI scripting target window tertentu tanpa raise (butuh izin Accessibility)
osascript <<'EOF'
tell application "System Events"
tell process "Safari"
set frontmost to false -- jangan rebut fokus
perform action "AXPress" of button 1 of window 1
end tell
end tell
EOF
# Jadwalkan / keystroke ke app target tanpa memindahkan kursor global
import subprocess
def app_keystroke(app: str, keys: str):
"""Kirim keystroke ke app tertentu tanpa mindahin kursor fisik."""
script = f'tell application "System Events" to tell process "{app}" to keystroke "{keys}"'
subprocess.run(["osascript", "-e", script], check=True)
Aturan main (HARD):
- Sekali minta izin Accessibility + Automation (
System Settings → Privacy). Setelah granted → diam. - Default: target app by name,
frontmost: false. Janganactivatekecuali operator eksplisit minta. - JANGAN screenshot/keylog diam-diam. Screenshot lewat m15 multimodal hanya saat diminta.
- Aksi destruktif (hapus file, kirim email, post) → R9 gate sekali.
Reference helper: tools/desktop_control.py (background AppleScript runner + app targeting + dry-run).
2. NVIDIA Robotics — Isaac Sim / Omniverse
Isaac Sim = simulator robot di atas Omniverse (USD scene + PhysX + RTX). Drive via Python (omni.isaac.kit), headless di server GPU.
# Headless Isaac Sim (jalan di VPS GPU). Jangan buka GUI buat batch.
from omni.isaac.kit import SimulationApp
sim = SimulationApp({"headless": True}) # WAJIB pertama, sebelum import omni lain
from omni.isaac.core import World
from omni.isaac.core.robots import Robot
world = World(stage_units_in_meters=1.0)
world.scene.add_default_ground_plane()
robot = world.scene.add(Robot(prim_path="/World/robot", name="r1"))
world.reset()
for _ in range(240): # ~4s @ 60Hz
world.step(render=False) # render=False → cepat, no RTX cost
sim.close()
Pola: assemble scene (USD) → set robot articulation → step physics → record observasi. Headless + render=False = murah & cepat untuk training/eval. RTX render hanya untuk demo/video.
3. Scene preparation automation
Scene = file USD (.usd/.usda). Build programmatically, jangan drag-drop manual:
from pxr import Usd, UsdGeom, Gf
stage = Usd.Stage.CreateNew("scene.usda")
UsdGeom.Xform.Define(stage, "/World")
ground = UsdGeom.Cube.Define(stage, "/World/ground")
ground.AddTranslateOp().Set(Gf.Vec3d(0, 0, -0.5))
ground.GetSizeAttr().Set(10.0)
stage.GetRootLayer().Save()
tools/scene_prep.py = helper assembling USD scene (ground, lighting, asset placement, randomization) — dipakai bareng Isaac Sim (m19) dan juga referensi untuk render pipeline (m18). Domain randomization (lighting/pose/material) penting buat sim-to-real.
4. Mobility & physical movement
- Sim default: latih gerak via Isaac Lab (RL) atau scripted articulation di Isaac Sim. Aman, tak terbatas.
- Real robot: jembatan ROS2 (
rclpy) publish ke/cmd_veldll. Setiap perintah ke robot fisik = R9 gate (bisa nabrak/rusak). Default mulai dari sim, validasi, baru sim-to-real.
# ROS2 bridge — kirim velocity (real robot → R9 gate dulu)
import rclpy
from geometry_msgs.msg import Twist
def drive(linear=0.2, angular=0.0):
rclpy.init(); node = rclpy.create_node("agent_drive")
pub = node.create_publisher(Twist, "/cmd_vel", 10)
t = Twist(); t.linear.x = linear; t.angular.z = angular
pub.publish(t); node.destroy_node(); rclpy.shutdown()
5. Cosmos Physical AI & meta-actions (v4.1)
NVIDIA Cosmos = world foundation models buat Physical AI: generate data sintetik, simulasi, dan latih policy robot tanpa ribuan jam data dunia nyata.
Pipeline Physical AI:
scene_prep (USD) → Cosmos/Isaac (generate variasi + simulasi fisika)
→ policy training (Isaac Lab RL) → eval di sim → sim-to-real (gated)
- Data generation: Cosmos bikin variasi skenario (lighting, layout, objek) dari sedikit seed → dataset besar buat training.
- Policy training: latih di sim (murah, paralel, aman) → validasi → baru real.
- Object manipulation: latih grasp/manipulasi di sim, domain-randomize (lihat
scene_prep.randomize), transfer ke robot nyata.
Meta-actions — safety reflex (WAJIB untuk gerak fisik)
Robot yang gerak butuh refleks berhenti, bukan cuma "maju". Tiap perintah mobility dibungkus guard:
# meta-action: deselerasi/STOP saat risiko terdeteksi — mendahului perintah gerak
def safe_drive(linear, angular, risk_fn):
if risk_fn(): # obstacle dekat / kehilangan sensor / area terlarang
return stop() # STOP menang atas perintah apa pun
return drive(min(linear, V_MAX), angular) # clamp ke batas aman
Hierarki: STOP > deselerasi > perintah operator. Ini analog Operational Rails crypto (governor) tapi untuk dunia fisik — refleks keselamatan yang gak bisa di-bypass auto_confirm. Real-robot tetap R9 gate di atas ini.
6. Vision-Language-Action (VLA) & multimodal reasoning (v4.1)
VLA = satu loop: persepsi → reasoning → aksi motor. Kamera/video masuk, model paham scene + instruksi bahasa, keluarin aksi (gerak/grasp).
PERCEIVE (camera/video/sensor) → GROUND (objek, posisi, instruksi bahasa)
→ PLAN (langkah aksi) → ACT (motor command, lewat safe_drive/meta-action) → OBSERVE (loop)
- Scene understanding: deteksi objek + relasi spasial + affordance ("gelas bisa diangkat").
- Object manipulation: dari instruksi bahasa ("ambil botol merah") → grasp pose → eksekusi (latih di sim dulu).
- Multimodal reasoning: gabung text + vision + audio + action di satu agent (robotics, industrial automation, healthcare). Input multimodal → keputusan tunggal yang konsisten.
- Persepsi visual/audio sisi input pakai m15 (multimodal: vision/Whisper); m19 nambah action di ujungnya.
Safety tetap di atas segalanya: aksi motor apa pun lewat meta-action (§5) — STOP/deselerasi menang atas perintah VLA. Model salah baca scene → refleks keselamatan yang nahan, bukan eksekusi buta. Real-robot = R9 gate.
Catatan
- Isaac Sim/Cosmos butuh GPU NVIDIA + driver; deploy headless via m2, drive jarak jauh.
- macOS control = background-first, no cursor steal — itu kontrak utama skill ini.
- Real-robot & destructive desktop = R9 gate. Sim & background app-drive = bebas.
- Combo: m19 + m2 (deploy GPU box) + m17 (planner buat multi-step scene/training) + x6 (debug sim).