m19

Kontrol mesin lokal & dunia fisik/simulasi: desktop macOS tanpa nyolong kursor, robotika NVIDIA (Isaac Sim/Omniverse), scene prep, dan mobility. Opsional, gated. Aksi yang nyentuh hardware/desktop

SKILL.md

GitHub

m19 — Desktop & Physical Control (NEW in v4.1)

Kontrol mesin lokal & dunia fisik/simulasi: desktop macOS tanpa nyolong kursor, robotika NVIDIA (Isaac Sim/Omniverse), scene prep, dan mobility. Opsional, gated. Aksi yang nyentuh hardware/desktop operator = punya risk gate sendiri (R9), terpisah dari governor dana.

Capability Surface Risk
macOS native control (background) AppleScript / osascript / Accessibility API desktop — R9 gate sekali
Robotics sim Isaac Sim / Omniverse (USD + Python API) sim only — aman
Scene preparation USD scene assembly aman
Mobility / movement Isaac Lab RL / ROS2 bridge sim default; real-robot = R9

1. macOS desktop control — TANPA nyolong kursor

Prinsip inti: background-first. Jangan pakai cliclick/PyAutoGUI yang ngerebut kursor fisik operator. Pakai jalur yang nge-drive app langsung:

# kirim aksi ke app spesifik via AppleScript — gak ganggu kursor/fokus operator
osascript -e 'tell application "Notes" to make new note with properties {body:"dari agent"}'

# UI scripting target window tertentu tanpa raise (butuh izin Accessibility)
osascript <<'EOF'
tell application "System Events"
  tell process "Safari"
    set frontmost to false   -- jangan rebut fokus
    perform action "AXPress" of button 1 of window 1
  end tell
end tell
EOF
# Jadwalkan / keystroke ke app target tanpa memindahkan kursor global
import subprocess
def app_keystroke(app: str, keys: str):
    """Kirim keystroke ke app tertentu tanpa mindahin kursor fisik."""
    script = f'tell application "System Events" to tell process "{app}" to keystroke "{keys}"'
    subprocess.run(["osascript", "-e", script], check=True)

Aturan main (HARD):

  • Sekali minta izin Accessibility + Automation (System Settings → Privacy). Setelah granted → diam.
  • Default: target app by name, frontmost: false. Jangan activate kecuali operator eksplisit minta.
  • JANGAN screenshot/keylog diam-diam. Screenshot lewat m15 multimodal hanya saat diminta.
  • Aksi destruktif (hapus file, kirim email, post) → R9 gate sekali.

Reference helper: tools/desktop_control.py (background AppleScript runner + app targeting + dry-run).

2. NVIDIA Robotics — Isaac Sim / Omniverse

Isaac Sim = simulator robot di atas Omniverse (USD scene + PhysX + RTX). Drive via Python (omni.isaac.kit), headless di server GPU.

# Headless Isaac Sim (jalan di VPS GPU). Jangan buka GUI buat batch.
from omni.isaac.kit import SimulationApp
sim = SimulationApp({"headless": True})          # WAJIB pertama, sebelum import omni lain
from omni.isaac.core import World
from omni.isaac.core.robots import Robot

world = World(stage_units_in_meters=1.0)
world.scene.add_default_ground_plane()
robot = world.scene.add(Robot(prim_path="/World/robot", name="r1"))
world.reset()
for _ in range(240):                              # ~4s @ 60Hz
    world.step(render=False)                      # render=False → cepat, no RTX cost
sim.close()

Pola: assemble scene (USD) → set robot articulation → step physics → record observasi. Headless + render=False = murah & cepat untuk training/eval. RTX render hanya untuk demo/video.

3. Scene preparation automation

Scene = file USD (.usd/.usda). Build programmatically, jangan drag-drop manual:

from pxr import Usd, UsdGeom, Gf
stage = Usd.Stage.CreateNew("scene.usda")
UsdGeom.Xform.Define(stage, "/World")
ground = UsdGeom.Cube.Define(stage, "/World/ground")
ground.AddTranslateOp().Set(Gf.Vec3d(0, 0, -0.5))
ground.GetSizeAttr().Set(10.0)
stage.GetRootLayer().Save()

tools/scene_prep.py = helper assembling USD scene (ground, lighting, asset placement, randomization) — dipakai bareng Isaac Sim (m19) dan juga referensi untuk render pipeline (m18). Domain randomization (lighting/pose/material) penting buat sim-to-real.

4. Mobility & physical movement

  • Sim default: latih gerak via Isaac Lab (RL) atau scripted articulation di Isaac Sim. Aman, tak terbatas.
  • Real robot: jembatan ROS2 (rclpy) publish ke /cmd_vel dll. Setiap perintah ke robot fisik = R9 gate (bisa nabrak/rusak). Default mulai dari sim, validasi, baru sim-to-real.
# ROS2 bridge — kirim velocity (real robot → R9 gate dulu)
import rclpy
from geometry_msgs.msg import Twist
def drive(linear=0.2, angular=0.0):
    rclpy.init(); node = rclpy.create_node("agent_drive")
    pub = node.create_publisher(Twist, "/cmd_vel", 10)
    t = Twist(); t.linear.x = linear; t.angular.z = angular
    pub.publish(t); node.destroy_node(); rclpy.shutdown()

5. Cosmos Physical AI & meta-actions (v4.1)

NVIDIA Cosmos = world foundation models buat Physical AI: generate data sintetik, simulasi, dan latih policy robot tanpa ribuan jam data dunia nyata.

Pipeline Physical AI:
  scene_prep (USD) → Cosmos/Isaac (generate variasi + simulasi fisika)
  → policy training (Isaac Lab RL) → eval di sim → sim-to-real (gated)
  • Data generation: Cosmos bikin variasi skenario (lighting, layout, objek) dari sedikit seed → dataset besar buat training.
  • Policy training: latih di sim (murah, paralel, aman) → validasi → baru real.
  • Object manipulation: latih grasp/manipulasi di sim, domain-randomize (lihat scene_prep.randomize), transfer ke robot nyata.

Meta-actions — safety reflex (WAJIB untuk gerak fisik)

Robot yang gerak butuh refleks berhenti, bukan cuma "maju". Tiap perintah mobility dibungkus guard:

# meta-action: deselerasi/STOP saat risiko terdeteksi — mendahului perintah gerak
def safe_drive(linear, angular, risk_fn):
    if risk_fn():                 # obstacle dekat / kehilangan sensor / area terlarang
        return stop()             # STOP menang atas perintah apa pun
    return drive(min(linear, V_MAX), angular)   # clamp ke batas aman

Hierarki: STOP > deselerasi > perintah operator. Ini analog Operational Rails crypto (governor) tapi untuk dunia fisik — refleks keselamatan yang gak bisa di-bypass auto_confirm. Real-robot tetap R9 gate di atas ini.

6. Vision-Language-Action (VLA) & multimodal reasoning (v4.1)

VLA = satu loop: persepsi → reasoning → aksi motor. Kamera/video masuk, model paham scene + instruksi bahasa, keluarin aksi (gerak/grasp).

PERCEIVE (camera/video/sensor) → GROUND (objek, posisi, instruksi bahasa)
→ PLAN (langkah aksi) → ACT (motor command, lewat safe_drive/meta-action) → OBSERVE (loop)
  • Scene understanding: deteksi objek + relasi spasial + affordance ("gelas bisa diangkat").
  • Object manipulation: dari instruksi bahasa ("ambil botol merah") → grasp pose → eksekusi (latih di sim dulu).
  • Multimodal reasoning: gabung text + vision + audio + action di satu agent (robotics, industrial automation, healthcare). Input multimodal → keputusan tunggal yang konsisten.
  • Persepsi visual/audio sisi input pakai m15 (multimodal: vision/Whisper); m19 nambah action di ujungnya.

Safety tetap di atas segalanya: aksi motor apa pun lewat meta-action (§5) — STOP/deselerasi menang atas perintah VLA. Model salah baca scene → refleks keselamatan yang nahan, bukan eksekusi buta. Real-robot = R9 gate.


Catatan

  • Isaac Sim/Cosmos butuh GPU NVIDIA + driver; deploy headless via m2, drive jarak jauh.
  • macOS control = background-first, no cursor steal — itu kontrak utama skill ini.
  • Real-robot & destructive desktop = R9 gate. Sim & background app-drive = bebas.
  • Combo: m19 + m2 (deploy GPU box) + m17 (planner buat multi-step scene/training) + x6 (debug sim).