multimodal-understanding

Analyze images and visual content using the NoDesk AI Gateway multimodal API. ALWAYS use this skill when the user mentions an image file path, image URL, or asks to analyze/describe any image. Do NOT

SKILL.md

GitHub

Multimodal Understanding

Analyze images via NoDesk AI Gateway. The API is OpenAI Vision-compatible.

Configuration

No setup needed. Scripts read the API key from ~/.deskclaw/nanobot/config.json, checking providers.custom → anthropic → openai (first non-empty key wins).

Workflow

Local file (most common)

When the user provides a local file path (e.g. /Users/.../photo.jpg, ~/Desktop/img.png):

bash <skill_dir>/scripts/multimodal-call.sh --file '<image_path>' '<user_prompt>' [max_tokens] [model]

The script auto-detects the image type and base64-encodes it. No manual conversion needed.

Image URL

When the user provides a public image URL (e.g. https://example.com/photo.jpg):

bash <skill_dir>/scripts/multimodal-call.sh --url '<image_url>' '<user_prompt>' [max_tokens] [model]

The URL must be publicly accessible. No download or base64 conversion needed.

Parameters

  • max_tokens: default 1000. Use 300 for short descriptions, 2000+ for OCR / detailed analysis.
  • model: default gpt-4o.

Present results

Extract choices[0].message.content from the JSON response and show it directly to the user.

Error Handling

HTTP Status Meaning Action
401 / 403 Invalid API key Ask user to verify their key
429 Quota exceeded or rate limited Tell user to check quota on gateway dashboard
Timeout Image too large or gateway busy Retry once; if still failing, tell user

Important

  • Local file path → use --file mode. Tell users: instead of pasting/dragging images, type the file path as text for best results.
  • Image URL → use --url mode. URL must be publicly accessible.
  • Each image costs ~200–800 prompt tokens. Multimodal and LLM chat share the same token quota.