libragent-plugin
Implement, verify, and deploy a host MediaAssist plugin that converts audio/image/video into text when the LLM rejects multimodal input (HTTP 400). Use when media-assist placeholders appear, assistPlu
SKILL.mdsrc-tauri/bundled_skills/libragent-plugin/SKILL.md
GitHubLibrAgent MediaAssist Plugin
When the model cannot accept audio/image parts, LibrAgent strips them (or runs a host plugin) and may show:
[media-assist] ... load @skill:libragent-plugin ...
This skill installs a durable host plugin under app local storage so later sessions (including Harbor on this machine) auto-convert media.
Do / Don't
| Do | Don't |
|---|---|
Implement manifest.json + run (stdin JSON → stdout JSON) |
apt/pip whisper mid-task as the only fix |
Verify with scripts/verify_plugin.py |
Deploy to system_skills/ or user_skills/ |
Deploy via media__deployAssistPlugin on Host isolation |
Call deploy/status from Docker/Harbor attach sessions (blocked) |
On Windows, ship run.cmd / run.bat / run.exe (optional run.py helper) |
Rely on a shebang-only extensionless run on Windows |
| In Docker/Harbor: convert with in-container CLI (ffmpeg/OCR) | Assume host plugin ACE works from attach mode |
Contract (interfaceVersion 1)
Plugin dir after deploy: {appDataDir}/harness-plugins/media-assist/v1/
manifest.json:
{
"interfaceVersion": 1,
"name": "my-media-assist",
"timeoutMs": 120000,
"modalities": ["audio", "image", "video"]
}
run (executable): read one JSON object from stdin:
{
"interfaceVersion": 1,
"modality": "audio" | "image" | "video",
"mimeType": "audio/wav",
"path": "/abs/path/to/media",
"maxOutputChars": 8000
}
Write one JSON object to stdout:
{ "ok": true, "modality": "audio", "text": "…", "notes": "optional" }
On failure: { "ok": false, "error": "…", "message": "…" }.
Templates: templates/manifest.json, templates/run (Unix), templates/run.cmd
(Windows). Stubs echo a clear note until you wire real ASR/OCR/VLM.
Workflow
- Status —
media__assistPluginStatus - Scaffold — copy templates into a workspace folder (e.g.
.libragent/media-assist-build/) - Implement — replace stub
runwith local ASR (audio), OCR/VLM (image), extract+ASR/OCR (video). Prefer already-installed host tools. - Verify —
python <skill-base-dir>/scripts/verify_plugin.py <plugin-dir> - Deploy — Only under Host isolation:
media__deployAssistPluginwithfilescontaining at leastmanifest.jsonandruncontent. Blocked in Docker/Harbor attach sessions (container breakout prevention). There, convert media inside the container with CLI tools (e.g. ffmpeg/whisper/tesseract) instead of deploying a host plugin. - Confirm —
media__assistPluginStatus→installed: true(Host isolation).
Acceptance
- Verify script exits 0.
- Status shows installed modalities.
- A later multimodal 400 is auto-assisted without re-entering this skill.