ai-media

Use when a creative goal must become a finished media file: pick and order generative-media models per modality — AI voiceover, image-to-video clips, score — then glue them with ffmpeg (mux, duck, lou

SKILL.md

GitHub

ai-media

You are the cross-modal director. You decide which generative-media model to call per modality, in what order, with what params, then assemble the pieces with ffmpeg into one finished file. You do not own a single provider's API surface and you do not prompt still images — you orchestrate and glue.

Pipeline shape — decide what the goal needs

Map the goal to modalities and an ordered step list, and lock that plan before you generate a single asset — media generation is slow and metered, so a re-roll of a 10 s Veo clip or a 90 s music track costs real money and minutes. Fixing the scene list, aspect ratio, target loudness and model per modality first is cheaper than discovering at mux time that your clips are 9:16 and your VO is the wrong sample rate. The "delegate to" column is where the actual call mechanics live — you pick the model and params, those skills run the call.

Goal Needs Ordered steps Delegate calls to
Narrated explainer stills + img→video + VO + music script → per-scene stills → clip per scene → VO → music → conform → concat → mix+duck → loudnorm → MP4 replicate-images, fal/replicate
Product teaser (1 hero) 1 still + img→video + music still → clip → music → mix → loudnorm → MP4 replicate-images, fal/replicate
Faceless short stills + img→video + VO + music + captions (explainer pipeline) + burn captions replicate-images; ../video-shorts/SKILL.md for the script
Just a voiceover VO only script → TTS → loudnorm —
Just a clip from a still img→video only still (input) → clip fal/replicate
Code-rendered explainer none of the above render from React/TS stop — route to remotion-video

If the video is rendered from data/code (charts, timelines, JSON-driven scenes), this is not your job → ../remotion-video/SKILL.md. You handle model-generated + ffmpeg-glued.

Modality 1 — Voice (TTS)

ElevenLabs Python SDK. The call is convert(text, voice_id, model_id, output_format); auth via ELEVENLABS_API_KEY.

from elevenlabs.client import ElevenLabs

client = ElevenLabs()  # reads ELEVENLABS_API_KEY
audio = client.text_to_speech.convert(
    text="Your narration script here.",
    voice_id="JBFqnCBsd6RMkjVDRZzb",
    model_id="eleven_multilingual_v2",     # final-quality VO
    output_format="mp3_44100_128",          # codec_samplerate_bitrate
)
with open("vo.mp3", "wb") as f:
    for chunk in audio:
        f.write(chunk)

Pick the model tier by what the job needs:

Model When Latency Cost lever
eleven_v3 most expressive, hero final VO — verify availability first (see caveat) higher most credits/char
eleven_multilingual_v2 high-quality multilingual VO (default for finals) medium medium
eleven_flash_v2_5 real-time / batch / scale / drafts ~75 ms cheapest

Do not hardcode eleven_v3 blind. It shipped to the API in alpha (Aug 2025) and the docs model list now carries it, but the text_to_speech.convert API reference still documents the default as eleven_multilingual_v2 and does not enumerate eleven_v3 as a guaranteed value. Before you build a final pass on it, confirm it returns from GET /v1/models for your key (or just call once and check) — otherwise default to eleven_multilingual_v2, which is the safe, always-available hero tier.

output_format is codec_samplerate_bitrate — e.g. mp3_44100_128, mp3_22050_32. Match the VO sample rate to your assembly target, do not master the VO loud and hope.

Bad → Good:

  • Bad: generate VO at mp3_22050_32, then mux onto a 48 kHz video — ffmpeg silently resamples, you get artifacts and a level mismatch.
  • Good: generate VO at the rate you will master at (e.g. mp3_44100_128), and set final loudness with loudnorm in assembly, not by cranking the TTS.

TTS is billed per character/token (~0.5–1 credit/char on the Flash/Turbo lines). ElevenLabs cut TTS API pricing up to 55% on 2026-05-07 (e.g. Flash on Creator $0.11→$0.05 / 1k tokens) — that figure is TTS-specific, not the Music cut. Pricing staling fast: these are point-in-time numbers from elevenlabs.io/pricing/api as of 2026-06-02 — re-check the page before quoting a budget. Shorter scripts and Flash on drafts are the cost levers.

Modality 2 — Image-to-video

The still is an input, not your output. Generate or edit the source image in ../replicate-images/SKILL.md, then animate it here. Reality check: every serious 2026 model does 1080p or native 4K — resolution is no longer the differentiating axis. The hard limit is per-generation duration (~5–15 s, model-dependent). Long pieces are one clip per scene, then concat — never one long take.

Durations below are from each vendor's own pages (as of 2026-06; see references/models-and-params.md for the citations) — they move with releases, so verify on the catalog before a final run:

Model Duration Aspect / max res Control surface Native audio Open-source
Google Veo 3.1 8 s / generation 16:9 / 9:16, up to 4K high yes — synced 48 kHz dialogue/SFX no
Kling 3.0 up to 15 s flexible, 4K strong identity/temporal, lip-sync no no
Runway Gen-4.5 2–10 s flexible best — motion brushes, camera control, reference image no no
MiniMax Hailuo 02 6 s or 10 s (1080p caps at 6 s) up to 1080p medium no no
Wan 2.6 up to 15 s up to 1080p first/last-frame control, A/V sync no (sync) yes (Apache)

Choose by the binding constraint: need synced dialogue → Veo 3.1; need precise camera/motion control → Runway Gen-4.5; need identity consistency across scenes or the longest single take → Kling 3.0 / Wan 2.6; need open-source/self-host → Wan 2.6; cost-sensitive 1080p → Hailuo 02. Endpoint ids and per-call mechanics live in ../fal/SKILL.md / ../replicate/SKILL.md (both rails carry these models). See references/models-and-params.md for endpoint ids and current limits.

Modality 3 — Music / score

Costs are per-minute and plan-dependent — treat them as approximate and verify on the vendor pricing page (figures as of 2026-06; sources in references/models-and-params.md):

Model Cost (approx, verify) Licensing story Control
ElevenLabs Music v2 per-minute, ~$0.15–0.50/min depending on plan (Music API pricing cut up to 50% at v2 launch — separate from the 55% TTS cut) cleanest — vendor states trained only on licensed data, cleared for commercial use (Believe collaboration named at launch) genre-switch mid-track
Suno v5 plan-based usage rights on paid plans post Nov-2025 label settlements (rights, not ownership) vendor blind-test benchmark ELO ~1293
Udio $30/mo Pro plan (commercial rights); no official public API — third-party gateways only UMG-licensed platform announced for 2026 —

Confirm commercial rights before you ship. Licensing differs per model and per plan; "I generated it" is not "I may sell the ad with it." For a clean commercial story with an official API, ElevenLabs Music v2 is the safe default — Udio has no first-party API, so do not plan a programmatic pipeline around it. The rest is the same fal/replicate call mechanics.

Assembly with ffmpeg

Four operations. Each is a copy-paste recipe; full filter graphs, caption burning and pitfalls are in references/ffmpeg-assembly.md.

(a) Mux VO onto video — map both streams, copy video, take the shorter duration:

ffmpeg -i scene.mp4 -i vo.mp3 \
  -map 0:v -map 1:a -c:v copy -shortest out.mp4

(b) Duck music under the VO — sidechaincompress keys the music off the voice so it drops when narration plays (pro DAW ducking, no manual keyframes):

ffmpeg -i vo.mp3 -i music.mp3 -filter_complex \
  "[1:a][0:a]sidechaincompress=threshold=0.03:ratio=8:attack=20:release=300[duck]; \
   [0:a][duck]amix=inputs=2:duration=longest[aout]" \
  -map "[aout]" -c:a aac mix.m4a

Cheaper static fallback when sidechain is overkill — fix the music low under a full VO:

ffmpeg -i vo.mp3 -i music.mp3 -filter_complex \
  "[1:a]volume=0.3[m];[0:a][m]amix=inputs=2:duration=longest[aout]" \
  -map "[aout]" mix.m4a

(c) Loudnorm to a target LUFS (two-pass) — measure, then apply. Target -14 LUFS for social/streaming, -16 for podcast-style VO. Normalize per track before mixing.

# pass 1: measure (read the JSON it prints)
ffmpeg -i mix.m4a -af loudnorm=I=-14:TP=-1.5:LRA=11:print_format=json -f null -
# pass 2: apply with the measured values
ffmpeg -i mix.m4a -af \
  loudnorm=I=-14:TP=-1.5:LRA=11:measured_I=-20.1:measured_TP=-4.2:measured_LRA=6.0:measured_thresh=-30.8:offset=0.5:linear=true \
  master.m4a

(d) Concat scenes — conform first. Same-codec/res/fps clips → fast demuxer with -c copy. Mismatched clips → re-encode and scale first, then concat. Never -c copy-concat mismatched clips — you get desync or a corrupt stream.

# all clips identical codec/res/fps:
printf "file '%s'\n" scene1.mp4 scene2.mp4 scene3.mp4 > list.txt
ffmpeg -f concat -safe 0 -i list.txt -c copy joined.mp4

# mismatched: conform each, then concat filter
ffmpeg -i s1.mp4 -i s2.mp4 -filter_complex \
  "[0:v]scale=1920:1080,fps=30,setsar=1[v0];[1:v]scale=1920:1080,fps=30,setsar=1[v1]; \
   [v0][1:a?][v1][1:a?]concat=n=2:v=1:a=0[v]" -map "[v]" joined.mp4

End-to-end worked pipeline (narrated explainer)

Ordered command list — generate once, assemble deterministically:

  1. Lock the plan — scene list, aspect (e.g. 16:9 1080p 30fps), target -14 LUFS, models chosen.
  2. Stills per scene → ../replicate-images/SKILL.md (one prompt per scene).
  3. Clip per scene (img→video, ~5–15 s each, model cap) via your chosen model on fal/replicate.
  4. VO → ElevenLabs convert(...) at the master sample rate.
  5. Music → ElevenLabs Music v2, length = total runtime, confirm rights.
  6. Conform every clip to 1920x1080/30fps/SAR 1.
  7. Concat the conformed clips → body.mp4.
  8. Loudnorm VO and music tracks (two-pass) to consistent levels.
  9. Mix + duck music under VO → mix.m4a.
  10. Final loudnorm the mix to -14 LUFS → master.m4a.
  11. Mux master.m4a onto body.mp4 with -shortest → final.mp4.

Emit this as a runnable script. scripts/verify.sh lints it (loudnorm present, conform-before-concat, final MP4 target).

Cost & regen discipline

  • Draft small, then final. Generate clips short/low-res and VO on Flash to lock timing and the cut; only the final pass spends on hero quality. Re-rolling locked scenes is the biggest waste.
  • Per-modality levers: shorter scripts (TTS per-char), fewer scene re-rolls (img→video), fewer music minutes (music is billed per minute — verify the current rate on the vendor pricing page).
  • Spend tracking as a discipline → ../fal/SKILL.md / ../replicate/SKILL.md for per-call cost; treat budget as a constraint you set before generating.

Anti-patterns

Anti-pattern Why it bites Do instead
Generating assets before locking the pipeline aspect/sample-rate/duration mismatches surface at mux time, forcing paid re-rolls lock scene list, aspect, LUFS, models first
One long video-gen call for the whole piece models cap at ~5–15 s; you fight the limit and waste rolls one clip per scene, then concat
Mixing tracks without per-track loudnorm VO buried or blasting over music; inconsistent levels two-pass loudnorm each track before mix
-c copy-concat of mismatched clips desync, corrupt stream, wrong frame timing conform res/fps/SAR, then concat
Music low set by ear / static only when VO needs space narration gets masked under the bed sidechaincompress keyed off the VO
Shipping generated music without checking rights "generated" ≠ "licensed to sell" — legal exposure confirm commercial rights per model/plan
Prompting/editing the still inside this skill duplicates replicate-images' job, worse prompts delegate the still, consume it here
Mastering loudness by cranking the TTS clipping, no true-peak control set level with loudnorm, not the generator