add-music
Generate AI background music for a clip based on mood analysis, with genre selection and audio mixing.
SKILL.mdAutoAnimation/.claude/skills/add-music/SKILL.md
GitHubPurpose
Generate background music that matches the emotional tone of a clip's dialogue. Analyze dialogue lines for mood, build a composition plan or text prompt, generate music via ElevenLabs or a backend provider, and mix it with dialogue audio using auto-ducking.
Steps
Analyze dialogue mood. Call
buildMusicPlanFromDialogue(dialogueLines, fps)frommusicAnalyzer.ts. This:- Detects themes from script text (motivation, love, adventure, humor, tech, etc.) using keyword matching
- Groups consecutive dialogue lines by emotion into segments (intro/middle/climax/outro)
- Maps each emotion to music styles using
EMOTION_MUSIC_MAP(e.g., Joy = uplifting/bright/warm, Anger = intense/driving/aggressive) - Returns a
MusicCompositionPlanwith global styles, negative styles, and per-section styles/durations
Choose generation method. Two paths:
- Composition plan (preferred): Use
createMusicCompositionPlan(prompt, durationMs)thengenerateMusic({ compositionPlan })fromelevenlabs.ts. The plan endpoint is free but rate-limited. - Simple prompt: Use
generateMusic({ prompt, durationMs, forceInstrumental })for a one-shot generation. - Advanced backend: Use
generateMusicAdvanced(options)frommusicGeneration.tswhich hits/api/music-generation/generateand polls for completion. Supports genre, mood, tempo, and instrumental flags.
- Composition plan (preferred): Use
Configure genre and mood. Available genres: ambient, cinematic, electronic, hip-hop, jazz, lo-fi, orchestral, pop, rock, r&b, acoustic, world, custom. Available moods: uplifting, energetic, calm, dark, dramatic, happy, melancholic, mysterious, romantic, suspenseful, neutral. Use
buildMusicPrompt({ genre, mood, tempo, customPrompt })to construct a prompt string.Mix audio tracks. Use
mixTracks(tracks, totalDurationSec, options)fromaudioMixer.ts. This:- Uses
OfflineAudioContextfor offline rendering - Supports per-track volume (0-1), mute, and stereo pan (-1 to 1)
- Auto-ducks music/SFX under dialogue (default -12dB reduction)
- Configurable attack (0.3s) and release (0.5s) fade times
- Extracts dialogue regions and schedules gain automation
- Uses
Store the result. Add the music audio URL to the voice/audio store and update the timeline. The orchestrator typically places music at frame 0 spanning the entire clip duration.
Key Files
src/services/musicAnalyzer.ts-- buildMusicPlanFromDialogue(), buildMusicPromptFromDialogue(), EMOTION_MUSIC_MAP, theme detection, beat detection bridgesrc/services/musicGeneration.ts-- generateMusicAdvanced(), MusicGenerationOptions, MusicGenre, MusicMood, GENRE_OPTIONS, MOOD_OPTIONSsrc/services/elevenlabs.ts-- createMusicCompositionPlan(), generateMusic(), MusicCompositionPlan, MusicSectionsrc/services/audioMixer.ts-- mixTracks(), AudioTrackSource, MixerOptions, auto-ducking logicsrc/stores/useAudioDesignStore.ts-- Audio design stateserver/routes/musicGeneration.ts-- Backend music generation endpoint
Common Issues
- Composition plan vs prompt mode:
force_instrumentalis only valid with prompt mode, not composition plans. For instrumental music with composition plans, use emptylines: []in sections (already the default). - Section duration limits: ElevenLabs requires integer
duration_msand each section must be 3-120 seconds.segmentToSection()clamps and rounds automatically. - Music too loud under dialogue: Adjust
duckAmountDbin mixer options. Default is -12dB. For clearer dialogue, use -14 to -16dB. - Credit gating: Both
generateMusic()andgenerateMusicAdvanced()are credit-gated. Check user credits before calling. - Backend unavailable:
isMusicGenerationAvailable()checks if the advanced backend has providers configured. Falls back to ElevenLabs direct.
Examples
// From dialogue analysis
import { buildMusicPlanFromDialogue } from '@/services/musicAnalyzer'
import { generateMusic } from '@/services/elevenlabs'
const plan = buildMusicPlanFromDialogue(dialogueLines, 30)
const { audioBlob, audioUrl, durationMs } = await generateMusic({ compositionPlan: plan })
// Simple prompt
const result = await generateMusic({
prompt: 'uplifting cinematic background music',
durationMs: 30000,
forceInstrumental: true,
})
// Mix with dialogue
import { mixTracks } from '@/services/audioMixer'
const mixed = await mixTracks(
[
{ url: dialogueAudioUrl, startTimeSec: 0, type: 'dialogue', volume: 1 },
{ url: musicAudioUrl, startTimeSec: 0, type: 'music', volume: 0.3 },
],
totalDurationSec,
{ autoDuck: true, duckAmountDb: -12 }
)