generate-voice
Generate TTS voiceover with ElevenLabs, including phoneme alignment, viseme mapping, and emotion-based mouth curvature for lip sync.
SKILL.mdAutoAnimation/.claude/skills/generate-voice/SKILL.md
GitHubPurpose
Generate a spoken voiceover from text using ElevenLabs TTS, extract phoneme alignment data, convert phonemes to visemes for lip sync, and build an emotion timeline from inline [emotion] cues. This is the core pipeline that turns a script into an animated talking character.
Steps
Prepare the script with emotion cues. Inline cues like
[happy],[sad],[angry]are placed before the words they apply to. These get converted to ElevenLabs v3 expressive annotations (e.g.,[happy]becomes<cheerfully>). The mapping lives inEMOTION_V3_MAPinsideelevenlabs.ts.Select a voice. Call
getElevenLabsService().getVoices()to list available voices. Each voice has avoice_id. The user picks one fromVoicesPanel.tsxor the orchestrator assigns one.Generate speech with alignment. Call
generateWithAlignment(text, voiceId, settings)on theElevenLabsServicesingleton. This hits the/text-to-speech/{voiceId}/with-timestampsendpoint and returns:audioBlob/audioUrl-- the MP3 audioalignment-- anElevenLabsAlignmentobject withcharacters,character_start_times_seconds,character_end_times_seconds, and optionallyphonemes,phoneme_start_times_seconds,phoneme_end_times_secondsduration-- total audio duration in seconds
Convert alignment to viseme timeline. Use
getLipSyncProcessor(fps).processAlignment(alignment)fromlipSync.ts. The processor:- Prefers phoneme-based mapping (ARPAbet phonemes via
PHONEME_TO_VISEME) - Falls back to character-based mapping (
CHAR_TO_VISEME) when phonemes are unavailable - Merges consecutive identical visemes
- Adds smooth transitions between viseme shapes (2-frame intermediate visemes)
- Returns
VisemeEvent[]withviseme,startTime,endTime,startFrame,endFrame
- Prefers phoneme-based mapping (ARPAbet phonemes via
Build the emotion timeline. Call
buildEmotionTimeline(rawScript, wordTimeline)fromemotionTimeline.ts. This:- Parses
[emotion]cues from the raw script using regex - Maps each cue to a word index in the clean (cue-stripped) text
- Looks up frame positions from the word timeline
- Returns
EmotionEvent[]withemotion,startFrame,endFrame
- Parses
Map emotion to mouth curvature. The emotion determines which set of 8 viseme sprites to use (3 curvatures x 8 visemes = 24 total sprites). Store the generated voice data in
useVoiceStoreand the dialogue line inuseMultiCharacterStore.
Viseme Reference Table
| Viseme | Mouth Shape | Phonemes (ARPAbet) |
|---|---|---|
| Rest | Closed, relaxed | SIL, SP, K, G, NG, HH |
| Aa | Wide open | AA, AE, AH, AY, AW |
| Ee | Wide smile | EH, EY, IY, IH |
| O | Rounded medium | AO, OW, OY |
| U | Pursed lips | UH, UW |
| M | Lips pressed | M, B, P |
| F | Teeth on lower lip | F, V |
| L | Tongue tip | L, TH |
| D | Tongue behind teeth | D, T, N, DH |
| R | Rounded retracted | R, ER |
| S | Teeth close | S, Z, SH, ZH, CH, JH |
| W | Rounded tight | W, Y |
Mouth Curvature by Emotion
| Curvature | Emotions |
|---|---|
| Upward | Joy, Satisfaction, Amusement, Laughter |
| Neutral | Anger, Sternness, Indignation, Surprise, Shock |
| Downward | Sadness, Fear, Disgust, Grief, Melancholy |
Key Files
src/services/elevenlabs.ts-- ElevenLabsService class, TTS generation, emotion cue conversion, voice listing, voice cloningsrc/services/lipSync.ts-- LipSyncProcessor, PHONEME_TO_VISEME mapping, IPA_PHONEME_TO_VISEME, viseme transition logicsrc/services/emotionTimeline.ts-- buildEmotionTimeline(), getEmotionAtFrame(), expression cue parsingsrc/stores/useVoiceStore.ts-- Voice/TTS state, caption configsrc/stores/useMultiCharacterStore.ts-- Multi-character dialogue state, dialogue linessrc/types/voice.ts-- Viseme, VisemeEvent, WordEvent, ElevenLabsAlignment typessrc/types/nanoBanana.ts-- CurvedVisemeKey, MouthCurvature (24-sprite system)
Common Issues
- Missing API key: The ElevenLabs API key must be set as
VITE_ELEVENLABS_API_KEYin the environment. CheckhasElevenLabsService()before calling. - No phoneme data: Some ElevenLabs models or languages may only return character-level alignment. The processor falls back to
CHAR_TO_VISEMEautomatically, but viseme quality is lower. - Emotion cues not stripped before TTS: The raw script with
[emotion]cues is sent to ElevenLabs, which converts them to v3 annotations. The cues are not spoken words -- they only affect delivery style. - Credit gating:
generateWithAlignmentis wrapped inwithCreditGate('elevenlabs-tts', ...). Ensure the user has sufficient credits. - IPA phonemes for non-English: Use
mapIpaPhonemeToViseme()fromlipSync.tsfor non-English languages that return IPA instead of ARPAbet.
Examples
// Generate voice with alignment
const service = getElevenLabsService()
const { audioUrl, alignment, duration } = await service.generateWithAlignment(
'[happy] Hey everyone! [excited] Welcome to my channel!',
'pNInz6obpgDQGcFmaJgB', // voice ID
{ stability: 0.5, similarityBoost: 0.75 }
)
// Convert to viseme timeline
const processor = getLipSyncProcessor(30) // 30 fps
const visemeTimeline = processor.processAlignment(alignment)
// Build emotion timeline
const emotionTimeline = buildEmotionTimeline(
'[happy] Hey everyone! [excited] Welcome to my channel!',
wordTimeline
)
// Query at a specific frame
const viseme = processor.getVisemeAtFrame(visemeTimeline, 45) // e.g. 'Ee'
const emotion = getEmotionAtFrame(emotionTimeline, 45) // e.g. 'Happy'