Create Clip via AI Orchestrator

Generate a complete animated short-form video from a text prompt. The orchestrator uses Gemini 2.0 Flash to create a structured ClipPlan, then executes it step-by-step across all platform stores.

Purpose

Automate the entire clip creation process: script writing, character selection, voice generation, lip sync, background selection, text overlays, motion graphics, stock media, sound effects, camera animation, and timeline assembly -- all from a single text prompt.

Pipeline Steps (27 Step Types)

The orchestrator builds a dynamic step list based on the ClipPlan content:

Step	Description
setup-canvas	Set aspect ratio, dimensions, FPS
setup-background	Lottie animation or AI-generated SVG scene
setup-characters	Match saved characters or mark for generation
generate-characters	AI character generation via NB2 pipeline
generate-voices	ElevenLabs TTS for each dialogue line (lip sync + alignment)
setup-dialogue	Build frame-based dialogue timeline from voice data
generate-music	AI background music from dialogue mood analysis
setup-text-overlays	Place title/subtitle/CTA/quote/watermark overlays
setup-shapes	Add decorative shapes (rectangle, circle, star, etc.)
setup-html-templates	Load HTML motion graphics templates with config
setup-motion-graphics	Setup kinetic typography / scene templates
generate-svg-objects	AI-generate animated SVG icons via Gemini 2.5 Pro
setup-stock-media	Search and place Pixabay images/videos
setup-smart-broll	AI B-roll suggestions and placement
setup-sound-effects	Sound effect selection and placement
setup-camera	Camera preset and keyframe setup
setup-auto-camera	Automated camera movement
setup-character-motion	Character motion keyframes
setup-gestures	Character gesture animations
setup-retention-hooks	Visual engagement hooks (progress bars, countdowns)
auto-animate-elements	Auto-animate static elements
sync-to-beat	Synchronize animations to music beats
quality-gate	Quality assessment check
generate-thumbnail	AI thumbnail generation
setup-captions	Configure caption style (word-by-word/sentence/karaoke)
finalize-timeline	Set total duration and seek to start
refine-plan	Chat-based plan refinement

Non-fatal steps (generate-music, setup-stock-media, setup-sound-effects) will not abort execution on failure.

Execution Flow

1. Prompt Phase -- User enters a text prompt describing the desired video
2. Planning Phase -- Gemini generates a ClipPlan JSON with all content structured
3. Review Phase -- User reviews and optionally edits the plan before execution
4. Execution Phase -- Steps execute sequentially, updating stores across the platform
5. Done Phase -- Clip is ready for preview and export

ClipPlan Interface

The plan generated by Gemini contains:

interface ClipPlan {
  canvas: { aspectRatio, fps, durationSeconds, mode }
  background: { type: 'lottie' | 'svg-generate', lottieQuery?, svgPrompt? }
  characters: [{ name, savedCharacterName?, generateNew?, position, scale, voiceName?, dimension? }]
  dialogue: [{ characterName, script, emotion?, animationName? }]
  textOverlays: [{ preset, content, startPercent, endPercent, fontFamily?, color? }]
  shapes: [{ type, position, size, color, startPercent, endPercent }]
  htmlTemplates?: [{ templateId, startPercent, endPercent, config }]
  motionGraphics?: [{ templateId, startPercent, endPercent, props }]
  svgObjects?: [{ description, position, scale, keyframes }]
  stockMedia?: [{ query, role, startPercent, endPercent, transition }]
  soundEffects?: [{ description, startPercent, endPercent }]
  retentionHooks?: [{ type, startPercent, endPercent }]
  camera?: { presetId?, keyframes? }
  cameraDirectives?: [{ type, startFrame, endFrame, intensity }]
  captions: { style, position, fontSize?, color? }
}

OrchestratorSettings

Configurable before generation:

• aspectRatio: '16:9', '9:16', '1:1', '4:3', '21:9'
• durationSeconds: Target duration (0 = auto)
• fps: 24, 30, or 60
• generateMusic: Enable/disable background music
• enableSVGGeneration: Enable/disable SVG object creation
• enableStockMedia: Enable/disable Pixabay search
• enableGoogleSearch: Enable Gemini grounding with Google Search
• selectedCharacterIds: Pre-select characters to use
• selectedAnimationNames, selectedTemplateIds: Pre-select assets
• contentType: 'generic', 'article', 'product', 'recipe', 'news', 'document', 'presentation'

Key Files

Purpose	Path
Orchestrator service	src/services/orchestrator.ts
Orchestrator store	src/stores/useOrchestratorStore.ts
Orchestrator types	src/types/orchestrator.ts
Step executors	src/services/orchestrator/steps/*.ts
Plan builder	src/services/orchestrator/planBuilder.ts
Chat refiner	src/services/orchestrator/chatRefiner.ts
Panel UI	src/components/panels/orchestrator/OrchestratorPanel.tsx
Dashboard entry	src/components/dashboard/DashboardOrchestratorPanel.tsx

Common Issues

• Plan generation fails: Check VITE_GEMINI_API_KEY is set. Gemini returns structured JSON; malformed responses are caught and retried.
• Voices not generating: Requires VITE_ELEVENLABS_API_KEY. Voice generation is per-dialogue-line; each produces alignment data for lip sync.
• Characters not found: The orchestrator fuzzy-matches savedCharacterName against saved characters. If no match, set generateNew: true in the plan.
• Stock media missing: Requires VITE_PIXABAY_API_KEY. Stock media steps are non-fatal and will skip on failure.
• Music generation fails: Non-fatal step. Requires ElevenLabs API key with music generation enabled.
• Cost tracking: Every API call logs a CostEntry with service, tokens/characters used, and estimated USD cost.

Example

Prompt: "Create a 30-second TikTok about the benefits of morning routines"

The orchestrator will:

1. Set canvas to 9:16 (TikTok), 30fps, ~30s duration
2. Select or generate a character
3. Write dialogue lines with emotion cues like [excited] and [thoughtful]
4. Generate TTS with ElevenLabs, extract phoneme alignment for lip sync
5. Build emotion timeline from script bracket cues
6. Add title card, tip overlays, and CTA text
7. Search Pixabay for relevant stock footage
8. Generate background music matching dialogue mood
9. Configure word-by-word captions at bottom
10. Assemble timeline with all layers synced