offline-voice
How the on-device narration engine (VieNeu-TTS) and voice cloning work in this system - install tiers, the worker protocol, and the verified traps around torchaudio, voice metadata parsing and duratio
SKILL.md.claude/skills/offline-voice/SKILL.md
GitHubOffline voice - VieNeu-TTS and voice cloning
The system has two narration engines running side by side. They are deliberately not merged.
gemini |
vieneu |
|
|---|---|---|
| Where it runs | Google API | The user's own machine |
| Cost | Per call | Free |
| Needs network | Yes | No |
| Voices | 30 preset | 14 Vietnamese preset + unlimited cloned |
| Voice cloning | No (API only accepts preset names) | Yes |
| Speed | A few seconds | RTF ~1.0 (about real time) |
| Setup | GEMINI_API_KEY |
pip install vieneu |
Keep both. Do not "simplify" by deleting one: a weak machine reads at roughly real time, which is unacceptable for a long script, while Gemini costs money on every preview click.
Why VieNeu and not F5-TTS / XTTS / Chatterbox
Checked against the requirement "Vietnamese + English, cloning, and shippable in an open-source repo":
- Chatterbox (MIT, excellent English) has no Vietnamese. Disqualified on language.
- F5-TTS code is MIT but the weights are CC-BY-NC, and every Vietnamese finetune is CC-BY-NC-SA. Disqualified on licence.
- viXTTS / XTTS-v2 inherit Coqui's CPML non-commercial licence. Disqualified on licence.
- VietTTS (dangvansam) is Apache-2.0 code but the installer is Linux-only, and its weights are CC-BY-NC too.
- VieNeu-TTS is Apache 2.0 on the code and on the v3 Turbo weights (
pnnbao-ump/VieNeu-TTS-v3-Turbo, trained from scratch on ~10k hours, not a finetune), plus Apache 2.0 on its codec dependencyOpenMOSS-Team/MOSS-Audio-Tokenizer-Nano. Vietnamese-first with English code-switching, torch-free on CPU via ONNX, clones from 3-8 seconds of reference audio.
If a future engine is evaluated, check the weights licence separately from the code licence. That is where every other candidate failed.
Licence trap inside the package itself. vieneu/assets/voices.json carries "license": "CC BY-NC 4.0" and "Model and voices are for non-commercial use only". That notice is scoped to that file, which is the v2 preset pack: 6 voices named Binh, Tuyen, Vinh, Doan, Ly, Ngoc, default Binh. We run mode="v3turbo", which reads voices_v3_turbo.json (14 Vietnamese voices, default Phạm Tuyên). The two sets have zero overlap - verified by comparing the key sets - so nothing on our path is NC-licensed. If anyone ever switches the engine mode away from v3turbo, or a preset named Binh/Tuyen/Vinh/Doan/Ly/Ngoc shows up in the picker, the commercial position changes and must be rechecked.
Install tiers - keep them separate
Two independent levels. Never collapse them into one check:
- Speech only:
pip install vieneu. Torch-free (ONNX Runtime), ~30 MB of wheels. This is enough for all 14 preset voices. - Cloning: additionally
pip install torch torchaudio, 2-3 GB.
start/doctor.mjs reports these as vieneu and vieneu-clone, and only asks about torch once vieneu is present. Forcing a 2-3 GB torch download on someone who only wants offline narration is the mistake to avoid.
The first synthesis after a server start downloads ~1 GB of model from HuggingFace and takes 30s; subsequent process starts take ~15s to load. Always surface this in the UI, otherwise the first click looks like a hang.
Known issues
Symptom: every transcription (video projects, Text to video, narration) fails with TypeError: open() got an unexpected keyword argument 'metadata_errors' from faster_whisper/audio.py.
Cause: PyAV 17+ dropped metadata_errors, but faster-whisper 1.2.x still passes it to av.open() (verified 2026-10: av 19.0.1 + faster-whisper 1.2.1). An unpinned pip install faster-whisper pulls the newest av.
Fix: .runtime/venv/Scripts/python.exe -m pip install "av>=11,<17" (16.1.0 verified). start/doctor.mjs now pins it in the faster-whisper fix, but a venv that already has av 19 must be downgraded by hand - the doctor's probe only checks import faster_whisper, which still succeeds.
Symptom: every VieNeu call (narration, Text to video, voice preview) returns 503 LOAD_FAILED for the rest of the server's life, yet the same Python loads Vieneu() and synthesizes fine from the command line.
Cause: one failed model load (seen once as External data path escapes model directory while the ~1GB model was still being fetched on first use) set loadFailure in ttsLocal.ts. That flag overrode every later probe, and the only way to clear it was probeLocalEngine(true) - which no route called. An agent mid-edit had no way out but asking the user to restart the backend.
Fix: the flag now expires after 2 minutes (the next call retries the real load), and GET /api/tts/engines?refresh=1 - the Voices page "Kiểm tra lại" button - clears it immediately. To check a machine by hand, run the worker with the server's env (HF_HOME and HUGGINGFACE_HUB_CACHE both = .runtime/models) and send {"id":1,"cmd":"synth","text":"Xin chào","voice":"Mai Anh","out":"<abs path>.wav"} on stdin.
Symptom: Cloning fails with ImportError: TorchCodec is required for load_with_torchcodec, or RuntimeError: Could not load libtorchcodec.
Cause: add_voice() reads the reference through torchaudio.load(). From torchaudio 2.9 every backend routes through torchcodec, which needs FFmpeg shared libraries - the usual Windows FFmpeg builds are static, so it can never load. Installing torchcodec does not fix it. Passing backend="soundfile" does not fix it either (verified: all three backends raise the same error).
Fix: monkeypatch torchaudio.load to use soundfile before importing vieneu. soundfile is already a vieneu dependency, so this adds nothing to install. Guard it in try/except so that a machine without torch still gets preset voices.
import torch, torchaudio, soundfile as sf
def _load(uri, *a, **kw):
data, sr = sf.read(str(uri), dtype="float32", always_2d=True)
return torch.from_numpy(data.T).contiguous(), sr
torchaudio.load = _load
Symptom: Voices are filed under the wrong region, or every male voice looks southern.
Cause: list_preset_voices() returns labels shaped Name — <Nam|Nữ> · <Bắc|Trung|Nam> · Phong cách <...>. Field 1 is gender, field 2 is region, and the word "Nam" means both "male" and "southern".
Fix: parse strictly by position, never by searching for the substring "Nam". Regression cases: Thái Sơn = gender nam / region nam; Mai Anh = gender nu / region bac.
Symptom: Cloned voices vanish after a restart, or a stray voice appears in the preset list and leaks into vieneu's own error messages.
Cause: tts.save_voices() (and add_voice(..., save=True)) rewrites vieneu/assets/voices_v3_turbo.json inside site-packages, permanently adding your test voice as a 15th "preset". This happened for real during development.
Fix: never call either. The store lives at assets/voices/ (library.json plus <id>/ref.wav) and voices are re-registered into the worker on demand with add_voice(..., save=False). Registration costs ~3s and is lost whenever the worker restarts, so track what the live process actually has. As defence in depth the worker drops any preset whose description does not parse into gender · region · style, and listLocalVoices() drops presets colliding with a store id - a contaminated install still yields a clean 14. To repair one by hand, delete the offending key from voices_v3_turbo.json or pip install --force-reinstall vieneu.
Symptom: A storytelling voice reads like a newsreader.
Cause: infer() has its own style= argument defaulting to "tu_nhien", which is independent of the voice you picked.
Fix: look up each preset's own style (get_preset_voice(name) returns {gender, style, description, ...} as real fields) and pass it into infer(). Only region has to be parsed out of the description string.
Symptom: Timeline drifts against the narration.
Cause: estimating duration from character count.
Fix: every duration is measured with ffprobe, exactly as on the Gemini path. This is not engine-specific advice - see the video-pipeline skill.
Architecture notes
apps/server/python/vieneu_worker.pyis a long-lived worker speaking JSON-lines on stdin/stdout. stdout is protocol only; all logging goes to stderr. It mustsys.stdout.reconfigure(encoding="utf-8")first thing or Vietnamese corrupts on the Windows cp1252 console.pingmust answer without loading the model - it backs the availability check.- One worker, requests serialised (the model is not re-entrant), 10-minute idle shutdown because it holds ~1 GB of RAM.
- Output is already 48 kHz mono, which equals the pipeline's
OUT_SAMPLE_RATE, so no resampling. It still goes through the sameTRIM_FILTERand gap-insertion path as Gemini insynthScript()- two engines with two joining strategies would drift audibly at the seams. - The Perth watermark is left at its default (on). It is inaudible and is the responsible default for cloned speech.
Reference audio guidance for users
3-8 seconds, clean speech, no background music, no noise. Longer is not better - the model only uses the head, so anything past ~8s is discarded. Below 3s the embedding is too weak and the clone stops resembling the speaker.