transcription-voice-system

Guide for working on the Transcription, Voice, and Wake Word system spanning Rust (Tauri), Python (FastAPI), and React layers. Use when editing Voice.tsx, use-transcription.ts, use-wake-word.ts, use-t

SKILL.md

GitHub

Transcription & Voice System

Mandatory: Keep This Skill Current

Any agent that makes significant modifications to this system MUST update this skill file before finishing. If you discover that the code has drifted from what this skill describes — fix the skill. Outdated guidance here causes real bugs (infinite render loops, state divergence, SIGABRT on quit). Treat this file as a living specification.

Architecture (4 Layers)

RUST (Tauri IPC)          — Whisper inference, audio capture, wake word (whisper-tiny)
PYTHON (FastAPI :22140)   — OpenWakeWord ONNX, AI audio tools, wake word settings
REACT (Vite :1420)        — Hooks, contexts, UI, session persistence
STORAGE                   — localStorage, SQLite, filesystem, Supabase (settings only)

Critical Rules — Read Before Touching Anything

1. Singleton Context Pattern (NEVER violate)

useTranscription() runs once inside TranscriptionProvider at the app root. All consumers use useTranscriptionApp() from TranscriptionContext.tsx. Never call useTranscription() directly in a page or component — the Rust side only allows one active audio stream; multiple hook instances cause state divergence.

// WRONG — creates independent state instance
const [state, actions] = useTranscription();

// CORRECT — reads from singleton context
const { state, actions } = useTranscriptionApp();

Same pattern applies: useLlmApp(), useTtsApp(). useWakeWord() is the one exception — it's called only in Voice.tsx and published to WakeWordContext via usePublishWakeWord().

2. useMemo on ALL Return Objects

Every hook returning [state, actions] must wrap both in useMemo. A plain object literal is a new reference every render, causing infinite loops through any useEffect or context consumer.

// Both state AND actions must be memoized
const state = useMemo(() => ({ ... }), [deps]);
const actions = useMemo(() => ({ ... }), [deps]);

Hooks that follow this pattern: useTranscription, useWakeWord, useTranscriptionSessions. Parity tests enforce this — see tests/parity/test_transcription_parity.py.

3. Never Use actions as a useEffect Dependency

Even with useMemo, always list the specific callback you call, never the entire actions object.

4. Session Writes Are Debounced

appendSegments() in sessions.ts buffers segments in memory and flushes to localStorage every 1s. Call flushNow() before finalizeSession() or component unmount. The useTranscriptionSessions hook registers a flush callback automatically.

5. Rust Shutdown Order Matters

graceful_shutdown_sync() in lib.rs must: stop wake word → join (2s) → stop recording → join (5s) → kill llama-server → drop TranscriptionState → kill sidecar. Dropping WhisperContext before joining threads causes SIGABRT from GGML's atexit handler.

6. Wake Word Keyword Is Read Dynamically

The wake word thread reads state.keyword from the Mutex on each detection window (not captured once at start). configure_wake_word persists to TranscriptionConfig.wake_keyword on disk.

File Map

Rust (desktop/src-tauri/src/transcription/)

File Purpose
commands.rs 26+ Tauri commands, state types, transcription loop
manager.rs WhisperContext wrapper (model load + inference)
audio_capture.rs CPAL mic → rubato 16kHz mono resampling
config.rs TranscriptionConfig persistence (JSON) — includes wake_keyword
downloader.rs HuggingFace model download with cancel support
hardware.rs RAM/CPU/GPU/Metal/CUDA detection (platform-specific)
model_selector.rs 3-tier model recommendation
wake_word.rs Whisper-tiny 2s window KWS thread

Python (app/)

File Purpose
api/wake_word_routes.py REST + SSE for OpenWakeWord engine
services/wake_word/service.py OWW singleton: sounddevice → ONNX → events
services/wake_word/models.py OWW model catalog + HF download
api/settings_routes.py (lines 124-155) Wake word settings GET/PUT
tools/tools/audio.py 4 AI tools: ListAudioDevices, RecordAudio, PlayAudio, TranscribeAudio

React (desktop/src/)

File Purpose
hooks/use-transcription.ts Whisper lifecycle, recording, download queue
hooks/use-wake-word.ts Dual-engine (whisper/OWW) state machine
hooks/use-transcription-sessions.ts Session CRUD over localStorage
contexts/TranscriptionContext.tsx Singleton — useTranscriptionApp()
contexts/WakeWordContext.tsx Publish/subscribe from Voice.tsx → toolbar
contexts/TranscriptionSessionsContext.tsx Shared session store
contexts/AudioDevicesContext.tsx Single source for mic devices
lib/transcription/sessions.ts localStorage CRUD with debounced writes
lib/transcription/types.ts All TypeScript types
pages/Voice.tsx Main page (5 tabs, ~2800 lines)
pages/WakeWord.tsx Wake word tab (engine picker, OWW models)
components/TranscriptOverlay.tsx Floating Tauri window (/#/overlay)
components/CompactRecorderWindow.tsx OS-resized compact recorder
components/TranscriptionMiniMode.tsx In-page floating panel
components/WakeWordOverlay.tsx Ambient glow indicator
components/WakeWordControls.tsx Status-bar control strip
components/recording/RecordingMicButton.tsx Reusable mic toggle
components/recording/RmsLevelBar.tsx Audio level meter
components/quick-actions/QuickTranscriptModal.tsx Quick record dialog
components/documents/NoteEditor.tsx Inline dictation

Provider Nesting Order (App.tsx)

DevTerminalProvider > DownloadManagerProvider > TtsProvider > LlmProvider
  > WakeWordProvider > TranscriptionSessionsProvider > PermissionsProvider
    > AudioDevicesProvider > TranscriptionProvider > TooltipProvider

TranscriptionProvider must be inside AudioDevicesProvider (it uses AudioDevicesContext).

State Ownership

State Storage Survives restart?
Transcription recording TranscriptionContext (React) No
Session history (500 max) localStorage key matrx-transcription-sessions Yes
Selected audio device localStorage key matrx-selected-audio-device Yes
Wake word engine pref SQLite via PUT /settings/wake-word Yes
App-level settings (enabled, auto-init, model, etc.) localStorage + Supabase cloud sync Yes
Whisper model files ~/{app_data}/models/*.bin Yes
Rust config (model, setup_complete, wake_keyword) ~/{app_data}/transcription.json Yes
OWW models ~/.matrx/oww_models/ Yes

No cloud sync for session data. Only settings sync via the generic AppSettings pipeline.

Dual Wake Word Engines

Whisper (Rust) OpenWakeWord (Python)
Backend Tauri IPC commands + events REST + SSE via FastAPI
Model ggml-tiny.en.bin (75 MB) ONNX models (~3 MB each)
Latency ~2s (full ASR on 2s windows) ~150ms (purpose-built KWS)
CPU High (runs full whisper) Low (small ONNX model)
Events Tauri: wake-word-detected, -rms, -mode, -error SSE: identical event names
Config Keyword in TranscriptionConfig Model/threshold in SQLite

useWakeWord abstracts both behind the same WakeWordHookState / WakeWordHookActions interface.

Recording Pipeline

1. startRecording() → Tauri IPC "start_transcription"
2. Rust: Opens AudioCapture (CPAL) → 16kHz mono via rubato
3. Rust: 2s silence calibration → emits "whisper-calibrated"
4. Rust: 5s chunk accumulation → ctx.transcribe_with_params()
5. Rust: Emits "whisper-segment" per segment (filtered for hallucinations)
6. React: useTranscription accumulates segments in state
7. React: sessionsActions.append() → debounced localStorage write
8. stopRecording() → sets flag → Rust flushes remaining audio
9. Rust: emits "whisper-stopped" → React tears down listeners

AI Polish Pipeline

Uses local LLM (llama-server sidecar), not cloud:

  1. useLlmPipeline.run("polish_transcript", { transcript })
  2. Structured output: { title, description, tags, cleaned }
  3. sessionsActions.applyPolish() preserves rawText on first run

Adding a New Feature — Checklist

  • New hook state → wrap state and actions in useMemo
  • New context → add provider to App.tsx in correct nesting order
  • New Tauri command → add to lib.rs generate_handler![], verify cargo check
  • New settings key → add to TS AppSettings, Python DEFAULT_SETTINGS, SECTION_KEYS
  • New AI tool → register in dispatcher.py + tool_schemas.py
  • Run uv run pytest tests/parity/test_transcription_parity.py to verify
  • Update this skill file if architecture changed

Common Pitfalls

Mistake Consequence Prevention
Call useTranscription() in a page Duplicate instance, state divergence Always useTranscriptionApp()
Plain object actions return Infinite render loops, API polling floods useMemo wrap — parity tests catch this
[actions] in useEffect dep Re-fires every render Use specific callback: [refreshStatus]
Use ReturnType<typeof useTranscription> for prop types TS2552 / TS2448 — name no longer in scope after context migration Import TranscriptionState, TranscriptionActions from @/hooks/use-transcription
useCallback declared after useEffect that references it TS2448 "used before declaration" — blocks release Always declare useCallback functions above any useEffect that references them
Drop WhisperContext before joining thread SIGABRT / macOS crash report graceful_shutdown_sync() joins first
Read localStorage on every render UI jank with 500 sessions Derive from React state; debounced writes
Forget flushNow() before finalize Up to 1s of segments lost finalizeSession() calls it internally

Tests

File Category What it verifies
tests/parity/test_transcription_parity.py Parity (fast) Context exists, IPC names match, useMemo wrapping, defaults match, cancel command
tests/smoke/test_wake_word.py Smoke (needs engine) Wake word routes, settings round-trip, model list, lifecycle

Run parity tests after any change: uv run pytest tests/parity/test_transcription_parity.py -v

Platform-Specific Notes

  • macOS: TCC microphone permission checked before audio stream opens (check_microphone_permission). Metal acceleration for whisper.
  • Windows: nvidia-smi path probing (C:\Windows\System32\, C:\Program Files\NVIDIA Corporation\NVSMI\). wmic fallback for GPU detection. No mic permission check needed.
  • Linux/WSL: WSL nvidia-smi bridge (/usr/lib/wsl/lib/nvidia-smi). DRM sysfs for AMD/Intel VRAM. ALSA/PulseAudio via CPAL.

Known Gaps (as of 2026-03-26)

  • Voice.tsx is ~2800 lines — tabs should be extracted to components/voice/*.tsx
  • English-only (language("en") hardcoded in Rust)
  • No partial/streaming results (full 5s chunks)
  • No cloud sync for session data
  • Wake word: whisper-tiny is CPU-expensive; sherpa-onnx KWS planned when Rust bindings ship