Agora Conversational AI Engine

Purpose

Build real-time voice AI features using Agora's Conversational AI Engine. This skill covers the full pipeline: RTC channel management, ASR (speech recognition), LLM integration, TTS (text-to-speech), and voice activity detection (VAD). Supports both standalone voice apps and voice features embedded in existing applications.

Architecture

User (mic) → Agora RTC Channel → Conversational AI Engine
                                    ├─ ASR (speech → text)
                                    ├─ LLM (reasoning)
                                    ├─ TTS (text → speech)
                                    └─ VAD (interruption detection)
                                 → Agora RTC Channel → User (speaker)

Latency: ~650ms voice-to-voice (ASR + LLM + TTS combined) Interruption: User can interrupt AI mid-sentence; VAD detects speech overlap

Core Concepts

Concept	Description
App ID	Project identifier from console.agora.io
App Certificate	Secret for token generation (never expose client-side)
Customer ID/Secret	REST API auth (Basic auth, base64 encoded)
Channel	Named audio room; users and agents join the same channel
RTC Token	Short-lived token granting channel access for a specific UID
Agent UID	The AI agent's identity in the channel (string or numeric)

Required Credentials

# Server-side only (never NEXT_PUBLIC_)
AGORA_APP_ID=            # From console.agora.io
AGORA_APP_CERTIFICATE=   # From console.agora.io
AGORA_CUSTOMER_ID=       # RESTful API credentials
AGORA_CUSTOMER_SECRET=   # RESTful API credentials

# Client-side (safe to expose)
NEXT_PUBLIC_AGORA_APP_ID=         # Same as AGORA_APP_ID
NEXT_PUBLIC_AGORA_AGENT_UID=333   # Agent's channel UID

Get credentials at: https://console.agora.io

• Create project → Get App ID + Certificate
• Account → RESTful API → Get Customer ID + Secret
• Free tier: 10,000 RTC min/mo + 300 Conversational AI min/mo

NPM Packages

npm install agora-rtc-react agora-token

Package	Purpose
agora-rtc-react	React hooks for RTC (useJoin, useLocalMicrophoneTrack, usePublish, useRemoteUsers)
agora-token	Server-side RTC token generation (RtcTokenBuilder)

Key Patterns

1. Token Generation (Server-Side)

import { RtcTokenBuilder, RtcRole } from "agora-token";

function generateToken(appId: string, cert: string, channel: string, uid: number | string) {
  const expiry = Math.floor(Date.now() / 1000) + 3600;
  return RtcTokenBuilder.buildTokenWithUid(
    appId, cert, channel, uid, RtcRole.PUBLISHER, expiry, expiry
  );
}

2. Start Conversational AI Agent (REST API)

POST {baseUrl}/{appId}/join
Authorization: Basic base64(customerId:customerSecret)
Content-Type: application/json

Base URL: https://api.agora.io/api/conversational-ai-agent/v2/projects

{
  "name": "unique-session-name",
  "properties": {
    "channel": "channel-name",
    "token": "agent-rtc-token",
    "agent_rtc_uid": "333",
    "remote_rtc_uids": ["user-uid"],
    "enable_string_uid": false,
    "idle_timeout": 30,
    "asr": {
      "language": "en-US",
      "task": "conversation"
    },
    "llm": {
      "url": "https://api.groq.com/openai/v1/chat/completions",
      "api_key": "groq-key",
      "system_messages": [{"role": "system", "content": "You are a helpful assistant."}],
      "greeting_message": "Hello! How can I help?",
      "failure_message": "Please wait a moment.",
      "max_history": 10,
      "params": {
        "model": "llama-3.3-70b-versatile",
        "max_tokens": 1024,
        "temperature": 0.7
      },
      "input_modalities": ["text"],
      "output_modalities": ["text", "audio"]
    },
    "vad": {
      "silence_duration_ms": 480,
      "speech_duration_ms": 15000,
      "threshold": 0.5,
      "interrupt_duration_ms": 160,
      "prefix_padding_ms": 300
    },
    "tts": {
      "vendor": "microsoft",
      "params": {
        "key": "azure-speech-key",
        "region": "eastus",
        "voice_name": "en-US-AndrewMultilingualNeural",
        "rate": 1.1,
        "volume": 70
      }
    }
  }
}

Response: { "agent_id": "xxx", "create_ts": 123, "state": "running" }

3. Stop Agent (REST API)

POST {baseUrl}/{appId}/agents/{agent_id}/leave
Authorization: Basic base64(customerId:customerSecret)

4. Next.js Client Component (SSR-Disabled)

Agora uses WebRTC (window APIs) — must disable SSR:

import dynamic from "next/dynamic";

const AgoraProvider = dynamic(async () => {
  const { AgoraRTCProvider, default: AgoraRTC } = await import("agora-rtc-react");
  return {
    default: ({ children }) => {
      const client = useMemo(
        () => AgoraRTC.createClient({ mode: "rtc", codec: "vp8" }), []
      );
      return <AgoraRTCProvider client={client}>{children}</AgoraRTCProvider>;
    },
  };
}, { ssr: false });

const VoiceChat = dynamic(() => import("./VoiceChat"), { ssr: false });

5. Voice Chat Component Hooks

import {
  useRTCClient, useLocalMicrophoneTrack, useRemoteUsers,
  useClientEvent, useIsConnected, useJoin, usePublish, RemoteUser
} from "agora-rtc-react";

// Join channel
useJoin({ appid: APP_ID, channel, token, uid }, true);

// Publish local mic
const { localMicrophoneTrack } = useLocalMicrophoneTrack(true);
usePublish([localMicrophoneTrack]);

// Listen for agent
const remoteUsers = useRemoteUsers();
useClientEvent(client, "user-joined", (user) => { /* agent connected */ });
useClientEvent(client, "user-left", (user) => { /* agent disconnected */ });

// Token renewal
useClientEvent(client, "token-privilege-will-expire", async () => {
  const newToken = await fetchNewToken();
  await client.renewToken(newToken);
});

// Render remote audio
remoteUsers.map(user => <RemoteUser key={user.uid} user={user} />);

TTS Providers

Provider	Config Key	Voices
Microsoft Azure	vendor: "microsoft"	en-US-AndrewMultilingualNeural, en-US-AriaNeural
ElevenLabs	vendor: "elevenlabs"	Custom voice IDs, eleven_flash_v2_5 model

Microsoft TTS Config

{ "vendor": "microsoft", "params": { "key": "...", "region": "eastus", "voice_name": "en-US-AndrewMultilingualNeural", "rate": 1.1, "volume": 70 } }

ElevenLabs TTS Config

{ "vendor": "elevenlabs", "params": { "key": "...", "voice_id": "XrExE9yKIg1WjnnlVkGX", "model_id": "eleven_flash_v2_5" } }

LLM Providers for Voice Agent

Any OpenAI-compatible API works:

Provider	URL	Model
Groq	https://api.groq.com/openai/v1/chat/completions	llama-3.3-70b-versatile
OpenAI	https://api.openai.com/v1/chat/completions	gpt-4o-mini
Together	https://api.together.xyz/v1/chat/completions	meta-llama/Meta-Llama-3.1-70B

VAD (Voice Activity Detection) Tuning

Parameter	Default	Purpose
silence_duration_ms	480	Silence before AI responds
speech_duration_ms	15000	Max user speech duration
threshold	0.5	Voice detection sensitivity (0-1)
interrupt_duration_ms	160	How quickly user can interrupt AI
prefix_padding_ms	300	Audio captured before speech detected

Lower silence_duration_ms = faster responses but more false triggers. Lower interrupt_duration_ms = easier to interrupt AI mid-sentence.

Best Practices

1. Never expose App Certificate or Customer Secret client-side — generate tokens server-side only
2. Use numeric UIDs for simplicity; string UIDs require enable_string_uid: true
3. Set idle_timeout to auto-disconnect agents after silence (saves Conversational AI minutes)
4. Keep system prompts voice-optimized — short responses, no markdown, conversational tone
5. Use dynamic() with ssr: false for all Agora components in Next.js
6. Implement token renewal via token-privilege-will-expire event to avoid disconnections
7. Test with the Agora Web Demo at webdemo.agora.io before building custom UI
8. Monitor usage at console.agora.io to stay within free tier limits

Common Pitfalls

Pitfall	Impact	Fix
SSR rendering Agora components	window is not defined crash	dynamic(() => import(...), { ssr: false })
Expired tokens	Silent disconnection	Listen for token-privilege-will-expire, renew proactively
Agent UID collision	Agent can't join channel	Use unique agent UIDs per session
No idle_timeout	Agent runs forever, burns minutes	Set idle_timeout: 30 (seconds)
Long LLM responses	Slow voice replies, bad UX	Set max_tokens: 256-512 for voice, use fast models
Missing enable_string_uid	Agent join fails silently	Set to true if agent UID contains letters

Reference Implementation

• Blog: https://www.agora.io/en/blog/build-a-conversational-ai-app-with-nextjs-and-agora/
• Repo: https://github.com/AgoraIO-Community/conversational-ai-nextjs-client
• Docs: https://docs.agora.io/en/conversational-ai/overview/product-overview