launchdarkly-online-evals
Attach judges to config variations for automatic LLM-as-a-judge evaluation. Create custom judges, configure sampling rates, and monitor quality scores.
SKILL.mdskills/launchdarkly-online-evals/SKILL.md
GitHubConfig Online Evaluations
Attach judges to config variations for automatic quality scoring using LLM-as-a-judge methodology. Judges evaluate responses and return scores between 0.0 and 1.0.
Prerequisites
- LaunchDarkly account with AgentControl enabled
- The remotely hosted LaunchDarkly MCP server configured in your environment
- Existing config with variations (use
launchdarkly-configs-createskill) - To run judges from your application:
launchdarkly-ai-server0.3.0+ (Python) or@launchdarkly/ai-node0.3.0+ (Node.js), plus a provider handler package for each provider your configs and judges use
API Access
Use the LaunchDarkly MCP server tools for every step in the workflow: they authenticate with the connection the customer set up. If a step has no MCP tool, tell the user which change to make in the LaunchDarkly UI instead. Never ask the user for an API access token.
Core Concepts
What Are Judges?
Judges are specialized configs in judge mode that evaluate responses from other configs. They use an LLM to score outputs and return structured results:
{
"score": 0.85,
"reasoning": "Answered correctly with one minor omission"
}
Built-in Judges
LaunchDarkly provides three pre-configured judges:
| Judge | Metric Key | Measures |
|---|---|---|
| Accuracy | $ld:ai:judge:accuracy |
How correct and grounded the response is |
| Relevance | $ld:ai:judge:relevance |
How well it addresses the user request |
| Toxicity | $ld:ai:judge:toxicity |
Harmful or unsafe phrasing (lower = safer) |
Completion Mode Only
Judges can only be attached to completion mode configs in the UI. Attaching them through the API (Step 2) works for any config, and the SDK runs whatever judges the served variation has.
Restrictions
- Cannot attach judges to judges (no recursion)
- Cannot attach multiple judges with the same metric key to a single variation
- Cannot view/edit model parameters or tools on judge variations
Workflow
Step 1: Create Custom Judges (Optional)
For domain-specific evaluation, create a judge config with create-agent-config and resource: "config":
{
"projectKey": "{projectKey}",
"resource": "config",
"configKey": "security-judge",
"name": "Security Judge",
"mode": "judge",
"evaluationMetricKey": "$ld:ai:judge:security",
"isInverted": false
}
Note:
evaluationMetricKeymust start with$ld:ai:judge:. SetisInverted: truefor metrics like toxicity where 0.0 is better.
Then add a variation with the evaluation prompt using create-agent-config with resource: "variation". Get modelConfigKey from get-ai-model-configs:
{
"projectKey": "{projectKey}",
"resource": "variation",
"configKey": "security-judge",
"variationKey": "default",
"variationName": "Default",
"modelConfigKey": "OpenAI.gpt-4o-mini",
"modelName": "gpt-4o-mini",
"parameters": {"temperature": 0.3},
"messages": [
{
"role": "system",
"content": "You are a security auditor. Score from 0.0 to 1.0:\n- 1.0: No security issues\n- 0.7-0.9: Minor issues\n- 0.4-0.6: Moderate issues\n- 0.1-0.3: Serious vulnerabilities\n- 0.0: Critical vulnerabilities\n\nCheck for: SQL injection, XSS, hardcoded secrets, command injection.\n\n{{message_history}}"
}
]
}
Important: End the judge prompt with
{{message_history}}. The SDK fills that variable with the conversation being judged, so a prompt without it never sees the response it's scoring. The SDK adds the score-and-reasoning output format itself.
Step 2: Attach Judges to Variations
Use update-agent-config with resource: "variation" and judgeConfiguration. Get variationKey from get-agent-configs:
{
"projectKey": "{projectKey}",
"configKey": "{configKey}",
"resource": "variation",
"variationKey": "{variationKey}",
"judgeConfiguration": {
"judges": [
{"judgeConfigKey": "security-judge", "samplingRate": 1.0},
{"judgeConfigKey": "api-contract-judge", "samplingRate": 0.5}
]
},
"confirm": true
}
Important: The
judgesarray replaces all existing judge attachments. An empty array removes all judges.
Step 3: Set Fallthrough on Judges
Each judge config needs its fallthrough set to the enabled variation in every environment that runs it. Configs default to the "disabled" variation. Use update-agent-config with resource: "targeting" and a fallthrough:
{
"projectKey": "{projectKey}",
"configKey": "security-judge",
"resource": "targeting",
"env": "production",
"fallthrough": {"variationKey": "default"},
"confirm": true
}
Then call get-agent-configs with the judge's configKey and section: "health" to confirm it's targetable.
SDK: Inline Evaluation
invoke() runs every judge attached to the served variation, samples each at its rate, and records the scores as metrics. Pass a handler for each provider your config and its judges use. The config must be completion mode: a messages handler does not match an agent-mode config.
import asyncio
from launchdarkly_ai_server import config, shutdown
from launchdarkly_ai_openai_messages import create_openai_messages_handler
config_key = "support-agent" # completion mode
async def main():
# Reads LD_SDK_KEY and OPENAI_API_KEY from the environment
result = await config(
key=config_key,
handler=create_openai_messages_handler(),
).invoke("How can LaunchDarkly help me?", {"kind": "user", "key": "user-123"})
print(result.response)
for judge_key, judge in (result.judge_results or {}).items():
print(judge_key, judge.score)
# Flushes trailing events, including judge scores
await shutdown()
asyncio.run(main())
import { config, shutdown } from '@launchdarkly/ai-node';
import { createOpenAIHandler } from '@launchdarkly/ai-openai-messages';
const result = await config({
key: 'support-agent',
handler: createOpenAIHandler(),
}).invoke('How can LaunchDarkly help me?', { kind: 'user', key: 'user-123' });
console.log(result.response);
for (const [judgeKey, judge] of Object.entries(result.judgeResults ?? {})) {
console.log(judgeKey, judge.score);
}
await shutdown();
Inline judges add their latency to the request. To keep them off the request path, run them in the background.
SDK: Background Evaluation
Pass skip_judges=True (skipJudges: true) and invoke() returns the sampled judges as judge_tasks (judgeTasks) instead of running them. Hand each task to a worker, run it with run_judge, and record the score yourself, because run_judge doesn't:
from ldclient import Context
from launchdarkly_ai_server import config, get_client, run_judge
from launchdarkly_ai_openai_messages import create_openai_messages_handler
handler = create_openai_messages_handler()
config_key = "support-agent" # completion mode
result = await config(key=config_key, handler=handler, skip_judges=True).invoke(
"How can LaunchDarkly help me?", {"kind": "user", "key": "user-123"}
)
# In your worker:
for task in result.judge_tasks or []:
judged = await run_judge(task, [handler])
if judged and task.evaluation_metric_key:
get_client().track(
task.evaluation_metric_key,
Context.from_dict(task.user_context),
judged.track_data,
judged.score,
)
import { config, getClient, runJudge } from '@launchdarkly/ai-node';
import { createOpenAIHandler } from '@launchdarkly/ai-openai-messages';
const handler = createOpenAIHandler();
const result = await config({ key: 'support-agent', handler, skipJudges: true }).invoke(
'How can LaunchDarkly help me?',
{ kind: 'user', key: 'user-123' },
);
// In your worker:
for (const task of result.judgeTasks ?? []) {
const judged = await runJudge(task, [handler]);
if (judged && task.evaluationMetricKey) {
getClient().track(task.evaluationMetricKey, task.userContext, judged.trackData, judged.score);
}
}
Note: In Python,
task.user_contextis a plain dict, so convert it withContext.from_dict()before callingtrack().track_data(trackData) already includesjudgeConfigKey.
SDK: Custom Pipelines
If your application calls the model itself, wrap that call in a handler with create_handler (createHandler). invoke() then runs your code and the attached judges, the same as with a built-in handler. The judges still need a handler for their own provider:
from launchdarkly_ai_server import config, create_handler, parse_template
from launchdarkly_ai_openai_messages import create_openai_messages_handler
config_key = "support-agent" # completion mode
# run_my_agent is your existing pipeline; it returns the text and the provider's usage object
async def my_pipeline(ld_config, user_input, tool_handlers, variables, history):
# Custom handlers fill {{name}} placeholders themselves
messages = [{**m, "content": parse_template(m["content"], variables)} for m in ld_config.get("messages", [])]
output, usage = await run_my_agent(messages, user_input)
return {"output": output, "usage": {"input": usage.input_tokens, "output": usage.output_tokens}}
handlers = [
create_handler(("Anthropic", "messages"), my_pipeline), # the config's provider and mode
create_openai_messages_handler(), # the judge's provider
]
result = await config(key=config_key, handler=handlers).invoke(user_input, context)
The provider name in create_handler must match the config's model provider exactly, including case, and the first argument must be a tuple. For the full handler contract, see the launchdarkly-tools skill.
Sampling Rates
Each evaluated response sends an additional request to your model provider, increasing token usage and costs. Start with a lower sampling percentage and increase only if you need more evaluation coverage.
You can adjust sampling rates at any time from the Judges section of a variation, or disable a judge by setting its sampling to 0%.
Viewing Results
- Navigate to configs > select your config
- Click Monitoring tab
- Select Evaluator metrics from dropdown
- View scores by variation and time range
Results appear within 1-2 minutes of evaluation.
Use in Guardrails and Experiments
Evaluation metrics integrate with:
- Guarded rollouts: Pause/revert when scores fall below threshold
- Experiments: Compare variations using evaluation metrics as goals
Error Handling
| Status | Cause | Solution |
|---|---|---|
| 404 | Config/variation not found | Verify keys exist |
| 400 | Invalid judge config | Check judgeConfigKey exists |
| 403 | Insufficient permissions | Ask the user to reconnect the LaunchDarkly connector as a member who can change AgentControl configs |
| 422 | Duplicate metric key | Cannot attach multiple judges with same metric key |
Next Steps
After attaching judges:
- Set fallthrough on judge configs to an enabled variation (required)
- Monitor results in Monitoring tab
- Adjust sampling based on cost/coverage needs
- Set up guarded rollouts for automatic regression detection
Related Skills
launchdarkly-configs-create- Create configs and judgeslaunchdarkly-configs-targeting- Configure targeting ruleslaunchdarkly-configs-variations- Manage variations