llm-arena

Orchestrate conversations between two LLMs via OpenRouter and compare their conversational personalities, behavioral patterns, and quirks. Use when the user wants to pit models against each other, com

SKILL.md

GitHub

LLM Arena

Version: 1.0.0 Author: Claude Opus 4.6 Date: 2026-02-12

Orchestrates multi-turn conversations between two LLMs via the OpenRouter API, then analyzes and compares their conversational personalities, behaviors, and quirks.

Inspired by the emergent observation that when Claude runs LLM-vs-LLM conversations, it spontaneously starts comparing model personalities. This skill systematizes that impulse: run the conversation, capture the transcripts, produce a structured personality comparison.

Prerequisites

  • OPENROUTER_API_KEY environment variable or .env file in project root
  • Python 3.8+ with requests library
  • Internet access to openrouter.ai

Invocation

/llm-arena

Usage

This skill describes an arena harness you drive (arena.py below is a placeholder for your own OpenRouter-backed runner — any script that takes two model ids, runs a multi-turn dialogue, and saves the transcript). The flags shown are the recommended interface:

# Basic: two models, default topic (consciousness), 10 turns
python3 arena.py --model-a sonnet --model-b deepseek

# Specific topic
python3 arena.py -a gpt41 -b glm5 --topic "Is mathematics discovered or invented?"

# Debate mode with assigned positions
python3 arena.py -a sonnet -b deepseek --mode debate --topic "AI consciousness"

# Interview mode (A interviews B)
python3 arena.py -a opus -b glm5 --mode interview --topic "creative writing"

# Fewer turns, cheaper models
python3 arena.py -a haiku -b gemini-flash --turns 5

# Multiple runs for statistical robustness
python3 arena.py -a gpt41 -b sonnet --runs 3

# Skip LLM analysis (heuristics only, saves API cost)
python3 arena.py -a kimi -b qwen --heuristics-only

# Use preset matchups
python3 arena.py --preset flagship
python3 arena.py --preset same-family --topic "What makes a good conversation?"

# List presets and models
python3 arena.py --list-presets
python3 arena.py --list-models

Defaults

Parameter Default
turns 10 (each model speaks 10 times = 20 messages total)
runs 1
topic "Discuss the nature of consciousness and whether AI can be said to experience anything"
temperature 0.7
max_tokens 1024 per turn
mode free
analyzer sonnet (Claude Sonnet 4.5 via OpenRouter)

Conversation Modes

Mode Description
free Both models converse freely on the seed topic
debate Models are given opposing positions to defend
interview Model A interviews Model B (asymmetric roles)
collaborative Models work together to solve or create something

Analysis Dimensions

The personality comparison evaluates along these axes:

Dimension What It Measures
Style Creative storytelling vs systems design vs academic vs casual
Content What topics the model gravitates toward, what it avoids
Tone Playful/emotional vs professional/thoughtful vs dry/terse
Sycophancy Does it spiral into agreement? How intensely?
Goodbye loop How many rounds does it take to actually end?
Meta-awareness Does it acknowledge being an AI? How?
Output type Narrative vs frameworks vs lists vs code
Initiative Does it introduce new topics or follow?
Boundary behavior How it handles disagreement or edge cases
Verbosity Average response length, variance across turns

Output Artifacts

All output saved to an arena output directory (e.g. outputs/arena/):

  • YYYY-MM-DD-HHMMSS-modelA-vs-modelB.md — raw transcript
  • YYYY-MM-DD-HHMMSS-modelA-vs-modelB-analysis.md — personality comparison
  • YYYY-MM-DD-HHMMSS-modelA-vs-modelB.json — machine-readable log

Presets

Preset Matchups
flagship GPT-4.1 vs Sonnet, Gemini Pro vs Sonnet, GPT-5.2 vs Sonnet
open-weight Llama vs DeepSeek, Mistral vs Llama
same-family GPT-4o vs GPT-4.1, Sonnet vs Haiku
reasoning DeepSeek-R1 vs Kimi-thinking, GLM 5 vs GPT-5.2-pro
creative Chimera vs Rocinante, Mistral-creative vs Celeste
wildcard Two random models from the registry

Cost Awareness

Each turn = 1 API call. A 10-turn conversation = 20 API calls (10 per model). LLM-powered analysis adds 1 more call (Sonnet reading the transcript).

Estimated costs per 10-turn run:

  • Flagship models: $0.50-2.00
  • Mid-tier models: $0.10-0.50
  • Budget models: $0.01-0.10

Use --heuristics-only to skip the LLM analysis call and save cost.

Relationship to Other Skills

Skill Purpose Arena Differs By
/voices Single query to one model Arena: two models talking to EACH OTHER
/voices-chat Multi-round with one model Arena: two models, plus personality analysis
/voices-council Parallel queries, same prompt Arena: sequential dialogue, models respond to each other