ai-scientist
SakanaAI AI-Scientist integration — automated scientific discovery via Hermes. Generate ideas, run experiments, write papers, review output — all through MCP tools.
SKILL.mdskills/research/ai-scientist/SKILL.md
GitHubAI Scientist — SakanaAI Integration
Overview
The AI Scientist is a fully automated scientific discovery system from SakanaAI. It runs through a complete research lifecycle:
- Idea Generation → LLM proposes novel research directions
- Experiment Execution → AI writes and runs PyTorch experiments (on GPU)
- Paper Writing → LaTeX paper generation with plots and citations
- Review → LLM-based peer review of the output
This skill connects the AI Scientist to Hermes via an MCP server exposing these capabilities as callable tools.
Prerequisites
- NVIDIA GPU (RTX 3090+ recommended) with CUDA 12.4+
- Python 3.11 (managed via repo venv)
- OpenRouter API key — required for all LLM calls
- LaTeX (pdflatex + chktex) — for paper generation (optional, can skip)
API Key Setup (First Step Before Anything Else)
The OpenRouter API key goes in exactly one place for Hermes-driven use:
${USER_HOME}\AppData\Local\hermes\.env
The line is already there commented out — uncomment it and add your key:
# Find this line:
# OPENROUTER_API_KEY=***
# Change to:
OPENROUTER_API_KEY=sk-or-v1-...
This single location feeds the MCP server, cron jobs, and gateway sessions because the Hermes config resolves ${OPENROUTER_API_KEY} into subprocess environments.
Optional — also set it in the AI Scientist project directory for manual CLI runs:
${MY_REPOS}\Documents\github\AI-Scientist\.env
Architecture
Hermes Agent
│
▼ (MCP tools)
AI Scientist MCP Server ─── stdio/SSE ───> hermes_mcp_server.py
│
├── Calls launch_scientist.py (full pipeline)
├── Runs subprocesses via AI-Scientist venv
└── Returns structured JSON results
MCP Tools Available
After adding ai-scientist to your Hermes mcp_servers config, these tools auto-register:
| Tool | Purpose |
|---|---|
list_templates |
List available experiment templates |
list_results |
List completed experiments |
generate_ideas |
Generate novel research ideas for a template |
run_experiment |
Run full pipeline: ideas → experiments → paper → review |
prepare_baseline |
Create baseline runs (required first step) |
review_paper |
LLM-based peer review of a generated PDF |
system_status |
Check GPU, venv, templates, API keys |
read_template_info |
Read template description and seed ideas |
run_command |
Execute arbitrary commands in the AI-Scientist venv |
Quick Start Workflow
0. Set API Key (required once)
Add to ${USER_HOME}\AppData\Local\hermes\.env:
OPENROUTER_API_KEY=sk-or-...
1. Check system readiness:
system_status()
2. List available templates:
list_templates()
3. Prepare a baseline experiment:
prepare_baseline(experiment="nanoGPT_lite")
4. Generate ideas (FREE OpenRouter model):
generate_ideas(
experiment="nanoGPT_lite",
model="openrouter/qwen/qwen3-coder:free",
num_ideas=5
)
5. Run the full pipeline:
run_experiment(
experiment="nanoGPT_lite",
model="openrouter/qwen/qwen3-coder:free",
num_ideas=1
)
6. Review results:
list_results(experiment="nanoGPT_lite")
review_paper(pdf_path="${MY_REPOS}/Documents/github/AI-Scientist/results/nanoGPT_lite/.../paper.pdf")
Verification
After setup, confirm the MCP server is live:
hermes mcp test ai-scientist
# Expected:
# ✓ Connected (985ms)
# ✓ Tools discovered: 9
hermes mcp list | grep ai-scientist
# ai-scientist ✓ enabled
Check imports, GPU, and templates:
from ai_scientist.llm import _is_openrouter_model
assert _is_openrouter_model('google/gemma-4-31b-it:free')
assert not _is_openrouter_model('gpt-4o')
import torch
print(torch.cuda.is_available(), torch.cuda.get_device_name(0))
# → True NVIDIA GeForce RTX 3090
from pathlib import Path
ts = [d.name for d in Path('templates').iterdir() if (d/'experiment.py').exists()]
print(len(ts)) # → 11
OpenRouter Model Support (FREE Models)
The LLM integration supports any OpenRouter model. Best FREE models ($0/token) per task:
| Task | Model | Context | Why |
|---|---|---|---|
| Idea Generation | openrouter/owl-alpha |
1M | Free, strong agentic model |
| Code Writing | openrouter/qwen/qwen3-coder:free |
1M | Code-specialized, structured output |
| Paper Writing | openrouter/owl-alpha |
1M | Long context, academic writing |
| Review | openrouter/nousresearch/hermes-3-llama-3.1-405b:free |
131K | 405B param, strongest free |
| Fallback | openrouter/qwen/qwen3-coder:free |
1M | Code + general all-rounder |
Also good:
openrouter/deepseek/deepseek-v4-flash:free— 1M ctx, fast, the model I'm running onopenrouter/nvidia/nemotron-3-super-120b-a12b:free— 1M ctx, NVIDIA's strongest open modelopenrouter/moonshotai/kimi-k2.6:free— 262K ctx, strong multimodalopenrouter/meta-llama/llama-3.3-70b-instruct:free— 131K ctx, reliable
Use the format: openrouter/provider/model-name:free (explicit OpenRouter prefix) or just provider/model-name (auto-detected).
Key Files
| File | Purpose |
|---|---|
hermes_mcp_server.py |
Hermes MCP server with all tools |
launch_scientist.py |
Main entry point (patched for OpenRouter) |
ai_scientist/llm.py |
LLM client factory (patched for generic OpenRouter) |
venv/ |
Python virtualenv with CUDA PyTorch |
templates/ |
Experiment templates (nanoGPT, grokking, 2d_diffusion, etc.) |
results/ |
Generated papers and experiment outputs |
Pitfalls
- GPU required: AI Scientist runs PyTorch experiments. CPU-only runs will be impractically slow. On this machine: RTX 3090 (24 GB) with CUDA 12.4 via PyTorch 2.6.0+cu124.
- OpenRouter API key: The single source of truth is
${USER_HOME}\AppData\Local\hermes\.env. The.env.examplefile in the repo has***as a placeholder — DO NOT put the real key there for Hermes-driven workflows, it must go in the Hermes.envso${OPENROUTER_API_KEY}resolution works in the MCP server config. - Aider integration: The experiment coding phase uses
aider-chatwhich needsOPENROUTER_API_KEYin the subprocess environment. Since the MCP server config passes it viaenv:, this is handled — but standalone CLI runs need it in the shell. - Review was hardcoded to OpenAI: The original
launch_scientist.pyhad two places hardcodingmodel="gpt-4o-2024-05-13"withopenai.OpenAI()(line 270 for review, line 301 for improvement review). Both were patched to use OpenRouter withopenrouter/google/gemma-4-31b-it:freeandapi_key=os.environ.get("OPENROUTER_API_KEY", ...). If you ever re-clone the repo, re-apply these patches. - Template baselines: Each template needs a baseline run (
prepare_baseline) beforerun_experimentwill work. The baseline createsrun_0/final_info.jsonwhich the experiment phase reads. NanoGPT also needs data preparation scripts (data/enwik8/prepare.py, etc.). - LaTeX (MiKTeX) on Windows: MiKTeX is installed at
${USER_HOME}\AppData\Local\Programs\MiKTeX\miktex\bin\x64\but NOT in the system PATH.shutil.which('pdflatex')(used bycheck_latex_dependencies()) will fail. The MCP server config injects PATH via theenv:block. Seereferences/latex-on-windows.md. - OpenRouter model name resolution: Models with
openrouter/prefix are handled explicitly. Models likegoogle/gemma-4-31b-it:free(bareprovider/model:freeformat) are auto-detected by_is_openrouter_model()because they contain/and don't match bedrock/vertex_ai/deepseek/gemini prefixes. The:freesuffix is passed through as part of the model name — OpenRouter handles it. - Long runs: Full experiments can take hours.
run_experimentlaunches asynchronously via threading. - Windows path note: All paths in the MCP server use forward-slash paths (
${MY_REPOS}/...) which work fine on MSYS2/bash and in subprocess calls. The Hermes config uses forward slashes for args and env vars consistently. **launch_scientist.py note: Thedo_idea()function referencesargs.enginefrom the global scope (line 254). This is a pre-existing bug that works in CLI mode but may fail if the function is imported — the fix would be to passengineas a parameter.
Reference Files
references/setup-and-usage.md— Venv setup commands, exact patch details, MCP config, verification steps.references/free-openrouter-models.md— Live-quoted catalog of 27 free OpenRouter models with per-task recommendations.references/latex-on-windows.md— MiKTeX PATH quirks, version info, config.yaml injection.