searxNcrawl

MCP server and CLI toolkit for web search and crawling, built on Crawl4AI + SearXNG. Returns model-friendly Markdown or JSON.

Quick start

# Docker (everything included — SearXNG + Playwright + MCP server)
cp .env.example .env && docker compose up --build
# MCP endpoint: http://localhost:9555/mcp

# pip (standalone)
python -m venv .venv && source .venv/bin/activate
pip install -e . && playwright install chromium

# uv (standalone)
uv sync && uv run playwright install chromium

Workflows

Crawl a page

crawl https://docs.example.com                    # markdown to stdout
crawl https://docs.example.com --json              # JSON with metadata + dedup stats
crawl https://example.com --remove-links           # strip URLs from output
crawl https://example.com -o output.md             # save to file
crawl https://a.com https://b.com --concurrency 5  # batch crawl

Crawl an entire site

crawl https://docs.example.com --site --max-depth 2 --max-pages 10 -o docs/
crawl https://docs.example.com --site --include-subdomains --json

Site crawl uses BFS strategy. One URL only (no batch). Output aggregates all pages.

Search via SearXNG

search "python tutorials"
search "Rezepte" --language de --max-results 5
search "AI news" --time-range week --categories news
search "rust async" --engines google,duckduckgo --safesearch 2

Requires a running SearXNG instance with JSON output enabled. Docker Compose includes one automatically.

Authenticated crawling (crawl-capture)

# Method A: Playwright login flow
crawl-capture --start-url https://example.com/login \
    --completion-url 'https://example.com/dashboard.*' \
    --output ./state.json

# Method B: Export from running Chrome via CDP
google-chrome --remote-debugging-port=9222 --user-data-dir="$HOME/.chrome-cdp"
# Log in manually, then:
crawl-capture --cdp-url http://127.0.0.1:9222 --list-sessions
crawl-capture --cdp-url http://127.0.0.1:9222 --cdp-session 2 --output ./state.json

# Use captured state
crawl https://example.com/private --storage-state ./state.json

Exit codes: 0 success, 2 timeout, 130 browser closed before completion.

Examples

Research a topic — search then crawl top results

search "FastAPI websockets" --max-results 5
# Pick the most relevant URLs from results, then:
crawl https://fastapi.tiangolo.com/advanced/websockets/ \
      https://websockets.readthedocs.io/en/stable/ \
      --remove-links -o research.md

Mirror a docs site for offline context

crawl https://docs.pydantic.dev/latest/ --site \
      --max-depth 3 --max-pages 50 -o pydantic-docs/
# Each page saved as a separate file in pydantic-docs/

Give Claude Code web search + crawl

Start the Docker stack, then add to your MCP config:

{
  "mcpServers": {
    "crawler": { "url": "http://localhost:9555/mcp" }
  }
}

Claude Code now has crawl, crawl_site, and search tools — no API keys needed.

Crawl a JS-heavy SPA

# Crawl4AI uses Playwright/Chromium — JS renders automatically
crawl https://app.example.com/dashboard --json --timeout 60

Search in a specific language

search "Maschinelles Lernen Einführung" --language de --max-results 10
search "料理 レシピ 簡単" --language ja --categories general

Extract structured data from a page

crawl https://example.com/pricing --json
# Returns: markdown content + extracted links as references[] + dedup metadata

MCP server setup

# STDIO transport (for Claude Code, Zed, VS Code, etc.)
python -m crawler.mcp_server

# HTTP transport
python -m crawler.mcp_server --transport http --port 8000

# HTTP with CORS
python -m crawler.mcp_server --transport http --cors-origins "http://localhost:3000"

# Docker
docker compose up --build   # HTTP at http://localhost:9555/mcp

Client config (STDIO):

{
  "mcpServers": {
    "crawler": {
      "command": "uv",
      "args": ["run", "--directory", "/path/to/searxNcrawl", "python", "-m", "crawler.mcp_server"],
      "env": { "SEARXNG_URL": "http://your-searxng:8888" }
    }
  }
}

Client config (HTTP/Docker):

{
  "mcpServers": {
    "crawler": { "url": "http://localhost:9555/mcp" }
  }
}

Reference

MCP tools

Tool	Key params	Notes
crawl	urls[], output_format, concurrency, timeout, remove_links, dedup_mode, storage_state	Batch crawl; returns markdown or JSON
crawl_site	url, max_depth (2), max_pages (25), include_subdomains, timeout (120)	BFS site crawl from seed URL
search	query, language (en), time_range, categories, engines, safesearch (1), max_results (10)	SearXNG metasearch; results capped at 50

Python API

from crawler import crawl_page_async, crawl_pages_async, crawl_site_async

doc = await crawl_page_async("https://docs.example.com", dedup_mode="exact")
docs = await crawl_pages_async(["https://a.com", "https://b.com"], concurrency=3)
result = await crawl_site_async("https://docs.example.com", max_depth=2, max_pages=10)
# result.documents, result.stats

Sync wrappers: crawl_page(), crawl_pages(), crawl_site().

CrawledDocument fields

request_url, final_url, status ("success"/"failed"/"redirected"), markdown, html, headers, references (extracted links), metadata (title, status_code, dedup metrics, guardrail signals), error_message.

Environment variables

Variable	Default	Purpose
SEARXNG_URL	http://localhost:8888	SearXNG instance URL
SEARXNG_USERNAME	(none)	Optional basic auth
SEARXNG_PASSWORD	(none)	Optional basic auth
MCP_PORT	9555	Docker HTTP port

Config file search order: ./.env → ~/.config/searxncrawl/.env → auto-copy from .env.example.

Dedup modes

• exact (default) — removes repeated markdown blocks; guardrail triggers if >60% sections removed
• off — no deduplication