anansi

Use the reviewed Anansi adaptive web scraping framework for authorized web fetch, extraction, crawl, selector healing, browser rendering, screenshot, and MCP-driven scraping workflows. Use when the us

SKILL.md

GitHub

Anansi

Use this skill for authorized adaptive web scraping with the reviewed Anansi source tree.

Scope

  • Source root: ~/.hermes/external-repos/anansi.
  • Package name: anansi-scraper; import package: anansi.
  • Capabilities: HTTP fetch, browser rendering, adaptive selector healing, crawl queue, structured extraction, screenshots, exports, and MCP server tools.

Safety Rules

  • Only scrape sites the operator is authorized to access.
  • Respect robots.txt, target terms, rate limits, privacy law, and data rights.
  • Do not start the MCP server, run browser automation, install Playwright browsers, configure proxies, or enable TLS/anti-bot impersonation unless the user explicitly asks for that operation.
  • Do not enable ANANSI_ALLOW_PRIVATE_NETWORKS=1 unless the user explicitly requests it for a trusted isolated host.
  • Prefer ANANSI_DISABLE_ANTIBOT=1 for ordinary fetch/extract use unless the user explicitly asks for authorized anti-bot handling.
  • Keep exports confined to the working project or Anansi's ~/.anansi/exports/ path.

Install When Needed

Use a local virtual environment in the user's working project:

. ~/.hermes/skills/devops/anansi/.venv/bin/activate

Browser support is separate and should be explicit:

playwright install chromium

TLS impersonation support is separate and should be explicit:

pip install "~/.hermes/external-repos/anansi[tls]"

First Probes

ANANSI_DISABLE_ANTIBOT=1 ~/.hermes/skills/devops/anansi/.venv/bin/python -m anansi --help
ANANSI_DISABLE_ANTIBOT=1 ~/.hermes/skills/devops/anansi/.venv/bin/python -m anansi.mcp_server.server --help

Common Commands

ANANSI_DISABLE_ANTIBOT=1 ~/.hermes/skills/devops/anansi/.venv/bin/anansi fetch https://example.com --output markdown
ANANSI_DISABLE_ANTIBOT=1 ~/.hermes/skills/devops/anansi/.venv/bin/anansi mcp