dl-crawl

20 crawlers avec TokenBucket, CircuitBreaker, cursor persistence

SKILL.md

GitHub

Crawlers — Datalake Souverain

20 crawlers with shared patterns:

  • AsyncGenerator for memory-efficient streaming
  • TokenBucket rate limiting per source
  • withRetry exponential backoff
  • CircuitBreaker prevents cascading failures
  • Cursor persistence resume on interrupt (dl_crawl_cursors)

Commands

pnpm crawl -- --source sirene --department 31 --max 5 --dry-run
pnpm crawl -- --source all --persist
pnpm enrich -- --siren 123456789
pnpm enrich -- --batch --limit 10 --type association
pnpm process -- --cross-ref --score --limit 1000

Key patterns

  • SIREN = universal key across all sources
  • Dry run: --dry-run flag on all CLIs
  • Audit trail: dl_crawl_runs + dl_crawl_cursors
  • Source config: src/config/sources.ts (20 sources + CircuitBreaker config)
  • OSINT tools: src/config/tools.ts (12 tools)

Похожие навыки: other

limits-colimits

Context management for Claude Code. Hooks maintain state via ledgers and handoffs. MCP execution without context polluti...

Автор: parcadei
3,392

ddttom-allaboutv2-claude-skills-block-collection-and-party

The Block Collection and Block Party are repositories for existing AEM blocks, build tools, code snippets, integration p...

Автор: mattnigh
20

nodnarbnitram-claude-code-extensions-claude-skills-esphome-confi

This skill should be used when the user asks to "create an esphome config", "set up an esp32 device", "configure esphome...

Автор: mattnigh
20