ingest
Ingest web content into Emma's llmwiki vault — scrape a single page or crawl a set of pages, compile them into schema-conformant wiki pages, and refresh the vault's retrieval indexes. Use when the use
SKILL.md.claude/skills/ingest/SKILL.md
GitHubIngest into the llmwiki vault
Vault root: /mnt/c/Users/emmaj/llmwiki. Read the vault's CLAUDE.md first — it is the schema authority (page types, frontmatter, folder layout, raw/ immutability). This skill only adds the mechanics.
Workflow
- Fetch — single page: WebFetch it. A set of pages: enumerate first (sitemap, index page, or link-follow with an explicit budget), confirm the list with the user if it exceeds ~20 pages, then fetch serially. MediaWiki sites: prefer the wikikb skill's pull machinery and respect its access-posture rules (robots.txt, AI-agent blocks, 403 = stop); never stealth-scrape a site that blocks automated clients.
- Archive raw — save fetched markdown under
raw/(immutable; never edit existing raw files). One file per source page, named after the page. - Compile — write wiki pages per vault schema: correct type (concept/entity/source), YAML frontmatter matching existing pages (copy the shape of a neighbor page in the target folder),
[[wikilinks]]to related pages,sources:carrying the origin URL. Updatelog.mdwith one line per ingest session. - Refresh retrieval — invoke the
retuneskill (owns the index+tune procedure and its failure modes). Until it completes, new pages are findable viallmwiki searchbut not viallmwiki ask. - Optional graph — for topics that benefit from entity/relation traversal, offer
llmwiki graph <vault-relative-path>(LLM-expensive; scope to the new pages only).
Hard rules
- Never edit
raw/after writing; never write memory or state intowiki/pages that isn't compiled knowledge. - Work-related sources: flag to the user instead of filing (vault policy).
- A failed tune leaves
askon the previous pipeline — safe; report and continue.