llms-txt-from-website
Generate `llms.txt` and `llms-full.txt` for a docs/product website from just a URL. Detect and reuse an existing `/llms.txt`, discover the docs source repo via “Edit this page”/GitHub links, fall back
SKILL.mdagent-marketplace/src/utils-human/skills/llms-txt-from-website/SKILL.md
GitHubllms.txt from Website
Use this skill when a user provides a website URL and wants a curated llms.txt plus a fuller llms-full.txt. It works without knowing the site framework or having a sitemap. The generator tries existing files first, then the docs repository, then a sitemap or bounded internal crawl.
Generate the files
Run the bundled script with the docs or product URL and an output directory:
python "<path-to-skill>/scripts/generate_llms_files.py" \
--url "https://docs.example.com/" \
--out "./llms-out"
The script creates a URL-derived subdirectory containing:
llms.txt: the curated manifest.llms-full.txt: the available full-text bundle.metadata.json: the source and generation details.
Add --json to print the metadata to stdout for debugging or chaining:
python "<path-to-skill>/scripts/generate_llms_files.py" \
--url "<url>" --out "./llms-out" --json
How the generator chooses sources
The script follows this order; use the first successful source and keep the generated metadata with the outputs.
- Existing files. It looks for
llms.txtat the supplied URL's root and the site origin, including an exact docs subpath. If it findsllms-full.txt, it downloads that too. Otherwise, it builds the full file from links inllms.txt, preferring Markdown endpoints and usinguvx markitdownfor pages it converts. - Docs repository. If no existing
llms.txtis found, inspect the homepage for “Edit this page”, “View source”, GitHub, repository, or similar links. The generator discovers a linked GitHub repository, shallow-clones it, and looks for docs sources (.md,.mdx,.rst). For large repositories,repomixpacks the selected sources. If discovery is unclear but a public docs repository likely exists, search the web for<project> docs github, then rerun with the relevant docs URL. - Sitemap or crawl. When repository discovery does not produce usable docs, the generator checks
robots.txtfor sitemap hints and tries/sitemap.xml. If it finds no usable sitemap, it crawls internal links from the supplied URL within the configured page and depth limits.
The repository path produces links to the live site when those pages resolve, and otherwise uses GitHub source links. By default, provenance stays in metadata.json; pass --include-source-links to include absolute source URLs beside manifest links.
Shape the outputs
llms.txt should help a reader choose relevant documentation quickly:
- Start with
# <Project/Docs name>and a one-sentence> summary. - Add a few lines of useful context without introducing headings.
- Group absolute links under
##sections, with each link as- [Name](URL). - Put non-essential links under
## Optional. - Keep local paths and provenance out of the manifest unless source links are requested.
llms-full.txt should include the documentation text available from the chosen source. Repository sources are packed from Markdown where possible; website pages are converted to Markdown and concatenated with clear separators.
Use Context7 for library documentation
For a known library or framework, use Context7 when its MCP tools are available to fill gaps or cross-check the generated material:
- Call
resolve-library-idwithlibraryNameandquery. - Call
query-docswith the returnedlibraryIdand a focusedquery. - Compare the result with the repository or crawl output and use it to identify missing references.
Script options
--max-pages: maximum pages to process during sitemap or crawl work; default60.--max-depth: maximum internal-crawl depth; default3.--max-links: maximum links inllms.txt; default30.--full-scope all|selected: include all repository docs sources or only the curated subset; defaultall.--max-full-bytes: repository full-text size limit before falling back toselected; default12 MB.--force-full: ignore--max-full-bytesand use the requested--full-scope.--no-crawl: stop if existingllms.txtand repository discovery both fail, instead of trying a sitemap or crawl.--include-source-links: include absolute source URLs beside links inllms.txt.--json: print generation metadata to stdout.