notion-hf-curator
End-to-end Notion-to-Hugging Face dataset and model curation workflow for SCBE repositories (export, QA, comparison, GitHub Actions, publishing).
SKILL.mdresearch/sphere-grid/skills/notion-hf-curator/SKILL.md
GitHubNotion-to-HF Curator
Use when a user asks to turn Notion content into an AI-ready dataset and sync to Hugging Face.
Prereqs
NOTION_TOKENorNOTION_API_KEY(Notion integration token).HF_TOKENandHFCLI installed (for publishing/validation).- In a clone of
SCBE-AETHERMOORE, run from repository root.
Pipeline workflow
Export Notion pages to dataset JSONL.
python scripts/notion_to_dataset.py --category all --output training-data/- Output files include
training-data/notion_export_<cat>_<yyyymmdd>.jsonlandtraining-data/metadata.json.
Push dataset to Hugging Face.
python scripts/push_to_hf.py --data-path training-data/notion_export_all_$(Get-Date -Format yyyyMMdd).jsonl --repo-id <owner>/scbe-aethermoore-training-data --token $env:HF_TOKEN- For repository-level updates, use
HF_TOKENor passhf uploaddirectly.
Run pipeline health checks.
python scripts/notion_pipeline_gap_review.py --output artifacts/notion_pipeline_gap_review.json --summary-path artifacts/notion_pipeline_gap_review.mdpython scripts/compare_notion_to_codebase.py --notion-jsonl training-data/notion_raw_clean.jsonl
Trigger GitHub Actions for repeatable runs.
gh workflow run notion-to-datasetgh workflow run cloud-kernel-data-pipeline -f ship_targets=hf,github
Optional browser validation (Playwright).
- Open target HF dataset/model pages and verify cards/metadata before/after publish.
Notes
- If dataset export records are 0, check Notion token scope and
scripts/sync-config.jsonentries. - Use
artifacts/notion_pipeline_gap_review.jsonand.mdfor the highest-priority remediation list. - Use
training/NOTION_CODEBASE_COMPARISON.mdto compare Notion coverage against repository code/docs.