book-citations
Deterministic footnote/endnote restoration and citation-form cleanup for movemental books: extract from PDF/EPUB (PyMuPDF / ebooklib), match anchors, stage [^n] markers, compile to noteref HTML + chap
SKILL.mdcursor/book-citations/SKILL.md
GitHubBook citations
Restore and normalize footnotes/endnotes for one book at a time. This is not an LLM placement task. Superscript detection and note bodies come from source geometry/markup; the agent only runs scripts, clears form defects, and places residuals that already cleared confidence gates (or human review).
Primary repo: books workspace with scripts/citations/
Database: Supabase vhaiiiykcukrlyvwlgip (movemental) — ask before destructive writes
Companion cleanup: [[book-faithful-cleanup]] (full PDF→Markdown recipe embeds this as Phase 5)
Invocation
/book-citations {book_slug} # full extract → match → form → integrate
/book-citations {book_slug} form-only # normalize [^n] form in corpus/cleanup MD
/book-citations {book_slug} phase5 # compile [^n] → noteref + assert vs notes JSON
/book-citations {book_slug} extract-only # refresh data/citations/{slug}.json
Run from the books repo root (or pass absolute paths). One book per run.
Prime directives
- Never guess a marker position. Wrong sentence > missing citation. Below threshold → review queue, not DB.
- Never invent note bodies or CSL. Empty
chapter_notes.bodyis a failure mode (the 5Q bug). - Idempotent. Re-run must not duplicate notes or double-insert markers (
already_has_marker). - Snapshot before every DB write (
recon.snap_chapter_content_citationsor equivalent). - Canonical staging form in Markdown: clean Pandoc
[^n]only — not[^n]n, bare6,^n, or[^n](Ibid.). - Canonical loaded form in HTML/DB:
<sup class="noteref" data-note="N">N</sup>+chapter_notesrows. - Scanned PDFs (no text layer): out of scope. Report and stop.
- Ask before destructive DB changes (wipe notes, replace book, delete chapters).
Target models
| Layer | Marker | Notes storage |
|---|---|---|
| Cleanup / corpus MD | [^n] (no trailing duplicate digit) |
data/cleanup/{slug}/db/notes/ and/or data/citations/{slug}.json |
| Supabase HTML | <sup class="noteref" data-note="N">N</sup> |
chapter_notes (body never empty; note_type ∈ footnote|endnote|sidenote) |
Corpus chNN-notes.md + empty citations: [] |
Formatting reference for prose only | Not the Hirsch/movemental citation design |
Scripts (do not reimplement)
| Job | Module |
|---|---|
| PDF extract (span geometry) | scripts/citations/pdf_extract.py |
| EPUB extract | scripts/citations/epub_extract.py |
| Normalize / anchors | scripts/citations/normalize.py |
| Match chapters | scripts/citations/match.py |
| Insert / idempotent load helpers | scripts/citations/load.py |
| Post-convert assert | scripts/citations/assert_notes_markers.py, convert_hook.py |
| Phase 5 compile MD→HTML | scripts/citations/generic_phase5_compile.py (or book-specific *_cleanup_pipeline.py) |
Form normalize [^n]n / bare / caret / linkish |
scripts/corpus_fix_citations.py |
Defect scan (dup_fn, glued, caret) |
docs/problem-examples-two/_scan_defects.py |
| Review queue CLI | scripts/citations/review_cli.py |
Install deps if missing: pip install pymupdf pymupdf4llm ebooklib beautifulsoup4 lxml rapidfuzz
Match thresholds
| Score | Action |
|---|---|
| ≥ 0.95 | Auto-apply marker insert |
| 0.80–0.95 | recon.citation_review_queue |
| < 0.80 | Reject + report |
| Ambiguous (anchor hits >1 place) | Reject regardless of score |
Progress checklist
book-citations:
- [ ] 0 Inventory source (text layer? PDF/EPUB path; prior JSON)
- [ ] 1 Extract → data/citations/{slug}.json (or author data/citations/)
- [ ] 2 Form cleanup on corpus + cleanup MD (dup_fn=0)
- [ ] 3 Match to chapters (seed/DB/working HTML or MD)
- [ ] 4 Apply notes + markers (snapshot first if DB)
- [ ] 5 Phase 5 compile / assert_notes_markers
- [ ] 6 Review queue residuals; convert_hook wired for future ingest
- [ ] 7 Report counts (extracted / matched / applied / queued / rejected / gaps)
Phase 0 — Inventory
- Resolve
book_slug,source_path(PDF or EPUB), priordata/citations/{slug}.jsonif any. - Confirm text layer (
fitz: emptyget_text()⇒ scanned ⇒ stop). - Note whether working tree is
data/cleanup/{slug}/, corpus seed, and/or live Supabase chapters. - Record page count, format, ISBN vs
books.isbnwhen edition risk exists.
Stop and report unless user said continue.
Phase 1 — Extract (no DB writes)
# PDF
python -c "from scripts.citations.pdf_extract import extract_pdf; ..."
# EPUB
python -c "from scripts.citations.epub_extract import extract_epub; ..."
Or book pipelines / phase1_ground_truth.py when body oracle is also needed.
PDF rules: modal body font size per page; superscript = 50–85% of body size + raised baseline + 60 chars preceding text** as anchor (verbatim). Footnote bodies at page foot and/or endnotes section.^\d{1,3}$; capture **
EPUB: real markup (epub:type="noteref", <sup>, id/href). Prefer EPUB when same edition exists.
Output: {book_slug, chapter_hint, note_number, marker_anchor_text, note_body, source_page} → data/citations/{slug}.json (Brad/author trees may use {Author}/data/citations/).
Reuse existing JSON when source bytes + extractor version unchanged.
Phase 2 — Citation form cleanup (Markdown)
Desired form: …place.[^2] — not [^2]2, bare "…God."6, ^2, or [^8](Ibid.).
python scripts/corpus_fix_citations.py --book {slug} # or full corpus
python scripts/corpus_fix_citations.py --dry-run --book {slug}
Passes (in order):
- Strip duplicate digits:
[^n]n→[^n](same number; use(?!\d)so[^4]4Thematches) - Wrap residual bare endnote digits after closing punct/quote →
[^n](abbrev false-positive filter) - Fix
[^n](...)linkish →[^n]; caret^n→[^n]when footnote-shaped
Gate: dup_fn=0 in _scan_defects.py / remediation RG5. Do not delete glued digits without restoring via extract+match.
Phase 3 — Match
- Resolve chapter via title normalize +
sort_order(DBchapter_numberfrom source is not authoritative). - Normalize both sides (
normalize_for_match: strip HTML, NFKC, collapse ws, smart quotes). - Locate anchor; exact then
rapidfuzz. - Bucket auto / review / reject per thresholds above.
Phase 4 — Load / apply
Markdown working tree (preferred during cleanup):
- Insert
[^n]at matched points (idempotent). - Persist note bodies under
data/cleanup/{slug}/db/notes/(_all.jsonand/or per-chapter). - Strip loose note-body paragraphs only after bodies are preserved in notes JSON.
Supabase:
- Snapshot chapter HTML.
- Upsert
chapter_noteson(chapter_id, note_number)— non-empty body only. - Insert
<sup class="noteref" data-note="N">N</sup>viainsert_marker/ skip ifalready_has_marker. - Remove redundant
<p><sup>N.</sup> …</p>only for preserved note numbers. - Fix empty-bodied shells (fill from extract or delete) — never leave empty rows.
Phase 5 — Compile + assert
python scripts/citations/generic_phase5_compile.py {slug}
# or book-specific pipeline phase5
python scripts/citations/assert_notes_markers.py --dir data/cleanup/{slug}/compiled_html
Compile maps [^n] → noteref. Report missing markers vs notes JSON. Numbering gaps are findings — do not renumber to hide them.
Phase 6 — Review queue + convert hook
- Residuals →
recon.citation_review_queue+review_cli.py(accept applies same insert). - Wire future ingest:
scripts/citations/convert_hook.py— extract on convert, emit noteref + notes in same transaction, fail if note bodies exist without markers. See reference.md.
Failure forms (CIT taxonomy)
| Form | Example | Fix |
|---|---|---|
| Clean Pandoc | .[^2] |
Keep |
| Duplicate digit | .[^2]2 / [^4]4The |
corpus_fix_citations pass 1 |
| Bare glued digit | "…God."6 |
Pass 2 or extract+match |
| Caret | ."^2 / .^10 |
Pass 3b |
| Linkish | [^8](Ibid.) |
Pass 3a |
| Stuck / glued | word12, .12 |
Match pipeline — do not delete |
Orphan [^n] no body |
refs without notes plan | Extract + notes JSON |
| Loose bodies in HTML | <p><sup>1.</sup> … |
Move to chapter_notes, then strip |
| Empty DB shells | 5Q-style | Fill or delete |
Evidence catalogs: docs/problem-examples/05-broken-citations.md, docs/problem-examples-two/05-broken-citations.md.
Anti-patterns
- Sending chapter text to a model to “find the footnotes”
- Treating corpus notes chapters as the final citation store for movemental books
- Deleting glued digits instead of restoring noterefs
- Renumbering notes to hide gaps
- Loading without snapshot
- Skipping form cleanup so Phase 5 compiles
[^n]ngarbage
Done report
Per book: extracted / matched / applied / queued / rejected / dup_fn / marker vs note equality / open gaps. Link review-queue IDs. State whether convert_hook is wired for the next ingest.
Additional resources
- Convert hook + assertion contract: reference.md
- Full cleanup recipe (Phases 0–7 +
/loop): skillbook-faithful-cleanup