gsm8k_answer_tools
Extract GSM8K final answers, validate calculator annotations, and label candidate solutions by answer correctness.
SKILL.mdbasicagent-skill/skill_library/paperbench/bbox/skill/gsm8k_training_verifiers/gsm8k_answer_tools/SKILL.md
GitHubGSM8K Answer Tools
Use this skill when working with GSM8K-style math word problem examples or candidate solutions from the verifier pipeline. It provides deterministic answer extraction, numeric canonicalization, calculator annotation checks, and correctness labels.
Do not use this skill as a full verifier. It does not judge reasoning quality beyond final-answer matching and calculator annotation validity.
Inputs
- JSONL rows with
questionandanswer. - Candidate solution strings that should contain a final
####answer marker. - Optional calculator annotations in the form
<<expression=result>>.
Outputs
- Canonical answer strings.
- Calculator validation records.
- Candidate label records with extracted answer, gold answer, and correctness.
Workflow
- Load GSM8K JSONL examples with
scripts/answer_tools.py load-jsonl. - Extract final answers from
####markers. - Canonicalize numeric strings by removing commas and normalizing integer-valued decimals.
- Validate calculator annotations with a restricted arithmetic evaluator.
- Label candidate solutions by comparing their extracted final answer to the ground truth answer.
Validation
Run:
python scripts/answer_tools.py self-test
python tests/test_answer_tools.py
The Distiller tree validator can also run the tests:
python /share/project/yuyang/workspace/Paper2Skills/Distiller/skills/module-to-skill/scripts/validate_skill_tree.py . --run-tests
Limitations
This skill intentionally follows the paper's final-answer labeling rule. A solution with flawed reasoning but the correct final answer is labeled correct for verifier-training purposes, while calculator validation remains a separate diagnostic.