mcp-ai-wpoos-model-foundry
Operational guide for the NV oOS Model Foundry (Proposal 067) — the cluster fine-tuning flywheel. Phase 1 shipped: the Corpus Foundry (Pro addon, addons/pro/includes/model-foundry/) — consent-gated, d
SKILL.md.agents/skills/mcp-ai-wpoos-model-foundry/SKILL.md
GitHubNV oOS Model Foundry — Corpus Foundry (Phase 1)
Operational playbook for the Corpus Foundry slice of the Model Foundry
(Proposal 067). Distilled from the Phase 1 implementation
(feature/model-foundry): the consent-gated pipeline that turns harness
traces, preference pairs, and docs-hub markdown into versioned,
trainer-agnostic fine-tuning corpora. Phases 2–6 (train-worker sidecar,
registry/serving, eval gate, automation) append to this skill as they land.
Key docs: proposal · implementation plan · feature doc · folder README.
1. Component map
| Component | File | Role |
|---|---|---|
WP_MCP_AI_Corpus_Schema_Registry |
addons/pro/includes/model-foundry/class-wp-mcp-ai-corpus-schema-registry.php |
Versioned row schemas + validation |
WP_MCP_AI_Corpus_Deduper |
…/class-wp-mcp-ai-corpus-deduper.php |
Exact → fuzzy → opt-in semantic dedup |
WP_MCP_AI_Corpus_Pii |
…/class-wp-mcp-ai-corpus-pii.php |
Deterministic PII scrub + provider-backed seam |
WP_MCP_AI_Corpus_Governance |
…/class-wp-mcp-ai-corpus-governance.php |
Consent, pipeline order, holdout split, manifest, guarded writes |
WP_MCP_AI_Preference_Pair_Store |
…/class-wp-mcp-ai-preference-pair-store.php |
Bounded FIFO DPO pair store |
WP_MCP_AI_Tool_Export_Trajectory_Corpus |
…/class-wp-mcp-ai-tool-export-trajectory-corpus.php |
export_trajectory_corpus |
WP_MCP_AI_Tool_Export_Preference_Pairs |
…/class-wp-mcp-ai-tool-export-preference-pairs.php |
export_preference_pairs |
WP_MCP_AI_Tool_Export_Plugin_Docs_Corpus |
…/class-wp-mcp-ai-tool-export-plugin-docs-corpus.php |
export_plugin_docs_corpus |
Registration: model-foundry-init.php hooks wp_mcp_ai_pro_tools; the
model_foundry module in addons/pro/includes/class-wp-mcp-ai-pro-module-registry.php
requires it. Slugs live in the ai_ml preset of
includes/helpers/class-wp-mcp-ai-tool-presets-helper.php.
2. Non-negotiable contracts
- Consent fails closed. Post meta
_wp_mcp_ai_training_consent='1'grants consent; absent/other →WP_Error( 'wp_mcp_ai_training_consent_required' )before any source data is read. Site-level override:wp_mcp_ai_model_foundry_assistant_consentedfilter (bool; null = use meta). The admin UI for the meta ships in Phase 3 — until then the filter is the operational path. - Pipeline order is fixed: per-case caps → exact dedup → fuzzy dedup → (opt-in) semantic dedup → PII scrub → deterministic holdout split → R12 manifest → guarded write. Never reorder without the ADR.
- Holdout split is
md5(canonical row) % 5 == 0→ holdout (~20%), computed after dedup. Holdout rows are never written intotrain.jsonl; the split is stable across exports (decontamination by construction). - Skip, never truncate. Named skip reasons (
skipped_too_large,skipped_no_user_input,skipped_low_margin,skipped_invalid_row) — a truncated row would poison the corpus. - Exporters never call the chat model. Semantic dedup is the only
provider-bound path (embeddings), opt-in via
semantic_dedup, capped, and filter-seamed (wp_mcp_ai_model_foundry_embed_text).
3. Row schemas
| Schema | Format | Shape |
|---|---|---|
chat_v1 |
openai_chat_jsonl |
{"messages":[{role,content},…]} |
trajectory_v1 |
openai_chat_jsonl |
tool-calling rows: assistant tool_calls + role:tool answers + final assistant content |
preference_v1 |
openai_preference_jsonl |
DPO: exactly prompt/chosen/rejected strings (R4, no lists) |
docs_v1 |
text_jsonl |
{"text", "meta":{source,path,hash,bytes}} |
New shapes get a new key, never a mutated one.
4. Output layout
wp-content/uploads/mcp-ai/model-foundry/<assistant_id>/<corpus_id>/
train.jsonl / holdout.jsonl / manifest.json / .htaccess / index.php
corpus_id = {schema}-{Ymd-His}. The manifest is written last so its
presence signals a complete corpus — shard metadata must be attached to the
manifest payload before the file write (the Phase-1 bug: the manifest file
was written with shards: [] because the envelope was updated after the
write — tests assert the file and envelope agree).
5. Filter seams
| Hook | Purpose |
|---|---|
wp_mcp_ai_model_foundry_assistant_consented |
Consent override |
wp_mcp_ai_model_foundry_embed_text |
Semantic-dedup embedding callable |
wp_mcp_ai_model_foundry_extra_scrub |
Provider-backed free-text PII scrub |
wp_mcp_ai_model_foundry_run_user_prompt |
User prompt for retrieval-less trace runs |
wp_mcp_ai_model_foundry_preference_pairs |
Contributed preference pairs |
wp_mcp_ai_model_foundry_trajectory_per_case_char_cap |
Trajectory row size cap |
wp_mcp_ai_model_foundry_docs_per_file_char_cap |
Docs per-file size cap |
6. Data-source facts (verified against the code)
- Trace runs (
WP_MCP_AI_Harness_Trace_Store): artifacts per run aretool_calls.jsonl,model_response.txt,retrieval.json(hasquery),profile.json,cost.json,self_refine.json,reasoning_trace.json,dspark.json. User prompts are NOT stored — the trajectory exporter uses the retrieval query, falling back to the run-user-prompt filter.list_runs()caps at 100 runs (newest first) — do not advertise a highermax_runs. - Preference pairs have no pre-existing store with full chosen/rejected
text (eval run stores keep summaries only) —
WP_MCP_AI_Preference_Pair_Store::record()is the intake API; chat-UI thumbs and eval-based rejection sampling wire in during later phases. - Docs source default is
wp_upload_dir()['basedir'] . '/nvoos-docs-hub/content'(NV_oOS_Docs_Hub_Plugin::uploads_docs_dir()when active). Explicitdirarguments must realpath inside the uploads basedir (symlink-escape hardened);.md/.markdown/.txtonly; dotfiles andvendor/node_modules/.gitskipped.
7. Adding a new exporter (checklist)
- New tool class in
addons/pro/includes/model-foundry/implementing the four interfaces (canonical envelope, two-gate sanitisation,manage_options, flagspro/read-only/local-only/idempotent/cacheable). - Register in
model-foundry-init.php(wp_mcp_ai_pro_toolsmap). - Add the slug to the
ai_mlpreset inincludes/helpers/class-wp-mcp-ai-tool-presets-helper.php(pattern 58 — the all-tools-accounted-for gate). - No coverage-manifest entry — the manifest test only scans
includes/tools+addons/pro/includes/tools; foundry tools follow the harness-slice precedent (folder outsidetools/, no manifest row). - Consent first:
WP_MCP_AI_Corpus_Governance::assert_consent()before any source read. - Route rows through
WP_MCP_AI_Corpus_Governance::build()— never hand-roll dedup/scrub/split/write. - Tests in
addons/pro/tests/test-model-foundry-*.php— include dry-run shape assertions, capability gate, consent fail-closed, skip accounting, and shard/manifest agreement. - Update the folder README + this skill +
docs/features/model-foundry.md.
8. Validation gates (the commands that ran green)
# phpcs (root standard covers Pro; the CI gate counts ERRORS):
php vendor/bin/phpcs --standard=phpcs.xml.dist --error-severity=1 --warning-severity=0 \
--report=summary addons/pro/includes/model-foundry/ <other changed files>
# PHPUnit — WP 6.9 (primary) and WP 7.1 (second validation), cross-worktree runner:
MSYS_NO_PATHCONV=1 docker run --rm \
-e WP_CORE_DIR=/var/www/html -e WP_DB_HOST=db -e WP_DB_NAME=wordpress_test \
-e WP_DB_USER=wordpress -e WP_DB_PASSWORD=wordpress \
-v oos-wp_wp_core:/var/www/html \
-v F:/GITHUB/mcp-ai-wpoos:/var/www/html/wp-content/plugins/mcp-ai-wpoos \
-v mcp-ai-wpoos-vendor:/var/www/html/wp-content/plugins/mcp-ai-wpoos/vendor \
--network oos-wp_default wordpress:6.9-php8.2-apache \
sh -c 'cd /var/www/html/wp-content/plugins/mcp-ai-wpoos && php -d memory_limit=1G vendor/bin/phpunit \
addons/pro/tests/test-model-foundry-governance.php \
addons/pro/tests/test-model-foundry-exporters.php --no-coverage'
# WP 7.1: swap WP_CORE_DIR=/tmp/wp71 + WP_DB_NAME=wordpress_test_wp71 and mount
# the mf-wp71-core volume (see the mcp-ai-wpoos-test-suite skill, "When /tmp/wp71 is gone").
Baseline: 33 tests / 156 assertions on both WP 6.9 and WP 7.1.3; regression
batch (test-harness-fine-tune-curriculum + test-workflow-suggestion-miner
test-tool-registry-coverage) 36/36 on both.
9. Gotchas
- Holdout-split flake in tests: with a single row, it lands in either
shard — assert shard-agnostically (sum of
rows.train+rows.holdout, or read whichever shard file exists). Never asserttrain.jsonlcontains a specific single row. - Filesystem fixture accumulation (test-suite pattern 71): tests scanning
the fixed docs-hub dir must wipe it first (
wipe_dir()+$cleanup_dirsintearDown), or passuniqid()-suffixed explicit dirs. - Manifest/shards ordering: attach
shardsto the manifest array beforewrite_corpus()encodes the file — the envelope and the file must agree. max_runshonesty: the trace store caps at 100 regardless of schema; keep tool schemas ≤ 100.
10. References
- Proposal + plan + feature doc (top of this file)
mcp-ai-wpoos-test-suiteskill — Docker runners, patterns 57/58/71docs/features/llm-harness.md— Layers A–H the foundry builds onmcp-ai-wpoos-toolkit-creationskill — full toolkit path (the foundry is a tool slice, not a toolkit: no SA Toolkit Manager surface, no WP-CLI commands yet)