intentumdiff-engine
Deep reference for the IntentumDiff semantic diff engine internals — the pipeline, data models, GumTree matching, change groups, and the change-group index-space contract. Use this whenever you touch
SKILL.mdskills/intentumdiff-engine/SKILL.md
GitHubIntentumDiff — Diff Engine Internals
Remember the boundary (see intentumdiff-architecture): new engine logic goes into the
Rust core (crates/rust-core-host). The Python modules below (analysis/, core/engine.py)
are a parity test-oracle, not where features should grow. When you fix a defect, decide
whether the authoritative fix is in Rust; use the Python side only for the oracle or a
pure-DTO-shell transform.
Migration reality — refinement / presentation / invariances are engine, and already largely
Rust. The maintainer's direction is that these belong in the Rust core, and much of it is
already there for the certified native/batch path: lib.rs has finalize_python_review_drafts,
suppress_low_signal_reorders_drafts, ~20 promote_* refinement/refactoring rules, and
invariance/style-only evidence. The Python analysis/* files are the transitional oracle +
fallback. So when you hit an engine bug: check the Rust core first (crates/rust-core-host/src/lib.rs)
— it may already be correct, in which case a Python analysis/ fix is just oracle parity (no
Rust change, no maturin rebuild — e.g. the reorder group's empty raw_change_indices at
lib.rs:5230). Fix authoritatively in Rust and keep the Python oracle in parity; don't grow new
logic in Python analysis/.
The pipeline
Source → [Preprocess] → Parse → Normalize → Diff → Analyze → [Enrich] → Render
- Normalize filters trivia and computes bottom-up
structural_hash. Style-only shortcut: if the filtered old/new roots hash equal →SemanticDiff(is_style_only=True, changes=[]). - Diff is GumTree two-phase (below).
- Analyze: move consolidation (
moves.py), refactoring detection (refactoring.py), classification (classifier.py), diff-analyzer plugins (stage 13.5), guardrails (guardrails.py), and presentation normalization (presentation.py) which builds the human-facingchange_groupsand suppresses noise.
Full stage-by-stage detail (matching heuristics, refactoring signatures, fuel, cross-file):
docs/ARCHITECTURE.md.
Data models (src/intentumdiff/core/models.py, frozen pydantic v2)
SemanticNode:id(unique within tree),node_type,label,position,structural_hash,children, optionalparent_type,type_info, andfacts(NodeFacts) — privacy-safe structural facts (param_count,returns,bodyempty/stub/substantive,is_async,is_generator,decorator_count) emitted by the Rust parser for definition nodes. Facts are counts/enums/bools only — never source text. Fact derivation is only as good as the tree:return_kindmaps the return VALUE node'snode_typethroughsemantic_literal_kindinrust-core-host/src/lib.rs— grammar-specific literal names (javadecimal_integer_literal, c/cppnumber_literal) must be in that map AND retained by the parser's semantic set (issue #72), and the return collectors matchreturn_statement | return_expression | return(scala/ruby). A parser that prunes the value degrades honestly toreturns: "value"(kotlin/swift line scanners still do).tests/unit/test_intent_facts_sufficiency.pyis the per-language contract.Change:change_type(ChangeType | str— plugins may add language-specific strings),old_node/new_node,refactoring_kind,confidence,description,text_diff(compact[-old][+new]leaf char diff).ChangeGroup: the review-level grouping.kind∈MOVED_CODE | REFACTORING | MEANINGFUL_CHANGE | IGNORED_STYLE | NOISE_SUPPRESSED;raw_change_indices(indices intoSemanticDiff.changes);old_labels/new_labels;old_node_ids/new_node_ids(the node identities it groups);refactoring_kind;metadata(carriesindex_space,suppressed_count,reason,rule_id, …).SemanticDiff:changes,change_groups,guardrail_violations,language,has_semantic_changes,is_style_only,parse_errors,metadata.
GumTree two-phase diff (core/engine.py)
- Top-down: priority queue by node height; equal
structural_hash→ anchor match (confidence=1.0); ties broken by positional closeness. - Bottom-up: unmatched containers matched by
dice(T1,T2) > min_similarity(0.5), counting matched descendant pairs.
Edit actions: Insert, Delete, Update (matched pair, labels differ), Move (matched,
different parents). _compute_matching runs once and is shared by diff + refactoring
detection.
Change groups and the index-space contract (READ BEFORE TOUCHING GROUPING)
This is the sharpest edge in the engine. Consumers (the VS Code extension, release notes)
index change_group.raw_change_indices straight into the final SemanticDiff.changes
array. But groups are built at different pipeline stages against different versions of
the changes list, and later stages re-sort/filter changes. A stale/cross-space index then
collides with an unrelated final change — the concrete symptom was a brand-new function
rendered as "Noise · below the meaningfulness threshold".
The contract and the reconciliation are documented in references/index-space-contract.md. The essentials:
- Every group carries
metadata.index_space∈presentation_input | mixed | final_changes(or a specialised evidence space likestyle_only_shortcut). _reindex_groups_to_final_changes(changes, groups)inanalysis/presentation.pyis the single boundary reconciliation: it rebuilds each group'sraw_change_indicesagainst the finalchangesby node identity (old_node_ids/new_node_ids) — the one key stable across re-sorts/filters. Applied last indiffer.py::_complete_final_diff.- Producer contract: a
ChangeGroupmust either carry node ids (so the reindex derives correct final indices) or be honestly empty. Never emit phantom positional indices (e.g.range(len(discarded_input))) and rely on a downstream pass to erase them — fix it at the producer. See thepresentation.generic_text_diffgroup for the canonical "owns no final change → emit[]" case. - Consumers must range-check every
raw_change_indicesdereference (0 <= i < len(changes)).
Presentation & noise suppression (analysis/presentation.py)
normalize_generic_text_for_reviewreplaces generic-parser token churn with clean line/character spans and emits aNOISE_SUPPRESSEDgroup that owns no final change._suppress_group/ style-context groups carry node ids and are safely reindexed.- Suppression preserves
suppressed_count/labels so the UI's "(N hidden)" summary and evidence counts stay correct even when indices are emptied.
Invariances (why a change is "ignored style / noise")
Rich prose explanations for style/noise/equivalence live in the invariance rules
(rules.yaml / invariances.py), surfaced as group metadata.reason. Examples:
integer-literal canonical equivalence, string-quote equivalence, CSS hex/rgb equivalence.
When you make something "ignored," attach an explainable reason — don't just drop it.
Where the change categories map to risk (used by the UI)
MEANINGFUL_CHANGE → Behavior; REFACTORING/MOVED_CODE → Internal;
IGNORED_STYLE/NOISE_SUPPRESSED → excluded; guardrail violations
(important/immutable) → critical/pinned. The extension derives risk from kind (it is
not a discrete engine field). See intentumdiff-release-notes for how this drives notes.
Testing engine changes
- Reproduce with the real engine:
SemanticDiffer().diff_strings(old, new, filename, language_hint=...)and inspectdiff.changes+diff.change_groups[*].raw_change_indices/metadata. - Confirm a suspected pre-existing failure by stashing only your engine files and re-running
the same tests (see
intentumdiff-dev-loop). - Rust changes need
maturin develop --release; pure-Pythonanalysis/changes do not.
DRY across the engine boundary (maintainer ruling, 2026-07-06)
One rule lives in ONE place. Two implementations of the same rule are drift waiting to happen (the same-id rename promoter existed in Rust finalize and had to be mirrored in Python for per-stage languages — every such mirror is PORT DEBT, not a solution):
- New engine logic goes in Rust. A Python mirror is only acceptable as a strangler transition step, and then it MUST: (a) cite its Rust twin by function name in a comment, (b) carry an open port issue (#37/#38/#39 pattern), (c) keep behavior identical — the scenario suites + corpus ratchets are the parity contract.
- Inside Rust, prefer shared helpers/traits over repeated match-arms and hunks: the SDK (crates/sdk) is the abstraction home for parser-side logic (#47); rust-core-host helpers (enclosing_entity_node, LIS, scope gates) are the home for matcher/refinement rules — extend them rather than cloning their logic at a new call site.
- Retire the Python copy the moment its language routes through the certified Rust path (RUST_CERTIFIED_LANGUAGES, issue #40).
The target boundary (maintainer, 2026-07-06)
Sources in -> finished SemanticDiff JSON out of the Rust core. Python is a thin API
wrapper — and deliberately so: with the whole engine behind that boundary, additional thin
API layers in OTHER languages (Node/TS, C ABI, browser WASM) can bind the same core with
the same oracle. presentation.py/refinement.py are per-stage-path transitional code; the
retirement route is the per-stage Rust finalize callable + certification (see the
finalize-routing issue), with python mirrors DELETED per language as routing lands.
Which pipeline actually served a diff? Probe the oracle before porting "missing" glue
Before porting a Python pass to Rust (or assuming a native path must fall back because a
pass is "missing"), run SemanticDiffer().diff_strings(...) on scenario fixtures and read
the FINGERPRINTS in the result — metadata["semantic_contract"] names the pipeline
(rust_finalize_review_v1 = the routed Rust finalize) and change_groups[].rule_id names
every rule that fired. If the pass you planned to port never appears in the oracle's
rule_ids for real inputs, it is dead code for that route and needs deletion, not porting.
Canonical example (#100, 2026-07-24): the live-server roadmap carried a "~300-line markdown
section-presentation port" for months — but .md is a certified routed language (#44):
the manifest resolves it to the markdown tree-sitter parser, sections/headings are real
nodes, and the ENGINE produces section moves/renames (refinement.entity_reorder_to_moved_code,
the refactoring grouping) inside the finalize the native chain already runs. The
_differ_presentation.py markdown passes only execute in the differ's language=="generic"
branch, which resolved .md files never enter. The "port" was deleting a special-case.