multi-tool-consensus-annotation
Runs a PANEL of independent cell-type voters (markers, atlas transfer, scANVI/scArches, popV, treeArches novelty, Census scVI-KNN), reconciled into one label space by a conservative, flag-heavy consen
SKILL.mdskills/multi-tool-consensus-annotation/SKILL.md
GitHubMulti-Tool Consensus Annotation
A framework for cell-type / cell-state annotation as a panel of independent voters. Each voter answers the annotation question with a different method and a different evidence base; a conservative, flag-heavy consensus reconciles their per-cell votes into one defensible label set. The design goal: no single method is the oracle, genuine cross-method corroboration raises confidence, and any disagreement or out-of-distribution signal is preserved rather than forced into a coarse label.
This skill is the index that composes the per-tool voter skills. Route to each voter's own skill for its implementation; use this skill to decide which voters to run, how to harmonize their vocabularies, how to read their agreement honestly, and how to fuse them into a frozen label set.
When this framework fits
- You have (or are building) an annotation pipeline where several methods each produce a per-cell label and you need one reconciled answer with a confidence you can defend.
- Your cohort has aging / treatment-remodeled states, so you need a novelty guard that can say "known lineage, novel state" instead of snapping every cell to a young/control label.
- Votes in different nomenclatures "never agree" and your agreement threshold is unreachable — the vocabulary dead-tally symptom (see Stage 1).
- You want to read an agreement matrix without being fooled by legs that are correlated by construction (see Stage 2).
The voter panel
Each voter contributes one per-cell vote. Voters differ in evidence base and in how independent they are from each other — the independence class matters when you count agreement (Stage 2).
| Voter | Question it answers | Per-tool skill | Independence class |
|---|---|---|---|
| Broad-marker consensus | Which coarse lineage, by canonical markers? | scanpy (score_genes / rank_genes_groups) |
coarse anchor (not a fine vote) |
| Cluster-level reference voting | Which public-atlas label does this cluster match (KNN over per-atlas scVI latents)? | scvi-basic + scvi-scarches-reference-mapping |
reference-seed (feeds the internal legs) |
| Public-atlas scANVI / scArches transfer | What public type is this, in the atlas nomenclature (whole data as pure query onto a frozen atlas)? | scvi-scarches-reference-mapping, scvi-scanvi |
independent corroborator |
| Internal control→treatment scANVI | Which control-defined state does treatment remodel? | scvi-scanvi |
correlated with the reference-seed |
| popV ensemble | What do many classifiers (KNN-on-scVI, SCANVI, SVM, RF, …) agree on? | (popV package; see scvi-framework for scvi-tools setup) |
independent corroborator |
| treeArches / scHPL open-set novelty | Does this cell fit ANY known type, or is it out-of-distribution? | treearches-hierarchy-learning |
authoritative OOD guard |
| CELLxGENE Census scVI-KNN | What does a large public atlas call this, via a projected KNN vote? | cellxgene-census-annotation |
supplementary (bounded tie-breaker) |
| Tissue-specific expert dictionary | For a known compartment (e.g. immune), which curated fine state? | curated marker dictionary via scanpy |
domain prior (compartment-scoped) |
Modality variants: for scATAC queries the reference-transfer voter is scembed-atac-annotation
(region2vec + KNN); for a reference-free marker-only panel of LLMs, mllmcelltype-consensus-annotation
is a self-contained alternative voter. A semi-supervised transfer voter fits when a good reference
exists; an ensemble (popV) adds robustness; the novelty guard (treeArches) earns its place whenever
aging or treatment can produce states absent from every reference.
Stage 1 — Harmonize vocabularies FIRST (a separate concern)
Every voter speaks its own native vocabulary: LungMAP celltype_level3, a popV panel's
ref_cell_type, CELLxGENE-ontology names ("classical monocyte", "CD8-positive, alpha-beta T cell"),
an internal marker dictionary's underscore labels. Before any tally, map every voter's label into
one canonical label space through crosswalk adapters. This is a distinct stage, kept out of the
fusion logic so the fusion stays vocabulary-agnostic.
Why it is load-bearing — the dead-tally bug. If two legs sit in different nomenclatures, their
labels never string-match, so they never agree, so an N-of-legs agreement threshold is unreachable
and the fine-type tally is silently dead. The fix is a label_synonyms map
{ normalize(native_label) → canonical_label } built from a crosswalk table (never an empty map),
applied to each leg before counting. Already-canonical legs (e.g. a popV panel harmonized at its
source) map to themselves; native-vocabulary legs (atlas transfer, Census) canonicalize here. Tokens
with no crosswalk row (Unknown, Novel, coarse lineages, treeArches Rejected) pass through
unchanged, so disagreement is preserved, never coerced.
Guard it. After canonicalization, assert that the genuinely reference-independent legs land in the same label space as the seed leg on a non-trivial fraction of confidently-labeled cells; a ~0% match signals the crosswalk failed to bridge vocabularies and the tally is dead.
Census is the worked example. cellxgene-census-annotation emits CELLxGENE-ontology names; its
crosswalk-to-canonical is a normalization adapter that is its own step. Treat each voter's vocabulary
map the same way — an adapter in front of the tally, not logic inside it.
Stage 2 — Vote independence & reading the agreement matrix
An "N legs agree" claim only means something if the N legs are genuinely independent. Two correlation traps recur:
1. Legs correlated by construction. A single evidence chain can wear several hats. In the
implemented lung recipe: the cluster-level reference vote seeds the internal scANVI's
scanvi_seed_label, which trains both the internal scANVI classifier and the treeArches tree —
so reference_vote → scanvi_seed → {scanvi_pred, treearches_pred} is one chain producing three
votes. Their mutual agreement is partly tautological. The legs that add genuinely
reference-independent corroboration are the public-atlas scArches transfer (whole data as pure
query onto a frozen atlas) and the popV ensemble (independent panel). When you read an agreement
matrix, the cells to watch are the independent-vs-correlated blocks: agreement inside the correlated
trio is expected; agreement between an independent leg and the trio is the signal.
2. Atlas-witness double-counting. Several methods can hit the same reference atlas — e.g. a
per-atlas reference vote, a Seurat transfer, and an scANVI transfer all onto LungMAP CellRef. That is
one atlas witnessed three ways, not three independent corroborators. Group such witnesses
(atlas_witness_groups: { lungmap: [...] }) so the group contributes ~1 (not 3) to a corroboration
count, and keep both the raw pairwise matrix and the down-weighted summary. A single-atlas-support
guard can additionally downgrade a high call to medium (and flag
uncertain_reference_disagreement) when its only corroboration is one atlas's witnesses with no
independent popV / scANVI backing.
Record this correlation structure in a provenance note (reasoning-trace) and surface it beside the
agreement matrix, so a reader interprets the matrix with the independence map in hand.
Stage 3 — The conservative fusion policy (the output contract)
The consensus emits, per cell, a small fixed vocabulary the whole project (and any cross-dataset integration) can join on:
annotation_confidence ∈ {high, medium, uncertain}basis_of_label ∈ {PI_curated, reference_transfer, de_novo, consensus}novelty_flag ∈ {none, uncertain_low_quality, uncertain_reference_disagreement, known_lineage_unknown_state, old_enriched_candidate_state, treatment_enriched_candidate_state, possible_doublet, possible_ambient}- plus the frozen
consensus_cell_type/consensus_cell_state, a continuousconsensus_entropy, and the raw fine-leg agreement countn_legs_agree.
Evaluate the rules in order — the ordering encodes the conservatism:
- Curated anchors first. A cell whose cluster carries a PI-curated control label keeps it at
high,basis_of_label = PI_curated. Human-curated anchors are authoritative and never overwritten. - treeArches-authoritative OOD → Novel. A treeArches distance/RE rejection routes the cell to a
de-novo state:
consensus_cell_state = Novel_<subcluster>,consensus_cell_type= its coarse lineage,basis_of_label = de_novo, and a condition-routednovelty_flag(old_enriched_candidate_statefor aged controls,treatment_enriched_candidate_statefor treatment arms, elseknown_lineage_unknown_state). Closed-simplex signals (an atlas-transfer "novel" flag, scANVI uncertainty) reinforce this routing but do not trigger it — treeArches is the authoritative OOD guard because its distance/RE rejection is threshold-invariant geometry, not a tunable softmax. - Ambiguity → Uncertain. A treeArches root-ambiguous call, or scANVI uncertainty with no
fine-leg majority, resolves to the coarse lineage (or
Uncertain) atannotation_confidence = uncertain,basis_of_label = consensus,novelty_flag = uncertain_reference_disagreement. - N legs agree → high / medium. When at least
min_legs_agreefine-type legs share the modal canonical label, that label is frozen atbasis_of_label = consensus. It reacheshighonly when an independent robustness signal concurs — gate on the ensemble's own confidence (popv_confidence == "high", which already folds any method-count rescale) and the internal scANVI's calibrated probability clearing its threshold — otherwisemedium. The single-atlas guard (Stage 2) downgradeshigh → mediumwhen the corroboration is one atlas's witnesses alone. - Weak majority → medium. Exactly two fine legs agreeing is a weak majority: take the label at
medium,basis_of_label = reference_transfer,novelty_flag = uncertain_reference_disagreement. - Split with no majority → supplementary tie-break, else Uncertain. On a tie, the supplementary
Census vote (or another bounded corroborator) breaks it toward the label it supports at
medium,basis_of_label = reference_transfer. With no supplementary vote, keep the coarse lineage and set the state toUncertain. The supplementary vote never elevates a call tohigh, never stands as the sole leg, and is ignored when it readsnot_run.
Supplementary vs load-bearing. Load-bearing legs (markers, reference vote, the two scArches
transfers, popV, treeArches) can carry a majority. The Census leg is supplementary: a bounded
tie-breaker / corroborator only, ignored when not_run. Keep that boundary explicit — a supplementary
leg must never be a cell's sole label source.
Flag-heavy by design. When in doubt, flag or reject rather than force a label. A high-confidence
call may not carry a disagreement flag (enforce the invariant: high ⟹ novelty_flag == none), and
quality seeds from QC (uncertain_low_quality, possible_doublet, possible_ambient) overlay onto
otherwise-unflagged cells. The flagged fraction (Novel + Uncertain), stratified by condition, is a
readout in its own right — a treatment-arm spike surfaces candidate remodeling.
Composition & sequencing
A workable order for a fresh dataset:
- Broad lineages by marker consensus (
scanpy) — the coarse anchor; do not force fine labels yet. - Cluster-level reference votes across public atlases (
scvi-scarches-reference-mapping) — the seed the internal legs build on. - Public-atlas scANVI/scArches transfer (
scvi-scanvi,scvi-scarches-reference-mapping) — the independent public-nomenclature vote; keepUnknowna valid outcome. - Internal control→treatment scANVI (
scvi-scanvi) — anchored on high-confidence controls, treatment arms as query. - popV ensemble — robustness across many classifiers; the second independent corroborator.
- treeArches / scHPL (
treearches-hierarchy-learning) — the open-set novelty guard. - Census scVI-KNN (
cellxgene-census-annotation) — the supplementary bounded vote. - Harmonize all vocabularies (Stage 1) → fuse (Stage 3) → freeze the label set → emit the
agreement matrix + flagged-fraction tables (Stage 2) and persist the policy via
reasoning-trace.
For an immune compartment, add a curated tissue-specific dictionary as a compartment-scoped fine leg alongside the public references. Reserve embeddings for geometry, QC, annotation, and discovery; leave the embedding for the measurement layer (pseudobulk DE, differential abundance) once labels are frozen.
Frozen .obs contract (what downstream consumes)
consensus_cell_type # frozen canonical fine/coarse type
consensus_cell_state # fine state or Novel_<subcluster>
annotation_confidence # high | medium | uncertain
basis_of_label # PI_curated | reference_transfer | de_novo | consensus
novelty_flag # none | uncertain_* | *_candidate_state | possible_doublet | possible_ambient
consensus_entropy # continuous uncertainty (from the internal scANVI leg)
n_legs_agree # raw count of fine-type legs sharing the modal canonical label
# every per-leg vote rides forward for audit (reference vote, atlas transfer, scANVI, popV,
# treeArches, census_label, ...)
Freeze once; do not recluster or rename after the freeze. annotation_confidence and
basis_of_label are the umbrella-harmonized axis that cross-dataset integration joins on;
novelty_flag is dataset-local and maps to an integration-level uncertainty_reason at the boundary.
Resources
- The conservative multi-leg policy and the leg-independence caveat are the recurring pattern behind the per-tool voter skills above; this skill is the framework that composes them.
When not to use
- Do not use for a reference-free multi-LLM consensus over marker genes. Use mllmcelltype-consensus-annotation.
- Do not use to run a single annotation method. Route to that method's own skill (scvi-scanvi, treearches-hierarchy-learning, cellxgene-census-annotation, ...); this skill is the framework that composes several.
- Do not treat the consensus as a way to manufacture agreement — its job is to preserve disagreement as Uncertain/Novel, not to coerce a label.
See also
scvi-scanvi— Voter; scANVI semi-supervised label transfer from seed cell-type labels (public or internal reference), always initialized from a trained scVI modelscvi-scarches-reference-mapping— Voter; scArches surgery to project a query onto a frozen atlastreearches-hierarchy-learning— Voter (authoritative OOD); open-set novelty / OOD rejection guardcellxgene-census-annotation— Voter (supplementary); pulls a CELLxGENE Census reference slice and takes a local KNN vote in its scVI embedding, without the tiledb stackscembed-atac-annotation— Voter (ATAC modality); region2vec KNN vote against a pre-trained HuggingFace databio reference, no retrainingmllmcelltype-consensus-annotation— Alternative voter / self-contained panel; reference-free multi-LLM marker consensusscanpy— Prerequisite (markers + coarse anchor); marker scoring, clustering,rank_genes_groupsreasoning-trace— Provenance; persist the fusion policy + correlation structure