cellxgene-census-annotation
Transfer cell-type labels from the CELLxGENE Census to a query scRNA-seq dataset: pull a reference slice keeping its scVI embedding, project the query into that space on CPU, then take a local sklearn
SKILL.mdskills/cellxgene-census-annotation/SKILL.md
GitHubCELLxGENE Census — Cell-Type Annotation via Reference Data
Purpose
The Census hosts standardized public single-cell atlases (human + mouse) with pre-computed scVI
embeddings, so you can transfer annotations from millions of reference cells without downloading the
underlying H5ADs. The robust route this skill teaches is a local scVI-projection + KNN vote:
pull a Census reference slice (keeping its obsm['scvi'] and cell_type), project your query into
that same scVI latent with a frozen forward pass, then vote with a local sklearn KNN. This runs on
CPU, needs no tiledb-vector-search, and survives Census/scvi-tools version drift.
Treat the Census call as one supplementary vote in a larger annotation panel — a bounded
tie-breaker / corroborator, never the sole label source. When the leg fails or the network is
unavailable, degrade cleanly to census_label = "not_run".
Version facts (checked 2026-07-05)
cellxgene-census1.18 — theexperimentalsubmodule carries the embedding-metadata and (optional) vector-search helpers.scvi-tools >= 1.1.0is mandatory (the <1.1 classifier-logits bug corrupts scANVI/label uncertainty); the projection route below is validated on scvi-tools 1.2 / 1.4.- Pin an LTS
census_versionfor reproducibility (e.g."2024-07-01"). Weekly builds are transient; LTS builds persist. Other LTS tags:"2023-07-25","2023-12-15". s3fsis needed to fetch the pretrained model (anonymous access to a public CZI bucket).
Core Workflow: local scVI-projection + KNN vote
The whole leg is one guarded function. The four steps are: (1) pull the reference slice keeping its scVI embedding, (2) resolve + download the pretrained Census scVI model, (3) project the full-gene query into that scVI space on CPU, (4) fit a local KNN on the reference and predict on the query.
Step 1 — Reference slice (keep obsm['scvi'] + cell_type)
import cellxgene_census
import cellxgene_census.experimental # submodule is not auto-imported
census_version = "2024-07-01" # LTS, pinned
organism = "mus_musculus" # lower_underscore form works for all Census calls
label_attr = "cell_type"
with cellxgene_census.open_soma(census_version=census_version) as census:
ref = cellxgene_census.get_anndata(
census,
organism=organism,
# TileDB QueryCondition syntax (Python operators), NOT pandas .isin():
obs_value_filter="tissue_general == 'lung' and is_primary_data == True",
obs_embeddings=["scvi"], # keep the pre-computed scVI embedding
obs_column_names=[label_attr, "tissue_general"], # correct kwarg (NOT `attributes=`)
)
ref_emb = ref.obsm["scvi"].astype("float32") # the KNN training features — DO NOT discard
ref_lab = ref.obs[label_attr].astype(str).to_numpy()
obs_embeddings=["scvi"] returns each reference cell's pre-computed scVI latent in obsm['scvi'].
Keep it — it is the space the query is projected into and the feature matrix the KNN trains on.
Step 2 — Resolve + download the pretrained Census scVI model (bare model.pt)
The Census scVI contrib model lives at
s3://cellxgene-contrib-public/models/scvi/<census_version>/<organism>/model.pt. It is a bare
~177 MB model.pt, not a tar archive — do not tarfile.extract it; the fix is to drop it into a
directory scvi-tools can load (<dir>/model.pt). Resolve the link through the experimental API
(it returns an s3:// URI that urllib cannot open directly) and fetch it with anonymous s3fs.
from pathlib import Path
def resolve_census_scvi_model(census_version: str, organism: str,
cache_dir: str = "objects/census_scvi_model") -> str:
"""Return a directory holding <dir>/model.pt for scvi-tools to load. Cache + skip re-download."""
cache = Path(cache_dir)
dst = cache / "model.pt"
if dst.exists() and dst.stat().st_size > 0:
return str(cache) # already cached
cache.mkdir(parents=True, exist_ok=True)
meta = cellxgene_census.experimental.get_embedding_metadata_by_name(
embedding_name="scvi", organism=organism, census_version=census_version)
link = meta["model_link"] # an s3:// URI, e.g. s3://cellxgene-contrib-public/...model.pt
_download_s3_object(link, dst)
assert dst.exists() and dst.stat().st_size > 0, f"model download produced no file at {dst}"
return str(cache)
def _download_s3_object(s3_uri: str, dst: Path) -> None:
"""Anonymous s3fs (no creds for the public CZI bucket); https-rewrite fallback."""
assert s3_uri.startswith("s3://"), f"expected s3:// URI, got {s3_uri!r}"
bucket_key = s3_uri[len("s3://"):]
try:
import s3fs
s3fs.S3FileSystem(anon=True).get(bucket_key, str(dst))
return
except Exception: # fall through to the https rewrite
import urllib.request
bucket, _, key = bucket_key.partition("/")
urllib.request.urlretrieve(f"https://{bucket}.s3.amazonaws.com/{key}", str(dst))
Step 3 — Project the FULL-GENE query into the Census scVI space (CPU, no training)
The model reconstructs from its own ~8000-gene Ensembl HVG set. Project a full-gene query object
whose var_names are Ensembl IDs, then let prepare_query_anndata pad/reorder to the model's gene
space. A forward-pass projection runs fine on CPU — there is no training and no GPU requirement.
import scvi
import torch
import numpy as np
# torch >= 2.6 defaults torch.load(weights_only=True); the Census checkpoint embeds numpy globals,
# so loading a trusted model needs weights_only=False.
_orig_load = torch.load
torch.load = lambda *a, **k: _orig_load(*a, **{**k, "weights_only": False})
model_dir = resolve_census_scvi_model(census_version, organism)
query.X = query.layers["counts"].copy() # raw integer counts
query.obs["batch"] = "my_query_batch" # placeholder for the unseen scArches batch
scvi.model.SCVI.prepare_query_anndata(query, model_dir) # pads to the model's ~8000 genes
vae_q = scvi.model.SCVI.load_query_data(query, model_dir, accelerator="cpu")
vae_q.is_trained = True # frozen encoder: forward pass only, skip fine-tuning
query_latent = vae_q.get_latent_representation().astype("float32")
Three caveats decide whether the vote is meaningful:
- Full-gene, not HVG-subset. A 2000-HVG query zero-fills ~75% of the model's genes and the projection collapses. Feed the full-gene (pre-HVG) query object.
- Ensembl
var_names. The model's var space is Ensembl IDs. Match on Ensembl; symbol-mapping puts the query in the wrong gene space and silently degrades the projection. - CPU is enough.
accelerator="cpu"keeps the GPU free for the heavy training legs; this is a frozen forward pass. Supply any placeholderbatchvalue — the query batch is unseen by design.
Step 4 — Local KNN vote (replaces find_nearest_obs / predict_obs_metadata)
Fit a plain sklearn KNN on the reference (obsm['scvi'], cell_type) and predict on the projected
query latent. Confidence is the winning class probability. This is the robust default — no
tiledb-vector-search dependency, no drifting Census experimental signature.
from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors=30, n_jobs=-1).fit(ref_emb, ref_lab)
census_label = knn.predict(query_latent).astype(str) # -> obs['census_label']
census_conf = knn.predict_proba(query_latent).max(axis=1) # -> obs['census_conf']
query.obsm["census_scvi"] = query_latent # SANITY UMAP/NN only, not final biology
k = 30 is a robust default (higher k = smoother, more confident vote). Larger references warrant a
larger k. Store obsm['census_scvi'] for a sanity UMAP; do not treat it as the analysis embedding.
Guarded degrade (the leg must never abort the pipeline)
Because Census is one supplementary vote, wrap the whole leg so any failure (no network, missing
model, API drift, gene-space mismatch) resolves to a not_run sentinel with a loud traceback rather
than crashing the run.
import traceback
def run_census(query, cfg) -> dict:
n = query.n_obs
not_run = {"census_label": np.full(n, "not_run", dtype=object),
"census_conf": np.full(n, np.nan, dtype=float),
"census_scvi": None, "ran": False}
if not cfg.get("enabled", True):
return not_run
try:
# ... Steps 1–4 above ...
return {"census_label": census_label, "census_conf": census_conf,
"census_scvi": query_latent, "ran": True}
except Exception as exc: # any failure is a clean skip (non-load-bearing leg)
print(f"WARNING: Census leg failed ({type(exc).__name__}: {exc}) -> census_label='not_run'")
print(traceback.format_exc())
return not_run
Downstream consumers treat census_label == "not_run" as absent: it contributes nothing to any
tally and never labels a cell on its own.
Standalone re-stamp (CENSUS_ONLY): recover just this leg cheaply
Once the heavy annotation ensemble has run, you often want to re-run only the Census leg — fix a
gene-space bug, bump the census_version, or recover a leg that skipped on a transient network
failure — without recomputing the expensive upstream votes. The pattern: load the existing annotated
object, run run_census on it, and merge census_label / census_conf / obsm['census_scvi'] in
place while asserting the other legs' columns do not move.
import os, anndata as ad
if os.environ.get("CENSUS_ONLY") == "1":
adata = ad.read_h5ad(out_h5ad) # the existing annotated checkpoint
guard = {c: adata.obs[c].astype(str).to_numpy().copy()
for c in adata.obs.columns if c.startswith("popv_") or c.startswith("scanvi_")}
census = run_census(adata, census_cfg) # re-run ONLY this leg
adata.obs["census_label"] = census["census_label"]
adata.obs["census_conf"] = census["census_conf"]
if census["census_scvi"] is not None:
adata.obsm["census_scvi"] = census["census_scvi"]
moved = [c for c, v in guard.items() if not np.array_equal(adata.obs[c].astype(str).to_numpy(), v)]
assert not moved, f"CENSUS_ONLY moved non-census columns: {moved}"
adata.write_h5ad(out_h5ad)
This is minutes of work versus hours for the full ensemble, and the guard assertion proves the re-stamp touched only the Census columns.
Vocabulary: Census speaks CELLxGENE ontology — canonicalize before any tally
census_label carries CELLxGENE-ontology cell-type names ("classical monocyte",
"CD8-positive, alpha-beta T cell"). Those strings will not string-match the labels from other
annotation legs (LungMAP celltype_level3, an internal marker vocabulary, a popV panel's
ref_cell_type), so a raw Census vote silently never agrees with anything and an N-of-legs
agreement threshold becomes unreachable — the "dead-tally" failure.
Map census_label into your shared canonical label space through a crosswalk normalization
adapter before it enters a consensus tally. This vocabulary harmonization is its own step, kept
separate from the fusion logic. See multi-tool-consensus-annotation for the framework that composes
this leg with the other voters and treats vocabulary harmonization as a first-class stage.
Native Census vector search (optional; version-sensitive)
The Census experimental module ships a server-side nearest-neighbour path
(find_nearest_obs + predict_obs_metadata). It requires pip install tiledb-vector-search (an
extra that is frequently absent), and its signature drifts across releases: the old attributes=
kwarg is gone — current builds take column_names= (or obs_column_names=). Use it only when that
dependency is present and pinned; the local-KNN route above is the robust default.
# Requires: pip install tiledb-vector-search (often NOT installed)
neighbors = cellxgene_census.experimental.find_nearest_obs(
embedding_name="scvi", organism=organism, census_version=census_version,
query=query, # must carry a matching obsm['scvi'] (project it first, as in Step 3)
k=30, nprobe=20,
)
predictions = cellxgene_census.experimental.predict_obs_metadata(
organism=organism, census_version=census_version, neighbors=neighbors,
column_names=["cell_type"], # NOTE: `attributes=` no longer exists; version-sensitive kwarg
)
If this path errors on a missing tiledb-vector-search or an unexpected kwarg, fall back to the
local-KNN vote — same inputs, no extra dependency.
Domain context
Census SOMA hierarchy:
census["census_data"]["mus_musculus"]
.obs # cell metadata (SOMADataFrame)
.ms["RNA"].var # gene metadata (Ensembl feature_id)
.ms["RNA"].X("raw") # raw counts
Pre-computed embeddings: scvi (continuous latent — the route this skill uses; good for
fine-grained states) and geneformer (token-based; better for broad classes, needs its own
tokenization). This skill's vote runs in the scvi space.
Confidence reading: KNN predict_proba max near or below ~0.5 flags a rare/ambiguous/novel cell.
A larger k yields smoother, more confident votes but blurs rare states. is_primary_data == True
removes technical duplicates (Census stores the same cell across multiple datasets).
Common pitfalls
- Treating
model.ptas a tarball → the S3 object is a baremodel.pt. Drop it into<dir>/model.ptand point scvi-tools at<dir>; nevertarfile.extract. urllibon thes3://model_link →model_linkis ans3://URI. Use anonymouss3fs(or rewrite tohttps://<bucket>.s3.amazonaws.com/<key>).- HVG-subset query → zero-fills the model's genes; project the full-gene object (Step 3).
- Symbol
var_names→ the model's space is Ensembl; match on Ensembl IDs. - Weekly
census_version→ pin an LTS tag ("2024-07-01") for reproducibility. torch.loadweights-only error (torch ≥ 2.6) → patchweights_only=Falsefor the trusted Census checkpoint (Step 3).- Raw Census labels in a tally → canonicalize via a crosswalk first (see the Vocabulary section).
Resources
- Census docs: https://chanzuckerberg.github.io/cellxgene-census/
- Census Python API: https://chanzuckerberg.github.io/cellxgene-census/python-api.html
- Census models / embeddings: https://cellxgene.cziscience.com/census-models
When not to use
- Do not omit is_primary_data == True filters — Census contains duplicate cells across datasets.
- Do not use pandas .isin() syntax in obs_value_filter — Census uses TileDB QueryCondition syntax.
- Do not project a 2000-HVG query subset through the Census scVI model — it expects the full ~8000-gene Ensembl HVG set and HVG-subsetting zero-fills most model genes, degrading the vote. Project the full-gene query.
- Do not treat census_label as a sole label source — it is a supplementary vote (bounded tie-breaker/corroboration), never load-bearing.
See also
single-cell-vector-search— Alternative; cross-atlas embedding similarity search (SCimilarity / Census)treearches-hierarchy-learning— Alternative / complement; open-set novelty / hierarchical multi-reference learningscvi-scanvi— Alternative; semi-supervised label transfer from partial labelsscvi-scarches-reference-mapping— Related (same projection primitive); scArches surgery to map a query onto a reference modelmulti-tool-consensus-annotation— Downstream; compose this vote with markers / scANVI / popV / treeArches into a frozen label (this leg is one voter)