anndata

AnnData — foundation for the scverse ecosystem. Use for format operations (.h5ad read/write, subsetting, concatenating), data-slot structure, and shape/index debugging. For analysis workflows use scan

SKILL.md

GitHub

AnnData: The scverse Data Container

Overview

AnnData is the shared data container for the scverse ecosystem (scanpy, scvi-tools, scvelo, cellrank, squidpy, muon). Understanding its structure is prerequisite knowledge for all single-cell Python work. Every tool in this skill library reads and writes AnnData objects.

When to use this skill:

  • Reading / writing .h5ad files, 10X MTX, loom, zarr
  • Subsetting, filtering, or concatenating datasets
  • Debugging index alignment errors, shape mismatches
  • Memory-efficient access to large files (backed mode)
  • Understanding where a tool stored its output (which slot)
  • Converting between sparse/dense, strings/categoricals

When NOT to use this skill alone:

  • For full scRNA-seq analysis → scanpy.md
  • For deep learning models → scvi-framework.md
  • For R↔Python format conversion → anndatar-seurat-scanpy-conversion.md
  • For multi-assay containers (RNA + protein/ATAC) → multimodal-anndata-mudata.md

AnnData Structure

AnnData object:  n_obs × n_vars  (cells × genes)
│
├─ .X              — expression matrix (n_obs × n_vars); sparse or dense
├─ .obs            — cell metadata DataFrame (n_obs rows); cluster, sample, QC metrics
├─ .var            — gene metadata DataFrame (n_vars rows); highly_variable, mean, std
├─ .uns            — unstructured dict; neighbors graph params, color palettes, DE results
│
├─ .obsm            — multi-dim cell arrays; 'X_pca', 'X_umap', 'X_scVI' (n_obs × k)
├─ .varm            — multi-dim gene arrays; 'PCs' loadings (n_vars × k)
├─ .obsp            — pairwise cell matrices; 'connectivities', 'distances' (n_obs × n_obs)
│
├─ .layers          — alternative expression matrices (same shape as X); 'counts', 'spliced'
│                     Use layers to store raw counts after normalization
└─ .raw             — snapshot of (X, var) before filtering; accessed with adata.raw.X

Mental model: Think of .obs as your cell-level metadata spreadsheet, .var as your gene metadata spreadsheet, .X as the expression matrix that links them, and everything else as additional matrices that hang off cells (.obsm) or genes (.varm) or pairs-of-cells (.obsp).

var_names convention: During build, keep Ensembl IDs as var_names (no duplicates, stable for gene-set tools). At the deliverable milestone, switch to gene symbols — preserve Ensembl in var['gene_id'] for a one-line switch-back:

adata.var["gene_id"] = adata.var_names.copy()    # preserve before switch
adata.var_names = adata.var["gene_name"].values   # symbols become active
adata.var_names_make_unique(join="-")             # Symbol-1 for any duplicate
# Switch back any time: adata.var_names = adata.var['gene_id']

See scrna-pipeline-conventions/references/gene-identifier-packaging.md for the milestone contract and verification.

Where tools store their outputs

Tool Output Location
sc.tl.pca() PCA coordinates adata.obsm['X_pca']
sc.tl.umap() UMAP coordinates adata.obsm['X_umap']
multiple UMAP sources (unsupervised + integration) X_umap_unsupervised, X_umap_integrated adata.obsm
scvi_model.get_latent_representation() scVI embedding adata.obsm['X_scVI']
sc.pp.neighbors() kNN graph adata.obsp['connectivities'], adata.uns['neighbors']
sc.tl.leiden() Cluster labels adata.obs['leiden']
sc.tl.rank_genes_groups() DE results adata.uns['rank_genes_groups']
sc.tl.score_genes() Cell scores adata.obs['<score_name>']
scv.tl.velocity() Velocity vectors adata.layers['velocity']
scv.tl.latent_time() Pseudotime adata.obs['latent_time']

Multiple UMAPs tip: If the pipeline produces both an unsupervised UMAP (PCA-based) and an integrated UMAP (scANVI/Harmony-based), do not overwrite the single X_umap key — rename both with a purpose suffix at packaging so collaborators select the right one by name. Document each in uns['embeddings']. See scrna-pipeline-conventions/references/embedding-deliverables.md.


Installation

pip install anndata        # Core package
# Usually installed as a dependency of scanpy or scvi-tools

Reading and Writing

import anndata as ad

# Read h5ad (standard format)
adata = ad.read_h5ad('data.h5ad')

# Read with backed mode (large files — X stays on disk)
adata = ad.read_h5ad('large_data.h5ad', backed='r')
# Must call .to_memory() before operations that need X in RAM

# Read 10X formats
adata = sc.read_10x_mtx('path/to/10x_output/')  # via scanpy
adata = ad.read_10x_h5('filtered_feature_bc_matrix.h5')

# Write
adata.write_h5ad('output.h5ad')
adata.write_h5ad('output.h5ad', compression='gzip')  # Smaller file, slower write

# Write zarr (cloud-friendly, chunked)
adata.write_zarr('output.zarr')

Subsetting

# Boolean mask on obs (cells)
t_cells = adata[adata.obs['cell_type'] == 'T cell']

# Multiple conditions
filtered = adata[
    (adata.obs['n_genes'] > 200) &
    (adata.obs['pct_mt'] < 20)
]

# By var (genes)
hvg_only = adata[:, adata.var['highly_variable']]

# By name
genes_of_interest = adata[:, ['CD14', 'CD3D', 'MS4A1']]

# Combined
subset = adata[
    adata.obs['leiden'] == '3',
    adata.var['highly_variable']
]

CRITICAL: View vs Copy

# Subsetting creates a VIEW — modifying it may modify the original
view = adata[adata.obs['cell_type'] == 'T cell']

# Make a copy to get an independent object
copy = adata[adata.obs['cell_type'] == 'T cell'].copy()

# Rule of thumb: always .copy() if you're going to modify the subset
# Otherwise you'll see "ImplicitlyConvertedView" warnings

Concatenation

import anndata as ad

# Concatenate along cells (add more cells — common for batch integration)
adata = ad.concat(
    [adata1, adata2, adata3],
    axis=0,
    join='inner',           # 'inner' = shared genes only; 'outer' = all genes, fill NaN
    label='batch',          # Adds a 'batch' column to obs
    keys=['batch1', 'batch2', 'batch3']
)
print(adata)  # Check shape

# Concatenate along genes (rare; add modalities)
combined = ad.concat([adata_rna, adata_protein], axis=1)

Common issue: After concatenation, obs_names may have duplicates.

# Check
print(adata.obs_names.is_unique)  # Should be True

# Fix if needed
adata.obs_names_make_unique()

Working with Layers

Layers store alternative expression matrices (same shape as X). Use them to preserve raw counts alongside normalized data.

# Store raw counts before normalization
adata.layers['counts'] = adata.X.copy()

# Normalize
sc.pp.normalize_total(adata, target_sum=1e4)
sc.pp.log1p(adata)
# Now adata.X is log-normalized, adata.layers['counts'] is raw

# scVI needs raw counts:
scvi.model.SCVI.setup_anndata(adata, layer='counts')

# scvelo needs spliced/unspliced:
adata.layers['spliced']   # from velocyto or STARsolo
adata.layers['unspliced'] # from velocyto or STARsolo

Raw Slot

# Save a snapshot of X before HVG filtering (for gene expression plots)
adata.raw = adata  # Freezes current X and var

# Later: plot gene expression using raw (even for genes filtered out by HVG)
sc.pl.umap(adata, color='SOME_GENE', use_raw=True)

# Access raw data directly
raw_X = adata.raw.X      # Expression matrix (n_obs × all genes)
raw_var = adata.raw.var  # All genes (not just HVGs)

# Restore raw to X (if you need to go back to full gene set)
adata = adata.raw.to_adata()

Memory Efficiency

# Backed mode: X stays on disk, obs/var in RAM
adata = ad.read_h5ad('50GB_atlas.h5ad', backed='r')

# Filter on metadata without loading X
filtered = adata[adata.obs['tissue'] == 'lung']

# Load only the filtered subset into memory
adata_lung = filtered.to_memory()

# Process in chunks (for very large files)
chunk_size = 5000
for i in range(0, adata.n_obs, chunk_size):
    chunk = adata[i:i+chunk_size].to_memory()
    # process chunk

# Use sparse matrices (most expression matrices are >95% zeros)
from scipy.sparse import csr_matrix, issparse
if not issparse(adata.X):
    adata.X = csr_matrix(adata.X)

# Convert string columns to categorical (saves memory, speeds up groupby)
adata.strings_to_categoricals()

Common Gotchas

Index alignment when adding external data

# WRONG — index positions may not match
adata.obs['new_col'] = external_series.values

# RIGHT — align on index
adata.obs['new_col'] = external_df.loc[adata.obs_names, 'value']

obs_names must be unique

# Check
assert adata.obs_names.is_unique, "Cell barcodes are not unique!"

# Fix
adata.obs_names_make_unique()

Modifying X after subsetting

# This may raise a warning about modifying a view
subset = adata[adata.obs['batch'] == '1']
subset.X = ...  # Warning!

# Fix: make a copy first
subset = adata[adata.obs['batch'] == '1'].copy()
subset.X = ...  # No warning

var_names must match between datasets for scVI / scArches

# Check before concat or scArches mapping
assert set(query.var_names) == set(ref.var_names), "Gene sets don't match"
# Or: subset to intersection
common_genes = query.var_names.intersection(ref.var_names)
query = query[:, common_genes].copy()
ref = ref[:, common_genes].copy()

Inspecting an Unknown AnnData Object

# Start here when handed an unfamiliar .h5ad
print(adata)                          # Shape, layers, obsm keys
print(adata.obs.columns.tolist())     # What cell metadata exists
print(adata.var.columns.tolist())     # What gene metadata exists
print(list(adata.obsm.keys()))        # What embeddings exist
print(list(adata.layers.keys()))      # What count matrices exist
print(adata.obs.dtypes)               # Check for unexpected types
print(adata.obs['cell_type'].value_counts())  # Check category distributions
print(adata.X.min(), adata.X.max())   # Normalized or raw? (raw: integers, norm: floats)

Integration Map: Which Skill Uses Which Slots

Skill Reads Writes
scanpy.md .X (normalized), .raw .obs (leiden, scores), .obsm (X_pca, X_umap), .uns (neighbors, DE)
scvi-basic.md .layers['counts'] (raw) .obsm['X_scVI']
scvi-scanvi.md .obsm['X_scVI'], .obs['cell_type'] .obs['scanvi_pred', 'scanvi_prob']
rna-velocity-trajectory.md .layers['spliced', 'unspliced'], .obsm['X_umap'] .layers['velocity'], .obs['latent_time'], .obsm['velocity_umap']
multimodal-anndata-mudata.md Multiple AnnData MuData container

Resources


When not to use

  • Do not use alone for full scRNA-seq analysis. Pair with scanpy.
  • Do not use for multi-assay containers (CITE-seq, multiome). Use multimodal-anndata-mudata.

See also

  • scanpy
  • anndatar-seurat-scanpy-conversion
  • scvi-framework