bio-reads-qc-mapping
Ingest, QC, and map reads with reproducible outputs. Use for raw read processing and coverage stats.
SKILL.mdskills/bio-reads-qc-mapping
GitHubBio Reads QC Mapping
Validate a read sample sheet, run read QC and trimming, and map reads to an optional reference with a restartable driver.
Instructions
Tool guides and versions: docs/README.md.
Validate
sample_sheet.tsvagainstschemas/sample-sheet.schema.jsonand inspect the plan before running anything:uv run --script skills/bio-reads-qc-mapping/scripts/run_reads_qc_mapping.py \ sample_sheet.tsv --out results/bio-reads-qc-mapping --threads 4 # Inspect run_manifest.json, then execute the same plan on a compute node: uv run --script skills/bio-reads-qc-mapping/scripts/run_reads_qc_mapping.py \ sample_sheet.tsv --out results/bio-reads-qc-mapping --threads 16 --executeread_typemust bepaired_short,single_short, orlong. Mapping runs only for rows with a non-emptyreference; a missing reference is not a mapping failure. The driver reuses a stage only when its declared outputs are non-empty and the stage's.donemarker exists. Run--executethrough the scheduler (sbatch) for real data, not on a login node.Know what the driver runs, and adapt the plan when the defaults do not fit:
- Short reads:
fastpv1.3.3+ with default adapter and quality trimming. Use BBDuk (BBTools container) instead when you need k-mer contaminant, spike-in, or host removal (docs/bbduk.md). - Long reads:
filtlongwith no thresholds, which passes every read through. Add--min_length,--keep_percent, or--target_basesfor the project (docs/filtlong.md). - Short-read mapping:
bwa-mem2 mem. Build the index first withbwa-mem2 index reference.fasta; the driver does not. - Long-read mapping:
minimap2 -ax map-ont. For PacBio HiFi usemap-hifi, and for PacBio CLR usemap-pb. - The driver writes SAM. Sort and index it (
samtools sort,samtools index), then compute per-reference coverage withsamtools coverageor CoverM.
- Short reads:
For long reads, use basecaller-aware QC first. For ONT, prefer Dorado trimming during basecalling or demultiplexing when starting from POD5 or BAM. For FASTQ-only input, use
chopperfor quality, length, and end trimming, orfiltlongv0.3.1 when selecting reads for assembly (v0.3.0 renamed the long options to--short_1/--short_2). UsePychopperfor full-length cDNA. UsePorechop_ABIonly as a documented fallback for adapter discovery, and record why.- For very large ONT FASTQ inputs, do not spend the first full read pass on
gzip -tor rawseqkit statsunless asked. Record rawstatmetadata, optionally check a small sample, make the first full pass the filtering step, then runseqkit statson the outputs. Pychoppercan write plain FASTQ even when the output path ends in.gz. Give its outputs plain.fastqnames unless you compress them yourself, and check gzip magic bytes before runninggzip -t. Rename mislabeled legacy outputs to.fastqbefore resuming.Pychopperreport plotting can fail after the reads are written. If the classified, unclassified, rescued, and read-stats outputs exist and are non-empty, resume downstream from them instead of rerunningPychopper.
- For very large ONT FASTQ inputs, do not spend the first full read pass on
Faster mappers, when hardware allows:
- Short reads on a GPU node: NVIDIA Parabricks
fq2bam(GPU BWA-MEM with sorting and duplicate marking). - Long reads on a GPU node: Parabricks
minimap2(presetsmap-ont,map-hifi,lr:hq). - The CPU and GPU forks
mm2-fastandmm2-gbare built on minimap2 v2.24, not current v2.30+. Use them only when the older base version is acceptable.
- Short reads on a GPU node: NVIDIA Parabricks
Record each tool, version, thread count, and any GPU device in the run log and
tasks/METHODS.md.
Input Requirements
Prerequisites:
- Tools declared in the project's pinned pixi environment; BBTools runs from the
bryce911/bbtoolscontainer. See docs/README.md. - A
bwa-mem2index next to each short-read reference. Inputs: sample_sheet.tsvwith exactly the columnssample_id,read_type,read1,read2,reference; relative paths resolve against the sheet's directory.- reads/.fastq.gz or reads/.fastq
- reference.fasta (optional)
Output
- results/bio-reads-qc-mapping/run_manifest.json (validated rows, planned commands, and per-stage
statusafter--execute) - results/bio-reads-qc-mapping/<sample_id>/ with trimmed reads (
reads.fastq, orreads_R1.fastqandreads_R2.fastq),fastp.json,fastp.html,mapped.sam, and theqc.doneandmapping.donemarkers - results/bio-reads-qc-mapping/mapping_stats.tsv (
mapping_statusisplannedbefore--execute, thencompletedorreused, andnot_requestedfor rows with no reference) - stdout: the last line is one JSON envelope
{ok, skill, out, manifest, warnings}(driver stdout contract in AGENTS.md) - Sample sheet contract: schemas/sample-sheet.schema.json
Quality Gates
- The sample sheet validates, and the plan covers every row exactly once.
- Post-QC read counts are plausible against the raw counts (
fastp.jsonorseqkit stats). - Mapping rate meets project thresholds, applied only to rows that supplied a reference.
- On execution failure, preserve logs and report the failed command; retry only after diagnosing the cause and recording the changed parameters. Report unmet biological thresholds as results; never tune parameters solely to pass a gate.
- Long-read QC records where trimming happened: basecaller or demultiplexer,
chopper,filtlong,Pychopper, or a documented Porechop_ABI fallback. - Huge ONT inputs have raw size and mtime recorded and no redundant full-file raw preflight.
- File type is checked by content, not suffix, before tools that expect gzip.
- Before accepting reused outputs downstream, run a light content check (
seqkit stats, FASTQ header sniff, or gzip magic).
Examples
The runnable fixture at fixtures/sample_sheet.tsv covers paired-end, single-end, and long reads, with mapping requested for two of the three rows.
Troubleshooting
Issue: Pychopper failed during report or stat plotting but output FASTQs exist
Solution: Treat it as a recoverable post-processing failure. Confirm the classified FASTQ is non-empty and readable, fix any misleading .gz suffix, run seqkit stats, and resume downstream from the existing Pychopper outputs.
Issue: bwa-mem2 mem fails immediately on a new reference
Solution: The index is missing. Run bwa-mem2 index reference.fasta once, then rerun with --execute; completed QC stages are reused.