dask
Use this repo skill for Dask, the Python parallel computing library, when working with lazy task graphs, schedulers, Dask Array, Dask DataFrame, Dask Bag/bytes IO, configuration, diagnostics, CLI usag
SKILL.mdskills/repositories/repo-skills/dask/SKILL.md
GitHubDask Repo Skill
Use this skill when a task involves Dask's public APIs, internal task-graph model, collection workflows, configuration, diagnostics, or contributor test/build practices. Dask provides lazy parallel collections and schedulers for Python analytics: delayed objects, arrays, dataframes, bags, low-level task graphs, and local/distributed execution integrations.
First Checks
- Import check:
python -c "import dask; print(dask.__version__)". - CLI check:
dask --helpshould showconfig,docs, andinfocommand groups. - Minimal compute check:
python scripts/dask_package_smoke.py --scheduler synchronous. - Install extras: use
dask[array]for array workflows,dask[dataframe]for dataframe workflows,dask[diagnostics]for local dashboard/profiling helpers, anddask[distributed]only when remote cluster APIs are needed.
Route By Task
- Core graphs and schedulers: Use
sub-skills/core-graphs-schedulers/SKILL.mdfordask.delayed,compute,persist,optimize, annotations, tokenization, HighLevelGraph, low-level task specs, scheduler choice, and graph debugging. - Array workflows: Use
sub-skills/array-workflows/SKILL.mdfordask.array, chunk planning,from_array, slicing,map_blocks,blockwise, reductions, overlap, rechunking, gufuncs, random arrays, linalg, FFT, stats, and array backend caveats. - DataFrame workflows: Use
sub-skills/dataframe-workflows/SKILL.mdfordask.dataframe, pandas-like APIs, CSV/Parquet/JSON/SQL IO, partitions/divisions, joins, groupby, shuffle, repartitioning, pyarrow strings, categoricals, anddask_exprquery planning. - Bag and bytes workflows: Use
sub-skills/bag-bytes-workflows/SKILL.mdfordask.bag,dask.bytes, text and byte IO, JSON-like records, Avro, fsspec URLs, compression,foldby, and small-file object pipelines. - Configuration, diagnostics, and CLI: Use
sub-skills/configuration-diagnostics-cli/SKILL.mdfordask.config, YAML config paths, environment variables,dask config,dask info, local profilers, progress bars, callbacks/cache, install extras, and contributor validation commands.
Shared References
- Read
references/installation-and-environment.mdwhen choosing extras, verifying imports, or diagnosing optional dependency availability. - Read
references/troubleshooting.mdfor cross-cutting failures that affect multiple Dask collections or schedulers. - Read
references/repo-provenance.mdbefore relying on this skill for a changed checkout; refresh the skill when commit, package version, or major evidence paths drift.
Dask Working Rules
- Keep graph construction lazy. Do not call
.compute()or.persist()inside collection methods or while defining a reusable graph unless the user explicitly wants materialization. - Use metadata, divisions, chunks, dtypes, and
metaobjects to infer output shape/schema instead of computing sample data. - Choose the smallest collection that matches the data model: delayed for arbitrary Python functions, array for NumPy-like blocked tensors, dataframe for pandas-like tabular data, bag for unordered Python records, and bytes for low-level file blocks.
- Prefer explicit scheduler and config scopes when reproducing behavior:
with dask.config.set({...}): ...andcompute(..., scheduler="synchronous")for deterministic local debugging. - Treat
array.query-planninganddataframe.query-planningas import-time configuration; set them before importing the relevant collection modules in a fresh process. - For dataframe and array APIs, validate
meta, chunks, divisions, and optional dependency requirements before changing algorithms.
Validation Pattern
- Run the root smoke script for package-level sanity.
- Run the owning sub-skill smoke script for the collection or support workflow.
- Add a tiny fixture or in-memory example that exercises the changed route.
- For repo contribution work, run the narrowest relevant pytest selection before broader suites.
- Record skipped native cases separately when they require network, GPU, distributed services, large data, or credentials.
Common Pitfalls
- Importing
dask.dataframewithout the dataframe extra can fail due to missing pandas or pyarrow. - Unknown chunks or divisions often make otherwise valid operations expensive, ambiguous, or unsupported.
- Multiprocessing schedulers require pickleable functions and guarded entry points; use threads or synchronous mode for quick debugging.
- Config files may be read from multiple paths;
dask config find <key>is the safest way to diagnose where a value comes from. - Optional GPU/cloud/storage backends are not part of the base package; keep those checks conditional and skip unsafe native cases unless dependencies and hardware are available.