data-download

Discover, evaluate, and download publicly available datasets from the internet. Infers data needs from a research question or task, selects authoritative sources, downloads reproducibly, validates fil

SKILL.md

GitHub

Data Download: Discover, Evaluate, and Acquire Research Data

Find and download data for: $ARGUMENTS

Overview

This skill turns a data need into a validated, documented, locally available dataset. It does NOT blindly download from the first URL found. It reasons from the research question to the right data, evaluates candidate sources, downloads reproducibly, validates the result, and records provenance.

The workflow is always: data need → source discovery → source evaluation → download → validation → provenance documentation.

Constants

  • DATA_DIR = data/ — Top-level data directory. Raw downloads go to data/raw/, processed outputs to data/processed/.
  • MANIFEST_FILE = data/DATA_MANIFEST.md — Provenance log for all downloaded datasets.
  • MAX_UNATTENDED_SIZE_MB = 500 — If a single file exceeds this size, pause and confirm with the user before downloading.
  • VERIFY_AFTER_DOWNLOAD = true — Always validate downloaded files (existence, size, format, openability).
  • PREFER_PUBLIC = true — When both public and authenticated sources exist, prefer the public source unless the authenticated source is materially better.

Override via argument, e.g., /data-download "US census tracts" — data dir: datasets/, max size: 2000.


Phase 0: Understand the Data Need

Before searching for data, classify what is actually needed.

Step 0.1: Infer Data Requirements

Extract from the user's request, the research question, or existing project files (research_contract.md, output/EXPERIMENT_PLAN.md, RESEARCH_PLAN.md):

Dimension Question to answer
Geography What study area? Country, state, city, bounding box, global?
Time range What period? Single year, multi-year, historical, real-time?
Spatial resolution Point, tract, county, grid cell, pixel? What resolution?
Temporal resolution Daily, monthly, annual, static snapshot?
Variables / bands What specific attributes, indicators, or spectral bands?
Data type Vector (points, lines, polygons), raster, tabular, network?
File format GeoJSON, Shapefile, GeoPackage, GeoParquet, GeoTIFF, COG, NetCDF, CSV?
Licensing Must it be open? CC-BY? Public domain? Commercial use?
Quality Research-grade? Official statistics? Authoritative boundaries?
Size How many features/pixels/rows? Will it fit in memory?

If the request is too vague (e.g., "get me some data"), STOP and ask the user to specify at least:

  • What geographic area?
  • What variables or themes?
  • What time period?

Step 0.2: Categorize the Data Need

Map the need to one or more data categories:

Category Examples Typical sources
Administrative boundaries Country, state, county, tract, district Census/TIGER, GADM, Natural Earth, OSM
Transportation / infrastructure Roads, railways, airports, utilities OSM, DOT, OpenStreetMap Overpass
Elevation / terrain DEM, slope, aspect, hillshade USGS 3DEP, SRTM, Copernicus DEM, ALOS
Land cover / land use NLCD, ESA WorldCover, MODIS LC MRLC, ESA, USGS, Google Dynamic World
Environmental / climate Temperature, precipitation, drought, soil PRISM, ERA5, NOAA, WorldClim, SoilGrids
Weather Observations, forecasts, station data NOAA ISD/GHCN, Open-Meteo, ERA5
Air quality PM2.5, ozone, AQI EPA AQS, OpenAQ, Copernicus CAMS
Hydrology Rivers, watersheds, streamflow, flood zones NHD, USGS NWIS, HydroSHEDS, FEMA NFHL
Disaster / hazard Hurricane tracks, wildfire, earthquake, flood NOAA IBTrACS, NIFC, USGS Earthquake, FEMA
Census / socioeconomic Population, income, housing, education US Census/ACS, WorldPop, IPUMS, Eurostat
Public health Disease rates, hospital locations, SVI CDC, HHS, EPA EJScreen, County Health Rankings
Remote sensing imagery Landsat, Sentinel, MODIS, VIIRS USGS EarthExplorer, Copernicus Open Access, NASA LAADS
Points of interest Schools, hospitals, parks, businesses OSM, HIFLD, local open data portals
Vegetation / agriculture NDVI, crop type, cropland extent MODIS, Sentinel-2, USDA CDL
Ocean / coastal Bathymetry, sea surface temp, shorelines NOAA NCEI, GEBCO, Copernicus Marine

Write the data requirement summary to data/DOWNLOAD_PLAN.md:

# Data Download Plan

**Task**: [user's request or research question]
**Date**: [today]

## Required Datasets

### Dataset 1: [descriptive name]
- Category: [from table above]
- Geography: [study area]
- Time range: [period]
- Spatial resolution: [resolution]
- Variables: [list]
- Format preference: [format]
- Priority: REQUIRED / NICE-TO-HAVE

### Dataset 2: [descriptive name]
...

## Constraints
- Licensing: [requirement]
- Size budget: [limit]
- Authentication: [any known requirements]

Phase 1: Discover Data Sources

Step 1.1: Source Search Strategy

For each required dataset, search for sources in this priority order:

  1. Official government portals — Census.gov, data.gov, USGS, NOAA, EPA, Eurostat, national mapping agencies
  2. Intergovernmental organizations — UN, World Bank, WHO, FAO, OECD
  3. Well-known research repositories — NASA Earthdata, Copernicus, PANGAEA, Zenodo, Figshare, Dryad
  4. Established open data projects — OpenStreetMap, Natural Earth, GADM, WorldPop, WorldClim
  5. Cloud-hosted public data — AWS Open Data, Google Earth Engine, Microsoft Planetary Computer, Source Cooperative
  6. Official APIs — Census API, NOAA API, EPA API, Open-Meteo, Overpass API
  7. Academic project pages — with clear documentation, DOI, and citation

NEVER use as primary source:

  • Random blog posts or personal websites
  • Undocumented file-sharing links
  • Preview/visualization tiles meant for display only (e.g., map tile servers are NOT data downloads)
  • Scraped HTML tables when a documented API or download endpoint exists
  • Sources without clear provenance or licensing

Step 1.2: Source Catalog

For each candidate source, record:

### Source: [name]
- URL: [download page or API endpoint]
- Provider: [organization]
- Format: [file format(s) available]
- Coverage: [geographic and temporal]
- Resolution: [spatial and temporal]
- License: [license type]
- Access: PUBLIC / API_KEY / LOGIN / INSTITUTIONAL / PAID
- Documentation: [link to docs/metadata]
- Last updated: [date or frequency]
- Stability: HIGH (government/institution) / MEDIUM (research project) / LOW (personal)
- Citation: [how to cite]

Phase 2: Evaluate and Select Sources

Step 2.1: Evaluation Criteria

Score each candidate source:

Criterion Weight What to check
Authority HIGH Is the provider an official or recognized institution?
Coverage match HIGH Does it cover the needed geography, time range, and variables?
Accessibility HIGH Is it publicly downloadable without login?
Format usability MEDIUM Is the format machine-readable and standard?
Documentation quality MEDIUM Are metadata, schema, and methodology documented?
Freshness MEDIUM Is the data current enough for the research need?
Resolution match MEDIUM Does the spatial/temporal resolution match the need?
Stability LOW Will the URL still work in 6 months?
Citability LOW Does it have a DOI, recommended citation, or clear provenance?

Step 2.2: Access Classification and Pause Logic

Classify each source by access type and act accordingly:

Access type Action
PUBLIC — direct HTTP download, no auth Proceed to download
PUBLIC API — free API, no key required Proceed to download via API
API_KEY — free but requires registration for key PAUSE: Tell user they need to register for an API key. Provide the registration URL. Ask if they have a key or want to use an alternative source.
LOGIN — requires account creation PAUSE: Explain that login is required. Provide registration URL. Ask user how to proceed.
INSTITUTIONAL — requires university/org credentials PAUSE: Explain institutional access requirement. Suggest public alternatives if available.
PAID — subscription or per-download cost PAUSE: Explain cost. Strongly recommend free alternatives. Only proceed if user explicitly confirms.
CLICK-THROUGH — requires manual license agreement PAUSE: Explain that a manual license agreement must be accepted in a browser. Provide the URL for the user to accept and download manually.
CAPTCHA — automated access blocked PAUSE: Explain that automated download is not possible. Provide the URL for manual download.

GUARDRAIL: Never fabricate, guess, or hardcode API keys, passwords, or tokens. Never attempt to bypass authentication, CAPTCHAs, or access controls. Never assume the user has credentials unless they explicitly provide them.

Step 2.3: Present Options When Tradeoffs Exist

If multiple sources exist with material tradeoffs, PAUSE and present the options:

I found multiple sources for [data need]:

1. [Source A] — PUBLIC, 30m resolution, 2021, GeoTIFF
   + No login required, direct download
   - Slightly older, coarser resolution

2. [Source B] — API_KEY required, 10m resolution, 2023, COG
   + Higher resolution, more recent
   - Requires free API key registration at [URL]

3. [Source C] — INSTITUTIONAL, 1m resolution, 2024
   + Best resolution and most current
   - Requires university login

Which source should I use? Or should I proceed with Source A (public, no login)?

Let the user decide. If no response and the public source is adequate, default to the public source.

Step 2.4: Size Check

Before downloading, estimate the file size if possible (from documentation, HTTP HEAD request, or API metadata).

If estimated size > MAX_UNATTENDED_SIZE_MB:

The download for [dataset] is approximately [X] MB.
This exceeds the [MAX_UNATTENDED_SIZE_MB] MB threshold.

Should I proceed? Options:
1. Download the full file ([X] MB)
2. Download a spatial subset (specify bounding box or region)
3. Skip this dataset for now

Phase 3: Download

Step 3.1: Directory Setup

from pathlib import Path

data_dir = Path('data')
raw_dir = data_dir / 'raw'
processed_dir = data_dir / 'processed'
raw_dir.mkdir(parents=True, exist_ok=True)
processed_dir.mkdir(parents=True, exist_ok=True)

Organize raw downloads into subdirectories by dataset name:

data/
├── raw/
│   ├── census_tracts_2020/
│   │   ├── tl_2020_us_tract.shp
│   │   ├── tl_2020_us_tract.dbf
│   │   └── ...
│   ├── nlcd_2021/
│   │   └── nlcd_2021_land_cover.tif
│   └── noaa_hurricanes/
│       └── ibtracs_NA.csv
├── processed/
│   └── [user creates these later]
└── DATA_MANIFEST.md

Step 3.2: Download Methods

Choose the download method based on the source type:

Method A: Direct HTTP Download (preferred for single files)

import requests
from pathlib import Path

def download_file(url, dest_path, chunk_size=8192):
    """Download a file with progress tracking and integrity check."""
    dest_path = Path(dest_path)
    dest_path.parent.mkdir(parents=True, exist_ok=True)

    response = requests.get(url, stream=True, timeout=60)
    response.raise_for_status()

    total = int(response.headers.get('content-length', 0))
    downloaded = 0

    with open(dest_path, 'wb') as f:
        for chunk in response.iter_content(chunk_size=chunk_size):
            f.write(chunk)
            downloaded += len(chunk)

    file_size = dest_path.stat().st_size
    if total > 0 and file_size != total:
        raise ValueError(f"Size mismatch: expected {total}, got {file_size}")

    print(f"Downloaded {dest_path.name} ({file_size / 1e6:.1f} MB)")
    return dest_path

For very large files or when requests is slow, use curl or wget:

curl -L -o data/raw/dataset/file.zip "https://example.gov/data/file.zip"
# or
wget -O data/raw/dataset/file.zip "https://example.gov/data/file.zip"

Method B: API Download (when structured query is needed)

import requests
import json

# Example: Census API
params = {
    'get': 'B01001_001E,NAME',
    'for': 'tract:*',
    'in': 'state:06',
    'key': API_KEY  # only if user provided it
}
response = requests.get('https://api.census.gov/data/2021/acs/acs5', params=params)
data = response.json()
# Example: Open-Meteo (free, no key)
params = {
    'latitude': 40.7128,
    'longitude': -74.0060,
    'daily': 'temperature_2m_max,precipitation_sum',
    'start_date': '2020-01-01',
    'end_date': '2023-12-31',
    'timezone': 'America/New_York'
}
response = requests.get('https://api.open-meteo.com/v1/forecast', params=params)
weather = response.json()

Method C: Direct Read (when library handles download internally)

import geopandas as gpd

# GeoJSON from URL
gdf = gpd.read_file('https://raw.githubusercontent.com/.../boundaries.geojson')

# Shapefile from zip URL
gdf = gpd.read_file('https://example.gov/data/tracts.zip')
import pandas as pd

# CSV from URL
df = pd.read_csv('https://example.gov/data/indicators.csv')
import xarray as xr

# NetCDF / Zarr from cloud
ds = xr.open_dataset('https://example.org/data/climate.nc')
# or from Zarr store
ds = xr.open_zarr('s3://bucket/dataset.zarr')

Method D: Archive Extraction

import zipfile
import tarfile
from pathlib import Path

def extract_archive(archive_path, dest_dir):
    """Extract zip or tar archive."""
    archive_path = Path(archive_path)
    dest_dir = Path(dest_dir)

    if archive_path.suffix == '.zip':
        with zipfile.ZipFile(archive_path, 'r') as z:
            z.extractall(dest_dir)
    elif archive_path.suffix in ('.tar', '.gz', '.tgz', '.bz2'):
        with tarfile.open(archive_path, 'r:*') as t:
            t.extractall(dest_dir)
    else:
        raise ValueError(f"Unknown archive format: {archive_path.suffix}")

    print(f"Extracted to {dest_dir}")
    # Optionally remove the archive after extraction
    # archive_path.unlink()

Method E: STAC / Cloud-Optimized Access (for remote sensing)

from pystac_client import Client

# Search a STAC catalog
catalog = Client.open('https://planetarycomputer.microsoft.com/api/stac/v1')
search = catalog.search(
    collections=['sentinel-2-l2a'],
    bbox=[-122.5, 37.5, -122.0, 38.0],
    datetime='2023-01-01/2023-12-31',
    query={'eo:cloud_cover': {'lt': 20}}
)
items = list(search.items())
print(f"Found {len(items)} scenes")
import rioxarray

# Read a Cloud-Optimized GeoTIFF with spatial subset
ds = rioxarray.open_rasterio(
    'https://example.org/data/dem.tif',
    overview_level=2  # use overview for quick preview
)
# Clip to area of interest
ds_clipped = ds.rio.clip_box(minx=-122.5, miny=37.5, maxx=-122.0, maxy=38.0)

Method F: OpenStreetMap via Overpass API

import requests

# Query Overpass API for specific features
overpass_query = """
[out:json][timeout:60];
area["ISO3166-1"="US"]["admin_level"="2"]->.searchArea;
(
  node["amenity"="hospital"](area.searchArea);
  way["amenity"="hospital"](area.searchArea);
);
out center;
"""
response = requests.get(
    'https://overpass-api.de/api/interpreter',
    params={'data': overpass_query}
)
data = response.json()

Or use osmnx for network data:

import osmnx as ox
G = ox.graph_from_place("Manhattan, New York", network_type='drive')
gdf_nodes, gdf_edges = ox.graph_to_gdfs(G)

Step 3.3: Rate Limiting and Courtesy

  • APIs: Respect rate limits. Add time.sleep(1) between sequential API calls unless documentation allows faster.
  • Bulk downloads: Use a 1-second delay between sequential file downloads from the same server.
  • Large files: Use chunked download (Method A) instead of loading the entire response into memory.
  • Retries: Retry failed downloads up to 2 times with exponential backoff. If still failing, report the error and stop.
import time

def download_with_retry(url, dest_path, max_retries=2):
    for attempt in range(max_retries + 1):
        try:
            return download_file(url, dest_path)
        except Exception as e:
            if attempt < max_retries:
                wait = 2 ** attempt
                print(f"Retry {attempt + 1}/{max_retries} in {wait}s: {e}")
                time.sleep(wait)
            else:
                raise RuntimeError(f"Download failed after {max_retries + 1} attempts: {e}")

Phase 3.5: Human Checkpoint — Data Synthesis

Honor the HUMAN_CHECKPOINT flag in CLAUDE.md (default: true). This skill downloads — it does not generate data — but a few code paths still create derived or synthetic artifacts. When HUMAN_CHECKPOINT is true, PAUSE and request explicit user approval before any of the following; when false, log the decision to output/PROJ_NOTES.md and proceed.

Trigger Show before pausing
The requested dataset does not exist in any authoritative source and you are about to construct a substitute by combining other sources (e.g., synthesizing a "population × land cover" raster from two unaligned products) Component sources, alignment / resampling / reprojection recipe, target schema, and the analytical risk if the synthesis is later treated as primary data
You are about to generate a simulated dataset (random points, synthetic boundaries, climate scenario realizations, demonstration data) for an experiment Generator (function + parameters + seed), N, spatial extent, intended use, and explicit confirmation that it will be tagged synthetic: true in the manifest
You are about to call a parametric API (Open-Meteo, Census ACS, Overpass) with parameters you inferred — not parameters the user gave — that materially shape the result (bbox larger than asked, time range padded, variable list expanded) Inferred vs requested parameters side-by-side and how they change the resulting dataset
You are about to spatially subset, temporally aggregate, or otherwise transform a raw download before placing it in data/raw/ The transform, why it is being done before raw storage (default: it shouldn't be — raw must stay raw)
A download failed and you are about to substitute an alternative source whose schema differs (different variable names, units, resolution) Original vs substitute schema diff, unit / projection conversions implied, and downstream claims that depend on field equivalence

Synthetic and substitute datasets must be recorded in data/DATA_MANIFEST.md with Source: SYNTHESIZED (or SUBSTITUTED), the recipe, and a Synthesis approved by user: YYYY-MM-DD line. Do not list a synthesized dataset under the same heading as authoritative downloads.


Phase 4: Validate Downloads

Every download must be validated. Never assume a download succeeded just because no exception was raised.

Step 4.1: Basic File Checks

from pathlib import Path

def validate_download(file_path, min_size_bytes=100):
    """Basic validation: file exists and is not empty/corrupt."""
    p = Path(file_path)
    assert p.exists(), f"File not found: {p}"
    size = p.stat().st_size
    assert size > min_size_bytes, f"File too small ({size} bytes): {p}"
    print(f"OK: {p.name} ({size / 1e6:.2f} MB)")
    return True

Step 4.2: Format-Specific Validation

Format Validation Tool
CSV / TSV Read header, check row count, check expected columns pd.read_csv(path, nrows=5)
GeoJSON / Shapefile / GeoPackage Read, check CRS, check feature count, check geometry types gpd.read_file(path, rows=5)
GeoTIFF / COG Check bands, CRS, bounds, nodata rasterio.open(path)
NetCDF / HDF Check variables, dimensions, time range xr.open_dataset(path)
ZIP archive Test archive integrity, list contents zipfile.ZipFile(path).testzip()
Parquet / GeoParquet Read schema, check row count gpd.read_parquet(path) or pd.read_parquet(path, nrows=5)
import geopandas as gpd

def validate_vector(file_path):
    """Validate a vector geospatial file."""
    gdf = gpd.read_file(file_path, rows=10)
    print(f"  Features: {len(gpd.read_file(file_path))}")
    print(f"  CRS: {gdf.crs}")
    print(f"  Geometry types: {gdf.geom_type.unique()}")
    print(f"  Columns: {list(gdf.columns)}")
    print(f"  Bounds: {gdf.total_bounds}")
    return gdf
import rasterio

def validate_raster(file_path):
    """Validate a raster file."""
    with rasterio.open(file_path) as src:
        print(f"  Bands: {src.count}")
        print(f"  Size: {src.width} x {src.height}")
        print(f"  CRS: {src.crs}")
        print(f"  Bounds: {src.bounds}")
        print(f"  Nodata: {src.nodata}")
        print(f"  Dtype: {src.dtypes}")
    return True

Step 4.3: Content Sanity Checks

After format validation, check content against the download plan:

  • Geographic coverage: Does the bounding box roughly match the requested study area?
  • Temporal coverage: Does the time range match what was requested?
  • Expected variables: Are the expected columns, bands, or layers present?
  • Row / feature count: Is the count in a plausible range?
  • CRS present: Is a coordinate reference system defined (for spatial data)?
  • No obvious corruption: No all-null columns, no zero-byte layers, no garbled text.

If any check fails, report the problem and suggest whether to re-download, try an alternative source, or ask the user.


Phase 5: Document Provenance

Step 5.1: Update the Data Manifest

Append an entry to data/DATA_MANIFEST.md for every downloaded dataset:

## [Dataset Name]

| Field | Value |
|---|---|
| **Source** | [organization / portal name] |
| **URL** | [exact download URL or API endpoint] |
| **Access date** | [YYYY-MM-DD] |
| **License** | [license name or "see URL"] |
| **Citation** | [recommended citation, if available] |
| **Geographic coverage** | [description or bounding box] |
| **Temporal coverage** | [time range or "static"] |
| **Spatial resolution** | [resolution] |
| **Format** | [file format] |
| **Local path** | `data/raw/[folder]/[filename]` |
| **File size** | [size in MB] |
| **Variables** | [key columns / bands / layers] |
| **CRS** | [EPSG code] |
| **Notes** | [any caveats, processing notes, or known issues] |

Step 5.2: Create/Update Manifest Header

If data/DATA_MANIFEST.md does not exist, create it with:

# Data Manifest

All datasets used in this project. Each entry records provenance, access date, and local path.

**Raw data**: `data/raw/` — original downloaded files, never modified.
**Processed data**: `data/processed/` — cleaned, reprojected, or subsetted versions.

---

Step 5.3: Per-Dataset Metadata (optional)

For datasets downloaded via API or with complex provenance, write a metadata.json in the dataset subdirectory:

{
  "name": "us_census_tracts_2020",
  "source": "US Census Bureau TIGER/Line",
  "url": "https://www2.census.gov/geo/tiger/TIGER2020/TRACT/",
  "access_date": "2026-04-10",
  "license": "Public Domain",
  "geographic_coverage": "United States",
  "temporal_coverage": "2020",
  "crs": "EPSG:4269",
  "format": "Shapefile",
  "files": ["tl_2020_06_tract.shp", "tl_2020_06_tract.dbf", "tl_2020_06_tract.shx", "tl_2020_06_tract.prj"],
  "notes": "California tracts only (FIPS 06)"
}

Source Reference: Common Data Sources by Category

This section provides starting points. Always verify URLs are current before downloading — government portals reorganize periodically.

Administrative Boundaries

Source Coverage Format Access
US Census TIGER/Line US (all levels) Shapefile, GeoJSON PUBLIC
GADM Global (admin 0-5) GeoPackage, Shapefile, GeoJSON PUBLIC
Natural Earth Global (simplified) Shapefile, GeoJSON PUBLIC
Eurostat GISCO Europe GeoJSON, Shapefile, TopoJSON PUBLIC
OSM (via Overpass or Geofabrik) Global PBF, Shapefile, GeoJSON PUBLIC

Elevation / Terrain

Source Resolution Coverage Access
USGS 3DEP 1m–10m US PUBLIC
SRTM (NASA) 30m, 90m 60°N–56°S PUBLIC (Earthdata login for 30m)
Copernicus DEM (GLO-30) 30m Global PUBLIC
ALOS World 3D 30m Global PUBLIC (JAXA registration)

Land Cover

Source Resolution Coverage Access
NLCD (MRLC) 30m US PUBLIC
ESA WorldCover 10m Global PUBLIC
MODIS MCD12Q1 500m Global PUBLIC
Google Dynamic World 10m Global PUBLIC (Earth Engine)
USDA Cropland Data Layer 30m US PUBLIC

Environmental / Climate

Source Type Resolution Access
PRISM Climate Group Temp, precip, etc. 4km PUBLIC
WorldClim Climate normals ~1km PUBLIC
ERA5 (Copernicus CDS) Reanalysis 0.25° API_KEY (free registration)
NOAA Climate Normals Station observations Point PUBLIC
MODIS LST Land surface temperature 1km PUBLIC
SoilGrids (ISRIC) Soil properties 250m PUBLIC
Open-Meteo Weather/climate ~1-11km PUBLIC API (no key)

Disaster / Hazard

Source Type Access
NOAA IBTrACS Hurricane/cyclone tracks PUBLIC
USGS Earthquake Catalog Earthquake events PUBLIC API
NIFC / GeoMAC / MTBS Wildfire perimeters PUBLIC
FEMA NFHL Flood hazard zones PUBLIC
FEMA Disaster Declarations Disaster events PUBLIC API
NOAA Storm Events Severe weather reports PUBLIC
NASA FIRMS Active fire detections PUBLIC
Copernicus EMS Emergency mapping products PUBLIC

Census / Socioeconomic

Source Coverage Access
US Census / ACS (API) US (tract, county, state) PUBLIC API (key recommended)
US Census TIGER (geometry) US boundaries PUBLIC
IPUMS US + international microdata LOGIN (free registration)
WorldPop Global population grids PUBLIC
County Health Rankings US counties PUBLIC
CDC PLACES US health indicators PUBLIC
EPA EJScreen Environmental justice indicators PUBLIC

Remote Sensing

Source Platform Access
USGS EarthExplorer Landsat, NAIP, DEMs LOGIN (free)
Copernicus Open Access Hub Sentinel-1/2/3/5P LOGIN (free) → migrating to Copernicus Data Space
NASA Earthdata MODIS, VIIRS, ICESat, etc. LOGIN (free)
Microsoft Planetary Computer Sentinel, Landsat, DEM, more PUBLIC (STAC API)
Google Earth Engine Multi-source archive LOGIN (free for research)
AWS Open Data (USGS/Sentinel) Landsat, Sentinel COGs PUBLIC (S3)

Guardrails: Common Mistakes This Skill Prevents

Mistake How this skill prevents it
Downloading from untrusted random sites Phase 1 Step 1.1 enforces source priority order
Attempting to bypass login/authentication Phase 2 Step 2.2 forces a PAUSE for any non-public source
Hardcoding API keys into scripts Phase 2 guardrail: never embed credentials in code
Downloading a 50 GB global file when 200 MB regional subset suffices Phase 2 Step 2.4 enforces size check and subset suggestion
Mixing raw and processed data Phase 3 Step 3.1 enforces raw/ vs processed/ separation
Failing to record where data came from Phase 5 requires manifest entry for every download
Not checking if downloaded files are valid Phase 4 validates every download
Confusing map tiles with downloadable data Phase 1 Step 1.1 explicitly warns against this
Downloading outdated versions when newer exist Phase 2 Step 2.1 checks freshness
Scraping fragile HTML when a documented API exists Phase 1 Step 1.1 prefers documented endpoints
Proceeding when user must choose among tradeoffs Phase 2 Step 2.3 forces a PAUSE with options
Ignoring licensing Phase 2 evaluation records license for every source
Downloading without checking CRS Phase 4 Step 4.2 validates CRS for spatial data

Outputs

  • data/raw/[dataset_name]/ — Raw downloaded files, never modified
  • data/processed/ — User-created processed versions (this skill does not write here)
  • data/DATA_MANIFEST.md — Provenance log for all downloaded datasets
  • data/DOWNLOAD_PLAN.md — Requirements and source evaluation (if complex multi-dataset download)
  • data/raw/[dataset_name]/metadata.json — Per-dataset metadata (for API or complex downloads)

Key Rules

  • Large file handling: If the Write tool fails due to file size, immediately retry using Bash (cat << 'EOF' > file) to write in chunks. Do NOT ask the user for permission — just do it silently.

  • Data need first. Never download before understanding what is needed and why.

  • Authoritative sources first. Government and institutional sources beat random websites.

  • PAUSE for authentication. If login, API key, payment, CAPTCHA, or click-through is required, stop and ask the user. Never fabricate credentials or bypass access controls.

  • PAUSE for major tradeoffs. If multiple sources exist with material differences in resolution, coverage, access, or size, present options and let the user decide.

  • PAUSE for large files. If a single file exceeds MAX_UNATTENDED_SIZE_MB, confirm before downloading.

  • Validate every download. Check existence, size, format readability, CRS, and content plausibility.

  • Document everything. Every downloaded dataset gets a manifest entry with source URL, access date, license, and local path.

  • Preserve raw data. Never overwrite raw downloads. Keep originals in data/raw/, create processed versions in data/processed/.

  • Prefer subsets over full archives. If only California is needed, do not download all 50 states.

  • Respect rate limits. Add delays between sequential API calls and bulk downloads.

  • Do not fabricate data. If a download fails, report the failure — do not create synthetic substitutes without telling the user.

  • Do not scrape when APIs exist. Prefer structured download endpoints over HTML parsing.

  • No secrets in code. Never write API keys, tokens, or passwords into scripts, notebooks, or committed files. Use environment variables or prompt the user.

Composing with Other Skills

This skill provides data for the rest of the research pipeline:

/data-download "what data is needed"   ← you are here
/spatial-analysis "research question"  → analyze the downloaded data
/lit-review                            → literature context (separate concern)
/paper-write                           → document data sources in Methodology
/paper-figure                          → visualize the downloaded data
/submit-check                          → verify data citations and DOIs

Integration points:

  • To spatial-analysis: Downloaded data lands in data/raw/. The spatial-analysis skill reads from there (or from paths the user specifies).
  • To deploy-experiment: This skill handles all external data — custom study area data, external covariates, boundary files, remote sensing imagery.
  • To paper-write / submit-check: The DATA_MANIFEST.md provides source URLs, citations, and access dates for the Methodology section and submission checklist.
  • Knowledge base: Read skills/knowledge/spatial-methods.md for CRS guidance when downloading spatial data.