modal
Serverless GPU cloud for Python with per-second billing on A100/H100/L40S. Use for burst ML training (>13B parameter models, multi-GPU), parallel evaluation runs, scheduled batch jobs that exceed loca
SKILL.md.agents/skills/modal/SKILL.md
GitHubModal — Serverless GPU Cloud
When to use this skill
Use when you need to:
- "Train a 13B+ model that won't fit on my M4 Mac GPU"
- "Run a parallel hyperparameter sweep on 8× A100"
- "Process the full multi-nation curriculum corpus at scale"
- "Schedule a recurring GPU job (nightly HTR re-train)"
- "Serve a model on GPU for inference (without a permanent instance)"
Overview
Modal is a serverless Python cloud that runs code on on-demand GPUs (A100, H100, L40S, A10G, T4) with per-second billing. No Kubernetes, no cluster management — you decorate a function, and it runs in the cloud.
Key features:
@app.function(gpu="H100")decorator — run any Python function on a remote GPU@app.webhook()decorator — expose a function as an HTTP endpoint@app.asgi()/@app.wsgi()— mount a full ASGI/WSGI app (FastAPI, etc.) without writing a servermodal.Volume— shared network-attached storage (similar to S3)- Image caching — Docker images are built once and cached across runs
- Snapshotting — pause a long-running function and resume
- Multi-region — choose the GPU region (us-east, eu-west, ap-south)
When NOT to use this skill
- The job fits on the local M4 Mac (use the local
bunchlochenvironment) - The job is I/O-bound, not GPU-bound (use Dagster + Cloudflare Workers instead)
- The job requires 24/7 uptime (Modal cold-starts ~5-15s; use a dedicated server)
Cianfhoghlaim integration (PRESERVED from the docs)
The Cianfhoghlaim burst-training story is:
┌─────────────────┐
│ MacBook M4 │ ← daily fine-tuning
│ (bunchloch) │ 3B-7B models, <2hr jobs
└────────┬────────┘
│ exceeded capacity
▼
┌─────────────────┐
│ Modal H100 │ ← burst training
│ (serverless) │ 13B+ models, multi-GPU
└────────┬────────┘
│ trained artifacts
▼
┌─────────────────┐
│ Garage S3 │ ← sync via rclone
│ (cloud) │
└────────┬────────┘
│ downloaded
▼
┌─────────────────┐
│ llama-swap │ ← local serving
│ (bunchloch) │
└─────────────────┘
The MacBook M4 (bunchloch) handles daily fine-tuning of
3B-7B models (Unsloth + TRL). For 13B+ full-parameter
training or multi-GPU sweeps, the project uses Modal H100
on-demand (~$3/hr). Trained artifacts are synced to Garage S3
via rclone, then downloaded back to bunchloch for llama-swap
serving.
The same pattern applies to:
- HTR / OCR ensemble inference — Modal L40S for batch processing of the multi-nation curriculum corpus
- Embedding fine-tuning — Modal A100 for BGE-M3 domain adaptation
- Parallel eval runs — Modal A10G for parallel RAGAS evaluation across 100s of test cases
Core patterns
Basic GPU function
import modal
app = modal.App("cianfhoghlaim-htr-finetune")
image = (
modal.Image.debian_slim(python_version="3.12")
.pip_install("torch", "transformers", "datasets", "peft", "trl")
)
@app.function(gpu="H100", image=image, timeout=7200)
def train_htr(epochs: int = 3) -> str:
"""Fine-tune a HTR model on a 13B parameter base."""
from transformers import ...
# ... training code ...
model.save_pretrained("/mnt/models/cianfhoghlaim-htr-v1")
return "/mnt/models/cianfhoghlaim-htr-v1"
@app.local_entrypoint()
def main(epochs: int = 3):
result = train_htr.remote(epochs=epochs)
print(f"Trained model: {result}")
Run: modal run --detach train.py --epochs 3
Volumes (shared storage)
volume = modal.Volume.from_name("cianfhoghlaim-models", create_if_missing=True)
@app.function(gpu="A100", volumes={"/mnt/models": volume}, timeout=3600)
def evaluate(model_id: str, test_set: str) -> dict:
"""Evaluate a model on a test set; results stored in the volume."""
# ...
volume.commit() # persist
return {"accuracy": 0.94, "loss": 0.08}
Webhook (HTTP endpoint)
@app.function(gpu="L40S", timeout=300)
@app.webhook(method="POST")
def ocr_endpoint(request: dict) -> dict:
"""OCR a PDF via vision model; return extracted text."""
pdf_bytes = base64.b64decode(request["pdf_base64"])
# ... call vision LLM ...
return {"text": extracted_text, "pages": n}
ASGI app (FastAPI / Hono)
fastapi_app = FastAPI()
@fastapi_app.get("/health")
def health():
return {"status": "ok"}
@app.function(gpu="A10G", timeout=600)
@modal.asgi()
def serve():
return fastapi_app
Snapshotting (pause + resume)
@app.function(gpu="H100", timeout=86400, snapshot=True)
def long_training():
"""Train for 24 hours; pause + resume across Modal restarts."""
# ... long training ...
Scheduled job (cron)
@app.function(gpu="A100", schedule=modal.Period(hours=12), timeout=3600)
def nightly_htr_retrain():
"""Re-train HTR model every 12 hours."""
# ... retrain ...
CLI
# Run a function locally (or in the cloud if `image=`)
modal run script.py
# Run detached (don't block terminal)
modal run --detach script.py
# Deploy a webhook
modal deploy script.py
# List recent runs
modal run list
# Inspect logs
modal run logs <run-id>
# Volume management
modal volume create cianfhoghlaim-models
modal volume ls cianfhoghlaim-models
Best practices
- Use
Image.debian_slim(python_version="3.12")for fast cold starts —ubuntuimages are 2-3× larger - Pin dependencies —
pip_install("torch==2.5.0", ...)notpip_install("torch") - Use
Volumefor model artifacts > 100MB — local disk is ephemeral - Snapshot long jobs —
snapshot=Truelets you pause + resume across Modal restarts - Use
--detachfor long jobs — don't block your terminal - Cache expensive setups with
@app.enter()— runs once per container, not per function call - Set timeouts aggressively —
timeout=3600for an hour, nottimeout=86400by default
Common pitfalls
- Cold start latency — first call to a function can take 30-60s (image pull + setup). Subsequent calls are fast
- GPU quota — Modal has per-account GPU limits; request quota increases via the dashboard
- S3 transfer costs — for multi-GB model artifacts, sync via rclone (cheap) not via Modal (expensive)
- Snapshots cost money while paused — pay-per-second applies to paused snapshots too
- Image caching is per-account — if you have multiple Modal accounts, you re-build images for each
Cross-references
.agents/skills/unsloth/SKILL.md— for local Unsloth + TRL fine-tuning (the upstream skill).agents/skills/dagster/SKILL.md— for the orchestration layer; Modal is invoked from Dagster assets.agents/skills/dlt/SKILL.md— Modal is the deployment target for thedeploy-modalrecipe- The Cianfhoghlaim
infrastructure/stacks/modal/stack (planned; not yet provisioned)
Resources
- Modal docs: https://modal.com/docs
- Modal examples: https://github.com/modal-labs/modal-examples
- Cianfhoghlaim burst-training workflow: see the
cianfhoghlaim/agents/meaisinfhoghlaim/ocr/ensemble_gradio.pyfor a working example