airflow
Build and troubleshoot Apache Airflow DAGs, operators, sensors, schedules, and retry behavior for data pipelines.
SKILL.mdskills/airflow/SKILL.md
GitHubApache Airflow AI Skill Guide
Overview & Engine Architecture
Airflow schedules DAGs of tasks executed by workers; the scheduler parses DAG files, the metadata DB stores run state, and executors (Local/Celery/Kubernetes) run task instances. Agents write idempotent tasks, set explicit retries/timeouts, avoid top-level heavy I/O in DAG files, and pass data via XCom sparingly (or external storage).
DAG file -> scheduler -> executor/workers
|
metadata DB (runs, XCom)
|
task logs / sensors
When to use this skill
- Time-based or data-aware batch pipelines
- Orchestrating dbt, Spark, warehouse SQL, ML batch jobs
- Backfills with clear logical dates
Operational directives
- Keep DAG top-level code fast (imports + structure only).
- Tasks must be idempotent for a given
data_interval/ logical date. - Set
retries,retry_delay, andexecution_timeoutintentionally. - Prefer pushing large payloads to object storage over big XComs.
- Never commit connection passwords; use Airflow Connections / secrets backend.
Minimal DAG
from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.bash import BashOperator
with DAG(
dag_id="orders_daily",
start_date=datetime(2026, 1, 1),
schedule="@daily",
catchup=False,
default_args={"retries": 2, "retry_delay": timedelta(minutes=5)},
tags=["orders"],
) as dag:
extract = BashOperator(
task_id="extract",
bash_command="python /opt/airflow/jobs/extract_orders.py --date {{ ds }}",
)
dbt_run = BashOperator(
task_id="dbt_run",
bash_command="cd /opt/dbt && dbt build --select marts.* --vars '{run_date: {{ ds }}}'",
)
extract >> dbt_run
Useful CLI
airflow dags list
airflow dags test orders_daily 2026-08-26
airflow tasks test orders_daily extract 2026-08-26
Common failures
| Symptom | Cause | Fix |
|---|---|---|
| DAG not appearing | import error / parse fail | check scheduler logs |
| Zombie / stuck tasks | worker death | timeouts; health checks |
| Huge backfill load | catchup=True | limit; clear carefully |
| Sensor hanging | wrong poke / mode | reschedule mode; timeouts |
Best practices
- One business pipeline per DAG id; stable task ids for clear history.
- Use datasets/data-aware scheduling when producers/consumers share tables.
- Pin provider package versions with Airflow constraints.
- Alert on SLA misses and failed task emails/Slack callbacks.
Limitations
- Not a streaming engine; pair with Kafka/Flink for continuous event processing.
- Executor/deployment topology (MWAA, Composer, K8s) changes ops details.
- This skill does not replace capacity planning for workers/metadata DB.
Related skills
@prefect- alternative Python-native orchestration@dbt- SQL models often invoked from Airflow@spark- heavy distributed tasks