data-pipeline-etl
ETL/ELT data pipelines, data lakehouse architecture, Apache Spark/DuckDB transformations, and data quality checks.
SKILL.mdskills/data-pipeline-etl/SKILL.md
GitHubData Pipeline (ETL/ELT) Protocol
- Extract raw data into immutable bronze/raw storage layers.
- Transform using DuckDB or Spark with idempotent partition overwrites.
- Validate data quality rules (Great Expectations, Soda) before publishing to gold analytics layers.