dagster-asset-sync
Layer 6 of the knowledge-sync-loop: validate the ~833 Dagster assets in the 5-layer defs/ tree + ingest them into the dagster_assets Cognee cluster. Use when the user asks 'how do I validate Dagster a
SKILL.md.agents/skills/dagster-asset-sync/SKILL.md
GitHubDagster Asset Sync (Layer 6)
The Layer 6 of the knowledge-sync-loop architecture. Validates the 5-layer Dagster asset graph + ingests it into the
dagster_assetsCognee cluster + emits metadata via thedagster_sync_healthDagster asset.
Why Layer 6 exists
The 5-layer pull-based sync architecture (from
2026-08-15-knowledge-sync-loop-v1) covered:
| Layer | What it syncs |
|---|---|
| 1. paths | pre-v7 path drift in source files |
| 2. ccc | the codebase semantic index |
| 3. cognee | openspec changes + specs + skills |
| 4. skills | agent SKILL.md metadata + references |
| 5. mcp | 14 MCP server health |
The biggest gap: the Dagster asset graph (~833 assets across
the 5-layer orchestration/defs/ tree) wasn't validated by any of the
5 layers. The 2026-08-15-retroactive-pre-v7-cleanup-v1 change closes
this gap by adding Layer 6.
What sync:dagster does
mise run sync:dagster
# → bash scripts/sync/dagster.sh
# → stedding/sync-reports/dagster-{date}.md
The script:
- Walks the 5-layer
orchestration/defs/tree:1_ingestion/,2_materials/,3_model_lifecycle/,4_asset_generation/,5_agent_ops/ - Counts
@asset,@asset_check,@sensordecorators per layer - Counts YAML defs (the Cianfhoghlaim Component YAML files)
- Counts the Cianfhoghlaim Component classes at
orchestration/components/ - Writes a per-group report to
stedding/sync-reports/dagster-{date}.md - Returns 0 if the total asset count is > 0 (suspicious if 0)
The new Dagster asset: dagster_sync_health
Lives at orchestration/defs/sync_assets.py. Emits:
| Metadata | Description |
|---|---|
asset_count |
Total @asset decorators across the 5 layers |
sensor_count |
Total @sensor decorators across the 5 layers |
group_count |
5 (the canonical 5-layer convention) |
broken_asset_count |
0 if healthy (placeholder for future expansion) |
Plus a dagster_sync_alert companion asset that fires when
broken_asset_count > 0 + a dagster_assets_sensor that fires on
every orchestration/defs/ file change.
The new Cognee cluster: dagster_assets
Ingested by scripts/sync_dagster_assets_to_cognee.py. The script:
- Walks
orchestration/defs/+ parses each .py file viaast - Extracts
@asset,@asset_check,@sensordecorator usage - Renders a per-layer markdown summary
- Adds the summary to the
dagster_assetsCognee cluster
After sync, the Cognee graph has 8 typed clusters (was 7):
cognee_docs,openspec_changes,openspec_specs,agent_skills- (5 existing) +
dagster_assets(new)
The consolidated marimo dashboard: sync_health.py
The Dagster tab in notebooks/sync_health.py reads the latest
stedding/sync-reports/dagster-{date}.md and surfaces:
- The per-layer breakdown table
- The total asset count vs the expected baseline (~833)
- The health status (OK if
total_assets > 0) - A full-report expander
Run via:
uv run marimo edit notebooks/sync_health.py
The new CCC concept guide: dagster-asset-graph
The 21st concept guide in .cocoindex_code/guides.yml. Maps queries
about "Dagster asset graph", "5-layer defs/", "Cianfhoghlaim Components" to the
canonical files (the 5 layer folders + the 5 Cianfhoghlaim Component classes).
Quick routing
| If you want to... | Do this |
|---|---|
| Validate the Dagster asset graph | mise run sync:dagster |
| Re-ingest into Cognee | mise run sync:dagster && uv run python scripts/sync_dagster_assets_to_cognee.py |
| See the asset health in Dagster UI | Open dagster_sync_health asset at orchestration/defs/sync_assets.py |
| See the asset sync dashboard | uv run marimo edit notebooks/sync_health.py (Dagster tab) |
| Read the latest report | cat stedding/sync-reports/dagster-{date}.md |
| Run all 6 sync layers | mise run sync:all |
Cross-references
openspec/changes/2026-08-15-retroactive-pre-v7-cleanup-v1/(this change)openspec/specs/retrospective-cleanup/spec.md(the capability spec)scripts/sync/dagster.sh(the Layer 6 sync script)scripts/sync_dagster_assets_to_cognee.py(the Cognee ingest)orchestration/defs/sync_assets.py(thedagster_sync_healthasset)notebooks/sync_health.py(Dagster tab) (the dashboard).cocoindex_code/guides.yml(the 21st CCC concept guide).agents/skills/knowledge-sync-loop/SKILL.md(the parent sync loop skill).agents/skills/dagster/SKILL.md(the core Dagster skill)openspec/specs/dagster-5-layer-component-architecture/spec.md(the 5-layer convention spec)