Organizing eBFE Models

Purpose

When the user asks to organize eBFE or BLE downloads, use this skill to transform downloaded FEMA eBFE/BLE study area files into a standardized 4-folder structure, regardless of the original archive organization patterns.

Standardized Output Structure

Every organized eBFE study area should have this structure:

{StudyAreaName}_{HUC8}/
├── HMS Model/              # HEC-HMS hydrologic models
│   ├── {ProjectName}.hms
│   ├── {ProjectName}.basin
│   ├── *.dss
│   └── ... (all HMS-related files)
├── RAS Model/              # HEC-RAS hydraulic models
│   ├── {Model1}/
│   │   ├── {Model1}.prj
│   │   ├── {Model1}.g##
│   │   ├── {Model1}.p##
│   │   └── ... (all RAS project files)
│   └── {Model2}/...
├── Spatial Data/           # GIS, terrain, geodatabases
│   ├── Terrain/
│   ├── *.tif, *.hdf
│   ├── *.gdb/
│   └── ... (all spatial data)
└── Documentation/          # Reports, metadata, inventories
    ├── *.pdf
    ├── *.xlsx (inventories)
    ├── *_metadata.xml
    └── ... (all documentation)

Input Patterns

Handle these downloaded archive patterns:

Pattern 1: Multiple 1D models in variable wrapper folders (80-200 MB) Pattern 2: ModelURLs.txt links file (1 KB) Pattern 3: Single 2D model in nested zip (5-15 GB) Pattern 4: Compound HMS + RAS in nested zips (8+ GB)

See feature_dev_notes/eBFE_Integration/RESEARCH_FINDINGS.md for complete pattern documentation.

File Classification Rules

HMS Model/ Folder

Include files with these extensions or patterns:

• .hms - HEC-HMS project
• .basin - Basin model file
• .met - Meteorology file
• .control - Control specifications
• .run - Run configuration
• .results - HMS results
• .dss in Hydrology/ path
• .log, .out in HMS folders

Path indicators: Hydrology/, HMS/

RAS Model/ Folder

Include files with these extensions or patterns:

• .prj - HEC-RAS project (validate: contains "Proj Title=", "Geom File=")
• .g## - Geometry file
• .p## - Plan file
• .f## - Flow file
• .u## - Unsteady flow file
• .c## - Sediment file
• .b## - Bridge/culvert file
• .bco## - Boundary condition override
• .IC.O## - Initial conditions override
• .x## - Cross section index
• .rasmap - RAS Mapper project
• .dsc - DSS catalog (if in RAS folder)
• .dss (if in RAS folder, not HMS folder)

Exclude: .prj files in Features/, Shp/, gis/ (shapefiles, not HEC-RAS)

Organization within RAS Model/:

• If single model: Place files directly in RAS Model/
• If multiple models: Create subfolder per model (use .prj stem as folder name)

Spatial Data/ Folder

Include files with these extensions or patterns:

• .tif - GeoTIFF raster
• .hdf - Terrain HDF (if NOT .g##.hdf or .p##.hdf)
• .vrt - Virtual raster
• .gdb/ - File geodatabase
• .shp, .shx, .dbf, .prj - Shapefiles (in Features/, Shp/, gis/ folders)
• Folder named Terrain/ (entire folder)
• Folder named Spatial/, GIS/, Shp/, Features/

Exception: .g##.hdf (geometry preprocessor) and .p##.hdf (results) stay with RAS Model/

Documentation/ Folder

Include files with these extensions or patterns:

• .pdf - Reports, guides
• .xlsx, .xls - Inventories, metadata spreadsheets
• .xml - Metadata
• Files with "Inventory" in name
• Files with "metadata" in name
• Files with "Report" in name
• README* files

Agent Deliverables

REQUIRED for Every Organization:

1. ✅ Organized model in 4-folder structure
2. ✅ agent/model_log.md - Agent work log
3. ✅ MANIFEST.md - Organization manifest
4. ✅ NEW: Generated organize_{modelname}() function in feature_dev_notes/eBFE_Integration/generated_functions/
5. ✅ NEW: DSS validation results (if DSS files present)
6. ✅ NEW: Suggested compute test command for user
7. ✅ NEW: Haiku subagent results check (if compute test run)

Workflow

Step 1: Analyze Archive Structure

# Inspect the downloaded archive
# - List all files
# - Identify nested zips
# - Detect pattern (1-4)
# - Count .prj files (RAS) and .hms files (HMS)

Step 2: Extract Recursively

# Extract Models.zip
# Find nested .zip files (_Final.zip, RAS_Submittal.zip, etc.)
# Extract nested zips in place
# Preserve directory structure during extraction

Step 3: Classify and Organize

# Recursively walk extracted content
# For each file:
#   - Determine classification (HMS/RAS/Spatial/Docs)
#   - Move to appropriate standardized folder
#   - For RAS models with multiple projects:
#       - Group files by .prj stem
#       - Create subfolder per model

# CRITICAL INTEGRATIONS (to create runnable HEC-RAS models):

# 1. Move Output/ folder INTO HEC-RAS project folder
#    eBFE separates Output/ (pre-run HDF files) from Input/ (project)
#    HEC-RAS expects HDF files in project folder
for model_folder in ras_model.glob('**/Input'):
    output_folder = model_folder.parent / 'Output'
    if output_folder.exists():
        # Move all Output/*.hdf and other files INTO Input/
        for output_file in output_folder.rglob('*'):
            if output_file.is_file():
                shutil.copy2(output_file, model_folder / output_file.name)

# 2. Move Terrain/ INTO HEC-RAS project folder
#    eBFE places Terrain/ as sibling to Input/ (breaks .rasmap references)
#    HEC-RAS expects Terrain/ in project folder
for model_folder in ras_model.glob('**/Input'):
    terrain_folder = model_folder.parent / 'Terrain'
    if terrain_folder.exists():
        # Move Terrain/ INTO Input/Terrain/
        shutil.copytree(terrain_folder, model_folder / 'Terrain', dirs_exist_ok=True)

Step 4: Validate Organization

# Verify each folder:
#   - HMS Model/: Check for .hms files
#   - RAS Model/: Verify .prj files are valid HEC-RAS projects
#   - Spatial Data/: Check for terrain files
#   - Documentation/: Ensure reports are present
#
# Report summary:
#   - X HMS projects found
#   - Y RAS projects found
#   - Z terrain files found
#   - N documents organized

Step 4: Correct DSS File Paths (CRITICAL - VALIDATE EXISTENCE)

# CRITICAL: Fix DSS file path references in HEC-RAS files AND verify files exist
# eBFE models have:
#   - Absolute paths from original system: C:\eBFE\... (doesn't exist on user system)
#   - Wrong relative paths: DSS\Input\file.dss (file is actually in same folder)
# These cause "DSS path needs correction" GUI popups that break automation

import re

# Step 1: Find all DSS files that ACTUALLY EXIST in organized structure
dss_files = list(ras_model_folder.glob('**/*.dss'))
dss_lookup = {dss.name: dss for dss in dss_files}  # Lookup by filename

print(f"Found {len(dss_files)} DSS file(s):")
for dss in dss_files:
    print(f"  - {dss.relative_to(ras_model_folder)}")

# Step 2: Find HEC-RAS files that reference DSS (.u##, .prj, .p##)
hecras_files = []
hecras_files.extend(ras_model_folder.glob('**/*.u[0-9]*'))   # Unsteady flow files
hecras_files.extend(ras_model_folder.glob('**/*.p[0-9][0-9]'))  # Plan files
hecras_files.extend(ras_model_folder.glob('**/*.prj'))  # Project files

# Step 3: For each HEC-RAS file, validate and correct DSS references
for hecras_file in hecras_files:
    content = hecras_file.read_text(encoding='utf-8', errors='ignore')
    modified = False

    # Find all "DSS File=" lines
    dss_file_pattern = re.compile(r'DSS File=(.+?)(?:\n|$)', re.IGNORECASE)

    for match in dss_file_pattern.finditer(content):
        old_path = match.group(1).strip()
        dss_filename = Path(old_path).name  # Extract just the filename

        # Check if this DSS file exists in our organized structure
        if dss_filename in dss_lookup:
            actual_dss_path = dss_lookup[dss_filename]

            # CRITICAL: Verify the DSS file actually exists
            if actual_dss_path.exists():
                # Calculate correct relative path from HEC-RAS file to actual DSS location
                rel_path = actual_dss_path.relative_to(hecras_file.parent)
                rel_path_str = str(rel_path).replace('\\', '/')

                # Only replace if current path is wrong
                if old_path != rel_path_str:
                    old_full = f"DSS File={old_path}"
                    new_full = f"DSS File={rel_path_str}"
                    content = content.replace(old_full, new_full)
                    modified = True
                    print(f"  Corrected in {hecras_file.name}:")
                    print(f"    Old: {old_path}")
                    print(f"    New: {rel_path_str} (✓ verified exists)")
            else:
                print(f"  ⚠️ WARNING: DSS not found: {dss_filename}")
        else:
            print(f"  ⚠️ WARNING: DSS not in organized structure: {dss_filename}")

    if modified:
        hecras_file.write_text(content, encoding='utf-8')

# Result: Model opens without GUI popups, all DSS paths verified
# Handles cases like:
#   DSS\Input\UPG_precip.dss → UPG_precip.dss (if file is in same folder)
#   C:\eBFE\...\file.dss → ../DSS/file.dss (if file is in subdirectory)

Step 5: Validate DSS Files (If Present)

# Validate DSS pathname contents (not just file paths)
from ras_commander.dss import RasDss

# Find all .dss files in RAS Model/
dss_files = list(ras_model_folder.glob('**/*.dss'))

for dss_file in dss_files:
    # Get catalog
    catalog = RasDss.get_catalog(dss_file)

    # Check each pathname
    for pathname in catalog['pathname']:
        result = RasDss.check_pathname(dss_file, pathname)
        if not result.is_valid:
            # Document invalid pathnames in model_log.md
            pass

Step 6: Suggest Compute Test (If RAS Model Present)

# If RAS model organized, suggest optional compute test to user
# This validates terrain, land use, and DSS files are correct

print("=" * 80)
print("OPTIONAL VALIDATION: Compute Test")
print("=" * 80)
print("To verify terrain, land use, and DSS files are correct,")
print("run at least one plan:")
print()
print("  from ras_commander import init_ras_project, RasCmdr")
print(f"  init_ras_project(r'{ras_model_folder}', '{version}')")
print("  RasCmdr.compute_plan('01', num_cores=2)")
print()
print("If the plan executes successfully, terrain/DSS files are valid.")
print("=" * 80)

Step 7: Check Results with Haiku Subagent (If Compute Test Run)

# After compute test completes (if user runs it), check results for errors
from ras_commander.hdf import HdfResultsPlan

# Launch haiku subagent to check results
Task(
    subagent_type="notebook-output-auditor",
    model="haiku",
    description="Check HEC-RAS results for errors",
    prompt=f"""
    Check HEC-RAS results for errors in: {hdf_file}

    Extract compute messages and check for:
    - Error messages
    - Warning messages
    - Convergence issues
    - Numerical instabilities
    - Missing data

    Write findings to: {agent_folder}/compute_test_results.md
    """
)

Step 8: Generate Deterministic Function

REQUIRED: After organizing each model, generate a Python function for future use.

# Generate organize_{modelname}() function
# Write to: feature_dev_notes/eBFE_Integration/generated_functions/{studyarea}_{huc8}.py

def organize_{clean_name}(
    downloaded_folder: Path,
    output_folder: Optional[Path] = None
) -> Path:
    """
    Organize {StudyArea} ({HUC8}) eBFE model.

    Pattern {X}: {Pattern description}

    Generated by ebfe_organize_models agent on {date}.
    """
    # Specific organization steps discovered by agent
    # Include pattern detection, extraction, classification
    # Use lessons learned from this organization
    pass

Deliverable: Write generated function to feature_dev_notes/eBFE_Integration/generated_functions/organize_{studyarea}_{huc8}.py

Step 9: Generate Manifest and Model Log

# Create organization manifest: MANIFEST.md (in project root)
# Document:
#   - Original archive pattern detected
#   - Files organized by category
#   - RAS project inventory (model names, types, sizes)
#   - HMS project inventory (if any)
#   - Terrain files inventory
#   - Document inventory
#   - DSS validation results
#   - Suggested compute test command

# Create agent work log: agent/model_log.md (in project root)
# **REQUIRED**: Every organized model MUST have agent/model_log.md
# Document:
#   - Agent actions taken
#   - Files classified and moved
#   - Decisions made during organization
#   - Issues encountered and resolutions
#   - DSS validation results
#   - Generated function location
#   - Timestamps for all operations

Output Format

REQUIRED: agent/model_log.md

CRITICAL: Every organized eBFE model MUST have agent/model_log.md documenting agent work.

Location: {StudyAreaName}_{HUC8}/agent/model_log.md

Template:

# Agent Work Log - {StudyAreaName}

**Agent**: ebfe_organize_models
**Date**: {YYYY-MM-DD HH:MM:SS}
**Study Area**: {Name} ({HUC8})
**Pattern**: {Pattern 1/2/3/4}

## Actions Taken

### Archive Extraction
- Extracted: {source_archive}
- Nested zips: {list any nested zips extracted}
- Total files extracted: {count}

### File Classification
{For each file category, list files moved}

#### HMS Model (X files, Y MB)
- {list key files}

#### RAS Model (X files, Y GB)
- {list models and key files}

#### Spatial Data (X files, Y MB)
- {list terrain, shapefiles, etc.}

#### Documentation (X files, Y MB)
- {list reports, inventories}

### Decisions Made
- {Document any classification decisions}
- {Document handling of ambiguous files}
- {Document terrain handling approach}

### Issues Encountered
- {List any problems}
- {Document resolutions}

### Validation Checks
- [x] All files classified
- [x] RAS .prj files validated (not shapefiles)
- [x] Terrain location documented
- [x] Manifest created

## Next Steps for User
{Provide clear next steps, e.g., testing with ras-commander}

MANIFEST.md Template

# eBFE Model Organization Manifest

**Study Area**: {Name}
**HUC8**: {HUC8}
**Original Pattern**: {Pattern 1/2/3/4}
**Organized Date**: {YYYY-MM-DD}

## HMS Model

{X} HEC-HMS projects found:
- {ProjectName}.hms - {StormFrequencies}

## RAS Model

{Y} HEC-RAS projects found:

### {ModelName1}
- Type: {1D/2D}
- Plans: {#}
- Size: {MB/GB}
- Terrain: {Self-contained/External/Not needed}

### {ModelName2}
...

## Spatial Data

{Z} terrain/spatial files:
- {TerrainFile1} - {Size}
- Geodatabase: {.gdb folders}
- Shapefiles: {count}

## Documentation

{N} documents:
- {Report1}.pdf - {Size}
- {Inventory}.xlsx
- {Metadata}.xml

## Notes

{Any special observations or issues encountered}

Example Usage

# Organize a downloaded study area
from pathlib import Path

# Input: Raw downloaded and extracted files
source = Path("D:/eBFE/raw_downloads/12040102_Spring_Models_extracted/")

# Output: Organized into 4 folders
target = Path("D:/eBFE/organized/SpringCreek_12040102/")

# Agent organizes the files
# Result: target/ now has HMS Model/, RAS Model/, Spatial Data/, Documentation/

Special Handling Notes

Nested Zips

• Recursively extract _Final.zip, RAS_Submittal.zip, etc.
• Delete intermediate zips after extraction (optional)
• Preserve meaningful folder names

Shapefile .prj vs HEC-RAS .prj

• Shapefiles: .prj in Features/, Shp/, gis/ → Spatial Data/
• HEC-RAS: .prj with "Proj Title=" content → RAS Model/

Self-Contained vs External Terrain

• If Terrain/ folder in model directory → stays with RAS Model/, copy to Spatial Data/
• If .rasmap references external terrain → document in manifest

Multiple RAS Projects

• Create subfolder per .prj file using stem name
• Group related files (.g##, .p##, .f##) with matching basename
• Keep directory hierarchy for watershed organization if meaningful

Templates and Validation

For generated function templates, DSS validation examples, and compute test workflows, see references/templates-and-validation.md.

Cross-References

Skills (related workflows):

• ebfe_validate_models -- Use downstream to validate organized models

Agents (delegate when needed):

• ebfe-organizer -- Delegate for complex multi-model eBFE organization

Primary sources:

• ras_commander/ebfe_models.py -- RasEbfeModels implementation
• docs/ebfe_models.md -- Complete eBFE documentation