batch-processing
Large-scale batch data processing, chunked streaming, checkpointing, and resume-on-failure protocols.
Файлы1
Открыть в GitHubSKILL.mdskills/batch-processing/SKILL.md
GitHubBatch Processing & Data Chunks Protocol
Process massive datasets efficiently without exhausting memory.
1. Chunked Processing
Process items in chunks of 500 to 5,000 items. Commit database changes after each chunk.
2. Checkpointing & Resumption
Record the last processed ID in a persistent checkpoint store to resume immediately upon restart after failures.