batch-processing

Large-scale batch data processing, chunked streaming, checkpointing, and resume-on-failure protocols.

SKILL.md

GitHub

Batch Processing & Data Chunks Protocol

Process massive datasets efficiently without exhausting memory.


1. Chunked Processing

Process items in chunks of 500 to 5,000 items. Commit database changes after each chunk.


2. Checkpointing & Resumption

Record the last processed ID in a persistent checkpoint store to resume immediately upon restart after failures.