m12
Superagent skill m12 — Batch Operations & Parallel Execution
SKILL.mdskills/superagent/m12/SKILL.md
GitHubm12 — Batch Operations & Parallel Execution
Load ONLY when: batch, parallel, mass, bulk, queue, multiple wallets, multiple accounts, many addresses, 100x, scrape, farming
When to batch
- N > 5 independent ops with same shape
- Each op is idempotent OR retryable
- Total time at sequential exceeds patience (>30s)
When NOT to batch
- Ops share mutable state (nonce, balance, file write)
- Rate limits force serial
- One failure should halt the rest
Pattern A — JS p-limit (recommended)
import pLimit from 'p-limit';
const limit = pLimit(5); // 5 concurrent max
const tasks = wallets.map(w => limit(() => process(w)));
const results = await Promise.allSettled(tasks);
const ok = results.filter(r => r.status === 'fulfilled').map(r => r.value);
const fail = results.filter(r => r.status === 'rejected').map(r => r.reason);
console.log(`OK: ${ok.length}, FAIL: ${fail.length}`);
Promise.allSettled not Promise.all — one failure won't kill the batch.
Pattern B — Python asyncio.gather with semaphore
import asyncio
sem = asyncio.Semaphore(5)
async def bounded(coro):
async with sem:
return await coro
results = await asyncio.gather(
*[bounded(process(w)) for w in wallets],
return_exceptions=True
)
ok = [r for r in results if not isinstance(r, Exception)]
fail = [r for r in results if isinstance(r, Exception)]
Pattern C — Worker pool (CPU-bound)
from concurrent.futures import ProcessPoolExecutor, as_completed
with ProcessPoolExecutor(max_workers=4) as pool:
futures = {pool.submit(work, item): item for item in items}
for fut in as_completed(futures):
try:
print(fut.result())
except Exception as e:
print(f"FAIL {futures[fut]}: {e}")
Rate-limit aware batching
Telegram: ~30 msg/sec global, 1/sec per chat. Use token bucket.
class TokenBucket {
constructor(rate, capacity) {
this.tokens = capacity;
this.capacity = capacity;
this.rate = rate; // tokens/sec
this.last = Date.now();
}
async take() {
while (this.tokens < 1) {
const elapsed = (Date.now() - this.last) / 1000;
this.tokens = Math.min(this.capacity, this.tokens + elapsed * this.rate);
this.last = Date.now();
if (this.tokens < 1) await new Promise(r => setTimeout(r, 50));
}
this.tokens -= 1;
}
}
const bucket = new TokenBucket(25, 25); // 25/sec
for (const msg of messages) {
await bucket.take();
bot.sendMessage(msg.chat, msg.text).catch(e => console.error(e));
}
Web3 specific — parallel send WITHOUT nonce collision
ONE wallet → serial only (or pre-assign nonces). MANY wallets → fully parallel safe.
// Many wallets, parallel mint
const limit = pLimit(10);
const txs = wallets.map(w => limit(async () => {
try {
const tx = await contract.connect(w).mint();
return await tx.wait();
} catch (e) {
return { error: e.message, wallet: w.address };
}
}));
const out = await Promise.all(txs);
Progress tracking
let done = 0;
const total = items.length;
const interval = setInterval(() => {
console.log(`[${done}/${total}] ${((done/total)*100).toFixed(1)}%`);
}, 5000);
const tasks = items.map(it => limit(async () => {
try { return await work(it); }
finally { done++; }
}));
await Promise.allSettled(tasks);
clearInterval(interval);
Python equivalent — use tqdm:
from tqdm.asyncio import tqdm_asyncio
results = await tqdm_asyncio.gather(*[bounded(work(i)) for i in items])
Error aggregation pattern
const errors = [];
const results = await Promise.allSettled(tasks);
for (const r of results) {
if (r.status === 'rejected') {
errors.push({ reason: r.reason.message || r.reason });
}
}
if (errors.length) {
fs.writeFileSync(
`errors-${Date.now()}.json`,
JSON.stringify(errors, null, 2)
);
console.log(`${errors.length} errors → errors-*.json`);
}
Resume from failure (CRITICAL for long batches)
Save state every N items so SIGINT doesn't lose hours of work.
import fs from 'fs';
const STATE = './batch-state.json';
let state = fs.existsSync(STATE)
? JSON.parse(fs.readFileSync(STATE))
: { done: [], failed: [] };
const remaining = allItems.filter(x =>
!state.done.includes(x.id) && !state.failed.includes(x.id)
);
const limit = pLimit(5);
for (const item of remaining) {
limit(async () => {
try {
await work(item);
state.done.push(item.id);
} catch (e) {
state.failed.push(item.id);
} finally {
fs.writeFileSync(STATE, JSON.stringify(state)); // checkpoint
}
});
}
For 100+ items, save every 10 — not every 1 — to avoid I/O bottleneck.
Chunked processing (memory-safe)
Process 1M-row file? Don't load it all.
import readline from 'readline';
import fs from 'fs';
const rl = readline.createInterface({
input: fs.createReadStream('big.csv'),
crlfDelay: Infinity
});
const BATCH = 1000;
let buffer = [];
for await (const line of rl) {
buffer.push(line);
if (buffer.length >= BATCH) {
await processBatch(buffer);
buffer = [];
}
}
if (buffer.length) await processBatch(buffer);
Output format for batch jobs
[BATCH START] N=300 concurrency=10
[10/300] 3.3% eta ~2m
[50/300] 16.7% eta ~1m40s
...
[DONE] ok=287 fail=13 → errors-1748150400.json
Always report: total, success, fail, error file path. Kakak wants the number, not the story.