m12

Superagent skill m12 — Batch Operations & Parallel Execution

SKILL.md

GitHub

m12 — Batch Operations & Parallel Execution

Load ONLY when: batch, parallel, mass, bulk, queue, multiple wallets, multiple accounts, many addresses, 100x, scrape, farming


When to batch

  • N > 5 independent ops with same shape
  • Each op is idempotent OR retryable
  • Total time at sequential exceeds patience (>30s)

When NOT to batch

  • Ops share mutable state (nonce, balance, file write)
  • Rate limits force serial
  • One failure should halt the rest

Pattern A — JS p-limit (recommended)

import pLimit from 'p-limit';

const limit = pLimit(5);  // 5 concurrent max
const tasks = wallets.map(w => limit(() => process(w)));
const results = await Promise.allSettled(tasks);

const ok = results.filter(r => r.status === 'fulfilled').map(r => r.value);
const fail = results.filter(r => r.status === 'rejected').map(r => r.reason);
console.log(`OK: ${ok.length}, FAIL: ${fail.length}`);

Promise.allSettled not Promise.all — one failure won't kill the batch.

Pattern B — Python asyncio.gather with semaphore

import asyncio

sem = asyncio.Semaphore(5)
async def bounded(coro):
    async with sem:
        return await coro

results = await asyncio.gather(
    *[bounded(process(w)) for w in wallets],
    return_exceptions=True
)
ok   = [r for r in results if not isinstance(r, Exception)]
fail = [r for r in results if isinstance(r, Exception)]

Pattern C — Worker pool (CPU-bound)

from concurrent.futures import ProcessPoolExecutor, as_completed

with ProcessPoolExecutor(max_workers=4) as pool:
    futures = {pool.submit(work, item): item for item in items}
    for fut in as_completed(futures):
        try:
            print(fut.result())
        except Exception as e:
            print(f"FAIL {futures[fut]}: {e}")

Rate-limit aware batching

Telegram: ~30 msg/sec global, 1/sec per chat. Use token bucket.

class TokenBucket {
  constructor(rate, capacity) {
    this.tokens = capacity;
    this.capacity = capacity;
    this.rate = rate;          // tokens/sec
    this.last = Date.now();
  }
  async take() {
    while (this.tokens < 1) {
      const elapsed = (Date.now() - this.last) / 1000;
      this.tokens = Math.min(this.capacity, this.tokens + elapsed * this.rate);
      this.last = Date.now();
      if (this.tokens < 1) await new Promise(r => setTimeout(r, 50));
    }
    this.tokens -= 1;
  }
}

const bucket = new TokenBucket(25, 25);  // 25/sec
for (const msg of messages) {
  await bucket.take();
  bot.sendMessage(msg.chat, msg.text).catch(e => console.error(e));
}

Web3 specific — parallel send WITHOUT nonce collision

ONE wallet → serial only (or pre-assign nonces). MANY wallets → fully parallel safe.

// Many wallets, parallel mint
const limit = pLimit(10);
const txs = wallets.map(w => limit(async () => {
  try {
    const tx = await contract.connect(w).mint();
    return await tx.wait();
  } catch (e) {
    return { error: e.message, wallet: w.address };
  }
}));
const out = await Promise.all(txs);

Progress tracking

let done = 0;
const total = items.length;
const interval = setInterval(() => {
  console.log(`[${done}/${total}] ${((done/total)*100).toFixed(1)}%`);
}, 5000);

const tasks = items.map(it => limit(async () => {
  try { return await work(it); }
  finally { done++; }
}));

await Promise.allSettled(tasks);
clearInterval(interval);

Python equivalent — use tqdm:

from tqdm.asyncio import tqdm_asyncio
results = await tqdm_asyncio.gather(*[bounded(work(i)) for i in items])

Error aggregation pattern

const errors = [];
const results = await Promise.allSettled(tasks);

for (const r of results) {
  if (r.status === 'rejected') {
    errors.push({ reason: r.reason.message || r.reason });
  }
}

if (errors.length) {
  fs.writeFileSync(
    `errors-${Date.now()}.json`,
    JSON.stringify(errors, null, 2)
  );
  console.log(`${errors.length} errors → errors-*.json`);
}

Resume from failure (CRITICAL for long batches)

Save state every N items so SIGINT doesn't lose hours of work.

import fs from 'fs';

const STATE = './batch-state.json';
let state = fs.existsSync(STATE)
  ? JSON.parse(fs.readFileSync(STATE))
  : { done: [], failed: [] };

const remaining = allItems.filter(x =>
  !state.done.includes(x.id) && !state.failed.includes(x.id)
);

const limit = pLimit(5);
for (const item of remaining) {
  limit(async () => {
    try {
      await work(item);
      state.done.push(item.id);
    } catch (e) {
      state.failed.push(item.id);
    } finally {
      fs.writeFileSync(STATE, JSON.stringify(state));  // checkpoint
    }
  });
}

For 100+ items, save every 10 — not every 1 — to avoid I/O bottleneck.


Chunked processing (memory-safe)

Process 1M-row file? Don't load it all.

import readline from 'readline';
import fs from 'fs';

const rl = readline.createInterface({
  input: fs.createReadStream('big.csv'),
  crlfDelay: Infinity
});

const BATCH = 1000;
let buffer = [];
for await (const line of rl) {
  buffer.push(line);
  if (buffer.length >= BATCH) {
    await processBatch(buffer);
    buffer = [];
  }
}
if (buffer.length) await processBatch(buffer);

Output format for batch jobs

[BATCH START] N=300 concurrency=10
[10/300] 3.3%   eta ~2m
[50/300] 16.7%  eta ~1m40s
...
[DONE] ok=287 fail=13 → errors-1748150400.json

Always report: total, success, fail, error file path. Kakak wants the number, not the story.