evo-parallel-determinism
Provides patterns and utilities to ensure parallel TF-IDF results are identical to the sequential baseline, addressing floating-point ordering, tie-breaking, and pickling compatibility.
SKILL.mdtasks-evolved/parallel-tfidf-search/environment/skills/evo-parallel-determinism/SKILL.md
GitHubParallel Determinism
Overview
Ensures parallel results match sequential exactly by handling:
- Floating-point accumulation order
- Tie-breaking in result sorting
- Pickling compatibility for multiprocessing
- Safe context selection (fork vs spawn)
Key Principles
- FP Ordering: Iterate query terms in same order as sequential (dict insertion order)
- Tie-breaking: Sort by (-score, doc_id) for deterministic ordering
- Pickling: Define NamedTuples/dataclasses at module level
- Context: Use 'fork' on Linux for performance (copy-on-write)
Usage
import sys
sys.path.insert(0, '/app/environment/skills/evo-parallel-determinism/scripts')
from utils import (
deterministic_score_accumulation,
stable_sort_results,
get_safe_mp_context
)
Functions
deterministic_score_accumulation(query_vector, doc_vector)- FP-safe dot productstable_sort_results(results)- Sort with deterministic tie-breakingget_safe_mp_context()- Get appropriate multiprocessing context