dior

Quantifies how sensitive a language model benchmark's reliability and ranking stability are to specific design choices, such as the selection of scenarios, subscenarios, examples, and few-shot prompts

SKILL.md

GitHub

dior

Efficient Benchmarking of Language Models — Perlitz et al. (2023) (arXiv:2308.11696, 2023)

What this evaluates

Quantifies how sensitive a language model benchmark's reliability and ranking stability are to specific design choices, such as the selection of scenarios, subscenarios, examples, and few-shot prompts.

Datasets

  • HELM — total ?; splits: test (-1)

Metrics

  • DIoR (primary) — range: [0, 1]
    • Decision Impact on Reliability measures the change in benchmark reliability (e.g., ranking stability or score variance) when a specific design choice is perturbed. It is computed via bootstrapping simulations that repeatedly resample the design choice and measure the probability of rank switches or score deviations across models.

Input / output format

Input: A benchmark's design configuration (e.g., set of scenarios, subscenarios, examples, or few-shot prompt seeds) and the resulting model scores/rankings across these configurations.

Output: A DIoR reliability score indicating the sensitivity of the benchmark's outcomes to the perturbed design choice, typically expressed as a probability or stability metric.

Scoring recipe

def compute_dior(design_config, model_scores, n_iterations=1000):
    reliability_scores = []
    for _ in range(n_iterations):
        perturbed_config = bootstrap_sample(design_config)
        new_scores = evaluate_models(perturbed_config)
        reliability_scores.append(calculate_rank_stability(model_scores, new_scores))
    return 1 - mean(reliability_scores)

Common pitfalls

  • Treating DIoR as a direct measure of model capability rather than benchmark design sensitivity.
  • Assuming that reducing the number of scenarios or subscenarios saves compute without penalty; the paper shows it drastically drops reliability.
  • Using full cross-product evaluation for few-shot prompts instead of uniform sampling across the example-prompt space, which the paper shows yields lower reliability for the same compute budget.

Evidence (verbatim from paper)

proposes Decision Impact on Reliability (DIoR) to quantify how design choices affect reliability.

Citation

@misc{perlitz2023efficientbenchmarking,
  title={Efficient Benchmarking of Language Models},
  author={Perlitz et al. (2023)},
  year={2023},
  note={arXiv:2308.11696}
}
  • arXiv: 2308.11696