MENLI

Evaluates the robustness and alignment with human judgment of reference-based and reference-free evaluation metrics for machine translation and summarization, particularly under adversarial conditions

SKILL.md

GitHub

MENLI

MENLI: Robust Evaluation Metrics from Natural Language Inference — Chen et al. (2022) (arXiv:2208.07316, 2022)

What this evaluates

Evaluates the robustness and alignment with human judgment of reference-based and reference-free evaluation metrics for machine translation and summarization, particularly under adversarial conditions.

Datasets

  • WMT15-17 / WMT20-21 — total ?; splits: test (-1)
  • RealSum — total ?; splits: test (-1)
  • SummEval — total ?; splits: test (-1)
  • PAWS / XPAWS / SEadv / Rank19 — total ?; splits: adversarial (-1)

Metrics

  • Pearson correlation (primary) — range: [-1, 1]
    • Measures the linear correlation between metric scores and human judgment scores (e.g., DA, MQM, or LitePyramids ratings) at segment or system level.
  • Kendall correlation — range: [-1, 1]
    • Measures ordinal association between metric scores and human judgments on coherence, consistency, fluency, and relevance criteria.
  • Accuracy — range: [0, 1]
    • Relative frequency that a metric correctly ranks the preferred candidate higher than the incorrect candidate in adversarial pairwise comparisons.

Input / output format

Input: Source text (or reference) and candidate translation/summary. For adversarial tasks, pairs of candidates (one correct, one incorrect) are provided.

Output: A continuous scalar score indicating quality/relevance, or a binary preference decision for adversarial pairs.

Scoring recipe

def compute_menli(premise, hypothesis, nli_model, formula, direction):
    e, c, n = nli_model.predict(premise, hypothesis)
    score = apply_formula(e, c, n, formula)
    if direction == 'bidirectional':
        score = (score_fwd + score_bwd) / 2
    return score

def evaluate_against_human(menli_scores, human_ratings):
    return pearsonr(menli_scores, human_ratings)

def adversarial_accuracy(correct_scores, incorrect_scores):
    return mean([c > i for c, i in zip(correct_scores, incorrect_scores)])

Common pitfalls

  • Directionality matters: ref/src → cand vs ref/src ← cand vs bidirectional averaging yield different rankings; ref-free summarization only uses src → cand.
  • Multi-reference aggregation: using max vs mean aggregation for reference summaries significantly changes correlation values on SummEval.
  • NLI probability normalization: e, c, n must sum to 1; different fine-tuning datasets (SNLI, MNLI, ANLI, XNLI) drastically affect robustness.

Evidence (verbatim from paper)

In general, we check the metrics' preference between the two candidates and calculate accuracy: the relative frequency that the metrics correctly choose among the two alternatives. On MT standard benchmarks, we evaluate the metrics on both segment-level (where we correlate metrics scores to human judgements for individual sentences/segments in the datasets) and system-level (where we correlate the average metric scores to the average human scores over the segments generated by each system), using Pearson correlation as the performance indicator.

Citation

@misc{chen2022menli,
  title={MENLI: Robust Evaluation Metrics from Natural Language Inference},
  author={Chen et al. (2022)},
  year={2022},
  note={arXiv:2208.07316}
}
  • arXiv: 2208.07316