MENLI
Evaluates the robustness and alignment with human judgment of reference-based and reference-free evaluation metrics for machine translation and summarization, particularly under adversarial conditions
SKILL.mdskill-factory/output/MENLI/SKILL.md
GitHubMENLI
MENLI: Robust Evaluation Metrics from Natural Language Inference — Chen et al. (2022) (arXiv:2208.07316, 2022)
What this evaluates
Evaluates the robustness and alignment with human judgment of reference-based and reference-free evaluation metrics for machine translation and summarization, particularly under adversarial conditions.
Datasets
- WMT15-17 / WMT20-21 — total ?; splits: test (-1)
- RealSum — total ?; splits: test (-1)
- SummEval — total ?; splits: test (-1)
- PAWS / XPAWS / SEadv / Rank19 — total ?; splits: adversarial (-1)
Metrics
Pearson correlation(primary) — range: [-1, 1]- Measures the linear correlation between metric scores and human judgment scores (e.g., DA, MQM, or LitePyramids ratings) at segment or system level.
Kendall correlation— range: [-1, 1]- Measures ordinal association between metric scores and human judgments on coherence, consistency, fluency, and relevance criteria.
Accuracy— range: [0, 1]- Relative frequency that a metric correctly ranks the preferred candidate higher than the incorrect candidate in adversarial pairwise comparisons.
Input / output format
Input: Source text (or reference) and candidate translation/summary. For adversarial tasks, pairs of candidates (one correct, one incorrect) are provided.
Output: A continuous scalar score indicating quality/relevance, or a binary preference decision for adversarial pairs.
Scoring recipe
def compute_menli(premise, hypothesis, nli_model, formula, direction):
e, c, n = nli_model.predict(premise, hypothesis)
score = apply_formula(e, c, n, formula)
if direction == 'bidirectional':
score = (score_fwd + score_bwd) / 2
return score
def evaluate_against_human(menli_scores, human_ratings):
return pearsonr(menli_scores, human_ratings)
def adversarial_accuracy(correct_scores, incorrect_scores):
return mean([c > i for c, i in zip(correct_scores, incorrect_scores)])
Common pitfalls
- Directionality matters: ref/src → cand vs ref/src ← cand vs bidirectional averaging yield different rankings; ref-free summarization only uses src → cand.
- Multi-reference aggregation: using max vs mean aggregation for reference summaries significantly changes correlation values on SummEval.
- NLI probability normalization: e, c, n must sum to 1; different fine-tuning datasets (SNLI, MNLI, ANLI, XNLI) drastically affect robustness.
Evidence (verbatim from paper)
In general, we check the metrics' preference between the two candidates and calculate accuracy: the relative frequency that the metrics correctly choose among the two alternatives. On MT standard benchmarks, we evaluate the metrics on both segment-level (where we correlate metrics scores to human judgements for individual sentences/segments in the datasets) and system-level (where we correlate the average metric scores to the average human scores over the segments generated by each system), using Pearson correlation as the performance indicator.
Citation
@misc{chen2022menli,
title={MENLI: Robust Evaluation Metrics from Natural Language Inference},
author={Chen et al. (2022)},
year={2022},
note={arXiv:2208.07316}
}
- arXiv: 2208.07316