bleurt

Evaluates the correlation between automatic text generation scores and human quality ratings. Probes a model's ability to accurately rank or score generated translations and data-to-text outputs again

SKILL.md

GitHub

bleurt

BLEURT: Learning Robust Metrics for Text Generation — Sellam et al. (2020) (arXiv:2004.04696, 2020)

What this evaluates

Evaluates the correlation between automatic text generation scores and human quality ratings. Probes a model's ability to accurately rank or score generated translations and data-to-text outputs against human judgments, including robustness to domain/quality drift and few-shot adaptation.

Datasets

  • WMT Metrics Shared Task (2017-2019) — total ?; splits: train (-1), test (-1)
  • WebNLG 2017 — total 4677; splits: train (-1), test (-1)

Metrics

  • Kendall's Tau ($\tau$) (primary) — range: [-1, 1]
    • Measures the ordinal association between predicted scores and human ratings. Computed as the number of concordant pairs minus discordant pairs, divided by total pairs.
  • Pearson correlation ($r$) / Direct Assessment (DA) — range: [-1, 1]
    • Linear correlation coefficient between predicted and human scores. DA is a scaled version of human assessment scores used as the official WMT metric.

Input / output format

Input: Source sentence and generated target sentence (or multiple reference sentences), paired with human quality ratings.

Output: A scalar quality score for the generated text.

Scoring recipe

def compute_agreement(predictions, human_ratings):
    # predictions and human_ratings are lists of floats of equal length
    tau = kendalltau(predictions, human_ratings).correlation
    pearson = pearsonr(predictions, human_ratings).correlation
    return tau, pearson

Common pitfalls

  • When multiple references are available, compute the metric for each reference and report the maximum score.
  • Training and test splits vary by year; for WMT, the test set from the previous year is used for training/validation.
  • Quality drift experiments use a skew factor α that heavily biases training data toward low ratings and test data toward high ratings.

Evidence (verbatim from paper)

We evaluate the agreement between the automatic metrics and the human ratings. For each year, we report two metrics: Kendall's Tau $\tau$ (for consistency across experiments), and the official WMT metric for that year (for completeness). The official WMT metric is either Pearson's correlation or a robust variant of Kendall's Tau called DARR, described in the Appendix.

Citation

@misc{sellam2020bleurt,
  title={BLEURT: Learning Robust Metrics for Text Generation},
  author={Sellam et al. (2020)},
  year={2020},
  note={arXiv:2004.04696}
}
  • arXiv: 2004.04696