lambre

Evaluates the morphosyntactic well-formedness and grammaticality of generated natural language text. It measures how closely a generated sentence or corpus adheres to language-specific dependency rule

SKILL.md

GitHub

lambre

Evaluating the Morphosyntactic Well-formedness of Generated Texts — Pratapa et al. (2021) (arXiv:2103.16590, 2021)

What this evaluates

Evaluates the morphosyntactic well-formedness and grammaticality of generated natural language text. It measures how closely a generated sentence or corpus adheres to language-specific dependency rules extracted from treebanks.

Datasets

  • (no dataset; pure metric skill)

Metrics

  • L'AMBRE (primary) — range: [0, 1]
    • Segment-level: weighted average of the percentage of applicable morphosyntactic rules satisfied within a segment. Corpus-level: macro-average of the empirical satisfiability counts for each applicable rule across the entire corpus.

Input / output format

Input: A text segment or corpus, accompanied by a set of language-specific morphosyntactic rules and a dependency parser/tagger.

Output: A single floating-point score between 0 and 1.

Scoring recipe

def compute_lambre(text, rules, parser, corpus_level=False):
    tree = parser.parse(text)
    applicable = [r for r in rules if is_applicable(r, tree)]
    satisfactions = []
    for r in applicable:
        sat = count_satisfied(r, tree)
        total = count_applicable_instances(r, tree)
        satisfactions.append(sat / total if total > 0 else 0)
    if corpus_level:
        return mean(satisfactions)
    return weighted_average(satisfactions)

Common pitfalls

  • Requires a robust dependency parser capable of handling noisy or ungrammatical generated text, as parsing errors directly lower rule satisfaction counts.
  • Segment-level and corpus-level scores use different aggregation methods (weighted average vs. macro-average of empirical counts), making them non-interchangeable.
  • Rule coverage and accuracy depend heavily on the quality of the automatically extracted morphosyntactic rules from the source treebank.

Evidence (verbatim from paper)

In this section, we present L'AMBRE, a metric to gauge the morphosyntactic well-formedness of generated natural language sentences. ... The final score is a weighted average of the scores of individual rules. Our score lies between [0,1], where 1 and 0 represent that rules are perfectly satisfied or not satisfied at all respectively.

Citation

@misc{pratapa2021lambre,
  title={Evaluating the Morphosyntactic Well-formedness of Generated Texts},
  author={Pratapa et al. (2021)},
  year={2021},
  note={arXiv:2103.16590}
}
  • arXiv: 2103.16590