posh

Evaluates how well automated metrics and vision-language models can identify granular errors (attribute/relation misattachments) in detailed image descriptions and correctly rank paired descriptions a

SKILL.md

GitHub

posh

PoSh: Using Scene Graphs To Guide LLMs-as-a-Judge For Detailed Image Descriptions — Amith Ananthram et al. (2025) (arXiv:2510.19060, 2025)

What this evaluates

Evaluates how well automated metrics and vision-language models can identify granular errors (attribute/relation misattachments) in detailed image descriptions and correctly rank paired descriptions against human judgments.

Datasets

Metrics

  • macro F1 (primary) — range: [0, 1]
    • Token-level F1 score for identifying annotated mistake and omission spans in candidate descriptions. Credits or penalizes predictions per token.
  • pairwise accuracy — range: [0, 1]
    • Percentage of pairs where the metric correctly predicts the better text or a tie. Uses a tie threshold inferred from the gold tie rate.
  • Spearman rank ρ (primary) — range: [-1, 1]
    • Rank correlation between the metric's score difference (m(text1)−m(text2)) and the human gold score difference.
  • Kendall's τ — range: [-1, 1]
    • Rank correlation measuring the concordance between metric predictions and human gold judgments on paired descriptions.

Input / output format

Input: Image, candidate description text, gold reference description text, and scene graph components extracted from the text.

Output: Per scene graph component: a presence score (1-5) derived from weighted token logits. Per pair: a preference label (better/tie) or rank correlation value. Per span: a binary/mask score for mistakes/omissions.

Scoring recipe

def score_posh(image, candidate, reference):
    # 1. Extract sentence-level scene graphs & resolve co-references
    graphs = extract_scene_graphs(candidate, reference)
    # 2. Prompt LLM (qwen-3-14b) for templated presence questions
    logits = prompt_llm(graphs, candidate, reference)
    # 3. Compute presence scores via weighted avg over token logits for 1-5
    scores = weighted_avg_token_logits(logits, range=[1,5])
    # 4. Apply threshold of 2 for object presence
    presence = scores >= 2.0
    # 5. Compute granular macro F1 on token spans
    f1 = compute_macro_f1(predicted_spans=presence, gold_spans=annotated_mistakes)
    # 6. Compute coarse metrics for pairs
    acc = pairwise_accuracy(metric_diff, gold_diff, tie_thresh=gold_tie_rate)
    rho = spearman_corr(metric_diff, gold_diff)
    tau = kendall_tau(metric_diff, gold_diff)
    return f1, acc, rho, tau

Common pitfalls

  • The tie threshold for pairwise accuracy is not fixed; it must be inferred from the gold tie rate in the dataset.
  • Granular F1 operates at the token level, so results are highly sensitive to the tokenizer used for the candidate and reference texts.
  • CapArena contains significantly simpler images than DOCENT; high performance on CapArena does not guarantee robustness on complex, dynamic scenes.

Evidence (verbatim from paper)

We evaluate granular metrics on this benchmark using macro F1 where we credit/penalize a model for predicting each annotated/unannotated token. Our coarse judgments are in the form (text1, text2, score) where score indicates how much better or worse text1 is than text2. We evaluate each coarse metric m by calculating its 1) pairwise accuracy (whether it picks the better text or a tie, using a tie threshold inferred from the gold tie rate) and 2) Spearman rank ρ and Kendall's τ correlations between m(text1)−m(text2) and score, a common practice in machine translation metric evaluation (Kocmi et al., 2021).

Citation

@misc{ananthram2025posh,
  title={PoSh: Using Scene Graphs To Guide LLMs-as-a-Judge For Detailed Image Descriptions},
  author={Amith Ananthram et al. (2025)},
  year={2025},
  note={arXiv:2510.19060}
}
  • arXiv: 2510.19060