polos
Evaluates the alignment and quality of generated image captions relative to reference captions and source images. It measures how well a learned metric correlates with human judgments, specifically pr
SKILL.mdskill-factory/output/polos/SKILL.md
GitHubpolos
Polos: Multimodal Metric Learning from Human Feedback for Image Captioning — Wada et al. (2024) (arXiv:2402.18091, 2024)
What this evaluates
Evaluates the alignment and quality of generated image captions relative to reference captions and source images. It measures how well a learned metric correlates with human judgments, specifically probing hallucination robustness and open-vocabulary caption evaluation.
Datasets
- Polaris — total 131000; splits: full (-1)
Metrics
Polos(primary) — range: [0, 1]- A scalar score output by the Polos model, trained via the M²LHF framework to align with human feedback. It jointly models text-image and text-text relationships using parallel SimCSE and CLIP embeddings.
Input / output format
Input: Image, reference caption ($x_{\mathrm{ref}}^{(1)}$), and candidate caption ($x_{\mathrm{cand}}$).
Output: A single scalar score representing caption quality/alignment.
Scoring recipe
def compute_polos_score(image, ref_caption, cand_caption):
# Extract features using parallel encoders
img_feat = clip_encoder(image)
ref_feat = simcse_encoder(ref_caption)
cand_feat = simcse_encoder(cand_caption)
# Pass through trained M²LHF head to get alignment score
score = model_head(img_feat, ref_feat, cand_feat)
return score
Common pitfalls
- CLIPScore family metrics tend to overestimate scores, especially for longer captions.
- Reference-with-image metrics (e.g., RefCLIP-S, RefPAC-S) may fail to effectively compare references and candidates.
Evidence (verbatim from paper)
Fig.5 and Fig.6 provide additional comparisons between the CLIPScore family and Polos. We observed that the CLIP-Score family tends to overestimate scores. Specifically, RefCLIP-S and RefPAC-S may not effectively compare references and a candidate. Although CLIP-S could not show overestimation, this does not imply adequacy in caption evaluation.
Citation
@misc{wada2024polos,
title={Polos: Multimodal Metric Learning from Human Feedback for Image Captioning},
author={Wada et al. (2024)},
year={2024},
note={arXiv:2402.18091}
}
- arXiv: 2402.18091