cv-vqa-yes-no-logit-scoring
Extracts calibrated yes/no probabilities from a VQA model by masking all logits except yes/no token variants and renormalizing via softmax.
SKILL.mdskills/cv/vqa-yes-no-logit-scoring/SKILL.md
GitHubVQA Yes/No Logit Scoring
Overview
Visual Question Answering models can score image-text alignment by asking "Does this image match [description]? Answer yes or no." But taking argmax gives a binary answer, not a confidence score. This technique masks all vocabulary logits except the yes/no tokens (including space-prefixed variants like yes), applies softmax over just those 4 tokens, then sums the yes-variants to get a calibrated probability. This converts any VQA model into a continuous image-text scorer without fine-tuning.
Quick Start
import torch
class VQAScorer:
def __init__(self, model, processor):
self.model = model
self.processor = processor
tok = processor.tokenizer
self.yes_ids = [tok.convert_tokens_to_ids(t) for t in ['yes', ' yes']]
self.no_ids = [tok.convert_tokens_to_ids(t) for t in ['no', ' no']]
@torch.no_grad()
def score(self, image, question):
inputs = self.processor(images=image, text=question, return_tensors='pt').to('cuda')
logits = self.model(**inputs).logits[:, -1, :]
mask = torch.full_like(logits, float('-inf'))
for tid in self.yes_ids + self.no_ids:
mask[:, tid] = logits[:, tid]
probs = torch.softmax(mask, dim=-1)
p_yes = sum(probs[0, tid].item() for tid in self.yes_ids)
p_no = sum(probs[0, tid].item() for tid in self.no_ids)
return p_yes / (p_yes + p_no)
scorer = VQAScorer(model, processor)
score = scorer.score(image, f'Does this image show "{desc}"? Answer yes or no.')
Workflow
- Feed image + yes/no question to VQA model
- Extract logits from the last generated token position
- Mask all logits to -inf except yes/no token IDs (including space variants)
- Softmax over masked logits
- Sum yes-variant probabilities and normalize
Key Decisions
- Token variants: Include both
yesandyes(space-prefixed) — tokenizers handle them differently - Last token: Use logits at position -1 (the next-token prediction position)
- Question phrasing: "Answer yes or no" constrains the model to the target tokens
- Multi-criteria: Ask multiple questions (fidelity, text presence, style) and combine scores