cv-learned-distance-metric
Trainable nonlinear distance metric that transforms (v1-v2) and (v1-v2)^2 through a linear layer before computing squared norm
SKILL.mdskills/cv/learned-distance-metric/SKILL.md
GitHubLearned Distance Metric
Overview
Euclidean and cosine distances treat all embedding dimensions equally. A learned distance metric passes the difference vector and its element-wise square through a linear transformation, then computes the squared norm. This approximates a Mahalanobis-like distance but with nonlinear capacity from the squared terms, learning which dimensions and interactions matter for similarity.
Quick Start
import torch
import torch.nn as nn
class LearnedMetric(nn.Module):
def __init__(self, emb_dim=64):
super().__init__()
self.linear = nn.Linear(emb_dim * 2, emb_dim * 2, bias=False)
def forward(self, v1, v2):
d = v1 - v2
d2 = d.pow(2)
x = self.linear(torch.cat([d, d2], dim=-1))
return x.pow(2).sum(dim=-1)
metric = LearnedMetric(emb_dim=128)
dist = metric(embedding_a, embedding_b)
Workflow
- Train embedding model with a fixed or learned metric
- Initialize
LearnedMetricmatching embedding dimension - Train jointly with contrastive or triplet loss
- At inference, use the learned metric for nearest-neighbor retrieval
Key Decisions
- No bias: bias-free linear keeps distance symmetric and zero for identical inputs
- Concat [d, d^2]: captures both linear (Mahalanobis-like) and nonlinear distance components
- Joint training: train metric alongside the embedding model end-to-end
- vs Mahalanobis: Mahalanobis is linear-only; this adds nonlinear capacity with minimal parameters