cv-sigmoid-normalized-rmse

Sigmoid-transformed normalized RMSE that maps error from [0,inf) to a bounded (0,1] similarity score using R2-score ratio

SKILL.md

GitHub

Sigmoid Normalized RMSE

Overview

Standard RMSE is unbounded and hard to interpret across different scales. Normalize RMSE by dividing by the baseline error (predicting the mean), then apply a sigmoid transform 2 - 2/(1+exp(-x)) to map the result to (0, 1]. Perfect predictions score 1.0, predictions at baseline quality score ~0.63, and worse-than-baseline predictions approach 0. This creates a bounded, interpretable metric suitable for averaging across heterogeneous data series.

Quick Start

import numpy as np
from sklearn.metrics import r2_score

def sigmoid(x):
    return 2 - 2 / (1 + np.exp(-x))

def sigmoid_normalized_rmse(y_true, y_pred):
    r2 = r2_score(y_true, y_pred)
    normalized_error = max(0, (1 - r2)) ** 0.5
    return sigmoid(normalized_error)

y_true = [10, 20, 30, 40, 50]
y_pred = [12, 18, 33, 38, 52]
score = sigmoid_normalized_rmse(y_true, y_pred)  # ~0.89

Workflow

  1. Compute R2 score between predictions and ground truth
  2. Convert to normalized error: sqrt(max(0, 1 - R2))
  3. Apply sigmoid transform: 2 - 2/(1 + exp(-error))
  4. Result is in (0, 1] — higher is better

Key Decisions

  • Why sigmoid: bounds the metric to (0, 1], making it averageable across different scales
  • R2-based normalization: divides by variance of true values, making it scale-invariant
  • sqrt(1-R2): equivalent to RMSE/std(y_true), the coefficient of variation of error
  • Use case: comparing prediction quality across series with different magnitudes

References