Benchmark methodology

Someone finds a benchmark — a vendor PDF, a conference slide, an analyst "industry average" — and asks whether the operation is above or below it. The honest answer, most of the time, is: you cannot tell from the number alone, and presenting the comparison as if you can is how bad decisions get funded.

Benchmarks sell certainty. Operations run on definitions, scope, and survivorship. When those do not match, the gap between your metric and theirs measures incomparability, not performance.

This skill structures an honest comparison: what would need to be true for the benchmark to apply, what usually is not true, and when an internal baseline is the better reference.

The four mismatch classes

Before any "we are X% below industry" statement, check all four. Most external comparisons fail at least two.

Mismatch	What it means	Typical symptom
Scope	Different markets, channels, segments, or product complexity	Your email-heavy B2B queue compared to a vendor's voice retail average
Survivor bias	The benchmark population excludes failures	"Top quartile programmes" that dropped out; published CSAT from responders only while you report all surveys sent
Definition	Same label, different formula	Their FCR is same-day close; yours is no reopen in seven days
Selection	The benchmark is voluntary, paid, or self-reported	Customers who buy benchmarking software skew larger and more mature

Document which mismatches apply. If you cannot verify a match on definition and scope, do not put the comparison on a headline slide — at most, a footnote with caveats.

When external benchmarks are usable

External benchmarks are occasionally worth citing when all of the following hold:

1. Definition is published or obtainable — numerator, denominator, filters, date field, not just a label.
2. Scope matches yours — channel mix, customer type, and geography are close enough that you would expect the same structural drivers.
3. Sample method is known — who is in the panel, what response rate, what period, whether outliers were trimmed.
4. The comparison serves a decision — pricing an outsource, setting a plausible range for a new programme, sanity-checking an order-of-magnitude gap.

Even then, treat the benchmark as a band, not a target. Report your figure, their figure, the mismatches you could not resolve, and a statement of what you would need to believe for the gap to imply underperformance.

Never invent industry numbers. If the source is not in hand with a citation the user can verify, do not fill the gap with a plausible-sounding average.

When internal baselines beat external ones

Internal comparisons are often more decision-ready than external benchmarks:

Internal baseline	Use when
Your own prior period	Tracking improvement or regression with stable definitions
Your own best cohort	Same operation, same definitions — top team or top month as an achievable reference
Your own pre-change window	Before/after a policy, tooling, or staffing change
Matched segments	Same channel and queue over time, or A/B on a controlled rollout

Internal baselines fail when definitions drift or scope shifts — which is exactly why the metric registry matters. A stable internal series beats a mismatched external one every time.

Say this plainly when someone asks for an industry slide: "We can't match their definition; here is our trend against our own Q1 baseline instead."

How to run an honest comparison

1. Write your definition first — registry entry or equivalent, not the dashboard label.
2. Extract theirs — from methodology appendix, not the headline. If methodology is missing, stop; the benchmark is not usable for comparison.
3. Build a reconciliation table — row per dimension: scope, channel, denominator, date field, response handling, exclusions. Mark match / partial / unknown.
4. Quantify what you cannot reconcile — "Their CSAT excludes neutral; ours includes them — expect us to read lower by roughly the neutral share" only if you can compute that share from your data; otherwise say "direction unknown".
5. Choose the reference — external band with caveats, internal baseline, or explicit "not comparable".
6. State the decision the comparison enables — if no decision changes, the slide does not belong in the pack.

Board and vendor contexts

Board packs — external benchmarks belong in an appendix, never as one of the three headline numbers, unless you have verified definition and scope match and the board understands the residual uncertainty.

Vendor and outsource RFPs — vendors supply benchmarks optimised to win. Ask for methodology, raw sample description, and whether their "similar clients" include failed implementations. Compare proposed SLAs to your historical attainment on your definitions, not their brochure.

Goal-setting — "reach industry median" without a matched definition sets a target you may already exceed or can never reach. Prefer targets derived from your own distribution: median, top quartile of your teams, or improvement from a fixed baseline period.

Traps

• Single-number worship. "Industry average 82%" with no source, no year, no scope.
• Ranking on unadjusted numbers. Benchmark panels rarely match your channel mix.
• Response-rate blindness. Higher CSAT with half the response rate is not winning.
• Survivorship in "best practice" stories. Case studies omit the programmes that churned off the platform.
• Using a benchmark to avoid owning a definition. "We're fine, we're near average" when nobody agrees what the internal metric means.

Present results to the user

1. Your metric definition — formula, denominator, scope, as used in the comparison.
2. Their metric definition — quoted or summarised from source, with citation.
3. Reconciliation table — match / partial / unknown per dimension.
4. Verdict — comparable with stated uncertainty, not comparable, or comparable only for order-of-magnitude.
5. Recommended reference — external band, internal baseline, or both, with which decision each supports.
6. What not to claim — explicit list of conclusions the data does not support.