EI-301b · Module 1

Scoring Methodology

3 min read

Scoring methodology determines how vendor attributes are converted into numeric scores. The two dominant approaches are evidence-based scoring (each score point requires specific evidence — benchmark results, SLA documents, audit reports) and assessment-based scoring (evaluators assign scores based on their professional judgment). Evidence-based scoring is more defensible but more time-consuming. Assessment-based scoring is faster but more susceptible to bias. The VANGUARD approach uses evidence-based scoring for the top three weighted criteria and assessment-based scoring for the remainder.

Do This

  • Require evidence for high-weight criteria — if reliability is weighted at 30%, the score must be backed by uptime data, not impressions
  • Use multiple evaluators and average scores — individual bias is reduced when three or more independent evaluators score the same vendor
  • Document the evidence behind every score — the scorecard is only as defensible as its evidence trail

Avoid This

  • Score based on vendor demos alone — demos are curated performances, not representative of production behavior
  • Allow a single evaluator to score all vendors — one person's preferences become the organization's decision
  • Use fractional scores (4.3, 3.7) — they imply precision that the methodology does not support. Use whole numbers.

Benchmark testing is the gold standard for technical capability and performance criteria. Run the vendor's product through your actual workload — not a synthetic benchmark, not the vendor's own benchmark, but your data, your queries, your latency requirements. The gap between vendor-reported performance and benchmark-tested performance is often significant. Organizations that skip independent benchmarking pay for the gap in production.