Selecting Evaluation Criteria

Vendor evaluation criteria must be selected before you start evaluating — not discovered during the evaluation. Criteria selection bias is the most common scorecard failure: organizations choose criteria that favor the vendor they already prefer, then claim the evaluation was objective. The VANGUARD scorecard methodology uses six standard criteria categories for AI vendor evaluation: technical capability (does the product do what we need?), reliability and performance (does it do it consistently?), pricing and total cost of ownership (what does it actually cost, including hidden costs?), vendor viability (will this vendor exist and support this product in 3 years?), integration and ecosystem fit (does it work with our existing stack?), and compliance and security (does it meet our regulatory and security requirements?).

1. Start with Requirements, Not Products List what your organization needs from the vendor category before looking at any specific vendor. This prevents criteria selection bias. If you need sub-100ms inference latency, that criterion exists before you know which vendor meets it. If you need SOC 2 Type II compliance, that criterion exists independent of any vendor's certification status.
2. Weight by Business Impact Not all criteria are equally important. Assign weights that sum to 100%. A mission-critical AI inference API might weight reliability at 30%, technical capability at 25%, pricing at 20%, vendor viability at 10%, integration at 10%, and compliance at 5%. The weights must be set before scoring — changing weights after seeing vendor scores is reverse-engineering a desired outcome.
3. Define Scoring Scales For each criterion, define what a 1, 3, and 5 look like on a 5-point scale. "Reliability: 1 = <95% uptime, 3 = 99-99.9% uptime, 5 = >99.99% uptime with published SLA." Defined scales prevent subjective scoring drift. Two evaluators scoring the same vendor should arrive at the same score if the scales are well-defined.