EI-201b · Module 2

Credibility Scoring Systems

3 min read

Not all sources are equally reliable, and reliability changes over time. A credibility scoring system assigns each source a numeric reliability rating based on historical accuracy, then uses that rating to weight signals during analysis. The VANGUARD credibility model uses a 1-5 scale: 5 = verified accuracy above 95% over 12+ months (e.g., official vendor changelogs), 4 = consistently accurate with occasional errors (e.g., well-established analyst firms), 3 = generally reliable but requires corroboration (e.g., industry newsletters), 2 = mixed track record, useful for early signals but never trusted alone (e.g., social media), 1 = unverified or known to be unreliable, tracked for coverage but never cited without corroboration.

Do This

  • Score every source when you add it to your network — start at 3 and adjust based on observed accuracy
  • Re-score sources quarterly based on verified accuracy over the trailing 12 months
  • Use credibility scores to weight signals — a score-5 source saying something happened is near-certain; a score-2 source requires corroboration

Avoid This

  • Trust a source because it is well-known — reputation and accuracy are not the same thing
  • Assign scores based on intuition — track actual accuracy rates against predictions
  • Drop low-credibility sources entirely — they may catch early signals that reliable sources report later

Credibility scores should be transparent to your briefing consumers. When you cite a signal, include the source credibility: "Per [Source Name] (credibility: 4/5), Vendor X has depreciated API v2 effective June 30." This transparency lets decision-makers apply their own judgment to the reliability of the intelligence. It also creates accountability — if a score-4 source is consistently wrong, your readers will notice before you do.