Inside the Bayesian Scoring Formulation for AI Models
A mathematical walkthrough of Superrank's Dirichlet prior Bayesian rating formulation, designed to prevent low-sample distortion on AI leaderboards.
In public AI evaluation directories, one of the most persistent statistical hazards is the small-sample bias. When a new foundation model launches with only three or four reviews, enthusiastic early adopters often assign perfect 10/10 scores. In an unweighted arithmetic mean system, that model instantly eclipses established frontier systems evaluated by hundreds of independent practitioners.
To resolve this structural vulnerability, Superrank implements Bayesian shrinkage derived from empirical Bayes theory. Rather than calculating raw arithmetic averages, every score displayed on the Superrank Leaderboard is governed by the formulation: WR = (v / (v + m)) * R + (m / (v + m)) * C.
In this formulation, v represents the total verified community votes, R represents the raw arithmetic mean of those votes, m represents the minimum confidence threshold fixed at 25 votes, and C represents the catalog-wide prior mean fixed at 7.0.
When an entity has zero or very few votes, the weighting heavily favors the catalog prior C. More importantly, under Superrank Rule 1, any model with fewer than 25 verified votes displays 'Not enough ratings yet', preventing unvetted entities from appearing at the summit of public leaderboards.
As empirical data accumulates and v surpasses 100 or 1,000 verified ratings, the ratio v / (v + m) approaches 1.0, and the Bayesian score organically converges toward the true empirical mean. This mathematical foundation ensures our rankings remain completely transparent, tamper-resistant, and free from artificial promotion.