How not to sort by average rating

How not to sort by average rating

By Evan Miller

February 6, 2009 (Changes)

Problem: You are a web programmer. you have users. your users rate stuff on your site. you want to put the highest-rated stuff at the top and lowest-rated at the bottom. you need some sort of "score" to sort.

Wrong solution #1: Score = (positive ratings)-(negative ratings)

Why it is wrong: Suppose one item has 600 positive ratings and 400 negative ratings: 60% positive. suppose item two has 5,500 positive ratings and 4,500 negative ratings: 55% positive. this algorithm puts item two (score = 1000, but only 55% positive) abve item one (score = 200, and 60% positive ). wrong.

Sites that make this mistake: Urban Dictionary

Wrong solution #2: Score = average rating = (positive ratings)/(total ratings)

Why it is wrong: Average rating works fine if you always have a ton of ratings, but suppose Item 1 has 2 positive ratings and 0 negative ratings. suppose Item 2 has 100 positive ratings and 1 negative rating. this algorithm puts item two (tons of positive ratings) below item one (very few positive ratings ). wrong.

Sites that make this mistake: Amazon.com

Correct Solution: Score = lower bound of Wilson score confidence interval for a Bernoulli Parameter

say what : we need to balance the proportion of positive ratings with the uncertainty of a small number of observations. fortunately, the math for T His was worked out in 1927 by Edwin B. wilson. what we want to ask is: given the ratings I have, there is a 95% chance that the "real" fraction of positive ratings is at least what? Wilson gives the answer. considering only positive and negative ratings (I. e. not a 5-star scale), the lower bound on the proportion of positive ratings is given by:

(for a lower bound use minus where it says plus/minus .) here P is the observed fraction of positive ratings, Z α/2 is the (1-α/2) quantile of the standard normal distribution, and n is the total number of ratings. the same formula implemented in Ruby:

Require 'statistics2 'def ci_lower_bound (Pos, N, confidence) if n = 0 return 0 end z = statistics2.pnormaldist (1-(1-confidence)/2) phat = 1.0 * POS/N (phat + z * z/(2 * n)-z * Math. SQRT (phat * (1-phat) + z * z/(4 * n)/(1 + z * z/n) End
 

Pos   Is the number of positive ratings,   N   Is the total number of ratings, and   Confidence   Refers to the statistical confidence level: Pick 0.95 to have a 95% chance that your lower bound is correct, 0.975 to have a 97.5% chance, etc. the Z-score in this function never changes, so if you don't have a statistics package handy or if performance is an issue you can always hard-code a value here   Z . (Use 1.96 for a confidence level of 0.95 .)

Now for any item that has a bunch of positive and negative ratings, use that function to arrive at a score appropriate for sorting on, and be confident that you are using a good algorithm for doing so.

Sites that are enlightened: Reddit, Yelp, Digg

Other applications

the Wilson score confidence interval isn' t just for sorting, of course. it is useful whenever you want to know with confidence what percentage of people took some sort of action. for example, it cocould be used to:

• detect spam/abuse: what percentage of people who see this item will mark it as spam?
• Create a "best of" list: what percentage of people who see this item will mark it as "best "?
• Create a "most emailed" list: what percentage of people who see this page will click "email "?

Indeed, it may be more useful in a "top rated" list to display those items with the highest number of positive ratings Per page view, download, or purchase, Rather than positive ratings per rating. please people who find something mediore will not bother to rate it at all; the act of viewing or purchasing something and declining to rate it useins useful information about that item's quality.

Changes

• Nov. 13,201 1: Fixed statistical confidence language and altered code example accordingly
• Feb. 15: clarified the statistical power example
• Feb. 13 II: "Other Applications"
• Feb. 13: general clarification, plus a link to the relevant Wikipedia article.
• Feb. 12,200 9: the example in "wrong solution #1" was erroneous. It has been fixed.

References

Binomial proportion confidence interval (Wikipedia)

Agresti, Alan and Brent A. Coull (1998), "approximate is better than 'extract' for Interval Estimation of Binomial proportions ," The American Statistician, 52,119-126.

Wilson, E. B. (1927), "Probable Inference, the law of succession, and statistical inference ," Journal of the American Statistical Association, 22,209-212.

back to Evan Miller's home page