A Crowded Space

Response Rate is a Quality Signal

I’m sure everyone here has received several hundred of these emails: “How would you rate this book / freelancer / coffee maker /_etc_, on a scale of 1 through 5 stars?” 

If you know me, you know that I’ve ranted about how these questions don’t get to a true understanding of high quality.  They might help identify some bad actors, but they are terrible at identifying the highest quality in a marketplace. 

However, I recently learned that some marketplaces are using the quantitative score in addition to the response rate to get a more interesting signal of quality.  See this excellent paper by Chris Nosko and Steven Tadelis for a thorough academic perspective. (hat tip to Andrei Hagiu for pointing this out to me)

I was excited to learn about this – it makes a ton of sense.  The mental reaction when I get one of those “how would you rate xyz?” emails is to almost always immediately ignore it.  The only time I will take the time to respond is if I had a truly terrible or a truly wonderful experience.  

It’s no surprise that many marketplaces have a response rate to that feedback question in the single digit percentages.  I’d guess that a typical response rate is between 5-20% for email questions. Marketplaces get much higher response rates if the question is asked via SMS or in-app.  What I did not appreciate until recently is that the response rate will vary widely based on the quality of the good or service. 

Consider this scenario: 

“How would you rate Game of Thrones?”  – result: 4.9 stars, 500 ratings out of 1,000 possible responses.   

“How would you rate Lord of the Rings?” – result: 4.9 stars, 500 ratings out of 5,000 possible responses. 

Which one is better? 

Most platforms would probably say that they are the same.  However, I think it’s reasonable to say that Game of Thrones is much “higher quality” based on the response rate.  If 50% of people that watched thought it was worth spending a minute to rate the show, that is an incredibly strong indication that it’s a well-loved show.  It is creating that exceptionally wonderful experience that causes people to invest additional time to provide a high rating.  

On the other hand, a very low response rate is likely an indicator that there are actually a lot more silent people that would rate things somewhere in the middle, perhaps a 3 or 4 star rating, but didn’t think it was worth the time to enter a rating.  3 stars, just ok, not worth a single click.  

Consider this scenario:

4.9 stars, 100 ratings, 20% response rate

5.0 stars, 100 ratings, 10% response rate

This is where I think things really start to get interesting.  I could easily see the higher response rate being a more important indicator of quality than the actual star rating itself. 

As usual, these platforms are all special snowflakes and everyone will need to decide if and how to implement a measure of response rate into your overall quality and matching algorithms.  

I put together this trusty 2x2 to help think about the categories. 

Please reach out if you have any more experience using response rate for quality.  Would love to hear from you.