The Delusion of Customer Testimonials
I’m in the fortunate position of being able to see a lot of pitches from a lot of different startups. One of the things these pitches have in common is testimonials from lots of happy customers. It’s not very surprising given the point of the pitch deck.
But, I believe the testimonials can be leading to a larger form of delusion in startups.
While helpful for the pitch deck, it is important for entrepreneurs and investors to avoid falling victim to testimonial delusion. Testimonial delusion is when we extrapolate a few good comments to mean all is good. As investors, we run the risk of being swayed by a false sense of social proof. For founders, the overwhelmingly positive testimonials could lead to a false illusion of market fit.
Here are the issues as I see them for typical startups:
Conversion Selection Bias
A lot of people come to your site and never convert. The fact that someone converted already means they’re positively inclined towards your product. Something about your site and messaging really resonated with the visitor. It’s the other 90%+ of visitors that didn’t convert that you need to know about.
User Selection Bias
If you have 100 users and need 3 testimonials, who are you going to ask? Most founders ask their most engaged users. SImilarly, if you were to send out a survey to all of your users, you’ll get sample bias as the most engaged and happy customers will happily spend time to fill in a survey but your “passive” (see NPS) customers will likely just ignore the survey. This further leads you to be talking with only the most enthusiastic fans.
Misaligned Incentives
The user submitting a testimonial has some incentive to say very positive things and hardly any incentive not to. It’s all upside for the testimonial giver. One popular SEO tactic is to give testimonials to every vendor you work with in order to get a backlink and go up the search rankings. Additionally, it just feels good emotionally to compliment someone and in return receive a high degree of gratitude.
Rating Bias
I’ve talked about this before with 5-star feedback systems. Basically, there is some simple rating bias that humans inherently have – we all tend to be nice people. You’ll see that the vast majority of feedback systems do not operate on a bell curve. They are A-centered or 5-star centered. This is in part because humans are terrible at assessing the absolute value of things. If you ask people to rate things on a scale of 1-5, you end up with lots of 4s and 5s. If you ask people, “Is A better than B?” you’ll get a much clearer picture.
——————
So, I believe testimonials are great and are sometimes useful, but I believe we all need to exercise caution when using them to make any decisions.
A few things we can all do a little better.
Focus on the Quantitative
It’s very tempting to use anecdotal testimonials and punchy quotations. A few of those 140-character ones fit perfectly on slides. However, we should all really be looking more at quantitative metrics and objective behavioral criteria. I see people report sometimes on surveys they ask customers, “How much would you be willing to pay for xxx?” We all know that the reality of what people pay and what they say they will pay are two very different numbers.
Random Customer Sampling
When evaluating the real picture, first start by talking to a random sampling of customers. Get a list of customers and choose at random to call the user directly and get their input.
Random Prospect Sampling
To address the customer conversion bias, we can get a list of prospects and try to contact them just to get input and feedback on the product. Can always do this through things like UserTesting, Mechanical Turk, or FeedbackArmy as well in a quick and dirty fashion. Or use SurveyMonkey’s Audience feature. Make sure to run some baseline tests on competitive websites or other similar services so you can compare your site to the competition. It’s so cheap and easy to do.
Allow for Anonymity
People are much more willing to be honest (and harshly critical) when they have anonymity. Just look at Secret vs Facebook. Give users the chance to be anonymous with you - see if you get different input.
Survey Completion Percentage
A focus on getting the highest participation rates in surveys will help eliminate sampling bias. I’m personally a big fan of SMS-based single question surveys. The completion rate for these can be north of 80%. I think this is one of the most underutilized tactics out there.
Use Standard Survey Design
I’m a believer in Net Promoter Score. It’s easy to compare and benchmark your company against the industry. On the other hand, I still see tons of companies using custom surveys and questions and varying collection methods. The more you standardize your survey, the more useful it will be to you to benchmark against your industry. Check out Delighted App for an easy way to launch your own.
Any other good ideas? Let’s hear ‘em.