Experimentation / CRO

Experimentation and Testing Programs acknowledge that the future is uncertain. These programs focus on getting better data to product and marketing teams to make better decisions.

Research & Strategy

We believe that research is an integral part of experimentation. Our research projects aim to identify optimization opportunities by uncovering what really matters to your website users and customers.

Data and Analytics

90% of the analytics setups we’ve seen are critically flawed. Our data analytics audit services give you the confidence to make better decisions with data you can trust.

Ecommerce UX Battle: Benchmarking 4 Beauty & Cosmetics Mobile Websites

Which popular beauty and cosmetics website has the best user experience?

This is a conversion-focused benchmark analysis of four competing beauty and cosmetics websites:

We selected sites from among the top 25 in their category, according to Alexa rankings. Our analyses measure overall user experience of each site as a composite of 5 separate UX dimensions.

This post details our process, the results, and the takeaways for businesses in this and similar verticals.

Table of contents

What is competitive UX benchmarking and why does it matter?

Generally speaking, benchmarking is a way to discover the best performance one can achieve. Benchmarking is widely used to test and compare organizations and products in a particular industry.

Competitive UX benchmarking tests aspects of a website and compares them to competitors. The testing is done manually, by real people, who are specially recruited for this purpose.

If you’re optimizing a website to improve user experience, affect product or brand positioning, or generally to change user behavior, Competitive UX benchmarking helps you to understand how your website is perceived among the sea of options available to consumers.

5 dimensions of UX benchmarking

  1. Appearance. How does the site’s look-and-feel compare to its competitors?
  2. Clarity. How do users perceive the value and benefits compared to its competitors?
  3. Credibility. Do users trust the site more or less than those of competitors?
  4. Loyalty. Are users more likely to return to a site or to its competitors?
  5. Usability. Do users think the site is more or less usable?

5 dimensions of UX benchmarking

When your revenue stream depends on your website, you need to be able to answer two questions:

  • Is your UX putting you behind your competition? If so, where?
  • How and where should you prioritize your website testing?

UX benchmarking helps to identify the areas where the user experience could be improved:

  • Is the message clear enough?
  • How do users perceive clarity compared to general usability?
  • Are users able to find what they are looking for?
  • Is credibility an issue?

(Speero Agency regularly performs these analyses for clients, which help prioritize issues and develop testing hypotheses.)

SHOW ME THE RESULTS

Our UX Benchmarking Method

The CXL benchmarking methodology has been developed during a series of studies conducted by our CXL Institute UX Research Team in collaboration with the Speero Agency.

We first teamed up with Jeff Sauro and his team at MeasuringU to benchmark 5 bicycle websites. We later modified the methods to include conversion-focused metrics to benchmark 5 nutrition websites, plus many many more as the basis for our ecommerce best practice guidelines.

This study is in the same vein of work published by TryMyUI and Userzoom. The CXL methodology is described in more detail on our Competitive UX benchmarking page.

Competitive UX benchmarking process

CXL benchmarking uses a standardized process to calculate a conversion-focused, validated, and standardized survey metric.

We ask participants to perform concrete tasks on a target website and collect feedback about their experience immediately after task completion.

Tests are remote and unmoderated; participants perform tasks on their own mobile device. Each person is asked to test all the websites under comparison (within-subjects study).

For this study, we recruited and surveyed 108 people (55 men, 53 women).

The participants were asked to browse the test website as if they were shopping for, in this case, beauty items.

Testing the UX

Here is one of the tasks the participants were asked to perform:

  1. Find a lipstick for $25 or less.
  2. Once found, compare it to similar ones and choose the one you would like to buy. Add it to the shopping cart.
  3. Imagine you want to buy something as a gift for your friend. Find an item you think they would like and add it to the shopping cart.
  4. Go to your cart and complete the purchase.
    Credit Card#: 1111 2222 3333 4444 CVV: 111 Expiration Date: 06/01/2017.
  5. Your task will end when you see an error message after submitting the proxy credit card information.

The task completion was confirmed via URL validation.

After testing, we administered a survey to each participant.

Collecting user feedback

In our standard benchmarking process, the survey is composed of statements and questions covering the following aspects:

  1. Appearance
  2. I found the website to be attractive.
  3. The website has a clean and simple presentation.
  4. Clarity (our additional question, not part of SUPR-Q)
  5. I clearly understand why I should buy from this website instead of its competitors.
  6. Credibility (Trust)
  7. I feel comfortable purchasing from this website.
  8. I feel confident conducting business with this website.
  9. Loyalty
  10. How likely are you to recommend this website to a friend or colleague?
  11. I will likely visit this website in the future.
  12. Usability
  13. The website is easy to use.
  14. It is easy to navigate within the website.

Participants evaluated each statement or question on a 10-point Likert scale.

Additionally, the survey contained two open-ended questions:

  • Was there anything that frustrated you about your experience on the website you performed the task on?
  • What did you like about the website experience?

From the open-ended questions, we extracted specific feedback to formulate hypotheses on how to improve the websites.

Why we chose this process

  • Conversion-focused: Developed with a UX dimension quantifying user’s perception of a website’s value proposition, or why they should consider a site compared to its competition;
  • Quantitative: based on 100+ user data points for each site;
  • Generalizable and transferable: applicable to any website—ideal for relative context and understanding how scores relate to each other when measuring before-and-after design changes, or comparisons to competitors;
  • Multidimensional: includes the main factors for measuring website user experience and the general quality of a website;
  • Standardized, normalized, and validated: developed through extensive testing on a massive user-testing database (see the peer-reviewed paper on its foundation);
  • Repeatable: ideal for quantifying a baseline for comparison against later design changes.

Understanding the data

With the data from the survey questions across participants, we calculate percentile ranking for each website on each UX dimension subcomponent and for a global metric.

Note: Percentile rankings are not only relative to each other, but also to the 84 sites currently in our database. The figures below represent sample visualizations of the data.

The battle results

1. Appearance

How does the site’s look and feel compare to its competitors?

The statements we used to evaluate Appearance were:

  • I found this website attractive.
  • The website had a clean and simple presentation.

Takeaways

All sites scored above average compared to other sites in our database. This suggests a higher barrier-to-entry or increased expectations among users in this space.

Sub-dimensions and insights

Based on the evaluative statements for each category, we can infer sub-dimensions in the form of questions. Qualitative feedback offers common descriptions that help answer each:

1.1. What makes a beauty or cosmetics website attractive?

Bold, high-contrast colors and large, high-resolution product images:

I loved their overlay and their packaging. The items looked gorgeous. (Sephora)
The website was easy to navigate and visually appealing. Looked high-end and was presented nicely. (Fresh)

1.2. What makes a beauty or cosmetics website unattractive?

Cluttered, noisy, distracting design:

Too many adds, pop-ups are terrible, too much marketing while I was trying to search for something else. (Lush)
It seemed cluttered. (Lush)
Wasn’t as responsive and didn’t seem as professional and polished. (Lush)

1.3. What makes a beauty or cosmetics website clean and simple?

Relevant filters, intuitive categories:

There’s a lot more selection and you can filter the price range and sort by price in a meaningful way. (Sephora)
Easy and simple to figure out, clear categories. (Sephora)

1.4. What makes it cluttered and confusing?

Poor UX design:

I thought the graphics on the front page were a little overwhelming. The filter by price was not out in front. (Clinique)
It was not scaled to a page. You had to scroll. There was too much going on during checkout. (Clinique)

2. Clarity

How do users perceive the site’s value & benefits compared to its competitors?

The statement we used to evaluate Clarity was:

  • I clearly understand why I should buy from this website instead of its competitors.

Takeaways

All websites are highly visual, featuring stunning images and very little text. They are perceived as exceptionally clear.

Sub-dimensions and insights

2.1. How does a website show that it is the best place to buy beauty products?

Simplicity, value, quality:

The website is beautifully designed. You can see exactly what the colors are of any makeup that you’re interested in purchasing. It’s fun to look at the various products. (Sephora)
I loved their overlay and their packaging. The items looked gorgeous. (Sephora)
Website looked nice and I felt like the products were top quality. (Sephora)

3. Usability

Do users think the site is more or less usable compared to competitors?

The statements we used to evaluate Usability were:

  • The website is easy to use.
  • It is easy to navigate within the website.

Takeaways

Getting usability right appears to be a major challenge. Three out of four among the tested mobile sites achieved their lowest score on this dimension. While Sephora and Fresh perform above average on Usability, Lush is clearly not on par.

Sub-dimensions and insights

3.1. What makes a beauty and cosmetics website easy to use?

Ease of use, speed, good filtering/sorting:

It was very easy to search and was also very well organized. (Fresh)
It was easy to search and check out. (Fresh)
Very easy to navigate. (Fresh)
The website was very fast and easy to use. (Sephora)

3.2. What makes it hard to use?

Slow to load, poor navigation, filtering, or sorting:

Search bar was hard to find, checkout didn’t let me use my quick fill-in feature. (Lush)
It was pretty hard to search around. It kept taking me back to the home page. (Lush)
The whole website didn’t fit on my phone’s screen. I had to scroll over to the right to see the button for my cart. (Lush)

3.3. What makes a beauty and cosmetics website easy to navigate?

Helpful categories, filters, and an easy checkout:

I liked that when I pushed ‘add to bag’ it automatically took me to my bag and I didn’t need to navigate to find it to check out. (Clinique)
The navigation was great. It was set up in broad categories that you could explore. Very nice looking and functional website. (Lush)

3.4. What makes navigation difficult?

Misleading, not intuitive:

I didn’t like it that the first thing was an email signup when I thought it was a registration sign up. I also didn’t like all the various offers coming all over the place. I would prefer a separate tab for “your offers.” where I could get samples or other points and things like that. The offers got in the way of a clean shopping and checkout process and made it a slight hassle. (Clinique)

4. Credibility

Do users trust the site more or less than the competitors?

The statements we used to evaluate Credibility were:

  • I feel comfortable purchasing from this website.
  • I feel confident conducting business with this website.

Takeaways

Three of the four websites are perceived as highly credible. Lush lags behind, scoring slightly below average.

Sub-dimensions and insights

4.1. What makes cosmetics shoppers feel comfortable buying from this site?

Fast, professional, easy to navigate, attractive:

It was very well designed. I like the addition of adding pictures and making it feel more homely. It was also pretty easy to navigate. (Fresh)

4.2. What keeps them from feeling comfortable?

Slow speed, too many popups, cluttered design, unclear options in cart or at checkout:

Most of the website was very cluttered and felt very chaotic. It was very hard to navigate compared to the other websites. (Clinique)
There were pop up ads that were annoying. (Clinique)
I didn’t like that the lipsticks didn’t show the packaging but only the lip smears. (Lush)
I don’t like the colors and they asked more information like my phone number. (Lush)

4.3. What gives cosmetics shoppers the confidence to buy from this site?

Ease of use, transparency on prices:

There’s a lot more selection and you can use the filter for price range and sorting for price in a meaningful way. (Sephora)

4.4. What undermines their confidence?

Marketing messages, interruptions:

Too many adds, pop-ups are terrible, too much marketing while I was trying to search for something else. (Lush)

5. Loyalty

Are users more likely to return to the site or to your competitors? (NPS included)

The statements we used to evaluate Loyalty were:

  • How likely are you to recommend this website to a friend or colleague?
  • I will likely visit this website in the future.

Takeaways

Loyalty is the UX dimension that separated these sites the most. The four websites show loyalty results spread all over the upper 50% of the bell curve. Lush and Fresh trail their competitors on this UX dimension.

Sub-dimensions and insights

5.1. Why would cosmetics shoppers recommend a beauty site to a friend?

Ease of use and product discoverability:

The pull-down menus from the top was elegant looking but not fussy. Loaded quick and to where I wanted. (Sephora)
I liked the ease of locating everything and quickly limiting it to just my needs. (Sephora)

5.2. What would keep them from recommending it?

Hard-to-find products, slow, confusing:

Search bar was hard to find, checkout didn’t let me use my quick fill out feature. (Lush)
While they had a sort ability, they didn’t have a true filter option. (Lush)

5.3. Which aspects of a beauty site bring users back again and again?

Ease of use and product discoverability:

There was nothing that I found to be frustrating. I’m a long time Lush customer so I regularly buy products from them online. I like their website.  (Lush)

5.4. Which aspects keep them from coming back?

Low-end design, hard-to-find products:

Wasn’t as responsive and didn’t seem as professional and polished. (Lush)
The categories weren’t as nice as the very first site. It felt like I had less choice here and the most obvious products were only the bestsellers.  (Clinique)

GLOBAL UX – Who Won?

After this analysis, we can declare a clear winner among the 4 websites we tested: Sephora. Sephora stood out on all five dimensions we tested. Clinique and Fresh have room for improvement, while Lush trails its three competitors.

Download the Full BeautyUX Report here.

Conclusion

Comparing two different “objects” at glance is difficult. Whether these objects are companies, websites, products, or services, UX benchmarking offers a rigorous comparison and quantitative results.

This benchmarking study compared four stunning websites—all scored high on Appearance and Clarity. Yet, despite their looks, we could separate them based on usability dimensions, and a clear winner emerged.

Benchmarking is a starting point to understand what you are doing better than your competitors and where you can improve, perhaps by learning from them.

What comes after benchmarking? The next step is in-depth research to assess which aspects of your online presence can be improved. At Speero Agency, we use the ResearchXL framework as a routine part of the client onboarding process.

If you want to learn more, book a free strategy session with our agency team.

Related Posts

Who's currently reading The Experimental Revolution?