With millions of capsules sold online every month, having a trustworthy measure of success for new features is critical. We set up an AB Testing framework for Nespresso Spain to track any relevant update on our e-commerce the most reliable way.

Starting with data

The first step in any experiment is to diagnose a measurable pain point. We used three sources of information to identify our main issues:

• Customer care tickets related to purchase issues.
• Web sessions recordings and heatmaps in SessionCam.
• Data-analysis of our product metrics in Google Analytics.

We monitored these three channels continuously to track progress and uncover any additional issues. While it may be tempting to address every issue reported by a user, having a measurable product metric (such as visits, click-through rate, add-to-cart, or purchases) helped us prioritize which problem to solve first.

Framing the issue

Having established the baseline performance of any page or feature on our site allowed us to identify several solutions to address the opportunity and evaluate whether the expected impact was achieved through a new design. The beginning of each test started with a hypothesis statement. Here's an example.

Hypothesis statement

Some examples

AB Testing is for learning. The featured cases allowed us to:

• Safely deploy new designs while monitoring for any unforeseen issues.
• Measure the actual impact of changes before committing to implementing a solution on a larger scale.
• Focus on the elements that have a significant impact.

Capsules Product List Page — Original vs New We measured whether a more visual design and navigation increased add-to-bag and coffee discovery.

Machine Product List Page — Adding machine Subscription visibility We monitored the effect of adding subscription badges on the visibility of machines with available subscription plans.