How to Do Shopify A/B Testing

An eCommerce business doesn’t want a subpar website, email marketing campaign, or app. 

All of those areas need to be exceptional and operating at their full capacity. To get to that place, there needs to be time for evaluation and strategizing. A/B testing, or split URL testing, does just that by offering a side by side glance at which aspects are performing best with customers. 

Shopify points out that when using A/B testing in the context of marketing, you show 50% of website visitors version A of your asset (let’s call this the “control”), and 50% of visitors version B (let’s call this the “variant”). The version that results in the highest conversion rate wins.

For example, a business may be able to see that one website design tested higher simply because there were more calls to action available to customers. The most advantageous part of A/B testing is that it enables a business to make better marketing decisions based on existing customers. This allows a business to take the data and apply it immediately to its designs. Doing so can help businesses avoid shopping cart abandonment among customers and create a more efficient platform. 

A/B testing can be applied in many ways—from design to copy, but how does it translate on Shopify’s platform?

Why A/B Testing Is Important

The results from A/B testing are a guide post for your business. A/B testing helps provide valuable information. Whether it’s perfecting design, improving Shopify marketing efforts, or discovering that investing in specific ads isn’t generating enough revenue, A/B testing helps your team to make improvements that are needed in a competitive eCommerce world. 

What Are Types Of Shopify A/B Testing?

There are a lot of ways that A/B testing can be customized on your Shopify store. Ideally, a split test should be isolated to one area of a landing page (or whatever area a business wishes to expertiment). Some areas to consider that can be tested on Shopify’s platform are:

Call to Action (CTA)

• Discount Codes: Should be simple to apply and easy to remember. An A/B test of your discount code will help illuminate any issues that your customers are having with applying them.
• Newsletter Opt-in: Are customers able to sign up for your newsletter with ease? 
• Click-Throughs: What does your overall click-through rate look like for the effort your team is putting into ads? 
• Learn More: Shoppers may not be ready to jump into a purchase just yet. Are you giving them the chance to learn more about your brand and the product? Spotify has the right idea by including a Learn More button as part of their CTA. This especially works well since they are a subscription service where customers want to learn about the terms and conditions of the contract.

Product Placement

Your products may not have the visibility you think they do. In some cases, something as small as changing the location of a product description will help improve visibility. If your eCommerce store is overloaded with copy then your products aren’t being seen. Test where products are currently and then compare them to the outcome of other placements.

Technical Analysis 

Is your online store taking too long to load? Nothing can deter a potential customer from your page faster than it taking your shop too long to load. Is the checkout process efficient? Is payment confirmation showing up in a timely manner?

Customer Surveys

Are you seeing a return on your efforts of sending out customer surveys? Is the form user-friendly? Are customers able to fill in the survey easily from a mobile device? 

It’s important that each test is run individually to result in accurate information. Then, additional tests can be planned.

How Long Should The A/B Test Run?

Best practices for the length of the A/B test are two business cycles which is roughly two to four weeks.

This timeframe will allow your Shopify A/B test to take several things into account. You can use a Shopify abandoned cart app to create a window for shoppers that abandoned cart or wanted more time to consider their purchase. Your team will also be able to track the traffic sources that your business is generating. The wait time will also enable you to have insight on your company gaining Shopify traffic from newsletters over the weekend.

How Do You Do An A/B Test On Your Shopify Website?

Before your team sets out to conduct the Shopify A/B test there are some preliminary things to do:

Calculate your test readiness. Is your business ready for A/B testing? By using this calculation tool, you will be able to gauge if your eCommerce store will get the optimal results if you were to conduct A/B testing. 

Simply enter your current conversion rate In the “Baseline Conversion Rate” field, and then how much you want to increase it in the “Minimum Detectable Effect” field.

The sample size for test variations will of course vary depending on your inputs.

This bit is the most important. You need to know that your Shopify A/B test results are going to be accurate–completely representative of your users and your marketing efforts. For example, your team doesn’t want to optimize your Shopify website design completely based on misinformation.

Establish a primary goal. (i.e. increase the click-through rate)

Set a secondary goal. (i.e. increase revenue per user, or RPU)

Choose your test subject. (i.e. landing page, the click-through rate on email, etc.)

Form your hypothesis. This quick hypothesis formula is great to have on hand for A/B testing: 

Because you see [insert data/feedback from research]

You expect that [change you’re testing] will cause [impact you anticipate] and

You’ll measure this using [data metric]

Set up the test. With all of the above-mentioned pieces in place, it’s time to set up your Shopify A/B test. 

What Are The Shopify A/B Testing Tools?

There are several Shopify apps that make A/B testing easy. Thank you, Shopify! A/B testing can be used on Shopify by simply choosing one of the page builders with built-in A/B testing capabilities. Page builders are offered through Shopify inexpensively on a monthly payment basis. This testing tool will allow your team to have plenty of time to run the split test.

The Shopify app market has several options to help streamline the A/B testing process: 

• Shogun page builder
• Neat A/B testing
• Always Be Testing 
• A/B Splits
• PageFly page builder

These are great for basic A/B testing because they allow you to customize the page, set it, and come back in an allotted time to review the results.

Analyzing the Results

Don’t let the data fool you. There will ultimately be a winner so to speak at a glance, but what is the data really saying? It’s important to segment the findings so that you can have an accurate aerial view of what those numbers mean for your business. 

Krista Seiden, Analytics Advocate at Google talks about the importance of digging deeper into all aspects of the data in order to learn. 

“The most overlooked aspect of A/B testing is learning from your losers. In fact, in the optimization programs I’ve run, I make a habit of publishing a ‘failures report’ where I call out some of the biggest losers of the quarter and what we learned from them.”

Applying the Results

When the two to four week window closes on the A/B test, it’s time to evaluate the findings and move forward with the changes, if any. The test will most likely shed light on one, if not all of the following areas:

Revenue loss or gain. Revenue tracking of the utmost importance. Did the test show that the money invested in Pinterest ads has actually generated loss? Was that one high-performing Facebook ad a one-off, or can you apply that lesson to other ads? That bit of information won’t just help your business save money, but will allow that allocation to be spent more wisely.

Design insight. A/B test results may show that the videos being posted on your business Facebook page aren’t leading to click-through. You’re able to see that opting for more readable relevant ads is the best way forward.

Overall brand. The results of the Shopify A/B test revealed that there’s a disconnect between click-through and actual purchases. Abandoned cart emails aren’t yielding a profit either. With there being so many holes in the marketing strategy, it may mean that your team needs to reassess the brand and the brand voice. Customers are visiting your eCommerce website, but may not be relating to your brand.

Shopify A/B Testing Highlights

A/B testing can help your business aim higher by highlighting areas that need to be changed. Having insight into how customers are relating to your design, brand, etc. can act as a suggestion on ways to remedy areas of your eCommerce website. A CTA button color can be tweaked, a website redesigned, or an email marketing campaign overhauled. 

The goal with testing is that it helps you evaluate what works best and creates space for your team to create alternatives to the areas that don’t serve your business. A/B testing on Shopify is simple with its many applications. 

Before you begin, make sure that your team knows why your business is testing and if you’ve got the right amount of website visitors to run a successful test. Here’s to testing, improving, and growing as a business.

For more helpful tips on how to use ManyChat, please check out our free course.