Improve Your A/B Tests with Five Simple Questions

Conversion optimization isn’t easy and remember, there is no quick way to success.

It’s a must to have a proper process and the right resources in place. Stay away from tactics, and focus on a smart process instead.

Brian Dean has put together a great list of CRO techniques I recommend to check out. Especially helpful for those who are new to A/B Testing.

However, don’t take things for granted.

Remember to always test so-called best practices. What works on website A doesn’t have to work on website B.

In this post I reveal five important questions to answer before setting up your next A/B Test.

Paying close attention to each of these questions dramatically enhances the chances of success in your A/B Test efforts.

1. Where Should I Run My Next A/B Test?

Think twice before you start A/B Testing without a pre analysis first. Your success lies in your preparation.

In order to make a clear impact on your bottom line, you need enough traffic and conversions.

Do you run a website with less than 10.000 visitors each month and no more than 200 conversions? In general these numbers are too low for A/B Testing.

Focus on growing your website first. Expert reviews can help you to get the basics right.

“In general, the higher your monthly number of visitors and conversions, the more A/B Tests you can run.”

A few subquestions to ask first:

1. What is the macro conversion that I want to optimize?
2. What do my traffic numbers look like?
3. How many conversions do I get in a month?
4. What’s my current conversion rate and how does it differ per (landing) page?
5. On what pages can I make the most impact?
6. Are there any political constraints I need to keep in mind?
7. What about technical issues that are present?

Make sure to start with a thorough data analysis first before moving to the next step.

At a minimum, use your quantitative website data to find great spots for future testing.

It’s a best practice to focus on pages that many people visit before they convert.

At first I recommend to start with one page tests (more easy to set up and analyze).

If you are more experienced, you can start running tests on multiple pages or site-wide tests as well.

They can deliver a great ROI, but might take more time to develop and analyze.

2. What Elements Are Involved?

You could literally test thousands of different things.

However, it’s crucial that you have a reason behind the elements you run your tests on.

Simply changing a color from red to green doesn’t make sense.

The “What” is closely related to the “Why”. Something I will talk about soon.

The more elements you add/remove/change in one test, the more difficult it is to learn from it. Interpreting results can get really difficult or simply impossible.

At least you have to make sure that all changes are in line with the same hypothesis.

Stick with running A/B tests first and run multivariate tests later on if you wish to.

3. Why are You Running This Test?

Running A/B Tests without asking this “Why” question is not a smart thing to do. It’s actually very stupid. :-)

It’s time to define your hypothesis:

“A tentative assumption made in order to draw out and test its logical or empirical consequences.”

The more research and data you have to base your hypothesis on, the better it will be.

Logical sense can definitely help, but don’t think you have all the solutions right in your head.

Your visitors will proof you wrong.

How to come up with an effective hypothesis?

1. Carefully analyze your quantitative data (e.g. Google Analytics)
2. Use Hotjar (heatmaps, video recordings, form and funnel analysis)
3. Run qualitative surveys
4. Talk to customer service department
5. Involve your colleagues (psychologists can be a great help!)
6. Visit competitor websites
7. Read through prior research reports
8. Study prior A/B Tests results
9. Keep your end goal in mind
10. Learn from peers (never take best practices from granted, always test them)

There are many more actions you can take to enhance the chances of succes.

These 10 points will definitely be a great help for you to formulate effective hypothesis.

4. When will My Test Run?

Tests, timing and planning are more crucial than you probably think.

There are a few things you need to keep in mind here.

1. Set your test length before you start running your A/B test (one week as a minimum)
2. Always run a test with full week cycles
3. Keep in mind the conversion cycle
4. Don’t let your test period interfere with (major) website changes
5. Run a pre-test
6. Run a post-test
7. Calculate your numbers upfront (how much traffic do you expect, uplift needed for statistical significance)
8. Don’t let your tests run for months (approximately four weeks as a maximum)

These tips will help you to set the testing period right and boost your chances of running successful A/B Tests.

5. Who is Involved in This Test?

I like to talk about two aspects here:

• Target audience of your test (“pre-segmentation”)
• Post-segmentation (segments you will analyze after the test is finished)

Target Audience

Do you want to test on all devices or just on desktop and tablet?

There are many reasons why you should want to test on mobile and desktop separately:

1. Different things work (or might not work) per device
2. Difference in traffic volumes
3. Difference in mobile traffic (iPhone vs Android)
4. Optimize for different outcome
5. More tests and in a faster way

In general I recommend to start with the device type where you can make the most impact on your bottom line.

In addition, in some cases you only want to target specific traffic sources.

And if you have lots of traffic and conversions, you don’t have to run the test on 100% of your traffic.

This will save traffic for running multiple experiments at the same time!

Post-segmentation

I recommend to define your segments upfront. This instead of searching for a significant effect within a specific segment afterwards.

After you have done a lot of testing and get more experience, you know there is always a significant effect to be found.

As long as you keep on digging your data you can come up with something.

However, this is not how A/B testing works.

This is it for now. In future posts I will dive in much more detail in each of those areas.

Do you have any tips about effective A/B Testing that you like to share?

One last thing... Make sure to get my automated Google Analytics 4 Audit Tool. It contains 30 key health checks on the GA4 Setup.

Get Free Access to The Google Analytics 4 Audit Tool