Should I be A/B Testing?
If you go to enough free lunches where people try and sell you things you would have heard that AB Testing, CRO or Optimization is the way of the future. “It is what you HAVE to be doing right now if you want to make money.” Let’s be honest there are about 10 other things that claim the same thing, so is it really worth your while to be AB testing.
Below I have outlined what I think you need to check off before you can start any sort of optimization project.
Do you trust your analytics?
I’ve seen people who have started AB testing when their data is a mess. They think if they have the conversion going ok, then it’s all golden. To this I say, how do you know what to AB test, if you aren’t tracking things correctly. Any time or effort is more valuable getting your tracking set up correctly and having the team trust the data then trying to test anything.
Does the team care about the data?
Once you have data you can trust, do people actually care. You might present that the new homepage design has increase bounce rate by 20%, but if the big boss says they like the new colours and it’s staying that way, no AB testing is really going to help you in this situation, and you should probably start looking other places for a job. They will never care about data.
Have you made changes because of data?
Caring and actually taken action are two different things - that is why we have climate change. Before you start investing time into optimization and AB testing can you point to a time that your organization has made a change due to some data they were presented? There is plenty of analysis that can be done on historical data that would show changes that need to be made based on conversion rates, click through rates etc. If you have presented countless of these reports and nothing has changed, then AB testing might just go down the same path.
Have you analysed the data you have?
The first stop to answer a business question shouldn’t be AB testing. First, it should be to look at what data you have, and what hypothesis it is guiding you towards. If you can’t prove your hypothesis with the data, you have then go to test. What I usually find is if I look at the data first I come up with even more questions/hypothesis’s I want to test vs what I originally started with. I can get more specific about the questions I’m asking, i.e. Do people like music? vs Do people like indie new wave music?
Have you made a change and reported on impact?
Brining the last two together, before getting into AB testing you try first making a change based on the data you have and then analyse the impact that change had. This is easier with AB testing, but if you aren’t doing it currently with changes made, then why would anyone care about the impact from a test. This behaviour needs to be baked into the organization if you want AB testing to be successful.
Do you have a Testing Strategy?
Now once the organization is embracing a data-driven approach, you can start creating your AB testing plan. It’s like you have painted the fence and washed the car, and now you are ready to learn karate. I wrote another article about how to go about setting up a strategy here: https://www.meliorum.com.au/blog/how-to-do-cro-without-being-a-wanker
I would recommend using a free AB testing tool to start, like Google Optimize. It has a low barrier to entry and lets you learn what features you need and don’t need before you invest in something larger.
If you can answer yes to all the questions above - then I say YES do AB testing. Otherwise, I would look at using that time to build a stronger foundation in the business before you go down that path.
