The myth of A/B testing in a digital world…and why it doesn’t work

A/B testing has long been considered the pinnacle of marketing analytics and the best way to maximize marketing budgets. In fact, 60% of businesses are already utilizing A/B testing, and another 34% plan to utilize it, making it the most popular CRO (Conversion Rate Optimization) technique. The principle behind A/B testing is sound: the results should be fair and reliable.

If you’re still doing single-channel A/B testing, you should reconsider your marketing approach – and all you know about marketing analytics.

Why do marketers love A/B testing if it doesn’t always love them back?

Data from A/B testing frequently feels like it has the most weight when it comes to defending marketing choices. It’s a scientific controlled procedure after all, and controlled, scientific data sets are what it should produce. And regardless of how poorly it may be carried out, marketers have historically had numerous reasons to support A/B testing in their marketing plans. Testing A/B provides:

Ease of use

A/B testing is a straightforward idea that’s also rather simple to put into practice. This implies that it can be used by even the smallest kitchen-table business to test its content and other marketing initiatives. It’s pretty easy to start, and the entry barrier is minimal.

Model validity

Of course, model validity is an issue for marketers. By tracking the precision and effectiveness of content, advertisements, and other marketing tactics, we can better defend our choices and expenditures. And because A/B testing has been around for so long, the results still have that air of legitimacy.

It works – in theory

It is possible to anticipate that A/B testing of variables presented to a randomized sample of a particular audience will produce genuinely insightful data on marketing performance. However, the way this method is currently being used, particularly on the large, algorithm-driven media platforms, doesn’t allow for meaningful A/B testing. This causes it to produce inaccurate findings.

The validity myth: how big media platforms make A/B testing a waste of time

A/B testing is a model that, while theoretically sound, is useless when used with popular media platforms. In actuality, just 28% of marketers say they are happy with their conversion rates now that A/B testing has been used. Why? Sadly, there are a few main causes of the inefficiency:

Lack of control

It is impossible to have complete control over a third-party media platform. Simply put, it is not yours. And, until you can fully control for every variable, as in a double-blind trial in a laboratory, your A/B testing results will be meaningless.

It Isn’t actually random

The “best” viewers for each social media post or digital advertisement are selected by algorithms, and the biggest platforms have the most potent algorithms.

Your results are never completely random since your information is always filtered through these algorithms rather than just being dispersed widely to your target audience.

The algorithm has its own purpose

A/B testing cannot override an algorithm. Profitability has been built into media platforms like Google Ads and Facebook. Their algorithms will always test and present your content to the most likely audience to perform the desired action.

This cannot be overstated. Media outlets prioritize profit since that is how they were designed.

This means that the platform’s profit motive takes precedence over your goal of testing variations of your content on truly random consumers, resulting in biassed and inaccurate data.