As professionals who live and breathe testing, we do get a ton of requests to “increase click throughs, increase revenue and increase engagement” on a variety of sites. In most cases these requests don’t come with a lot of context.

Clients often tend to think that we optimizers have some sort of magic formula that can unerringly take them to big conversion lifts.

DISCLAIMER: There is no magic formula.

But there is a tried and tested empirical process that turns “suppositions” into more actions, more dollars, more profits and just more for marketers and businesses in general.

And this process starts with a hypothesis.

Businesses starting out with optimization may already have a very clear idea of what has their visitors stymied. The hypothesis is cut and dried. And the gains from the tests are tangible and immediate.

But what happens when you run out of all the obvious options? Do you let go off optimization and rely on increasing your marketing spend by thousands to see proportional improvement in revenue?

Nope. There’s no limit to better, and no limits to testing. So you dig deeper and you read between the lines. Let’s show you how.

How to Generate an A/B Testing Hypothesis, When You’re Done with the Obvious:

1. Obtain and assimilate list of business objectives that the client has – both short and long-term

The great thing behind your client’s or your own business objectives is that after you’ve defined them you are halfway done with the measurement plan. As long as you know what you want to track you will also know how to track it.

There’s a really well done, comprehensive measurement template that takes a bit of time to populate but the results are worth the effort.

You can make a copy of this document from Fresh Egg and make it your own by adding in your numbers and objectives.

2. Check your data and then analyze it

You can’t really improve something if you’re measuring the wrong things. Use Google Analytics crawlers and checkers to make sure the tracking code is installed on all pages, get rid of any duplicate data, add the right filters and use dedicated views based on the objectives and particularities of each business.

Tools to consider:

http://www.gachecker.com/ – great for spotting any URLs that don’t have tracking code on.

Tag Assistant by Chrome – take this one with a grain of salt as there will be some non-standard configurations out there and it can tell you that Analytics is not installed when it actually is.

WASP Crawler – a saver in terms of event tracking and cross domain tracking validation.

What to look for in an analytics audit? Basically patterns and outliers in the users’ behavior.

In the initial audit, you will want to see the impact of each element on the site, what users are doing, where they are dropping off, any incompatibilities between the website and certain browsers, devices, resolutions, any bug that is worth fixing. After you have all this information put together, you will know where things go wrong but you will also need to find out the reason why things go wrong so go ahead and start working on the qualitative research part of the audit.

3. Set up qualitative research

Launch site polls and customer surveys. By the time you are done with the complete analytics research and insight generation, you will also have a sufficient volume of qualitative information to analyze for your audit.

Where do you launch these site polls? Do a quick funnel analysis and identify the drop-off points. If 70% of your users are leaving on the product pages, there might be something there that is not convincing them. What better way to find out than to ask them? A typical site poll looks something like this:

The customer survey is sent out to recent customers, after they have received their product, ideally. Try sending this to customers that are not older than 30-45 days as you want them to remember the experience they had with the website, with the shopping process and also with the actual product. If you send out the survey too soon, you will not get relevant information on the quality of the products or delivery. Good questions to ask:

• What can you tell us about yourself – good to find out their age, gender, location, etc. This will help put more context and validate the insights you have in the demographics reports in Analytics.
• What made your buy our products/become our subscriber/sign-up? – this is good to identify the main points that convince users to buy so you can emphasize them in further communication with your prospects.
• What doubts and hesitations did you have before buying/subscribing/signing up?
• What would you miss if you could not use our products/services anymore?
• What is the one big thing we are missing?

These questions are great as you get to know a lot more about your customers, and it can change how you present your message, and what to emphasize.

4. Conduct Heuristic Analysis

When we refer to heuristic research in the CRO realm, we mean a site walkthrough, really. A heuristic research is an experience-based assessment, meaning you simply apply your experience and understanding to the website and define some of the reasons why users are or are not converting. Analytics shows you the numbers while a heuristic analysis will show you why those numbers are like that.

There are certain elements to keep in mind when conducting a heuristic analysis.

Relevancy

The first thing to keep in mind when doing a heuristic analysis is that you should start from the SERP or from the banners or email or any of the other traffic mediums you are using to bring users to your website. Why is this important? Because relevancy has a lot to do with users’ expectation to the content they will find on your website. When they are clicking a link they are expecting something and it is your website’s role to deliver it.

For example, let’s say you are selling organic cotton t-shirts and users find you organically (pun intended) through a google search.

You can see there are some keywords that really stand out, such as organic, sustainable, comfort, care. These can highlight some of the elements users are looking for when searching for organic cotton t-shirts so let’s see what we find on the first result:

On the Thought landing page, we find further references to organic cotton and sustainable clothing. They’re doing a good job at it.

Clarity

Clarity simply refers to whether the offer is clear enough and users can easily understand it and how to buy it. The richest industry of examples on this subject has to be the world of SaaS. Many of the businesses out there present their tool in vague, “hype-y” terms that don’t really get the message across to users that are not very patient.

One example would be from Solteq who are sending Google Ads traffic to a landing page that has this headline “We want to simplify the digital world to make a better tomorrow” and that does not really say anything and is lacking a clear call to action. They showed up in our searches for “procurement management software” so you would see why the two don’t really meet.

On the other side of the spectrum, there’s Purchase Control who are keeping it simple and to the point. Their software helps you keep business spending under control, that’s what they say in the headline and in the ads.

There is a clear CTA that gets you a free demo so you can really understand the offer and what you’re supposed to do on that page.

Value

Generally speaking, the perceived value of a product will make users stay or move forward. If your product does not clearly communicate the value that is relevant to your users then they will just move to the next brand offering what you are.

This is not that complicated, it’s really about communicating what you have to offer and how that offer benefits your users. The less known your brand is, the stronger your value proposition should be.

Friction

In terms of user motivation, think of friction as a speed bump in the road for your customer journey. Any element that is causing anxiety, doubts, fear in the user’s mind will slow them down from buying or even entirely prevent them from doing it.

In terms of user experience and design, friction is represented by any element that is preventing users to take the action you want them to take. The most common example is the overlapping of live chat widgets with add to cart buttons or site polls overlapping checkout forms, etc.

To eliminate friction on the motivation side of users, just make good use of customer surveys and site polls and identify what needs to change.

Distraction

Distraction is caused by elements that are taking the user’s focus away from the main action and redirecting them to something less important.

One very simple example is that you remove navigation and blog posts and any other widgets from the cart and checkout page as you would not want users to leave those steps to read about your latest press appearance.

Try to keep your users focused on one call to action and reduce distractions to a minimum. Make sure you offer sufficient information but don’t pull people away from the landing page you paid money for them to see.

5. Mouse tracking and session replays analysis

You can use Hotjar for this.

The features you should use? Clickmaps, scroll maps and session replays.

Clickmaps are great to identify areas of interest, elements that appear clickable but aren’t, should be clickable but aren’t as well as the flipped side of this coin, with calls to action that are being ignored.

Scrollmaps show you how far down the page users scroll – this will help you identify and visuals stopping signs on your page (visual elements that give out the impression that the page is ending there).

You can also identify elements that are being overlooked by users as too few of them reach that point. There are many examples of testimonials or FAQs that were completely overlooked as they were close to the footer but users actually converted a lot more after seeing them. Sure enough, after moving some of those elements higher up you see an increase in conversion rates.

See which elements are contributing to your conversion rate and are being overlooked and test new layouts with that in mind.

Now that you have all this information at hand, make sure you have a winning hypothesis formula.

You can use something like this: 

1. Because we saw (qualitative & quantitative data)
2. We expect that (change) for (population) will cause (impact(s))
3. We expect to see (data metric(s) change) over a period of (x business cycles)

Let’s say we are writing a hypothesis based on information from Google Analytics and the scroll maps for the website selling organic cotton t-shirts. The statement will be something along the lines of this:

Because we observed a 70% drop off for mobile users and session replays with scroll maps indicate users are not reaching the “organic origins” section, we believe that by showing the section earlier to all visitors on the product pages, we will see an increase in “add to cart” rates. We will measure this through the add to cart goal conversion rate over a period of 2 business cycles.

Conclusion

And there you have it – A/B testing hypothesis generation that goes beyond the obvious and uncovers improvement opportunities that you would otherwise fail to capitalize on.

Start with a very thorough Analytics assessment and go from there.

Complex A/B Testing Hypothesis Generation: The Step-by-Step Guide For when You’re Done with the Obvious

PS. I love connecting dots between the numbers from analytics, the answers from surveys, the colors in heatmaps. It feels like being Sherlock and uncovering a great mystery 🙂