How to analyze test results with Google Analytics Custom Segments
This is Part 2 of the Custom Segments series, you can find Part 1 here.
Your A/B test has run for enough days and you are confident that you have enough data to start your analysis. Tools like Google Optimize, VWO or Optimizely have their own reporting, but more often than not they might be insufficient.
How to create Custom Segments to analyze Google Optimize results
In order to get the correct experiment data you will have to go to your Optimize account and select your experiment.
Next you will have to scroll down to Measurement and objectives and copy your Experiment ID:
Head over to your Analytics and open the Audience -> Overview report. Go ahead and create a new Segment and then go to Advanced -> Conditions.
Set the Filter to Users – Include. Click on the Dimension box and select Experiment ID with Variant.
Experiment ID with Variant is a dimension that links Optimize with Analytics. It can be divided into 2 parts:
- The first part is your experiment ID, ex: OkHF3fO4SOeTRLaWu4BdrA
- The second part is the variation number. The control will have number 0, the first variation will have number 1 and so on.
The final format of this dimension will be OkHF3fO4SOeTRLaWu4BdrA:0 for the Control and OkHF3fO4SOeTRLaWu4BdrA:1 for Variation 1.
Don’t forget to make this segment available only in the Current View.
Last but not least, give your Segment an easy to follow name, as an example I use Optimize – Control – Experiment Name.
Repeat the above steps to create your Variation segment. Be sure to change the 0 at the end of the Experiment ID with Variant dimension to 1 as well as the name of the segment to Optimize – Variation – Experiment Name.
How to use Custom Segments to analyze Google Optimize results
Your Segments are created, but you might be wondering what now? Well, you can use these to better understand user behaviour and dissect further into your data.
Since you have created your Segments in the Audience -> Overview report, you can start by looking at how many users took part on each of them.
IMPORTANT NOTE: Be sure to select the correct time period, otherwise your data will be faulty.
Let’s calculate the conversion rate of users that took part in this experiment.
First we’ll take a note of how many users each variation had, in our case 3204 users on Control and 2990 users on Variation.
The experiment in my example had one primary Goal and a secondary Goal. The primary goal was to increase the number of users who finish the funnel while the secondary goal was to increase the number of users who reach the following step in the funnel. These 2 were strictly related as the experiment took place on the page with the highest drop off rate.
Prior to the experiment I have created a Goal in Analytics to track the completion of my funnel (I suggest you do the same).
With the 2 segments in place it is very easy for me to gather the data that I am looking for: How many Goal Completion events have been tracked on Control as well as on Variation.
In my case there were 87 Goal Completions on Control and 72 on my Variation.
Using basic math we can calculate the conversion rate:
Control: 87 divided by 3204 times 100 equals 2.71, which means that the conversion rate is 2.71%.
Variation: 72 divided by 2990 times 100 equals 2.40, which means that the conversion rate is 2.4%.
Unfortunately, this experiment has proven to be a loser. Or is it?
Let’s jump down the rabbit whole
Google Analytics’ reports can make you wonder around for answers in an endless loop. I got rid of that with a simple ideology, if the following 2 reports don’t point out anything at all then I should move on.
(This really depends on the kind of test I am running, if the test involved really subtle changes than I can go with these reports. But if I tested a redesign than I would spend a lot of time on understanding user behavior on each variation as well as going down to the most detailed reports.)
Let’s edit the Control Segment to count only the users who have seen the Control from their mobile devices, like this:
Take note of how many users are being displayed in the right part and save the segment. Do the same for the Variation Segment.
Once you have edited and saved both of them you will then see the Goal completions of each one. Apply the previous calculation method and see what is the conversion rate of mobile device users. Do this for desktop and tablet users as well. Finally, repeat these steps for New Users and Returning users.
Once you have finished all these tedious steps you will have an overview of your experiment’s results, all done inside Analytics.
Of course there is an easier method and I am going to write an article about it. You can use Google Data Studio, a data visualization platform from Google. It allows you to set up a report and filter the data based on any dimension available in Analytics.
The idea behind this tutorial is not how to easily get test results, but rather to understand the process behind it and how to use Custom Segments in order to get the most value out of each and every experiment that you run.
This is the second part of my series on Google Analytics Custom Segments. In the next part I will explain how to create Advanced Custom Segments and Retroactive Goals. If you have forgotten to set up your goals before starting an experiment worry not. I will cover the easiest way to get all your results, with a very high confidence margin.
If you need help with your optimization process, you can always get in touch with us at [email protected] or just leave a comment down below. We’re always happy to hear from you!
