a relevant result<\/a> would be to go with option number 2, which is to run one test fully and then launch the next test.<\/p>\n\n\n\nHowever, this would definitely not be beneficial to your testing velocity and would incur considerable testing costs. Basically, you would probably be able to run somewhere between 6 and 12 tests a year, maximum, on most small to medium websites.<\/p>\n\n\n\n
Option number three doesn\u2019t present itself well either, as you\u2019re likely to have users that are exposed to one combination of the experiments but also combinations that get no users so that\u2019s far from ideal. <\/p>\n\n\n\n <\/figure>\n\n\n\nAs you can see in the image above, you will have users that will not be exposed at all to combination BB from the two experiments. <\/p>\n\n\n\n
If you would go with option number 1, you would be completely disregarding the interference between the tests and then you\u2019re exposing yourself to releasing an unvalidated user experience on your live environment. <\/p>\n\n\n\n
The main problem with running concurrent A\/B tests is that you increase the risk of a Type I error or false positives. This makes it harder to gain some actionable insights that you can rely on. <\/p>\n\n\n\n
We encounter that most of the businesses that are requesting testing multiple items at a time are smaller businesses that are the most impacted by high daily average variance. <\/p>\n\n\n\n
There isn’t a one-fits-all formula<\/strong><\/h3>\n\n\n\nThere doesn\u2019t seem to be a full agreement on this subject yet so you would still need to ask yourself what is more important to you and what will help you reach your business objectives. <\/p>\n\n\n\n
It could be that you treasure accuracy more or you treasure testing velocity more. There is that question on whether you\u2019re in the business of science or in the business of making money, which I can\u2019t really say I agree with.<\/p>\n\n\n\n
You need to ask yourself what the real possibility of interference is and how it would affect the results. The success of your A\/B testing program will be determined by the number of tests you run and the percentage of winning tests and their impact on your bottom line. Now, if you end up drastically cutting down the number of tests you run in order to avoid data being polluted, you risk not testing enough and then your CRO program will most likely fail. <\/p>\n\n\n\n
What we do with some of our clients with lower traffic volumes is that we define different conversion tracks and funnels and make sure we prevent the overlap of tests in that way. <\/p>\n\n\n\n
Going back to our example above, we would firstly monitor the impact of the change on the product page on the \u201cadd to cart\u201d conversion and then we would see from all the people on the cart how many are actually initiating the checkout on each of the variations. The funnel is a sum of steps and we think that by improving each step along the way, we maximize our clients\u2019 chances of getting more conversions – we prioritize the main conversion rate on each step of the funnel and then we analyze how all the changes combined impacted the overall conversion rate. <\/p>\n\n\n\n
It might not be the most orthodox way of doing it but our clients are able to be one step ahead of their competitors this way. In the end, their competition is not stopping so why should they?<\/p>\n\n\n\n
So, for the TL;DR, here is my take on running multiple tests at a time: <\/strong><\/h3>\n\n\n\nTry to test separate conversion paths – unless you\u2019re really testing conflicting hypotheses and fear a massive pollution effect, just do it and keep an eye on the results with advanced segmenting in Google Analytics. <\/li> Run tests on different segments of traffic – just create mutually exclusive tests or define your tested audience in such a way that there is minimum overlap. You will be able to do this with higher traffic volume websites. <\/li> Put your tests together and run them as a multivariate test – if you have the traffic for it and if you\u2019re testing for the same conversion (purchase, lead submit, etc). <\/li><\/ol>\n","protected":false},"excerpt":{"rendered":"With the increasing costs of website traffic and the decreasing attention span of users, the velocity of A\/B testing is becoming a critical metric. That’s when you start to wonder if\u2026 Read More »<\/a><\/p>\n","protected":false},"author":2,"featured_media":6790,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"footnotes":""},"categories":[124,113,114,1],"tags":[112,178],"yoast_head":"\nShould you run multiple A\/B tests at a time? - ONTRACK Digital - CRO Agency<\/title>\n \n \n \n \n \n \n \n \n \n \n \n \n \n\t \n\t \n\t \n \n \n \n \n \n\t \n\t \n\t \n