The Best Way to QA Experiments in Convert

Conversion rate optimization (CRO) is a structured, systematic approach to increasing the percentage of conversions on a website or mobile app. CRO typically focuses on persuading current traffic to take a desired action. This is done by generating hypotheses and validating them by comparing them through A/B testing/experimentation.

The key to the success of your experimentation is to start with an experiment setup that’s set up correctly. For that to happen, the experiment needs to go through a proper Quality Assurance (QA) process.

There are several approaches you can use to QA your A/B tests/experiments in Convert. You can:

• Use preview links.
• Make experiments live on QA audiences and force a variation through query parameters.
• 3. Make experiments live on QA audiences and set variation traffic to 100%.

For more details on the first two approaches, go to Convert’s help center articles. We will discuss the third approach below.

Why Did We Come Up with a Third Solution?

On using preview links, Convert has said: “Live previews work with a high accuracy but there are still cases when they can render unexpected results due to how technology works.” Also, preview links won’t validate your targeting and audience condition.

Making experiments live on QA audiences and forcing a variation by query parameters solves the aforementioned problem. But you still need to add extra query parameters to the URL along with the QA audience parameter when you force a variation. That’s annoying, it looks complex, and it could be missed—and, as a result, you might not be able to see the variation. Moreover, your interactions during QA will be recorded as a user, which might skew your results.

The best approach:

Making experiments live on QA audiences and setting variation traffic to 100% solves all the aforementioned problems. With this solution, you can have only one simple URL for QA the whole journey.

To take this approach, follow these steps:

1. Create an experiment-specific QA audience. You can use the following rule:

– Medium matches exactly QA_expX | X could be any identifier

You can replace “QA_exp31” with any text you’d like. I just used something that makes it easy to separate and understand why we have created this audience.

2. Now add the audience to the experiment, add the suffix “[QA]” to the experiment name, and make your experiment live by changing experiment status to “Active.”

3. Go to the “Report” tab of the experiment. Stop the control variant by clicking stop variation. That way, all traffic goes to the variation. If you have multiple variations, stop all variants except the one you want to QA right now. You can alter the variation traffic when you want to QA different variations.

When you click to stop it, you will see a warning popup that says: “Are you sure you want to pause the variation?” Click yes. And don’t worry, we’ll clone this experiment to get fresh results.

4. Open a new private or incognito window in your browser. Then open the following URL:

– http://www.yourwebsite.com?utm_medium=QA_exp31

The above URL is an example. You need to change it to match your website domain and the parameter you set in the QA audience you created earlier in step 1.

You will then be able to see the variation…even if your experiment is on the cart page. Your QA URL will be the same as the above. You just need to navigate to the cart page.

5. When QA is done, you can stop and clone the experiment, remove the QA audience, rename it as you want, and publish the experiment—you’ll get influence-free results.

Summary:

• Create an experiment-specific QA audience.
• Add the QA audience to the experiment.
• Add the suffix “[QA]” to the experiment name.
• Make your experiment live by changing experiment status to “Active.”
• Go to report and stop the control variant, so all traffic goes to the variation.
• Start QA on your variation with a simple QA audience URL.
• When QA is done, stop and clone the experiment to get influence-free results.