Este artigo está atualmente disponível em inglês e espanhol. Você pode usar esta documentação enquanto ela não estiver disponível em seu idioma.
A/B/X Tests are an essential tool in digital marketing to evaluate different versions of your campaigns and determine which performs best with a sample of your contacts, ultimately showing the best-performing content.
In this article, you will learn what A/B/X tests are, their benefits, and how to set them up in Connectif.
An A/B/X Test splits the Workflow traffic and, once finished, continues the contact through the winning branch. If you want to split the traffic of a workflow, we recommend reviewing the documentation on the "Split" node.
1. Properties
1.1. Properties of the "Start A/B/X Test" Node
- The "Start A/B/X Test" node, an Action type, is used to activate the start of an A/B/X test in a workflow. In the node interface, the test parameters are configured to help determine the winning branch.
- The "Start A/B/X Test" node allows testing different versions of content (Email, Popup, Inline, etc.) with a small sample of our contact list. This way, it is possible to analyze which content variation offers better results (more opens, clicks, purchases, etc.) for that small sample and show the best-performing content to the bulk of the contact list.
Using this node allows you to optimize content sending and achieve a better conversion rate.
1.1.1. Mode of Operation
To carry out these A/B/X Tests, we need to differentiate between two modes of operation: for Mass Actions or Transactional Actions.
- Mass Actions: This refers to marketing campaigns that involve taking action en masse for a large number of contacts (which could be our entire contact list) at a specific time. The best example is sending a promotional newsletter via email.
- Transactional Actions: These actions are taken individually and punctually. It depends only on the contact. An example could be showing a modal window to the contact when they browse our online store, offering them a discount. This modal window would obviously only be shown when the contact is browsing our website, so a trigger node is needed before the "Start A/B/X Test" node.
The mode of operation cannot be modified manually; the node itself will know which mode it should operate in based on the workflow structure. If we are clear about the mode of operation needed for the marketing campaign we want to implement, this indicator in the node configuration will help ensure we are setting up the flow correctly.
The differentiation of these two modes is necessary because the "Start A/B/X Test" node behaves differently depending on the scenario we want to carry out. Let's look at two examples:
- Example of a Mass Action
We want to send a promotional email to our entire list of 10,000 contacts. We have two different email designs and would like to send about 500 emails with design "A" to 500 contacts, and another 500 emails with design "B" to another 500 different contacts. We will wait one day to see which of the two designs offers better click statistics. After that day, the design that offers better results will be sent to the other 9,000 contacts.
-
- In this mode of operation, we need to first send 1,000 emails to our contacts, with 50% of them using design "A" and the other 50% using design "B", leaving the other 9,000 contacts on hold.
- During that day, we will count the clicks each content generates, and the next day, we will take the 9,000 contacts we left on hold and send them the email with the content that received the most clicks.
- The act of "leaving X contacts on hold" is what differentiates this Mass mode of operation from the Transactional mode of operation.
- Example of a Transactional Action
We want to show a push notification to contacts who abandon their shopping cart, giving them a discount code to return and complete their purchase. We have two different push notification designs and would like to test for about three days which design generates more cart recoveries. After those three days, the design with the higher cart recovery rate will be sent to everyone who abandons a shopping cart.
-
- A priori, we cannot anticipate how many people will abandon the cart in three days, so it wouldn't make much sense to send 500 push notifications with design "A" and another 500 with design "B".
- Instead, we need to send a push notification with design "A" or "B" chosen randomly (with a 50% chance for each) to all contacts who abandon a cart during those three days.
- Here, no contact is "put on hold"; during those three days, all contacts receive a push notification when they abandon the cart.
- After those three days, we will determine which design has a better conversion rate and, from then on, send the winning design to anyone who abandons a cart.
1.1.2. Restrictions
This node is special as it adds certain restrictions to the Workflow:
- Only one "Start A/B/X Test" node is allowed per flow.
- All outgoing connections from a "Start A/B/X Test" node always belong to a branch of the test.
- It is not possible to interconnect the outgoing branches of a "Start A/B/X Test" node.
- It is not possible to connect previous nodes to the "Start A/B/X Test" with a subsequent node, that is, with a node in a branch of the test.
- If a workflow has a "Start A/B/X Test" node, it will always be classified as a Classic workflow.
1.2. Phases of an A/B/X Test
According to the examples explained in the previous point, we can deduce that an A/B/X test has two phases:
- Sampling Phase, in which we test marketing actions with some of our contacts to gather metrics and statistics on which content performs better in some aspect.
- Final Phase, which comes after the sampling phase, where we already know which marketing action is the best and use it for the bulk of our contact list.
An A/B/X test always starts with a Sampling Phase. At a certain point, the winning marketing action is determined, and it moves to the Final Phase.
The timing of determining the winning option is crucial: it is important to choose the right moment to have enough sample size to draw relevant conclusions and reliably select the winning marketing action without waiting too long.
In Connectif, it is possible to see the phase of an A/B/X test at the bottom of the "Start A/B/X Test" node:
1.3. Properties of the "Evaluate A/B/X Branch" Node
- To determine the winning branch of an A/B/X Test, Connectif allows adding or subtracting points to the branches of a "Start A/B/X Test" node using the "Evaluate A/B/X Branch" node.
- This Action type node is used to evaluate any action within a branch of an A/B/X test. In the node interface, the points to add to the branch when it is executed are configured.
- To configure the "Evaluate A/B/X Branch" node, it must be connected to a branch of a "Start A/B/X Test" node.
- To configure it, simply indicate the number of points you want to add to the branch when this node is executed.
- Additionally, you can choose whether to add a fixed amount of points to the branch (literal value option) or add an amount that comes from some available data from previous workflow nodes (imported data option).
2. How to Use It in Connectif
2.1. Configuring the "Start A/B/X Test" Node
1. Once the "Start A/B/X Test" node is added to your workflow, access its configuration.
2. In the Branches section, indicate the number of branches or alternatives you want to test.
Branches are always automatically named with the letters A, B, C, D, and E. Contacts are always divided equally among all the branches, so if we have branches "A" and "B", the contact flow will be divided 50%. For example, if we have branches A, B, C, and D, the contact flow will be divided 25% each.
3. Configure the size of the Sampling Phase. This is where one of the main differences between each mode of operation lies.
- With the Mass mode, you must indicate what percentage of the total contacts to which the workflow applies should be part of the sample, and what percentage of contacts will remain on hold for the Final Phase.
- Optionally, you can indicate a maximum number of contacts to sample, regardless of the previous field.
For example, in a workflow with a mass email campaign for all known contacts, having 12,000 known contacts in total, if it is indicated that the percentage of contacts to sample is 5%, the sampling phase will send about 600 emails to 600 contacts, putting the remaining 11,400 known contacts on hold until the Final Phase.
- With the Transactional mode, you don't have to indicate what percentage of the contacts you want to be part of the sample since all contacts are part of it. Therefore, in this mode, you only need to indicate, if desired, a maximum limit of contacts to sample.
4. Finally, in Set the winning branch, determine when you want the A/B/X test to end its Sampling Phase and move to the Final Phase. Connectif provides three ways to do this:
- By fixed date and time: simply set a fixed date and time, and exactly at that moment, the A/B/X test will end its Sampling Phase and move to the Final Phase.
- After a specified time: similar to the previous option, but instead of setting a fixed date and time, a specific duration is set. For example, "when two days have passed since the Sampling Phase began".
The sampling phase begins when the workflow is activated, not when the "Start A/B/X Test" node is reached. It is important to consider this when configuring the time it takes to declare a winning branch, for example, if users reach the node after several hours or days.
- When the maximum contact limit is reached: that is, when the number of contacts we have set in the "Maximum contact limit to sample" field of the Sampling Phase configuration has been sampled.
5. The A/B/X test will move to the Final Phase as soon as the first of the three selected criteria is met.
If no criteria for a winner are met, the system will determine one after 90 days. Therefore, the maximum duration of a test is 90 days.
2.2. Configuring the Branches of an A/B/X Test
As mentioned earlier, within the configuration of the "Start A/B/X Test" node, you must indicate the number of different branches or alternatives you want to test. However, to configure which outgoing connection from the node corresponds to which branch, it must be configured from the general view of the workflow editor.
To do this, simply add the different nodes that will constitute each variant of the A/B/X Test to the workflow editor and connect them to the "Start A/B/X Test" node, as shown in the following example.
All outgoing connections from the "Start A/B/X Test" node always have an associated branch, but you can change the branch by clicking on the central circle of the connection containing the branch letter.
As a general rule, there should be one outgoing connection for each branch, but if you want more than one connection per branch, it can be done.
2.3. Determining the Winning Branch
Connectif provides a versatile and fully configurable mechanism to determine the winning branch. All branches of a Connectif A/B/X Test have a certain score, always starting at 0. It is up to the Connectif user to add or subtract points to the branches using the "Evaluate A/B/X Branch" node.
The main idea is to use trigger nodes in each branch to add or subtract points to the branch when certain events occur. For example:
- When opening an email: Add 1 point to the branch.
- When clicking on the email: Add 1 point to the branch.
- When unsubscribing from an email: Subtract 10 points from the branch.
- When purchasing: Add as many points to the branch as the number of products purchased.
It is important to note that the winning branch is not exactly the one with the most points but the one with the highest ratio of Points to the number of executions. For example, if an A/B/X Test has the following two branches:
- Branch A: Executed 521 times, assigned 128 points.
- Branch B: Executed 506 times, assigned 126 points.
The ratio for branch A is 0.246 points per execution, while the ratio for branch B is 0.249. Therefore, branch B would be the winner, even though it has fewer points than branch A.
There are several ways to determine which branch is the "winner," but in the end, it will always be the marketing team configuring the campaign who determines which metric of the two branches is the winner for the specific use case we are implementing.
2.4. Examples of Strategies with A/B/X Tests
In the workflow template directory of your Store, you have several pre-configured strategies using A/B/X tests that you can easily apply.
At the end of this article, you will find links to some of them. Additionally, here is a video on how to perform an A/B/X test to optimize your product recommendations:
Access the complete article on A/B/X Tests to optimize product recommender in this link.
Congratulations!
You have reached the end of the lesson.
Do you still have unresolved questions?
Remember that you have our Connectif specialists at your disposal. To contact them, simply open a support ticket by clicking on the blue "Help" button on your dashboard.
Keep learning!
To make the most of your Connectif account, we recommend continuing with the following articles: