A/B/X Testing


Este artigo está atualmente disponível em inglês e espanhol. Você pode usar esta documentação enquanto ela não estiver disponível em seu idioma.

This article explains the use of the "Start A/B/X Test" node to create campaigns. This node is used to try different strategies and see which one performs better within a percentage of our contacts, so that, after the test, we can fully implement them.

"Start A/B/X test" node

This node can be found in the action nodes section. It's grey by default, since it's part of the system integration of Connectif.




This is a special node, in the sense that it's use imposes certain restrictions to the workflow:

  • Only one "start A/B/X test" node allowed per workflow.
  • Every connection that comes out of the node belongs to a test path.
  • It's not possible to interconnect the paths coming out of the node.
  • It's not possible to directly connect nodes that came before "start A/B/X test" to nodes that are part of the branches.
  • If a workflow has this node, it will always be labeled as a classic workflow upon activation.

Inner workings

The main goal is, as previously discussed, to see which type of strategy or content is more effective among a sample of our contacts. When the test is done, you can then extend that strategy to whole segments of contacts from your list. By doing it this way, we'll optimice our content strategy and get a better conversion rate.

To create these tests, we need to know the difference between the two types: "All at once" and "On the go".

  • "All at once" actions: refers to campaigns that require an action from a huge amount of contacts at a determined time frame. A good example could be to send a promotional bulletin via email. 
  • "On the go" actions: these are actions that are executed in specific moments individually. They depend on only one contact. This could be the showing of a discount popup after a specific action on the site. Because of that, we need to have a trigger type node before the "start A/B/X test" node. 

We need to understand the differences, because the two types make the workflow behave in different ways depending on which one we use. We can see it with the next two examples:

  • Example of an "All at once" action:

We want to send a promo email to our whole list of 10.000 contacts. We have two different designs, so we want to send 500 each to two groups of contacts of also 500. We'll wait one day to see which design gets the better statistics in terms of clicks. Once that day is over, the email with the better numbers will be sent to the remaining 9.000 contacts.

With this example, we need to send 1.000 emails to our contacts. 50% will go to the first design and the remaining 50% will get the second design. The other 9.000 contacts will be waiting. During the one day interval, the system will keep track of the number of clicks for each email so the day after, the email with the best numbers will be sent to the remaining 9.000.

The main difference between this and the "on the go" method is that here, we'll leave contacts waiting.

  • Example of an "on the go" action:

We want to show a push notification to the contacts that abandon their cart. The push will contain a discount, to encourage them to complete the purchase. We have two different designs for the message and we'll like to try which one works better in a span of 3 days. After that, the design with better data will be the one that gets send to every contact that abandons a cart.

At first, we can't anticipate how many people will abandon their cart, so it doesn't make sense to send 500 of each design to two groups of contacts, like in the previous example. What we'll do is send one of the designs at random to everyone that abandons a cart, with a 50% chance of getting design A or design B. There's not waiting involved, since everyone in those three days will receive a notification with one design or the other. After that time passes, we'll determine which design has the best conversion rate and send it to everyone contact that performs the action.

A/B/X Test Phases

A/B/X Test are composed of two simple phases, as we can see from the previous examples:

  • Sampling phase, in which we try different strategies with some of our contacts that allow us to gather data that indicates which ones have performed better.
  • Final phase, which comes after the sampling, when we already know which strategy works better and are ready to use it on our complete contacts list.

An A/B/X test always starts in the sampling phase. The moment that it's determined which strategy is better, it goes to the final phase. That moment is key, and you have to choose the right one to make your own conclusions and select the best possible course of action. Remember that a good strategy is not guaranteed to work every single time, since good marketing strategies today could be out of date tomorrow. 

We can see in which phase of the test we are by looking at the bottom part of the "start A/B/X test" node:



Node configuration

The most important part of the configuration is at the top of the settings page. It's focused on the inner workings of the node, which we have explained in the previous section.



The type of test can't be changed manually, since it's the node itself that determines the type, according to the structure of the workflow. If we know which type we want for our strategy, here we can check if we're creating the type of workflow that we had in mind.

In the configuration, we can also select the number of paths or alternatives that we want to try. The paths are always automatically named after A, B, C, D or E. Contacts are equally spread among all the paths. So, if you only have two, A and B, they'll both get 50% of contacts. If you have four, A, B, C and D, each path will have 25% of the contacts.

Below this, we can change the size of the sampling. Here lies one of the main differences between the two types of test:

  • In the "all at once" type, we have to indicate which percentage of the total contacts we want to be part of the sampling, and which percentage will wait for the final phase (100% minus the percentage that was choose as sample). Alternatively, we can decide a maximum amount of contacts to sample, independent to the previous option. Let's see it with an example: we want to create a workflow for an email campaign that will reach all of our known contacts. If I have 12.000 known contacts and I select a 5% sample, that makes it so 600 emails will be sent to 600 contacts. That means the remaining 11.400 will be waiting for the final phase.  
  • In the "on the go" type, we don't have to indicate which percentage of contacts we want as part of the sampling. That's because, as it was explained previously, in this type all the contacts are part of the sample. In this case, we can only indicate a maximum amount of contacts to sample.



Lastly, in the bottom section of the settings, we can determine when the test will end the sampling phase and move on to the final phase. This will determine the winning path.

Connectif offers three ways of doing it:

  • Date and time achieved: we'll simply establish a fixed date and time. When it's reached, the test will stop the sampling and will go to the final phase. 
  • Time elapsed since sampling: similar to the previous option, but instead of establishing a date, it's an interval. For example: "two days since the sampling started".
  • Maximum limit of contacts reached: when the pre-established amount of contacts it's reached, the sampling phase will stop.

The test will move on to the final phase as soon as one of the three previous criteria gets fulfilled.

Test A/B/X Path

As we've stated before, we can choose the number of branches or alternatives that we want to try. To set up which connection goes with each path, we'll do it in the workflow overview. 



All the connections that come from the "start A/B/X test" always have a branch associated (A by default). We can change the letter of that path by clicking inside the circle that contains the letter.

As a general rule, there should always be one connection per path, but if we want more, we can do it, like in the following image:



How to determine the winning path

We've talked about how a path wins in the final phase, but we haven't discussed how it's actually determined. There are different ways to decide which one is the winning path, but, at the end of the day, we are the ones who decide which is the best option. 

Because of that, Connectif provides a tool that is easy to use to determine which one is the winning branch. Every path has points, always starting with 0. We, as the creators of the workflow, will add or deduct points to each paths via the "evaluate A/B/X Branch" node.

The main idea is to use trigger nodes in each path, for them to add or deduct points when certain events happen- For example:

  • On email open -> Add 1 point to the branch
  • On link click -> Add 1 point to the branch
  • On unsubscribe -> Deduct 10 points to the branch
  • On Purchase -> Add as many points to the branch as products were purchased

Lastly, keep in mind that the winning branch is not always the branch with more points, but the one that has a better point ratio in regards to the number of executions. So, if a test has the following two paths:

Branch A: Executed 521 times, 128 points assigned. Branch B: Executed 506 time, 126 points assigned.

The ratio on branch A is 0,246 for each execution. Meanwhile, the branch B ratio is 0,249. So, the winning branch would be B, even though it has less overall points than branch A.

"Evaluate A/B/X Branch" node

To add or deduct points to a branch, we'll use the node "evaluate A/B/X branch". We can find this node under the "actions" section, on the system category. 



The set up of this node is simple, since the only option is to add the number of points that we want to go to a branch when an action is executed.



To set this node up, it needs to be connected to a branch coming from the "start A/B/X test" node. Also, we can choose if we want to add a fixed amount of points (fixed value option) or if we want to add an amount that comes from any of the previous nodes in the workflow (imported data option).

Esse artigo foi útil?
Usuários que acharam isso útil: 1 de 1