A/B Testing in Javascript with Canvass

A/B testing is a popular method used to identify new changes or features which increase some metric (often the conversion rate in our industry). It works by creating a variant of the website which is served to only a subset of the users. Its performance is then measured against the current version of the website by comparing, for example, the conversion rate of users in each group. If the users who saw the variant spend more or more often, then we can be confident that the new change will be successful once delivered to everyone.

There are many services out there which provide A/B testing functionality (for example: Qubit, Maxymiser and Optimizely). The easiest way to use most of these services is to include a snippet of javascript at the top of your page. You can then build your variants in their online editor. The service takes care of the rest through the javascript snippet, such as pushing different users to a variant (traffic allocation) and tracking the performance of each (data tracking).

We decided to do things a little differently on the Vegas product by building our A/B tests directly in our codebase. This was for a few reasons:

1. We have quite a complex javascript front-end. Our older codebase uses React and Fluxible, and our newer one uses React, Typescript and Alt. Building variants in another service caused us issues when it was forcing us to modify the DOM outside of our application. We seemed to be spending more time fiddling around with hacks to make them work with our application rather than building the test itself.

2. We have great engineering standards and a good delivery pipeline which we didn’t want to bypass by using a third-party service. Instead, we wanted to use good coding standards, build trustworthy and well-tested features and use a smooth release process in A/B testing, too.

3. We take security seriously. Adding any third party javascript to your application should be done with thought. We found that by doing things internally we could omit these security concerns.

However, building a whole A/B testing service from scratch would take a lot of work. There are some really good service providers out there so we wanted to take advantage of the platforms that they offer, but still keep the benefits described above. To achieve this, we created Canvass.

Canvass provides a framework for us to write A/B tests in our React application, while optionally integrating with a provider (in our case, Qubit). Using Canvass, you define an experiment by its variants and when it should be displayed to a user. It provides an interface to the provider, where we can use them for the good stuff: Traffic allocation and data tracking.

Canvass helps your experiments and codebase stay simple, decoupled and easy to manage. It is event-based, which gives you a lot of flexibility when implementing a test. You simply listen to events fired from Canvass, and react to them accordingly. The easiest way to demonstrate is by example:

import canvass from 'canvass';
import Experiment from 'canvass/distribution/Experiment';

...
let myExperiment = new Experiment('MyExperiment', triggers, variants);

canvass.addExperiment(myExperiment);

This is how you register a new experiment in your application. Then, in your view layer, you would simply listen to the ACTIVE event and render the view accordingly:

canvass.on('MyExperiment.ACTIVE', () => {
    const display = canvass.getExperiment('MyExperiment').getVariant());
    // Render display object here
});

This leaves the responsibility of rendering the variant up to your application, which is exactly the decoupling we were looking for. This means Canvass becomes a simple, abstract and reusable library which can be used across any of our products that need to manage javascript A/B tests. It might also be useful for your web application, which is why we just open-sourced it on GitHub.

You can find out more about Canvass and how to use it at the Canvass GitHub repo.