A/B Testing

A/B Testing Experiments work using two sets of variables.

1. The variable you would like to test or change
2. The results you would like to measure

---

1.) Preparing the Metric

Before we start, we need to create a specific metric to be tested. This metric is created with the Data Slicer tool.

Note

All metrics and events MUST occur at least once in your app before the system will recognize it as a legitimate metric to be analyzed. In this example, we use the "ObjectLookedAt" transaction as a metric. To see this option, with the appropriate object name, you must first trigger the exact metric in your app, with the exact object name. In other words, run the app and look at an object named "car" to validate the event in the system.

1. Select "ObjectLookedAt" from the main event field.

2. Change the Any field to Certain mode. This will allow you to select a specific property associated with the ObjectLookedAt event.

3. Select the object you would like to track. In my example, I have a simple object called "cube" in my application.

4. Click the Save button and name your metric appropriately. I named mine "User gazed at Car".

Now we are ready to create the experiment.

---

2.) Creating the Experiment

To get started, first navigate to the Experiments Tab

Click on Create Experiment

After creating a Name and Description for your experiment, you must choose who will be inside this experiment. You can leave this as All and Devices, with no segment selected. This ensures all devices will be a part of this experiment.

Note

The Assignments are Sticky option sets whichever variable a user sees as permanent. For example, if you gave a user a red version of an object instead of a blue version, they would always see the red version.

Next, we set a Key Metric that will be tested. I have chosen "User gazed at Car" which created in step 1.

Now we must create a Tuning Variable. These are variables that are built in to your application which call our system to determine what to serve to a user.

Let's create a Tuning Variable called "CarColor".

Next, create a new Variant. This will be an alternative version of the variable that we serve to the segments that we chose above. The WEIGHT of a variant is the percent of users that this version of the variable will be served to.

Finally, we are ready to create the experiment. Click Create, and click Start on the summary page.

---

3.) Testing the experiment

In place of a car object, we will be turning a cube either red or blue, depending on the Tuning Variable result, and the measuring how often a user looks at the object. Remember, we set our experiment to be NOT sticky, which means we will be able to see different version of the variable every time we start up.

To start, I request the Tuning Variable from the server. Then I set the cube object to the resulting variable.

public string carColor;
public Material carMaterial;

IEnumerator Start () {
    // wait a few seconds for initialization to complete
    yield return new WaitForSeconds (2f);
    Debug.Log ("The color is: " + carColor);

    // Set our carColor string to the result of the tuning variable
    // "red" is our default value
    carColor = CognitiveVR.Tuning.getVar ("CarColor", "red");

    // Now we set the material color to the resulting string. This will be either "red" or "blue".
    if (carColor == "red") {
        carMaterial.color = Color.red;
    }
    if (carColor == "blue") {
        carMaterial.color = Color.blue;
    }

}

Now that we have the Tuning Variable set up, and the experiment running, our cube is randomly turn red or blue.

Now the experiment needs an input. We add the gaze input. To do this, add the DynamicObject component to your object. In this case, we called the cube "car". Be sure to tick the Track Gaze on Dynamic Object option.

4.) View Results

To view results, simply go to the Experiment Tab once again and select your experiment from the list. Your results will appear there after the system contextualizes the data.