A/B Testing
A/B Testing Experiments work using two sets of variables.
- The variable you would like to test or change
- The results you would like to measure
1.) Preparing the Metric
Before we start, we need to create a specific metric to be tested. This metric is created with the Data Slicer
tool.
Note
All metrics and events MUST occur at least once in your app before the system will recognize it as a legitimate metric to be analyzed. In this example, we use the "ObjectLookedAt" transaction as a metric. To see this option, with the appropriate object name, you must first trigger the exact metric in your app, with the exact object name. In other words, run the app and look at an object named "car" to validate the event in the system.
-
Select "ObjectLookedAt" from the main event field.
-
Change the
Any
field toCertain
mode. This will allow you to select a specific property associated with the ObjectLookedAt event. -
Select the object you would like to track. In my example, I have a simple object called "cube" in my application.
-
Click the
Save
button and name your metric appropriately. I named mine "User gazed at Car".
Now we are ready to create the experiment.
2.) Creating the Experiment
To get started, first navigate to the Experiments Tab
Click on Create Experiment
After creating a Name and Description for your experiment, you must choose who will be inside this experiment. You can leave this as All
and Devices
, with no segment selected. This ensures all devices will be a part of this experiment.
Note
The Assignments are Sticky
option sets whichever variable a user sees as permanent. For example, if you gave a user a red version of an object instead of a blue version, they would always see the red version.
Next, we set a Key Metric
that will be tested. I have chosen "User gazed at Car" which created in step 1.
Now we must create a Tuning Variable
. These are variables that are built in to your application which call our system to determine what to serve to a user.
Let's create a Tuning Variable
called "CarColor".
Next, create a new Variant. This will be an alternative version of the variable that we serve to the segments that we chose above. The WEIGHT
of a variant is the percent of users that this version of the variable will be served to.
Finally, we are ready to create the experiment. Click Create
, and click Start
on the summary page.
3.) Testing the experiment
In place of a car object, we will be turning a cube either red or blue, depending on the Tuning Variable
result, and the measuring how often a user looks at the object. Remember, we set our experiment to be NOT sticky, which means we will be able to see different version of the variable every time we start up.
To start, I request the Tuning Variable
from the server. Then I set the cube object to the resulting variable.
public string carColor;
public Material carMaterial;
IEnumerator Start () {
// wait a few seconds for initialization to complete
yield return new WaitForSeconds (2f);
Debug.Log ("The color is: " + carColor);
// Set our carColor string to the result of the tuning variable
// "red" is our default value
carColor = CognitiveVR.Tuning.getVar ("CarColor", "red");
// Now we set the material color to the resulting string. This will be either "red" or "blue".
if (carColor == "red") {
carMaterial.color = Color.red;
}
if (carColor == "blue") {
carMaterial.color = Color.blue;
}
}
Now that we have the Tuning Variable
set up, and the experiment running, our cube is randomly turn red or blue.
Now the experiment needs an input. We add the gaze input. To do this, add the DynamicObject
component to your object. In this case, we called the cube "car". Be sure to tick the Track Gaze on Dynamic Object
option.
4.) View Results
To view results, simply go to the Experiment Tab once again and select your experiment from the list. Your results will appear there after the system contextualizes the data.