Mobile App A/B Testing

Mobile App A/B Testing gives you conversion insight of two or more variants of an element and helps you in taking a decision about which one is better or which one is having a higher conversion rate. For example if you have two different Registration UI A and B in your app and not sure which one will be better to get user registered, A/B testing comes into picture here. You can start with two of these UI together and distribute these randomly to your app users. A/B Testing will give an insight of both of the UI variant and you should be able to know which one is having a higher conversion rate. If UI variant A is having higher conversion rate, you would like to make it a winner. Once A is made winner, all the traffic will be served to A only and you will have better performing element is being served in your app.

To start with A/B testing in your app with App42 platform, here are the steps to follow

bullet Login to AppHQ.

bullet Click on A/B Testing from AppHQ Menu panel and select your app.

CreateTest

bullet Click on Create Test button and Enter Test Name and other information. For example ‘Register Test’ is the Test Name for example explained here.

EnterDetails

bullet Once your test has been created you can create your test variants.

bullet Click on Add Variants button to add your variant name and % of traffic will be served to this test variant. In this case let’s say, we have two registration variant A and B and each have traffic distribution of 50% each.

AddVariants

bullet Click on save changes button.

bullet Now click on Add Variable button to add profile information in a key-value format that you want to attach with this variant. For example if Register screen A has color red, you can save the key as color and value as red in profile of A variant. Similarly you can add other profile information for both variants if you want.

AddVaraibles

bullet Once you are done with creation of test and its variants, call the following methods from your app to execute and see the result.

Execute Test

To execute given test, one has to call execute method inside ABTestService by passing the test name. This method will return an appropriate variant based on the distribution rule defined. For example, If A and B has 50 % distribution each, in first call this will return variant A and its profile info, however in second execution it will return variant B and its profile. Based on the response received, you can display UI and populate it based on its profile information as shown below in the snippet

  • create User Api for Android
  • create User Api for J2ME
  • create User Api for Android
  • create User Api for iOS
  • create User Api for Java
  • create User Api for .NET
  • create User Api for Unity
  • create User Api for Ruby
  • create User Api for Rest
  •  create User Api for WP7/WP8
  • create User Api for Flash
String testName="<Enter TestName>"; 
abTestService.execute(testName, new App42CallBack() {
public void onSuccess(Object response) 
{
	ABTest abTest  = (ABTest)response;
	System.out.println("Test Name  is :  "+abTest.getName());
	System.out.println("Type is :" +abTest.getType());
	System.out.println("Is Active : "+abTest.isActive());
	for(int i=0;i<abTest.getVariantList().size();i++)
	{
		System.out.println("Variant Name : "+abTest.getVariantList().get(i).getName());
		System.out.println("get profile list : "+abTest.getVariantList().get(i).getProfileJSON());	
	}
}
public void onException(Exception ex) 
{
	System.out.println("Exception Message"+ex.getMessage());
}
});
Coming Soon 
public class Callback : App42Callback  
{  
String testName="<Enter TestName>"; 
abTestService.Execute(testName, this);
	void App42Callback.OnException(App42Exception exception)  
	{  
		Console.WriteLine("Exception Message" + exception);  
	}  
	void App42Callback.OnSuccess(Object response)  
	{  
		ABTest abTest = (ABTest) response;  
		Console.WriteLine("Test Name  is :  " + abTest.GetName());
		Console.WriteLine("Type is :" + abTest.GetType());
		Console.WriteLine("Is Active : " + abTest.IsActive());
		for (int i = 0; i < abTest.GetVariantList().Count; i++)
		{
			Console.WriteLine("Name  : " + abTest.GetVariantList()[i].GetName());
			Console.WriteLine("Get profile list : " + abTest.GetVariantList()[i].GetProfileJSON());
		}
	}  
}   
String testName="<Enter TestName>"; 
ABTest abTest=abTestService.execute(testName);
System.out.println("Test Name  is :  "+ abTest.getName());
System.out.println("Type is :" + abTest.getType());
System.out.println("Is Active : "+ abTest.isActive());
Vector variantList = abTest.getVariantList();
for(int i=0;i<variantList.size();i++)
{
	System.out.println("Name : " + ((ABTest.Variant)variantList.elementAt(i)).getName());
	System.out.println("get profile list : "+ ((ABTest.Variant)variantList.elementAt(i)).getProfileJSON());	
}  
var testName = "<Enter TestName>",
abTestObj;
abTestService.execute(testName,{  
	success: function(object) 
	{  
		var result = JSON.parse(object);  
		abTestObj = result.app42.response.abtest;
        console.log("Test Name : " + abTestObj.name);
		console.log("Variant Name is : " + abTestObj.variant.name);
		console.log("ProfileJSON is : " + JSON.stringify(abTestObj.variant.profile));
	},  
	error: function(error) 
	{ 
		console.log("Exception is : " + error)
	}  
});     
String testName="<Enter TestName>"; 
ABTest abTest=abTestService.execute(testName);
System.out.println("Test Name  is :  "+abTest.getName());
System.out.println("Type is :" +abTest.getType());
System.out.println("Is Active : "+abTest.isActive());
for(int i=0;i<abTest.getVariantList().size();i++)
{
	System.out.println("Variant Name : "+abTest.getVariantList().get(i).getName());
	System.out.println("Get profile list : "+abTest.getVariantList().get(i).getProfileJSON());	
}  
String testName="<Enter TestName>"; 
App42Log.SetDebug(true);        //Prints output in your editor console
abTestService.Execute(testName, new UnityCallBack());
public class UnityCallBack : App42CallBack
{
    public void OnSuccess(object response)
    {
       ABTest abTest = (ABTest)response;
       App42Log.Console("Test Name : " + abTest.GetName());
       for (int i=0; i < abTest.GetVariantList().Count; i++) 
		{
			 App42Log.Console("Variant Name : " + abTest.GetVariantList()[i].GetName());
			 App42Log.Console("ProfileJSON  : " + abTest.GetVariantList()[i].GetProfileJSON());
		}
	}

	public void OnException(Exception e)
	{
		App42Log.Console("Exception : " + e);
	}
}
String testName="<Enter TestName>"; 
ABTest abTest = abTestService.Execute(testName);
Console.WriteLine("Test Name  is :  " + abTest.GetName());
Console.WriteLine("Type is :" + abTest.GetType());
Console.WriteLine("Is Active : " + abTest.IsActive());
for(int i=0; i < abTest.getVariantList().Count; i++)
{
	Console.WriteLine("Variant Name : "+abTest.GetVariantList()[i].GetName());
	Console.WriteLine("Get profile list : "+abTest.GetVariantList()[i].GetProfileJSON());	
}  
Not Available
Not Available
var testName:String="<Enter TestName>";
abTestService.execute(testName,new callback());
class callback implements App42CallBack
{	
	public function onException(exception:App42Exception):void
	{
		trace("Exception Message " + exception);
	}
	
	public function onSuccess(response:Object):void
	{					
		if(response is ABTest)
		{		
			var abtest:ABTest = ABTest(response);
			trace("Test Name  is :  "+abtest.getName());
			trace("Type is :" +abtest.getType());
			trace("Is Active : "+abtest.IsActive());
			for(var i:int=0;i<abtest.getVariantList().length;i++)
			{
				trace("Variant Name : "+abtest.getVariantList()[i].getName());	
				var object:Object = com.adobe.serialization.json.JSON.decode(Variant(abtest.getVariantList()[i]).getProfileJSON());
				trace("get profile list : "+Util.toString(object));
			}						
		}   	
	}
}
Not Available

Tracking Conversion/Goal Reached

If the user takes desired action in the shown variant version, goalReached method has to be called to track the conversion. For example, if the user has done the registration from variant A screen, it should consider as conversion and goalReached method needs to get called in this event by passing the variant name (A in this case). Snippet for the same is shown below.

  • create User Api for Android
  • create User Api for J2ME
  • create User Api for Android
  • create User Api for iOS
  • create User Api for Java
  • create User Api for .NET
  • create User Api for Unity
  • create User Api for Ruby
  • create User Api for Rest
  •  create User Api for WP7/WP8
  • create User Api for Flash
String testName="<Enter TestName>";
String variantName="<Name Of the variant>";
abTestService.goalAchieved(testName,variantName,new App42CallBack() {
public void onSuccess(Object response)
{
	ABTest abTest  = (ABTest)response;
	System.out.println("Test Name  is :  "+abTest.getName());
	System.out.println("Type is :" +abTest.getType());
	System.out.println("Is Active : "+abTest.isActive());
	for(int i=0;i<abTest.getVariantList().size();i++)
	{
		System.out.println("Variant Name : "+abTest.getVariantList().get(i).getName());
		System.out.println("get profile list : "+abTest.getVariantList().get(i).getProfileJSON());	
	}
}
public void onException(Exception ex)
{
	System.out.println("Exception Message " + ex.getMessage());
}
});  
Coming Soon
public class Callback : App42Callback  
{  
String testName="<Enter TestName>"; 
String variantName="<Name Of the variant>";
abTestService.GoalAchieved(testName, variantName, this);
	void App42Callback.OnException(App42Exception exception)  
	{  
		Console.WriteLine("Exception Message" + exception.GetMessage());  
	}  
	void App42Callback.OnSuccess(Object response)  
	{  
		ABTest abTest = (ABTest) response;  
		Console.WriteLine("Test Name  is :  " + abTest.GetName());
		Console.WriteLine("Type is :" + abTest.GetType());
		Console.WriteLine("Is Active : " + abTest.IsActive());
		for (int i = 0; i < abTest.GetVariantList().Count; i++)
		{
			Console.WriteLine("Name  : " + abTest.GetVariantList()[i].GetName());
			Console.WriteLine("Get profile list : " + abTest.GetVariantList()[i].GetProfileJSON());
		}
	}  
} 
String testName="<Enter TestName>"; 
String variantName="<Name Of the variant>";
ABTest abTest = abTestService.goalAchieved(testName, variantName);
System.out.println("Test Name  is :  "+ abTest.getName());
System.out.println("Type is :" + abTest.getType());
System.out.println("Is Active : "+ abTest.isActive());
Vector variantList = abTest.getVariantList();
for(int i=0;i<variantList.size();i++)
{
	System.out.println("Name " + ((ABTest.Variant)variantList.elementAt(i)).getName());
	System.out.println("Get profile list : "+ ((ABTest.Variant)variantList.elementAt(i)).getProfileJSON());	
}      
var testName = "<Enter TestName>",
variantName="<Name Of the variant>",
abTestObj;
abTestService.goalAchieved(testName, variantName,{  
	success: function(object) 
	{  
		var result = JSON.parse(object);  
		abTestObj = result.app42.response.abtest;
        console.log("Test Name : " + abTestObj.name);
		console.log("Variant Name is : " + abTestObj.variant.name);
		console.log("ProfileJSON is : " + JSON.stringify(abTestObj.variant.profile));
	},  
	error: function(error) 
	{ 
		console.log("Exception is : " + error)
	}  
});     
String testName="<Enter TestName>";
String variantName="<Name Of the variant>";
ABTest abTest=abTestService.goalAchieved(testName, variantName);
System.out.println("Test Name  is :  "+abTest.getName());
System.out.println("Type is :" +abTest.getType());
System.out.println("Is Active : "+abTest.isActive());
for(int i=0;i<abTest.getVariantList().size();i++)
{
	System.out.println("Variant Name : "+abTest.getVariantList().get(i).getName());
	System.out.println("Get profile list : "+abTest.getVariantList().get(i).getProfileJSON());	
}  
String testName="<Enter TestName>"; 
String variantName="<Name Of the variant>";
App42Log.SetDebug(true);        //Prints output in your editor console
abTestService.GoalAchieved(testName, variantName, new UnityCallBack());
public class UnityCallBack : App42CallBack
{
    public void OnSuccess(object response)
    {
       ABTest abTest = (ABTest)response;
       App42Log.Console("Test Name : " + abTest.GetName());
       for (int i=0; i < abTest.GetVariantList().Count; i++) 
		{
			 App42Log.Console("Variant Name : " + abTest.GetVariantList()[i].GetName());
			 App42Log.Console("ProfileJSON  : " + abTest.GetVariantList()[i].GetProfileJSON());
		}
	}

	public void OnException(Exception e)
	{
		App42Log.Console("Exception : " + e);
	}
}
String testName="<Enter TestName>"; 
String variantName="<Name Of the variant>";
ABTest abTest=abTestService.GoalAchieved(testName, variantName);
Console.WriteLine("Test Name  is :  " + abTest.GetName());
Console.WriteLine("Type is :" + abTest.GetType());
Console.WriteLine("Is Active : " + abTest.IsActive());
for(int i=0; i < abTest.getVariantList().Count; i++)
{
	Console.WriteLine("Variant Name : "+abTest.GetVariantList()[i].GetName());
	Console.WriteLine("Get profile list : "+abTest.GetVariantList()[i].GetProfileJSON());	
}         
Not Available
Not Available
var testName:String="<Enter TestName>";
var variantName:String="<Name Of the variant>";
abTestService.goalAchieved(testName, variantName, new callback());
class callback implements App42CallBack
{	
	public function onException(exception:App42Exception):void
	{
		trace("Exception Message " + exception);
	}
	
	public function onSuccess(response:Object):void
	{					
		if(response is ABTest)
		{		
			var abtest:ABTest = ABTest(response);
			trace("Test Name  is :  "+abtest.getName());
			trace("Type is :" +abtest.getType());
			trace("Is Active : "+abtest.IsActive());
			for(var i:int=0;i<abtest.getVariantList().length;i++)
			{
				trace("Variant Name : "+abtest.getVariantList()[i].getName());	
				var object:Object = com.adobe.serialization.json.JSON.decode(Variant(abtest.getVariantList()[i]).getProfileJSON());
				trace("get profile list : "+Util.toString(object));
			}						
		}   	
	}
}
Not Available

View Report and Make Winner

You can analyze the conversion of each variant in dashboard as shown below. This report will help you in making winner for variant among A and B. Variant which has highest conversion ratio is the recommended winner. You can make a variant winner from selecting the radio button against that particular variant. Once variant is selected as a winner, distribution for that variant would be 100% and all the request will be served to that variant only. For example if A is a winner in this case, execute the method will always return variant A as execution result.

ABTest

More Details?

bullet Go To ABTest API Usage Guide