AB Testing
Framework

for a Health & Fitness Mobile App Developer

#Analytics
#Data Management
#Real Time Integration
Get free consultation

Duration

The project was implemented over
3
months
ensuring an effective implementation of the AB testing framework.

client

Rapidly growing
EU
based
Mobile App Development company specializing in the Health & Fitness domain.

Tools

tools
tools
tools
tools

Project Overview

The client aimed to develop a robust AB testing framework to analyze and implement experiments based on Customer Lifetime Value (LTV), moving beyond the initial focus on Average Revenue Per User (ARPU) and Conversion Rate (CVR).

Challenge

Initially, the client relied on Firebase for AB testing, which limited their testing capabilities. They needed a more sophisticated approach to evaluate long-term customer value and make data-driven decisions. The challenge was to develop a framework that incorporated both Frequentist and Bayesian testing methodologies, enabling a deeper understanding of customer behavior and app profitability.

Solution

We developed an A/B testing engine designed to handle high-traffic environments while optimizing for both binary and continuous target variables. By implementing this engine in real-time, we were able to significantly improve the entire A/B testing process for our client. The engine’s ability to operate under heavy load, combined with its flexibility in targeting different types of variables, resulted in more accurate and efficient test outcomes. This advancement not only streamlined the decision-making process but also provided our client with deeper insights and more actionable data, ultimately enhancing their overall testing strategy.

solution

Implementation

01

Backend Admin Console:

Developed in-house by the client with guidance from Stellans, this served as the main control center for the AB testing system. Our team worked hand in hand with the Client, offering valuable insights and recommendations to optimize the functionality and usability of the Backend Admin Console.

02

Traffic Distribution Engine:

A python-based solution hosted on AWS Fargate, using DynamoDB for storing AB testing statistics and advanced features like caching, load balancing, and real-time monitoring.

03

Business Intelligence Layer:

By utilizing Looker Studio and Google BigQuery, we created a dashboard that served as the primary platform for daily analysis of A/B test outcomes and realized results.

Results

Time To Market

Significantly Reduced time to finish an AB test through an ability to optimize traffic allocation during tests

Improved Accuracy

As we moved from ARPU AB tests to LTV based tests, we have been able to receive much more accurate and correct results.

Usability

With the implementation of modern UI and BI platform that supports the AB testing process, we were able to significantly reduce load on all involved stakeholders that lead to overall improvement in results.

Related case studies

    Get a Free Consultation

    Let's talk about
    your project
    * You can attach up to 3 files, each up to 3MB, in doc, docx, pdf, ppt, or pptx format.