Intuit Reduces Experiment Runtime More Than 50% with Dash Enterprise

Introduction

Intuit is the global financial technology platform that powers prosperity for more than 100 million customers worldwide. The company behind TurboTax, Credit Karma, Quick- Books, and Mailchimp is continually innovating, and Intuit’s experimentation team is a key driver of those efforts. Among the company’s goals is to increase the pace of innovation by reducing experiment runtime by 50%, which led to its decision to adopt Dash Enterprise for building interactive experimentation tools.

Challenge

Intuit’s experimentation team is made up of data scientists and analysts, which is a core element of the company’s Analytics, Artificial Intelligence and Data (A2D) organization. Team members have several tools for building dashboards. Typically these tools have great graphing functionality and database connectivity, but present challenges for creating and communicating custom statistical analyses:

• Excludes statistical packages - Common dashboarding tools require data scientists to compute statistical analysis beforehand or jump through hurdles to integrate R, Python and custom algorithms.
• Not reusable – The analyst community uses R vignettes and Python Jupyter Notebooks to perform analysis and showcase machine learning methods or custom algorithms. But these tools are not ideal as reusable and interactive dashboards for the less code-savvy user.
• Decentralized - Without a centralized reporting hub, each experiment could be analyzed differently, making it difficult to ensure the consistent application of statistical methods across the organization.

Solution

With Dash Enterprise and some lightweight Python, Intuit’s experimentation team built a suite of experimentation apps. These apps enable users to design, analyze, and even simulate sequential experiments.

A Sample Size Estimation App enables users to design their experiment by specifying experiment standards and offering details of their metric. The app works equally as well for rate-based metrics like customer conversion or continuous metrics like revenue, time-on-site, or page loading times.

The Sequential AB Testing App is the central hub where any experiment can be analyzed. Users simply provide the database location of their experimental data and standards. The app has two main functions:

• Early Stopping of an experiment without p-value hacking, an error that leads to incorrectly concluding that an experiment’s results were not due to random chance. Users may declare whether a statistically meaningful difference was detected.
• Bayesian Inference, including effect size (How much lift against control are conditions generating?) and expected loss (What is the estimated down-side risk of choosing a winner?).

An experiment FAQ App allows users to simulate experiments to ground themselves on the fundamentals of experimentation.

Results

• Analyst Time Savings: With adoption by over 500 users, these tools are enabling all stakeholders at Intuit to design and analyze experiments, freeing up dozens of hours of analyst time each month for other projects.
• Supports Innovation: Dash Enterprise is a proving ground for Intuit’s statistical techniques. The experimentation team collaborates with Intuit’s developers to build the algorithms into internal tools.
• Centralized: Each experiment is properly analyzed. Each analysis has a dedicated, parametrized URL, making analyses easy to share within and beyond the team.
• Faster Experiments: In early results during the company’s “peak 3” tax season in October 2022, Intuit measured a savings of more than 70% because the Sequential AB Testing App detects “home run” and “strikeout” experiments so quickly.