Intuit Reduces Experiment Runtime More Than 50% with Dash Enterprise
- The experimentation team behind a leading global financial technology platform reduces experiment runtime more than 50% by leveraging Python and Dash Enterprise.
- Deploying interactive apps enables stakeholders across the organization to design and analyze experiments in a standardized way, resulting in significant time savings for analysts each month.
- Flexible use cases: The team also created simulation tools to teach the fundamentals of experimentation.
- Empowerment: A couple of data scientists built a suite of tools and services that has been adopted by more than 500 users within Intuit’s Analytics, Artificial Intelligence and Data (A2D) organization.
Intuit is the global financial technology platform that powers prosperity for more than 100 million customers worldwide. The company behind TurboTax, Credit Karma, Quick- Books, and Mailchimp is continually innovating, and Intuit’s experimentation team is a key driver of those efforts. Among the company’s goals is to increase the pace of innovation by reducing experiment runtime by 50%, which led to its decision to adopt Dash Enterprise for building interactive experimentation tools.
"Dash Enterprise marries the strengths of many tools into one — design beautiful dashboards and apply statistical algorithms behind the scenes. Adopting Dash Enterprise allows us to create tools and services that everyone at the company can leverage without the typical development cycle."
Russ Zaliznyak - Principal Data Scientist and Experimentation Team Lead, Intuit
Intuit’s experimentation team is made up of data scientists and analysts, which is a core element of the company’s Analytics, Artificial Intelligence and Data (A2D) organization. Team members have several tools for building dashboards. Typically these tools have great graphing functionality and database connectivity, but present challenges for creating and communicating custom statistical analyses:
- Excludes statistical packages - Common dashboarding tools require data scientists to compute statistical analysis beforehand or jump through hurdles to integrate R, Python and custom algorithms.
- Not reusable – The analyst community uses R vignettes and Python Jupyter Notebooks to perform analysis and showcase machine learning methods or custom algorithms. But these tools are not ideal as reusable and interactive dashboards for the less code-savvy user.
- Decentralized - Without a centralized reporting hub, each experiment could be analyzed differently, making it difficult to ensure the consistent application of statistical methods across the organization.
With Dash Enterprise and some lightweight Python, Intuit’s experimentation team built a suite of experimentation apps. These apps enable users to design, analyze, and even simulate sequential experiments.
A Sample Size Estimation App enables users to design their experiment by specifying experiment standards and offering details of their metric. The app works equally as well for rate-based metrics like customer conversion or continuous metrics like revenue, time-on-site, or page loading times.
The Sequential AB Testing App is the central hub where any experiment can be analyzed. Users simply provide the database location of their experimental data and standards. The app has two main functions:
- Early Stopping of an experiment without p-value hacking, an error that leads to incorrectly concluding that an experiment’s results were not due to random chance. Users may declare whether a statistically meaningful difference was detected.
- Bayesian Inference, including effect size (How much lift against control are conditions generating?) and expected loss (What is the estimated down-side risk of choosing a winner?).
An experiment FAQ App allows users to simulate experiments to ground themselves on the fundamentals of experimentation.
Intuit's sample size calculator data app
Sequential AB Testing data app
Intuit's experiment FAQ data app allows users to simulate experiments.
- Analyst Time Savings: With adoption by over 500 users, these tools are enabling all stakeholders at Intuit to design and analyze experiments, freeing up dozens of hours of analyst time each month for other projects.
- Supports Innovation: Dash Enterprise is a proving ground for Intuit’s statistical techniques. The experimentation team collaborates with Intuit’s developers to build the algorithms into internal tools.
- Centralized: Each experiment is properly analyzed. Each analysis has a dedicated, parametrized URL, making analyses easy to share within and beyond the team.
- Faster Experiments: In early results during the company’s “peak 3” tax season in October 2022, Intuit measured a savings of more than 70% because the Sequential AB Testing App detects “home run” and “strikeout” experiments so quickly.
Intuit is the global financial technology platform that powers prosperity for the people and communities the company serves. Serving more than 100 million customers worldwide with TurboTax, Credit Karma, QuickBooks, and Mailchimp, the company believes that everyone should have the opportunity to prosper.
More user stories like this
CIBC Drives ML-Powered Quant Analytics with Dash Enterprise and Databricks
How Crant Automated AI-driven Insight for Brand Building with Dash
NeuroBlu Analytics Platform Delivers Flexibility and Data-Driven Insight to Healthcare Stakeholders