Sidmach launches AI-driven Edustat platform built with Plotly Dash Enterprise and Azure Databricks

Introduction

Sidmach Technologies delivers custom technology solutions that have impacted more than 100 million lives across the African continent over the past 30 years.

Investing in Plotly’s Dash Enterprise and Microsoft’s Azure Databricks empowered the Sidmach data science team to generate new revenue streams and convert manual, paper-based processes into the fully featured EduStat platform, unlocking unique, exploratory data views and insights for governments, academics, funding agencies, students, and families.

Challenge

Sidmach has enjoyed a years-long partnership with the West African Examination Council (WAEC), the regional board that oversees all educational and public interest examinations in West Africa’s English-speaking countries.

The WAEC owns billions of rows of examination performance data, most of which cannot be found elsewhere. These data points span from the regional level down to the individual student level and have implications for public policy, international funding, and personal educational decisions — but barriers to access were in place:

• Manual, paper-based processes for requesting data were slow and could become costly if requesters didn’t know exactly which data to ask for.
• Billions of data records were sitting in warehouses, representing an opportunity for a major process overhaul and new revenue generation.
• A self-service solution would have to transcend the enhancement of WAEC’s existing services and actually equip data requesters with the freedom to explore, select, and pay for data on the same platform.

Solution

The Sidmach team began leveraging PowerBI, but found the EduStat mandate required more than traditional BI offerings, along with a robust future-proof data and analytics architecture to underpin the solution. They needed to get to market quickly. Equipped with Azure Databricks and Dash Enterprise, they leveraged a scalable production data app platform with the functionality and flexibility to create a custom end-user focused solution that delivers the following:

• Beyond BI: Developing the product quickly established that traditional business intelligence solutions would not accomplish the goal to present users with a self-service data platform.
• Developer flexibility: Sidmach data scientists started in a Databricks notebook environment, advancing from there to craft an interactive data application using the Dash Python framework to link UI components with their analytics code, resulting in a thoughtful and custom-built end user experience.
• Customer Success and Support: Sidmach’s team leveraged the guidance of Plotly’s Customer Success team to accelerate their delivery and maximize value from the platform.
• Easy app styling: With the built-in Dash Design Kit GUI, the data science team could quickly style the new app to match their client’s branding without involving front-end engineering, HTML, or CSS.
• Direct-to-user reporting: Dash Enterprise’s Reporting Engine allows each unique end user to save specific data views, export reports as interactive or static PDFs, and even revisit those data views later — a helpful function for expense approval workflows.

Results

Built with Dash Enterprise and underpinned with Azure Databricks, Sidmach delivers value to the WAEC and its downstream data customers via the EduStat platform:

• Faster time to delivery. By owning both the analytics and the application development workflows in Python, Sidmach’s data science team was empowered to deliver a working prototype app from design mockups in three weeks.
• 200 million rows of WAEC data are directly accessible in the web browser, with more on the way, for end users in government, academia, the education system, international funding agencies, industry, and the general public.
• Instant invoicing and transparent cost control for end users, allowing them to preview both the desired data and its prospective cost ahead of time and update it dynamically.
• One-click report generation of both static and interactive PDFs so end users can save, print, and share their selected data with research teams, budget holders, and other stakeholders.
• A curated user journey reflected in the app layout, created with Dash Design Kit, that provides a clean and thoughtful user experience to guide end users through exploratory data analysis.