EM Analytics
All Case Studies Start a Project

Home / Portfolio / Case Study

Healthcare · Data Platform Migration

Migrating a health insurer's Hadoop platform to Databricks on AWS

Replacing an expensive, failure-prone on-premises Cloudera Hadoop cluster with automated, governed data pipelines in the cloud.

Client
U.S. health insurer
Engagement
Project-based · 6 months
Team
1 data architect, 2 data engineers, 1 BI developer
Platform
AWS · Databricks

Delivered by our founder, Enmanuel Madera, in a previous role. Client details are kept confidential.

01The challenge

The insurer's healthcare data ran on an on-premises Cloudera Hadoop cluster that had become a bottleneck for the business:

  • Expensive to maintain. The cluster required ongoing hardware, licensing and specialist administration.
  • Slow on large data volumes. Processing delays meant data reached the business later than needed.
  • Unreliable pipelines. Some pipelines failed partway through, and others did not run at all.
  • Manual scheduling. Jobs were scheduled with cron, so failures and reruns needed manual intervention.

02Our approach

The data platform was rebuilt on AWS and Databricks, with every component defined as code:

  1. Cloud landing zone on Amazon S3. Healthcare data was moved off Hadoop and landed in S3 as the new storage foundation.
  2. Processing on Databricks. Pipelines were rebuilt as Databricks jobs running on scalable clusters, replacing the fixed-capacity Hadoop jobs.
  3. Governance with Unity Catalog. Processed data was published to Unity Catalog, giving one governed catalog with central access control for sensitive health data.
  4. Orchestration with Apache Airflow. Cron was replaced by Airflow, which schedules the Databricks jobs with dependencies and automatic retries.
  5. Infrastructure as code and CI/CD. Infrastructure, clusters and jobs were deployed with Terraform through Jenkins pipelines, so every environment is built the same way.
  6. Serving the business. Governed tables feed Power BI and Tableau dashboards, analysts and data scientists, and downstream systems.

03Architecture

Target architecture Data moves from the on-premises Cloudera Hadoop cluster into Amazon S3, is processed by Databricks jobs and clusters, and is published to Unity Catalog. From there it feeds Power BI and Tableau, analysts and data science, and downstream systems. Apache Airflow orchestrates the Databricks jobs, and Terraform with Jenkins CI/CD deploys the infrastructure and jobs. ON-PREMISES AWS CLOUD Apache Airflow schedules · dependencies · retries triggers jobs Cloudera Hadoop legacy cluster Amazon S3 landing zone Databricks jobs & clusters Spark processing Unity Catalog governed tables access control Power BI & Tableau Analysts & data science Downstream systems CONSUMERS Terraform + Jenkins CI/CD infrastructure, clusters and jobs deployed as code
Target architecture. Solid arrows show the data flow; dashed arrows show orchestration and deployment.

04Results

Automated scheduling

Airflow replaced cron, so jobs run in the right order with automatic retries instead of manual reruns.

Scalable processing

Databricks clusters handle large healthcare data volumes without the fixed capacity limits of the old cluster.

Governed health data

Unity Catalog gives one place to manage access to sensitive data across BI, analytics and downstream systems.

Repeatable deployments

Terraform and Jenkins deploy infrastructure and jobs the same way every time, reducing configuration drift.

Lower running costs

Moving off on-premises Hadoop reduced the monthly cost of running the platform.

05Technologies

Amazon S3DatabricksApache SparkUnity CatalogApache AirflowTerraformJenkinsPower BITableauCloudera Hadoop (legacy)

Running on an aging Hadoop platform?

We assess your current pipelines and design a migration path to the cloud that fits your data, compliance needs and budget.

Talk to Us About Your Migration