Transition of SQL Data Warehouse into Databricks

Why?
The data layers used by most of our report users today are not scalable for more than one year. With addition of new acquisitions as well as growing existing business into a data driven business, requires more history and larger capacity.

How?
By organizing the data from our systems through Kafka, and selecting the data that we need for creating history. Then running them in parallel and slowly transitioning from reading the data from the DWH to reading it from Databricks instead.

What? - Definition of Done
This initiative will be done once we are able to recreate the most important reports based on fetching the data from Databricks instead of the MySQL DWH.