The journey is indeed real, and it’s happening. Over many years, I’ve been deeply engrossed in building and managing data warehouses, with one aim in mind – to provide reliable reporting that empowers end-users to make informed decisions.

Throughout these years, I’ve witnessed a myriad of changes. We began with Microsoft SQL Server, Integration Services, and Analysis Services (focusing on multidimensional MDX at that time). However, we’ve now reached a stage where we simply can’t ignore the capabilities of platforms such as Azure, AWS, and Google.

About two years ago, I transitioned from on-premise environments to Microsoft Azure. While many elements of this transition went smoothly, there were some aspects that did not. Consequently, I’ve been exploring alternative solutions like Databricks, Snowflake, together with dbt (Data Build Tool). Each of these tools has its advantages and drawbacks, which I won’t delve into at the moment.

I found myself appreciating the open format Delta Lake, which aligns quite nicely with Databricks. However, getting a team mostly composed of individuals from a Data Warehousing background, primarily using SQL, to embrace Databricks seemed challenging. After all, you still need skilled individuals to implement and maintain these solutions.

Dbt integrates remarkably well with Databricks and allows users to employ SQL in creating the Data Lakehouse. It appears to be a solution, but it’s a double-edged sword – another tool, while offering a lot of simplicity, it’s also adding complexity to the landscape and only being a solution for the SQL Warehouse part in Databricks. Still, I love Databricks and their innovative approach!

Then there’s the pairing of dbt and Snowflake, which seems to operate flawlessly. Snowflake is very intuitive, especially for people coming from ‘traditional’ Data Warehousing. Unfortunately, many developers resist using Snowflake for various reasons, perhaps mainly because it’s not perceived as ‘cool’ as Databricks (we engineers do enjoy some coding, don’t we?).

Now, you might be wondering, what is a Data Lakehouse, and why is it beneficial? The Data Lakehouse is an innovative concept that combines the best elements of data lakes and data warehouses, offering structured and unstructured data handling, real-time and historical data capabilities, and high-level analytics, all in one place. Its open nature supports various data types, and its high-performance query capabilities enable sophisticated analytical tasks. The Data Lakehouse is designed to provide the benefits of a data warehouse, such as ACID transactions, schema enforcement, and BI workloads, with the low-cost scalable storage of a data lake. It empowers businesses to harness more insights from their data, improve decision making, and ultimately drive more value from their data assets.

So, what has been my focus for the past year? I’ve been exploring the Data Lakehouse concept, and after digesting the e-book “5 Steps to a Successful Data Lakehouse” by Bill Inmon, I’ve crafted a framework using pipelines, metadata, and notebooks to architect the Data Lakehouse, adhering to best practices. This versatile framework can run just about anywhere, be it Databricks, Azure Synapse, on-premise, or now, Microsoft Fabric.

This Data Lakehouse serves as the foundation upon which you can construct your machine learning models, Power BI reports, and more.

What about real-time data? It’s not an issue at all! There exist numerous ways to load and report on this data, some are straightforward, while others are more intricate.

Up to this point, my explanation may seem a bit abstract, but I’m eager to see this solution come to life in Microsoft Fabric. Microsoft is now also embracing the open Delta format. This platform where everything is combined might just be the ultimate solution I’ve been patiently awaiting.

If you are exploring ways to modernize your data platform, or perhaps adding some functionalities to your existing stack, I would be thrilled to discuss strategies for reaching the next step without having to replace your entire data platform or undergo a massive migration, unless that’s what you’re aiming for.

Indeed, I’ve developed a vast arsenal of tools, scripts, and notebooks ready to accelerate your insights from reporting. But most importantly, I want to initiate the conversation and assist you in conceptualizing the next step and how to reach it.