Low Code Lakehouse in Microsoft Fabric

 

The tech giant's one-stop-shop for data covers everything from data movement and data engineering to real-time analytics, offering a holistic experience for data integration. An end-to-end scenario tutorial is provided, aiding in enhancing your understanding of Fabric environment, and the different experiences and interfaces of this platform.

Take note though; Microsoft's analytics solution is currently in preview mode. However, this does not skimp on its functionalities. From dealing with structured data analytics needs in modern data warehouses to handling big data analytics need with data lakehouses, enterprises have traditionally worked in parallel with these two systems.

This method, however, leads to the creation of silos, unnecessary data duplication, and hence an increased total cost of ownership. Microsoft Fabric, or the unification of data stores, circumvents these issues by standardizing its data on a Delta Lake format, resulting in the elimination of silos, removal of data duplication, and, therefore, a drastic drop in total cost of ownership.

The flexibility granted by this solution empowers you to either maintain lakehouse or data warehouse architectures separately or combine them to optimize their benefits. Following the steps in the provided tutorial would lead to the construction of a lakehouse for a fictional retail organization from start to finish, using the medallion architecture.

This entails having layers of data- the bronze layer for raw data, the silver layer for validated and deduplicated data, and the gold layer for highly refined data. The same principle embodies the construction of a lakehouse for any organization in any industry.

Fictional Scenario and Architecture Overview

In a hypothetical setting, a model developer at 'Wide World Importers' completed the steps laid out in the tutorial. It starts off with creating a basic workspace to complex procedures like setting up end-to-end lakehouses for organizations. You would also get to learn to create Power BI reports for time-critical sales data analysis.

The lakehouse’s end-to-end architecture consists of several components. From the initial data source, Fabric enables smooth connection to Azure Data Services and other cloud-based platforms, making the data ingestion process streamlined and efficient.

The data ingestion process in the tutorial is facilitated by more than 200 native connectors incorporating a user-friendly drag-and-drop transformation of dataflow into the Fabric pipeline. Furthermore, Fabric's Shortcut feature helps connect to existing data, mitigating the need to copy or move it.

Transform and store is a significant step where an all-around analytics platform standardizes on the Delta Lake format. All Fabric engines are compatible and can access and manipulate the same dataset stored in OneLake, thus preventing any redundant copies of data.

OneLake offers the flexibility to set up lakehouses using a medallion architecture or a data mesh according to your organization's requisites. It also gives you the choice between a low-code or no-code environment for data transformation via dataflows or notebook/Spark for a more code-heavy experience.

 

Data consumption is another critical step. Power BI can pull data from the Lakehouse to visualize and create reports. Plus, when a Lakehouse is established, a corresponding Warehouse is also automatically created with a similar name. This Warehouse provides users with the TDS/SQL endpoint functionality for seamless connectivity and data querying from other reporting tools.

Beyond the Tutorial: Building on Real-World Applications

While the tutorial focuses on sample data from a hypothetical company, real-world applications would necessitate pulling data from various sources and line-of-business applications into a lakehouse. The applications would require different transformations stages, depending on the data model employed.

In real-life scenarios, data can originate from a plethora of sources and exist in diverse formats. For example, the source data could be in a Parquet file format and not possess a partitioned structure. In this case, the tutorial suggests setting up a pipeline that ushers the complete historical or one-time data into the lakehouse.

For data preparation and transformation, Fabric presents two different approaches. Notebooks/Spark is ideal for users who prefer a code-first experience and pipelines/dataflow for those who prefer a low-code or no-code experience. Data consumption is made easy by the built-in DirectLake feature of Power BI, which allows data to be queried directly from the lakehouse, along with creating reports and dashboards.

Moreover, the built-in TDS/SQL endpoint allows third-party reporting tools to connect to the warehouse and run SQL queries for analytics, thus potentially expanding the platform's reach even further.

Learn about Escape the Ordinary with a Low Code Lakehouse in Microsoft Fabric

 

Creating a low code lakehouse in the world of big data engineering can be challenging, particularly if you are not familiar with programming languages like Python. Enter the revolutionary all-in-one analytics solution that provides a unique solution. This solution, which shall be referred to as the superior data analytics platform hereinafter, provides a comprehensive suite of services. These services include everything from data movement to data science, real-time analytics, and business intelligence. It also unifies data storage and standardizes the Delta Lake format.

The tutorial guides you through an end-to-end scenario. It aids in developing a basic knowledge of the wonders of the superior data analytics platform. It explains the various experiences it entails and how they are integrated. Moreover, it makes you familiar with the professional and citizen developer experiences that come with working on this unified platform.

Organizations have been developing modern data warehouses and data lakehouses to meet their varied data analytics needs. The former deals with transactional and structured data analytics needs while the latter caters to big data or semi/unstructured data analytics needs. These two systems operate independently, leading to silos, data duplication, and increased total cost of ownership. This is where the superior data analytics platform plays its part. It eliminates silos, removes data duplication and significantly reduces the total cost of ownership.

Its flexibility allows implementation of either lakehouse or data warehouses, or the combination of both. The user is provided with the opportunity of simple implementation in order to extract the best of both architectures. In the tutorial, an example of a retail organization is taken, and a lakehouse is built for it from the start till end, using the medallion architecture. You can further use the same procedure to build a lakehouse for any organization in any industry.

Another segment of this tutorial explains the completion of certain steps by a developer at Wide World Importers, a company in the retail industry. These steps include signing up for a free trial of the superior data analytics platform and building and implementing an end-to-end lakehouse for your organization. You can create a workspace, ingest data into the lakehouse, load it and play around with different data modes. You can further connect your lakehouse using TDS/SQL endpoint. Optionally, you can orchestrate and schedule data ingestion and transformation flow with a pipeline.

The superior data analytics platform offers streamlining of data ingestion from Azure Data Services as well as other cloud-based infrastructures and on-premise data sources. This is done through a quick and easy connecting system. More than 200 native connectors are integrated to build insights using user-friendly drag and drop transformations with data flow. Further, the Shortcut feature prevents copying or moving of data by providing instant connection to existing datasets.

You have the versatility of transforming and storing data in your specified format. Power BI is available for data consumption. It also provides with a built-in TDS/SQL endpoint that functions via a Lakehouse. Thus, a Warehouse is automatically generated, whenever a Lakehouse is created. The Warehouse is of the same name as the Lakehouse and provides a simultaneous working experience with the TDS/SQL endpoint.

The tutorial makes use of the Wide World Importers (WWI) sample database for demonstrating an end-to-end lakehouse process. WWI is a wholesaler and distributor of novelty goods. To know more about its company profile and functionality, visit Wide World Importers sample databases for Microsoft SQL. The fabricated data is used in building the lakehouse and transform it through various stages of a Medallion architecture. Other than this, Sale fact table and its correlated dimensions have been used in the data and transformation flow along with the WWI data model.

In this data lake architecture, the previous data is stored in an Azure Data storage account in the Parquet file format. This is done for all the tables. However, in real-world scenarios, the data would typically originate from various sources and in diverse formats. In this tutorial, a pipeline has been set up to ingest the complete historical data or onetime data into the lakehouse. You can practice ingesting updated data for Oct and Nov and new data for Dec. Oct and Nov data is merged with the existing data and the new Dec data is written into the lakehouse table.

Finally, we have demonstrations showing two different approaches of data preparation and transformations namely, pipelines/dataflows and notebook/Spark. It also presents how the DirectLake feature of Power BI and a TDS/SQL endpoint can be used for data consumption and analytics respectively.

 

More links on about Escape the Ordinary with a Low Code Lakehouse in Microsoft Fabric

Guy in a Cube - YouTube

Escape the Ordinary with a Low Code Lakehouse in Microsoft Fabric · Discover the Untapped Settings in Power BI Desktop · The secret whereabouts of Adam & Patrick ...

What is a lakehouse? - Microsoft Fabric

Aug 30, 2023 — It's a flexible and scalable solution that allows organizations to handle large volumes of data using various tools and frameworks to process ...

Keywords

Low Code Lakehouse, Microsoft Fabric, Escape Ordinary, Microsoft Fabric Lakehouse, Lakehouse technology, Low Code architecture, Microsoft Fabric escape, Unique Microsoft Fabric, Advanced Low code, Uncommon Lakehouse Fabric