ATIF NOORUL HASANDATA MODERNIZATION CONSULTANT

Microsoft Fabric Enterprise
Data Platform & ETL Automation

Designed and implemented a scalable Microsoft Fabric platform that unified LMS OData, Oracle ERP, and shared-file data within a governed Lakehouse, automated incremental processing, and enabled high-performance enterprise reporting.

Cloud-Native Platform Bronze–Silver–Gold Incremental Processing Millions of LMS Records Automated Daily Refresh
1.

Business Challenge

The organization lacked a centralized enterprise data layer, while high-volume and unstable source systems limited reporting performance, reliability, and historical analysis.

Fragmented Enterprise Data Landscape

Business-critical data was distributed across LMS OData streams, Oracle ERP databases, and manually maintained Excel files.

Large-Scale Performance Limitations

Standalone BI tools struggled with high-volume OData feeds, creating system lag and restricting historical analysis.

Manual & Delayed Reporting

Manual and semi-automated consolidation created reporting gaps, human-error risk, and inconsistent corporate metrics.

Unreliable On-Premises Connectivity

Oracle and local file extraction depended on gateway availability, causing pipelines to stall or fail when systems were offline.

Executive Reporting Bottlenecks

Direct queries against unstable operational systems prevented scalable and responsive executive dashboards.

2.

Solution

A modern Microsoft Fabric data platform was implemented to centralize integration, automate ETL, improve data quality, and provide a trusted analytics foundation.

Centralized Fabric Lakehouse

Built Enterprise_Lakehouse with Delta tables to consolidate LMS OData, Oracle ERP, and shared-file data.

Medallion Data Architecture

Implemented Bronze, Silver, and Gold layers for raw ingestion, validation, cleansing, and analytics-ready models.

30-Day Incremental Loading

Used Dataflow Gen2 and Power Query Advanced Editor to process only new and modified records.

Delta Lake MERGE Automation

Developed NB_Incremental_Merge in PySpark to upsert staged records and prevent duplicate keys.

Fail-Safe Orchestration

Designed PL_Enterprise_Data_Integration with controlled dependencies and Wait activities to tolerate temporary gateway outages.

Enterprise Semantic Model

Built a Power BI semantic layer with advanced DAX, including TREATAS(), over optimized Gold Layer tables.

3.

Business Impact

70–90%

Faster ETL Processing

Automated Fabric pipelines, incremental refresh, and orchestration eliminated unnecessary full historical reloads.

80–95%

Less Manual Data Preparation

LMS OData, Oracle ERP, and Excel data were automatically integrated and transformed in the centralized Lakehouse.

60–80%

Faster Report Refresh

Incremental loading and Delta MERGE operations replaced complete dataset processing during every refresh.

Millions

Historical LMS Records Supported

The Lakehouse architecture overcame Power BI limitations associated with large OData datasets.

Single Source of Truth

Centralized and validated enterprise data improved reporting accuracy and eliminated discrepancies from disconnected systems.

Duplicate & Inconsistent Records Reduced

Delta Lake UPSERT logic retained the latest validated business records and prevented duplicate keys.

Reusable Enterprise Architecture

The Bronze–Silver–Gold framework accelerated future analytics initiatives and reduced development effort.

Fully Automated Daily Refresh

Business users gained current enterprise data with minimal operational intervention.

4.

Solution Architecture

Microsoft Fabric enterprise data platform architecture

LMS OData, Oracle ERP, and shared files flow through Fabric Dataflows Gen2 into the Enterprise_Lakehouse Bronze, Silver, and Gold layers before consumption through the Power BI semantic model.

5.

Automated Data Pipeline

Microsoft Fabric automated data pipeline

The pipeline combines resilient on-premises extraction, rolling 30-day incremental staging, PySpark MERGE notebooks, and parallel dimension-table processing for a reliable daily refresh.

6.

Technology Stack

Microsoft Fabric Data PipelinesFabric Dataflows Gen2Power BI GatewayFabric LakehouseDelta LakeSQL Analytics EndpointBronze–Silver–Gold ArchitecturePySpark NotebooksPower Query / MT-SQLPower BI DesktopPower BI ServiceDAXOracle ERPLMS ODataExcel / CSV
7.

Project Showcase

Live Power BI Dashboard

Replace this placeholder with the public Power BI embed URL after publishing the report.

Microsoft Fabric workspace creation and configuration
Microsoft Fabric workspace setup and configuration.
Microsoft Fabric Lakehouse data ingestion interface
Lakehouse ingestion workspace for bringing enterprise data into Microsoft Fabric.
LMS OData ingestion using Microsoft Fabric Dataflow Gen2
LMS OData ingestion workflow implemented with Dataflow Gen2.
Incremental staging and loading process in Microsoft Fabric
Incremental staging and loading process used to prepare source data for the Lakehouse.
Microsoft Fabric Dataflow Gen2 Lakehouse destination configuration
Dataflow Gen2 destination configuration targeting a Fabric Lakehouse table.
Microsoft Fabric Lakehouse staging table data
Validated staging data loaded into the Fabric Lakehouse.
Microsoft Fabric Dataflow Gen2 items for enterprise source ingestion
Reusable Dataflow Gen2 ingestion components supporting multiple enterprise datasets.
Microsoft Fabric Lakehouse curated tables
Curated Lakehouse tables organized for downstream analytics and reporting.
Microsoft Fabric notebook creation interface
Fabric notebook environment used for PySpark-based transformation and merge processing.
Microsoft Fabric notebook and Lakehouse integration
Notebook-to-Lakehouse integration supporting automated transformation workflows.
PySpark validation queries in Microsoft Fabric notebook
PySpark validation checks used to confirm staged and transformed record counts.
PySpark merge processing in Microsoft Fabric notebook
PySpark merge logic supporting controlled incremental updates between Lakehouse layers.
Microsoft Fabric automated enterprise data pipeline
Orchestrated Fabric pipeline coordinating ingestion, notebook processing, and downstream data preparation.
Bronze Silver Gold medallion architecture for Microsoft Fabric enterprise data platform
Bronze–Silver–Gold medallion architecture connecting source systems, curated Lakehouse layers, the Power BI semantic model, and reporting.
8.

GitHub Repository

Explore the Full Technical Implementation

The repository contains architecture, pipeline documentation, code samples, and detailed project implementation notes.

View GitHub Repository ↗