Azure Data Factory

This article articulates the access to the data in Lightning transformed using ETL Pipeline created in Azure Data Factory. As an example, an ETL pipeline is created in Azure Data factory that performs the delta operation for the customers that belong to the nation ‘Japan’ into the target table within the Raw Data Store. This data, in turn, will be accessed virtually within the Lightning platform.

Implementing an Azure Data factory & delta load pipeline

‘POCDataFactory01’ ADF consists of a delta load pipeline. The ‘Delta_Load_Pipeline’ connects to a customer table from the ‘Azure MSSQL’ database & filters out the customers that belong to the nation ‘Japan’ & loads the data into the RDS Table ‘customer_japan’ via delta load operation. If there is any new customer that belongs to Japan is entered into the source table then it will automatically get loaded into the target ‘customer_japan' table.

This target ‘customer_japan' table is connected to the Lightning environment virtually & can be used for further analysis/operations. The updated data will automatically get refreshed & the object doesn’t need to be refreshed manually.