Airflow
      Back to home
      1. Zetaris Support - Knowledge Base
      2. Zetaris Integrations
      3. Airflow

      Migrating Data Objects from one Zetaris instance to another using Airflow

      This page lists all the steps that needs to be followed in order to implement the data objects migration mechanism for Zetaris Lightning. This mechanism is useful to migrate objects (pipelines, data marts, views) from one Zetaris instance to another.

      Prerequisites:

      • Before running the dags, make sure to have Microsoft Teams and Slack connections set up in Airflow for notifications. Refer to the Version Control In Zetaris documentation for the setup steps.

      Setting up Airflow for Migration of Data Objects

      We need to create variables on the Airflow UI to provide parameters for the migration process.  Follow these steps:

      • On the Airflow UI, navigate to the Admin section and select "Variables" from the dropdown list.

      • Upload the Airflow Variables  content as a JSON file by selecting "Choose File," and then import it by clicking "Import Variables" in Airflow.

      Variable Name

      Value

      Email_Address

      <email_address_for_notifications>

      Migration_Data_Object_Container_Name

      <pipeline_container name or data_mart name or permanent_view name>

      Migration_Data_Object_Pipeline_Name

      <pipeline_name>

      Migration_Data_Object_Type

      permanent_views OR data_marts OR pipelines

      Option_For_Migration

      Any value from 1-5 or 7 (please refer to table below)

      Target_Lightning_API_URL

      <target-environment-api-url>

      Target_Lightning_Password

      <target-environment-user-password>

      Target_Lightning_User

      <target-environment-user-email>

      Screenshot 2023-07-28 at 13.25.39
      Variables in airflow

      Please Note:

      • By default, the installation of Airflow includes predefined variables related to Airflow configuration and the Source Zetaris Environment.

      • The table provided below contain either example values or parameter names, which should be replaced with appropriate and specific values.

      • In the case of migration of Pipeline Container / Permanent View / Data Mart, leave the variable ' Migration_Data_Object_Pipeline_Name' with a default value of a single space (' ').

      • On the Airflow UI, for migration mechanism, edit the variable values below in LOWER CASE before triggering the DAG.


      • Please refer to the table below for the different types of migration mechanisms with the specific required user inputs for the variables defined in the above step on the Airflow UI.

      Type of Migration (Level of Migration)

      Description

      Variable1
      Migration_Data_Object_Type


      Variable2
      Migration_Data_Object_Container_Name


      Variable3
      Migration_Data_Object_Pipeline_Name

      Container

      Migrate any Pipeline Container or Data Mart Container or a individual permanent view (allowed to enter only one option at any time)

      pipelines
      OR
      data_marts
      OR
      permanent_views

      <pipleine_container_name>

      OR

      <data_mart_container_name>

      OR

      <permanent_view_name>

      <single_space>

      Individual Pipeline

      Migrate any individual pipeline

      pipelines

      <pipleine_container_name>

      <pipeline_name>

      • Please refer to the table below for the different options of migration.

      Option of Migration

      Description

      1

      Data Pipeline Container

      2

      Individual Data Pipeline

      3

      Data Quality Pipelines Container

      4

      Individual Data Quality Pipeline

      5

      Data Mart

      7

      Permanent View
      • After setting up the airflow and defining the necessary variables, you are all set to execute the 'Migrate_Zetaris_Data_Objects' dag.

        Screenshot 2023-07-28 at 13.56.32


        Walkthrough of Data Objects Migration Mechanism (Video)