Version Control In Zetaris using Github and Airflow

This Airflow Directed Acyclic Graph (DAG) streamlines the automation of backup and restore processes for the Zetaris platform by utilizing a JSON configuration file. By simply modifying a single config.json file, you can effortlessly toggle between full and partial backup and restore operations.

📁 File Structure

zetaris-backup-restore/
├── dags/
│   ├── dag_backup_restore.py        # Main Airflow DAG for backup/restore
│   ├── BackupRestoreWrapper.py      # Python logic wrapper for API interactions
│   ├── config.json                  # Config to control backup/restore (edit this only)
│   ├── github_handler.py            # Handles GitHub push/pull logic
│   ├── zetaris_handler.py           # Handles Zetaris API operations
│   ├── __init__.py                  # Makes the folder a Python module
├── utils/
│   ├── logger.py                    # Custom logging setup (optional)
│   └── common.py                    # Shared utility functions
├── README.md                        # Complete setup instructions
└── requirements.txt                 # Python dependencies for the Airflow DAG

⚙️ Step 1: Create Your config.json

This table provides guidance for setting up the configuration file according to the specified instructions.

Place this file in the same folder as dag_backup_restore.py.

🔁 Full Backup Example

{
  "mode": "backup",
  "username": "your_email@zetaris.com",
  "password": "your_password",
  "api_url": "https://api.saas.example.zetaris.com",
  "orgid": "123456",
  "metastore": "metastore",
  "github_token": "ghp_xxxxxxxxxxxxxxxxxxxxx",
  "github_repo_url": "https://api.github.com/repos/zetaris/xxx-VersionControl",
  "backup_type": "full",
  "partial_options": null
}

🔁 Partial Backup Example

{
  "mode": "backup",
  "username": "your_email@zetaris.com",
  "password": "your_password",
  "api_url": "https://api.saas.example.zetaris.com",
  "orgid": "123456",
  "metastore": "metastore",
  "github_token": "ghp_xxxxxxxxxxxxxxxxxxxxx",
  "github_repo_url": "https://api.github.com/repos/zetaris/xxx-VersionControl",
  "backup_type": "partial",
  "partial_options": ["data_marts", "pipeline_containers"]
}

🔁 Full Restore Example

{
  "mode": "restore",
  "username": "your_email@zetaris.com",
  "password": "your_password",
  "api_url": "https://api.saas.example.zetaris.com",
  "orgid": "123456",
  "metastore": "metastore",
  "github_token": "ghp_xxxxxxxxxxxxxxxxxxxxx",
  "github_repo_url": "https://api.github.com/repos/zetaris/xxx-VersionControl",
  "backup_type": "full",
  "partial_options": null
}

🔁 Partial Restore Example

{
  "mode": "restore",
  "username": "your_email@zetaris.com",
  "password": "your_password",
  "api_url": "https://api.example.enterprise.zetaris.com",
  "orgid": "123456",
  "metastore": "metastore",
  "github_token": "ghp_xxxxxxxxxxxxxxxxxxxxx",
  "github_repo_url": "https://api.github.com/repos/zetaris/xxx-VersionControl",
  "backup_type": "partial",
  "partial_options": {
    "all_data_marts": null
  }
}

🕐 Step 2: Set the Schedule (Crontab Format)

To configure the DAG to execute daily at 7 AM, locate and open the `dag_backup_restore.py` file, then modify the following line:

schedule_interval='@daily'

To:

schedule_interval='0 7 * * *'

📧 Step 3: Set Up Email Notifications (Optional)

You can activate email notifications by modifying the `default_args` section in the main DAG script file. Additionally, to set up notifications for both Slack and email, you will need to define the relevant variables.

Notification_user_email = Variable.get('notification_user_email')

SLACK_WEBHOOK_URL = Variable.get('notification_webhook_url')

default_args = {
    'owner': 'airflow',
    'start_date': datetime(2024, 1, 1),
    'retries': 1,
    'email': [notification_user_email],            # Optional: failure notifications
    'email_on_failure': True,
    'email_on_retry': False,
    'on_failure_callback': lambda context: slack_notify(context, status="❌ Failed"),
    'on_success_callback': lambda context: slack_notify(context, status="✅ Succeeded"),

}

Ensure SMTP is properly configured in airflow.cfg or via environment variables.

🔐 Step 4: Use Airflow Variables (Optional)

To avoid hardcoding sensitive data like credentials, you can store them in Airflow variables:

airflow variables set zetaris_config '{"username": "...", "password": "..."}'

Then modify your DAG to load:

from airflow.models import Variable
import json

json_config = json.loads(Variable.get("zetaris_config"))

If you're using config.json, this step is optional.

✅ Step 5: Deploy and Trigger the DAG

1. Place all  files in your dags/ folder.
2. Restart Airflow services if needed:

   airflow dags reload
   

3. Trigger manually or wait for the scheduled run:

   airflow dags trigger zetaris_backup_restore
   

✅ Summary