2. Ingest metadata

Once a data source is registered in Zetaris Cloud Data Fabric it will ingest all table, column and constraints metadata.

2.1 Ingest all tables from the data source

REGISTER DATASOURCE TABLES FROM ORCL

This command will connect to ORCL database, and ingest all metadata(tables, columns, foreign key, index and all other constraints) into Schema Store

 

2.2 Ingest a table from the data source

REGISTER DATASOURCE TABLE "USER" [USER_ALIAS] FROM ORCL

This will register "USER" table as USER_ALIAS if alias is provided.

2.3 Update Schema

When changes were made to the target data source, a user can reflect them using update schema command:

UPDATE DATASOURCE SCHEMA ORCL

2.4 Ingest flat files in file store

A user can ingest flat files(CSV, JSON, ORC, Parquet)  in  file store, for example AWS S3, Azure Blob, or local(remote) file system

CREATE LIGHTNING FILESTORE TABLE pref FROM HR FORMAT (CSV | JSON)

OPTIONS (path "file path", header "true", inferSchema "true", [key value pair]);

 

(AWS S3)

CREATE . LIGHTNING . FILESTORE  TABLE customer FROM TPCH_S3 FORMAT CSV(JSON) OPTIONS (

  PATH "s3n://zetaris-lightning-test/csv-data/tpc-h/customer.csv",

  inferSchema "true",

  AWSACCESSKEYID "AKIAITGIWHBIPE3NU5GA",

  AWSSECRETACCESSKEY "EWfnuO/2E8UAA/5v89sxo6hTVefa5Umns0Qn6xys"

)

 

(Azure Blob)
CREATE  LIGHTNING  FILESTORE  TABLE customer FROM TPCH_AZBLB FORMAT CSV(JSON) OPTIONS (

  PATH "wasb://zettest-storage-container@zettesstorage.blob.core.windows.net/customer.csv",

  inferSchema "true",

  fs.azure.account.key.zettesstorage.blob.core.windows.net "bHLzau36KlZ6cYnSrvPzSJVniBDtu819nHTR/+hRyDZEVScQ3wuesst9P5/I7vqG+4czeimuHSrPe2ZtK+b+BQ=="

)

Key name for security key depends on Azure Blob container, refer to Azure Blob service.

 

2.5  Ingest a RESTful service

For the RESTful service, Lightning needs an endpoint, http method(GET | POST) and request type(URL Encoded | JSON)

CREATE LIGHTNING REST TABLE  SAFC_USERS FROM ORCL SCHEMA (

  uid Long,

  gender String,

  age Integer,

job String,

  ts String)

OPTIONS(

endpoint "http://localhost:9998/example/users",

  method "GET",

  requesttype "URLENCODED"

)

Other parameter for the API call, such as security key, can be provided in OPTIONS field.

2.6 Update description and materialized table for each relation in a data source

UPDATE DATASOURCE TABLE SET ORCL.movies  OPTIONS (
 
description"some description",

  materializedtable"FusionDB.movies"

)

2.7  List the tables from the data source 

Before registering one or more tables from a database, a user can list up all tables that the data source contains

LIST DATABASE TABLES OPTIONS (
  jdbcdriver"oracle.jdbc.OracleDriver",
  jdbcurl"jdbc:oracle:thin:@oracle-master:1521:orcl",
  username"scott",
  password"tiger",
  [key "value"]*

)