Scripts

Introduction

Each connector must come with a set of scripts such that customers can easily run the connector and schedule regular data extractions.

Standard Scripts

The following standard scripts are available in the devkit-connector.

Script	Description
run.ps1	Runs the complete process: extraction, transformation, and load. It calls the three scripts below.
extract_cdata.ps1	Runs a CData extraction by calling a CData job.
transform.ps1	Runs the transformations by calling dbt.
load.ps1	Triggers cache generation.

Note: Do not change these scripts, unless an extraction method is needed other than CData Sync, see Extractors.

Creating a Custom Extraction Script

If a custom extraction script is needed, extract_cdata.ps1 must be replaced by a new script. The name must be in a format extract_newmethod.ps1. The reference to the extraction script in run.ps1 must be changed to the new script name.

The script should adhere to the following guidelines:

• The structure is consistent with the scripts provided in the devkit-connector.
• Documentation includes instructions on how to configure the script. See Documentation.
• Configurations are not stored in the repo, but in a separate place. See config_template.json in the devkit-connector as an example.
• The script generates a log file.
• The script contains error handling.
• The script does not contain any manual steps, such that it can be scheduled to run at regular intervals.