Incremental extraction

Introduction

Depending on your source system, it is possible to run incremental data extraction with CData Sync.

Prerequisites

To run incremental data extraction with CData Sync, you must set up a local environment with a staging database. In this staging database, you can incrementally extract data from the source system and add the delta to the already extracted data. See the illustration below for an overview.

Staging database

Most common database types can be used as a staging database. Check the official documentation for your extraction tool whether your extract can output to the database you want to use as a staging database. For CData Sync see Destinations, and for Theobald Xtract Universal see Destinations.

Also, you have to be able to load the data from staging into the blob store. This can be done with a CData Sync job, if your database is supported as a source, see Sources.

Check Incremental Settings

For most source systems, CData Sync has preconfigured how incremental extractions should be performed. If that's not the case, the settings have to be edited manually to perform incremental extractions. For more on how to do this, see the official CData Sync documentation.

Running Incremental Extraction

Follow these steps to make sure incremental data extractions run properly.

1. Start the CData Sync Admin console and login as and administrator using the credentials defined during installation of CData Sync.
2. Go to Jobs and select the extraction job to display the job settings.

3. Go to the Advanced tab, and edit the Replication Option to deselect the Drop Table option. This enables incremental extraction.

The first time the CData Sync extraction job runs, a full extraction is performed and all source data is extracted. Every next time the CData Sync extraction job runs, only the data that has changed in the source is extracted and the delta is added to the already extracted data.

Note:

You need to create an additional CData Sync job to load the data from the staging database into the Azure blob store.

For more information on incremental data extraction with CData Sync, see the official CData Sync documentation.