Incremental extraction

By default, all extractions you can download for your app template are full extractions. To configure incremental extractions, you need to reconfigure the tables for which you want to enable incremental extraction in Xtract Universal Designer. Incremental extractions are mainly useful for transactional data tables, as these typically contain large amounts of data.

The section Input tables and fields in the documentation page of the app template contains a list of the transactional tables for the app template. Refer to App Templates.

Remarque :

When using full extractions, joins are sometimes applied to filter on properties that are not available in the table itself (for example, creation dates, which are usually stored at the header level. By joining to the header table, item-level tables can also be filtered).

With incremental extractions, only a single table can be extracted. This may result in more data being loaded during the initial run, which can increase execution time.

Astuce :

If you encounter an issue with an incremental extraction in Theobald Xtract Universal, you can reset the incremental settings by opening the extraction and selecting Delete CDC Resources.

This removes the record of the last successful extraction. The next time the extraction runs, it will perform a full load of the table. Subsequent extractions will then run incrementally again.

You can also use this option when modifying the extraction configuration (for example, when adding a new field).

1. Configuring incremental extraction for (transactional) tables

Note: You need to configure incremental extraction for transactional tables in Xtract Universal Designer.

Important: Make sure to configure incremental extraction for all the tables defined in the extraction_tables_incremental setting in the extraction script. Check out Configuring the extraction script for more information.

Follow these steps to configure an incremental extraction for a (transactional) table.

1. Remove the existing extraction.
2. Create a new extraction.
   1. Select your source connection.
   2. Define the same name as before for the extraction.
   3. Select Table CDC from the RFC extractors list.
   4. Sélectionnez OK.
3. In the Table CDC dialog, first select the Source table.
   Tip: Search the table and select it.
4. Make sure Extract table on first run is selected.
5. Make sure only the required fields are checked in the fields section.
   Note: The required fields are listed in the Input tables and fields section of the app template documentation page. App Templates.
6. Go to the WHERE clause tab and create the WHERE clause.
   Note: The condition for each table is listed in the app template documentation. App Templates.
   Note: You can use a date field to define a start date for the extractions.

   • Make sure the start date is the same in all extractions.
   • Select the specified start date carefully. You cannot make any changes to the WHERE clause after the table has been initialized.
   • Dates are specified in yyyymmdd format.
7. Select OK to save the settings and close the Table CDC dialog.
8. Select your newly added extraction and open the Destination settings.
   1. Select the correct destination configuration to be used.
   2. Make sure the File Name is set to Custom. Fill out the name of the to be extracted file, make sure to enter the table name (e.g. "CDPOS").
   3. Make sure Convert dates is turned off.

2. Configuring incremental extraction in Process Mining

Important: Configure the incremental extraction settings for the app you created, after the initial (full) dataset has been successfully loaded.

Suivez ces étapes :

1. Go to the Data transformations and select Manage input data. Check outManaging input data for more information.
2. For each table that you defined to be incremental, update the settings:
   1. Select Incremental as the Load type.
   2. Add the primary keys for the table.
      Note: The primary keys for each table are listed in the app template documentation. App Templates.
   3. For the timestamp field, select TS_TIMESTAMP.