How Data Source Syncing Works

Using a Data Source is a great way to auto-digest data into Creative Force. If you are not yet familiar with Data Source, you can refer to our other articles that guide you through the setup process:

What is a Data Source

How to Set Up a Data Source

After setting up a Data Source and a syncing schedule, the data will automatically be imported into Creative Force. Along the way, there might be cases where you would need to make updates or changes to the source file.

This article will explain the logic behind the syncing process and what actions should be taken.

There are three types of syncing processes.

The first sync will happen after you finish creating and setting up the mapping columns for the Data Source. The first sync is only to remember the mapping of the properties. Product data will not be imported. You must select ‘Update from Data Source’ to manually import the data.

If you don’t select ‘Allow Delta Updates for Data Source’ in your setup, all the subsequent syncing processes will be considered a Full Sync.

The Full Sync will scan all available files, including the main source file and delta files, to look for new values and import those into Creative Force.

Delta Syncs occur if you select ‘Allow Delta Updates for Data Source’,' and will run after the First Sync.

With Delta Sync, the system will scan the latest (Delta) files added to the Data Source, import new values, and update existing ones. It will not look for updates on the main file.

Note on File Size Limitations:

The total files for the syncing process should be less than 10,000.

To check the mapping columns, go to Studio Settings -> Data Source. Hover over the Data Source you want to check and click Actions -> Mapping Column.

When you have unmapped columns on a running Data Source, you can remap those and click ‘Save’. You will then see further sync options; the logic for these will be explained in the ‘Sync Options’ section of this article.

If your Data Source is using one of the following methods: FTP, Google Cloud Storage, Amazon S3, Brandquad, Akeneo, Airtable, or Feed URL, you can fetch new columns and sync to Creative Force.

Before adding new columns, you must ensure those columns are present in your main Data Source file as well as the latest delta file.

In the Data Source Mapping menu, you will see the ‘Fetch Columns’ button in the top right corner.

Clicking ‘Fetch Columns’ will populate new columns in the mapping setup. You can then adjust the column mappings as needed and click ‘Save’.

You will then see further syncing options, which will be explained in the ‘Sync Options’ section.

When you go to the Data Source Mapping menu, you will see the 'Add New’ button.

Enter the column name and select a corresponding property in Creative Force to map, then ‘Save’. You will then see further sync options, which will be explained in the ‘Sync Options’ section.

After saving the updated column mapping using the steps above, you will be prompted to select one of the following options:

By selecting this option, the updated column mapping will be automatically applied to all Data Source syncs or updates moving forward. Existing product records from previous syncs will remain unchanged.

Existing product records will not be retroactively updated to the changed mapping.

This works best if you are planning to capture a new data point moving forward and do not want to affect the existing product data.

By selecting this option, the updated column mapping will be synced to all existing Data Source records, including delta files, if applicable.

Once the sync is completed and the Data Source status returns to “Data Ready,” then products and samples can be manually updated with the new mapping applied by clicking on them and selecting “Update from Data Source”.

Completion time will vary based on size. Once synced, products may be manually updated from the Data Source.

Please note that when you have a large data source or a large number of delta files, the sync may take longer to complete.