What is a Data Source
Viviana avatar
Written by Viviana
Updated over a week ago

Overview

Within Creative Force, there are a number of different ways that you can bring in product data. You can choose to keep things simple with file import, or if you have a development team you can build out an API integration to connect another system like a PIM directly to Creative Force.

A third option would be to use what we call a “Data Source”, where you can have Creative Force look at a particular location like an FTP, AWS S3 bucket, or Google Cloud Bucket, and wait for new data to be uploaded to that location. Typically this is done by another software like a PIM that can set up daily, weekly, or monthly exports to a set location, effectively connecting the two systems without the need for a large development project.

If your Product Data System can't export data to a FTP, AWS S3 bucket, or Google Cloud bucket, some customers use scripts to extract data from the first system and upload it to the designated location.

Should every customer set up a data source?

Our Data Source functionality is available to all Enterprise customers, but it may not be the best fit for all customers based on a few different parameters.

If you want to have data imported to Creative Force in real-time, then an API integration from your other system would be best, or by file import. Depending on the size of your data source, it is recommended to do a daily, weekly, or monthly sync. We will dig into why a little later!

Equally, if you are not getting your data from another system, and either building it manually or getting it from a third-party client, then a data source may not be the best solution. A data source will work well when the data formatting and needs are the same and consistent. If those needs are changing regularly or prone to having issues (like inconsistent or poor data), then it may be best to stay with a file upload approach, as this is the most forgiving and easiest to correct after the fact. For instance, some commercial studios will shy away from a data source as they don't have 100% control over the data they are receiving and its quality.

However, if you are able to ensure consistent data and work with an update frequency of daily or weekly then a data source may be for you. Say for instance you are a Brand or Retailer that is getting your data seasonally, or in some set interval, then a data source is a great option to consider.

How is data brought into Creative Force and managed from the Data Source location?

Within Creative Force, you will define the frequency that Creative Force will scan the data source location to see if there are updates. We typically suggest setting the frequency and time of an update to a time when there is little activity within your studio, like the middle of the night or the weekend. So that the update is done when there is little activity on your instance.

Once that is set, Creative Force will scan the data source at the defined interval looking for updates that we can ingest.

We will not automatically import all the products that are in the data source location and put them in your Creative Force instance. Instead, we will place them in our database and store them for you to access at a moment's notice when you try to create a new job or update products from the data source. Going forward, the area that is between your Creative Force Instance itself and the data source location itself will be referred to as the “data source”,

You may be asking why don't we just import all the product data from the Data Source Location straight into Creative Force. It is a great question to ask! The reason we do not do this is because it can be quite overwhelming to have all of that data in at once in one big bundle in your Instance.

If you can imagine you have hundreds of thousands of products that are part of your data source location that you as a studio may want to work on during a season, it can be overwhelming to have all of that data in Creative Force as a default. Best practice is to break up that data into smaller, manageable jobs that are easily identifiable.

To create a job from Data Source you would first need to set up the Data Source and assign it to a client. You can find out how to do that here.

Once you've established the connection and assigned it to a client you are ready to start the Job import. You can either do this by creating a “Sync Job” that will populate the products based on the criteria you define, or by creating a Job “From Scan” and CF will then look up the products on the Data Source file.

Below we are covering the use case for both options:

Create Job by Syncing from Data Source

Only the initial Data source setup and definition of the criteria are required if you choose to create a job by “Syncing from Data Source”. After that, CF will check for updates in accordance with the schedule you determined in the Data Source Settings.

When your goal is to import products that must meet specific criteria, that is the recommended method. As soon as a new product that meets the set criteria is added to the data source file, it will automatically be added to the job with the next sync.

For instance, you might want to import only products associated with a specific Brand:

Creating job via Scan:

This is useful if your sample team is creating a job based on a rail, or perhaps a specific shipment of items that are received into the studio. Then can scan all the items on the rail/container, and CF will automatically build a job out of it, and pull data from the data source around those products:

When working with a Data Source, it is possible to run into some setup issues. For those, we have the following Trouble Shooting article which covers the most common issues.

Please feel free to reach out to our support team at any time via Intercom if you have any questions or run into issues!

Did this answer your question?