Amazon S3 Data Ingestion

can ingest your data from an Amazon S3 bucket - whether is it your storage or ours.

Requirements and preconditions

If your data provider fits the following conditions, we can ingest this data via S3 easily:

- Data should be in one of the following formats: tsv, csv, xls(x) files or its compressed versions (zip, gzip, tar.gz archives)
- Data set should be presented as a flat table (no tabulation, no data hierarchy, no images, column names in the first row, one table per file)
- Data structure (field names, field order and data types) should NOT be changed without warning
- Your data provider or your own software should be able to send reports to Amazon S3 buckets
- It must happen automatically and regularly

How to setup Amazon S3 data ingestion

Step 1. Contact our technical support and tell us what you want:
- Which data source do you need to connect?
- An example of your data frame is very helpful (for uncommon report types or data formats it is necessary)
- How often should this data be refreshed (typically - on a daily basis)?
- Update policy: should we insert, update or replace data
- Explain S3 folder structure
Step 2. We will confirm whether we can ingest your data or what needs to be changed in order to make it possible.
Step 3. Setup access:
- We can grant you access to our S3 bucket in us-east-1 region
- We will create a bucket for your company and provide access keys
- We can collect data from your S3
- You must provide us access keys
Step 4. Send test data with credentials given
Step 5. We will confirm whether we have successfully loaded the data
Step 6. Schedule reports from your data platform
That’s all! Your data platform sends us your data at the selected frequency and we do the rest of the job on our side.

Special options

In addition to the above we can offer the following additional options:

- One-time historical data download or reference data download if required for custom reports or data transformations
- For unstructured or atypical data formats (examples of data set needed), we offer a client data analysis in order to select the best data loading method.
- Data update frequency changes (if necessary)
- Report format or data structure update (if data source changes the set of fields available)