Using Bulk

Cloud Elements provides the ability to upload and download data in bulk from an endpoint in a normalized way. Cloud Elements leverages the provider bulk endpoints whenever available. When there is no bulk available from the provider, we provide a bulk service for uploading and downloading data from the endpoint. For uploads, we accept a file and then create objects at the endpoint on a record-by-record basis. For downloads, we search against the endpoint, and loop through all results until we retrieve all the data. Cloud Elements stores these files (encrypted) in our platform for a maximum of three days.

On this page

Bulk Download

Use bulk download to generate a csv or json file from a large number of records. Since the bulk download works asynchronously, you need to make two calls to download a file.

1. Create a Bulk Job

The POST /bulk/query creates a bulk job, which builds out a file in Cloud Elements. Use a CEQL query to specify the records included in the file. Since the job will take an unknown amount of time to complete, add a callback url to the bulk job. When the job completes it will make a callback to the url.

The bulk query API call is composed of the following parts:

  • Headers:
    • Elements-Async-Callback-Url: The webhook to be notified when a job completes. If you configured the Callback Notification Signature Key (event.notification.signature.key) when you authenticated an element instance, the bulk APIs will use the signature key to provide hash verification in the header of bulk jobs. For more on Cloud Elements Hash Verification, see Hash Verification.
  • Query:
    • q: A CEQL query for records to be included in the bulk job, for example: select * from contacts.

2. Download the Bulk File

After the job generated from the /bulk/query call completes, use GET /bulk/{id}/{objectName} to download the file. The file can be downloaded as a csv or json file.

  • Headers:
    • Accept: The format of the file to download, either text/csv or application/json.
  • Path:
    • {id}: The bulk job id generated by POST /bulk/query.
    • {objectName}: The object name that was used to generate the bulk query.

Bulk Upload

Unlike bulk downloads, using the bulk upload requires only a single call. The POST /bulk/{objectName} uploads a file which will then be processed by the bulk job.

For files that don't have column headers that match the field names in an endpoint, a transformation is required. The transformation provides a mapping from the column headers to field names. See Defining Virtual Data Resources and Transformations for more information about transformations.

The Bulk Upload is composed of the following parts:

  • Headers:
    • Elements-Async-Callback-Url: A webhook to be notified when the job completes. If you configured the Callback Notification Signature Key (event.notification.signature.key) when you authenticated an element instance, the bulk APIs will use the signature key to provide hash verification in the header of bulk jobs. For more on Cloud Elements Hash Verification, see Hash Verification.
  • Path:
    • {objectName}: The object that matches the file.
  • Body
    • metaData: Any metadata that supports the upload operation.
    • file: The file to be uploaded, either csv or json.

Note: For element that support upsert identifierFieldName is passed in the metaData to designate the field to be used for upserts.