BulkV3 User Guide

This document outlines the BulkV3 service from Cloud Elements. The document will cover important user information as well as compare/contrast our older BulkV2 service with the new one. 


Table Of Contents

  • What BulkV3 is & Why it Matters

  • Bulk Version Capabilities 

  • What BulkV3 is & Why it Matters

What: Bulkv3 is a generic bulk service for downloading large numbers of records. The service allows you to download any number of records (vendor API limit restricted). Bulkv3 works with any element with a GET (list) resource and pagination implemented to our standards. BulkV3 can be used on any Catalog, Community, or Custom Elements! 

Why: Most vendors offer no bulk APIs at all. This meant when users wanted to move many records, they would need to write code to paginate through the records. This could be 1 API call or 100,000 – now Cloud Elements Bulkv3 can handle this for you! For any element, you can use Bulkv3 download to create an Asynchronous job where Cloud Elements will download required records and generates a callback notification when done. 


Bulk Version Capabilities

BulkV3 comes with a variety of new capabilities. There are some limitations in BulkV3. This table describes the difference in capabilities offered by each service

Capabilities

Capability

BulkV2

BulkV3

CE Bulk Download 

Some Objects Supported

All Objects supported (list endpoints)

Bulk Download UI

Not Supported

Supported (coming soon)

CE Bulk Upload

Some Objects Supported

Not Supported (planned)

Native Bulk Download

Some Objects Supported

Not Supported

Native Bulk Upload

Some Objects Supported

Not Supported

Callback Notifications

Limited Support

Supported

API Limits

Not Supported

Supported

Server Downtime Resilience

Not Supported

Supported

Direct File Streaming

Not Supported

Supported

Continue From Last Job

Not Supported

Supported

CE Bulk Download: A Cloud Elements provided Bulk Download Service. This service does not use vendor bulk APIs, rather the standard GET (list) APIs available and paginates to form a result. 

CE Bulk Download UI: A Cloud Elements provided Bulk Download Service. This service is also exposed in the UI to help users get familiar with the Cloud Elements Bulk Download Service

CE Bulk Download: A Cloud Elements provided Bulk Upload Service. This service does not use vendor bulk APIs, rather the standard GET (list) APIs available and paginates to form a result. 

Native Bulk Download: A vendor-provided bulk service wrapped into the Cloud Elements platform. This takes advantage of vendor bulk API’s which generally are faster and have a lower impact on API limits.

Native Bulk Upload: A vendor-provided bulk service wrapped into the Cloud Elements platform. This takes advantage of vendor bulk API’s which generally are faster and have a lower impact on API limits. 

Callback Notifications: When the job is completed, send a POST notification to the specified CallbackURL. Typically this would be used to trigger some workflow in your code or a Cloud Elements Formula. For bulkv2 these notifications could take up to 1 minute. In Bulkv3 they are instant. 

API Limits: BulkV3 supports a few new fields that can modify how many records are download or how many API calls are made. This is to help prevent users from exceeding daily API limits. 

  • apiLimit: Throttles the bulk job when the API call count reaches a specified value. This field is helpful when service providers offer only a certain amount of calls per job.

  • limit: Restricts the bulk job to download a certain amount of records rather than downloading all records. When the job reaches a number of records equivalent to the specified limit, the job automatically stops and changes its state to COMPLETED.

  • continueFromJobId: This flag helps to initiate a bulk download job with respect to another job that was completed successfully. This property accepts another bulk job id as the value and when submitted, the new job considers the context of the old job (that will be identified with the given id) and starts download data from then.

Server Downtime Resilience: Sometimes vendor APIs just don’t work. To prevent a single API call from failing the whole job we’ve implemented an exponential backoff which will retry a request a total of 15 times  

Direct File Transfer: A common destination for bulk download data is a Cloud Storage Provider (box, dropbox, S3, GCS). Users are no longer required to write code or use a Formula to achieve this use-case. Users can now specify the docsHubDetails parameter in the Bulk Download request to automatically push data to a Cloud Storage Provider. 

Continue From Last  CE Bulk DownloadJob: This capability allows a user to automatically resume downloading from the last successful job. Users no longer need to manage timestamp for records downloaded in the past. This in conjunction with the API Limit features are intended to enable automatic daily sync of data without hitting API limits, without user intervention, and without custom code.