Connectors include basic record handling features such as Retry, Error Routing and Batching.
Retry
The Retry tab of connectors allows you to turn on Retry and select the number of times you want the connector to try to process a failed record. Retry will only be utilized if the record fails for any reason. Exponential backoff can also be leveraged on each retry attempt.
Error Routing
Error routing allows you to define a separate path for errored records to go down. If Retry is also enabled, the Retry will be attempted first. If, upon all retry attempts, the record has still failed, it will go down the Error path. Errored records can then be sent down a path that sends an alert, can be queued to try again later, etc. Simply go to the Error table and check the option to Route Errored Records.
Once that option is selected, you will see a new output path on the canvas that errored records will go down.
Batching
Batching allows the connector to hold records for a specified time period or number of records, whichever comes first. Go to the Batch tab of a connector and Enable Record Batching. You can provide the amount of time to wait before processing records and a Max Record Count. The connector will process the records either once the specified time has passed or the number of records waiting in the connector has been reached, whichever comes first.
For example, if you select a Max Wait Time of 1 hour and a Max Record Count of 1000:
- if you have 1000 records on the node after 15 minutes, those records will be processed as soon as the count reaches 1000.
- if after 1 hour, you only have 500 records on the node, those 500 records will be processed.
The count will never exceed the Max Count specified and the Max Time will never be exceeded.
Note that the Batching feature still processes records individually at the specified time/count. It does not put all data into a single record/request and submit it.