...
If the stream is restarted, a rollback is triggered to clean up incomplete transactions. The execution restarts from the last successfully processed transaction in the stream. For example, consider a stream that is processing 10 files. If the first 3 files are successfully processed and then an error occurs while processing the 4th file, the stream is aborted and Transaction Safety ensures that when the progress is saved and the stream is resumed, the processing continues from the last successful processing, i.e. from the 4th file.
The temporary state created during an ongoing transaction will be persisted for up to 40 days until it expires and is deleted from storage. This means that a stream needs to be restarted after a failed execution within 40 days in order to recover.
Transaction safety comes in three types: At-most-once, At-least-once, and Exactly-once.
...
The result is generated but duplicate results are possible due to multiple deliveries. The following functions use this method:
Exactly-once
The result is generated only once. No duplicates can be made.
...
- Amazon S3 collector
- SFTP collector
- Count
- Amazon S3 forwarder
- SFTP forwarder
- Interconnect collector
- Interconnect forwarder
- Data Aggregator
- Deduplicate
- Data Correction ( routed from Validate function only)
Transactions using Multiple Collectors
...