Transaction safety
The UsageCloud service defines a transaction as a unit of data (such as a file) being processed by a stream. A transaction is considered complete if the file is processed by the stream without errors. Individual transactions are limited to the specified data source. Transactions are separated into two types of categories:
File-based functions - Collectors such as Amazon S3 and SFTP define the input as files, so a transaction is defined as file processing without any errors.
Data-based functions - The Counter and Data Generator collectors generate input data on their own. In this case, transactions are regarded as processes themselves when the data generation has started.
The Data correction feature creates transactions by itself, and they are regarded as separate processes.
Transaction safety depends on the types of functions used in the stream and is only supported for functions that have state.
If a stream fails during a transaction, the stream is aborted, and all data is secured.
If the stream is restarted, a rollback is triggered to clean up incomplete transactions. The execution restarts from the last successfully processed transaction in the stream. For example, consider a stream that is processing 10 files. If the first 3 files are successfully processed, then an error occurs while processing the 4th file, the stream is aborted. Transaction Safety ensures that when the progress is saved and the stream is resumed, the processing continues from the last successful processing, i.e., from the 4th file.
The temporary state created during an ongoing transaction will be persisted for up to 40 days until it expires and is deleted from storage. This means that a stream must be restarted within 40 days after a failed execution to be recovered.
This section contains the following subsections: