Data Hub Forwarding Agent Transaction Behavior(3.1)

Owned by Kevin Wu
Last updated: May 18, 2023
1 min read

This section describes the transaction behavior of the Data Hub agent. For more information about general transactions, see, Transactions, in Workflow Monitor (3.1).

Emits

This agent does not emit anything.

Retrieves

The agent retrieves commands from other agents. Based on these commands the agent changes the state of the processed local file.

Command

Description

Begin Batch

When a Begin Batch message is received a local temporary file is created in MZ_HOME/tmp.

ConsumeWhen a consume message is received the agent will write the incoming data to the local temporary file.

Commit

When a Commit message is received, the agent will send the local temporary file to the staging directory in Impala. Once the file has been transferred, the agent will insert the transferred data into a temporary database table. The agent then copies the data in the temporary table to the selected database table and removes all temporary files and tables.

If the agent is recovering from an error, the agent will only remove the local temporary file.

RollbackIf a Rollback message is received, the agent will remove the local temporary file.