Management of Batch Collection Workflow States (4.2)

Owned by Sandra Dusic
Last updated: Aug 21, 2024 by Jenni Wallin
1 min read

A batch workflow containing a Kafka collection agent will remain in running state and wait for more data even if there is no more data to consume. This is a major difference compared to how batch workflows behave for other collection agents which will stop when the data from the last start cycle has been consumed, see Workflow Monitor (4.2) for more information about workflow execution states.

Note!

A batch will not be completed if:

  • the configured Batch Size has not been reached
    or

  • a hardcoded timeout of five (5) seconds has not been reached.

A Kafka collection workflow will only stop automatically if it aborts due to errors.

A workflow stop can also be triggered by evaluating MIMs like Workflow Throughput and Estimated Lag - where no lag means that there is no data left to process.

During other circumstances you can stop it via:

  • Desktop Online

  • mzcli.

  • the operational REST interface.

Note!

The 5 second timeout for completing a batch also applies when stopping a Kafka batch collection workflow through these interfaces.

 

Â