Batch-Based Real-Time Agents - Agent Configuration

This section describes how to configure the agents.

When you select one of the agents, the configurations include the same tabs that are displayed in a batch version of the configuration, with the exception of the Filename Sequence tab, which is not included as it is irrelevant in a real-time workflow. 

For details on the configuration of the agents, see the relevant section in the Desktop User's Guide:

Additional Configurations for Collection Agents

When the collection agents are included in a real-time workflow, there are also three additional tabs for each of the agent configurations: Decoder, Decompression and Execution. See the example image below.

Disk agent configuration dialog - Agent tab

Decoder Tab

In the Decoder tab you configure the settings related to the decoding of the collected data.

Disk agent configuration dialog - Decoder tab

Setting

Description

Setting

Description

Decoder

Click Browse to select from a list of available decoders created in the Ultra Format Editor, as well as the default built-in decoders:

  • CSV Format

  • JSON Format

  • MZ Tagged Format

Different settings are available depending on the Decoder you select.



Full Decode

 

MZ Tagged Specific Settings

Tagged UDR Type

Click the Add button to select from a list of available internal UDR formats stored in the Ultra and Code servers, to reprocess UDRs of an internal format and send them out.

If the compressed format is used, the decoder automatically detects this.

JSON Specific Settings

UDR Type

Click Browse to select the UDR type you want the Decoder to send out. You can either select one of the predefined UDRs or the DynamicJson UDR, which allows you to add a field of type any for including payload.

Unmapped Fields

If you have selected DynamicJson as UDR Type, you can select the option data in this field in order to include payload. If you have selected another UDR type that contains an any, or a map field, you can select to put any unmapped fields into the field you select in this list. All fields of any or map type in the selected UDR type will be available. If set to (None), any unmapped fields will be lost.

Schema Path

Enter the path to the JSON schema you want to use in this field.

CSV Specific Settings

UDR Type

Click Browse to select the UDR type you want the Decoder to send out. You can either select one of the predefined UDRs or the DynamicCsv UDR if the CSV format is not known.

Format

Select the CSV format you want to use; Unix, Mac, Windows, or Excel, or select to define your own customized format. If you select Custom, the following four settings will be enabled.

Delimiter

Enter the delimiter character(s) for the fields in the CSV.

Use Quote

Select this option if quotes are used in the CSV.

Quote

If Use Quote is selected, enter the type of quotes used in the CSV.

Line Break

Enter how line breaks are expressed in the CSV.

Decompression Tab

In the Decompression tab you specify if you want to decompress the files or not.

Setting

Description

Setting

Description

Compression Type

Select the required decompression algorithm:

  • No Compression: The agent will not decompress the files. This is the default setting.

  • GZIP: The agent will decompress GZIP formatted files. No configuration required.

  • LZO: The files are compressed using LZO compression. If you select this option, the Command Line field is displayed and populated by default with the command lzop -d -c -q. This command allows you to use the lzop unix command to decompress standard input with the LZO algorithm and write the output on standard output. You can enter an alternative lzop command if required. For further information on the lzop command, see https://www.lzop.org/lzop_man.php.

Execution Tab

In the Execution tab, you configure how often the workflow will be executed, how decoding errors will be handled and what action is to be taken when Cancel Batch messages are called. For furher information on Cancel Batch, see Batch-Based Real-Time Agents - Transaction Behavior.

 

Cancel Batch messages are sent:

  • If a decoding error occurs, and you have selected the Cancel Batch option in the agent configuration

  • If you have selected for files to be decompressed in the Decompression tab of agent configuration and it fails, e g because a file is corrupt or is not compressed.

 

Setting

Description

Setting

Description

Run Once

Select this option if you want the workflow to run once.

Repeat Every X Seconds

Select this option if you want the workflow to be run repeatedly with an interval of a specific number of seconds. The default value is 5 seconds.

If you have selected this option, when the agent encounters an error, the workflow does not abort. The error is reported in the System Log and the agent retries at the next repeat.

Decoding Error Handling

Select one of the error handling options to control how to react upon decoding errors:

  • Cancel Batch - Processing of the current batch is stopped and skips to the next batch.

  • Route Raw Data - Route the remaining, undecodable, data as raw data. This option is useful if you want to implement special error handling for batches that are partially processed.

Abort Immediately

If enabled, the workflow immediately aborts on the first Cancel Batch message from any agent in the workflow. The erroneous data batch is kept in its original place and must be moved/deleted manually before the workflow can be started again.

Abort After X Consecutive Cancel Batch

If enabled, the value of X indicates the number of allowed Cancel Batch calls, from the collection agent before the workflow will be aborted. The counter is reset between each successfully processed data batch. Thus, if 5 is entered, the workflow will abort on the 6th data batch in a row that is reported erroneous.

Never Abort

The workflow will never abort. However, as with the other error handling options, the System Log is always updated for each Cancel Batch message.

Additional Configurations for Forwarding Agents

When the forwarding agents are included in a real-time workflow, there is also one additional tabs for each of the agent configurations: File Closing Criteria. See the example image below.

Setting

Description

Setting

Description

(bytes)

When the file size has reached the number of bytes entered in this field, the file will be closed as soon as the current bytearray has been included, and stored in the storage directory. This means that the file size may actually be larger than the set value since the system will not cut off any bytearrays. If nothing is entered, this file closing criteria will not be used.

Volume (UDRs)

When the file contains the number of UDRs entered in this field, the file will be closed and stored in the storage directory. If nothing is entered, this file closing criteria will not be used.

Timer (sec)

When the file has been open for the number of seconds entered in this field, the file will be closed and stored in the storage directory. If nothing is entered, this file closing criteria will not be used.