Batch-Based Real-Time Agents - Agent Configuration

Owned by Jenni Wallin
Last updated: Mar 18, 2024 by Kevin Wu
6 min read

This section describes how to configure the agents.

When you select one of the agents, the configurations include the same tabs that are displayed in a batch version of the configuration, with the exception of the Filename Sequence tab, which is not included as it is irrelevant in a real-time workflow. 

For details on the configuration of the agents, see the relevant section in the Desktop User's Guide:

Additional Configurations for Collection Agents

When the collection agents are included in a real-time workflow, there are also three additional tabs for each of the agent configurations: Decoder, Decompression and Execution. See the example image below.

Disk agent configuration dialog - Agent tab

Decoder Tab

In the Decoder tab you configure the settings related to the decoding of the collected data.

Disk agent configuration dialog - Decoder tab

SettingDescription
Decoder

Click Browse to select from a list of available decoders created in the Ultra Format Editor, as well as the default built-in decoders:

  • CSV Format
  • JSON Format
  • MZ Tagged Format

Different settings are available depending on the Decoder you select.

Full Decode

This option is only available when you have selected a decoder created in the Ultra Format Editor.

Select this option to fully decode the UDR before it is sent out from the decoder agent. This action may have a negative impact on performance, since not all fields may be accessed in the workflow, making decoding of all fields in the UDR unnecessary. If it is important that all decoding errors are detected, you must select this option.

If this checkbox is cleared (default), the amount of work needed for decoding is minimized using "lazy" decoding of field content. This means that the actual decoding work may be done later in the workflow, when the field values are accessed for the first time. Corrupt data (that is, data for which decoding fails) may not be detected during the decoding stage, but can cause a workflow to abort later in the process.

MZ Tagged Specific Settings
Tagged UDR Type

Click the Add button to select from a list of available internal UDR formats stored in the Ultra and Code servers, to reprocess UDRs of an internal format and send them out.

If the compressed format is used, the decoder automatically detects this.

JSON Specific Settings
UDR TypeClick Browse to select the UDR type you want the Decoder to send out. You can either select one of the predefined UDRs or the DynamicJson UDR, which allows you to add a field of type any for including payload.
Unmapped FieldsIf you have selected DynamicJson as UDR Type, you can select the option data in this field in order to include payload. If you have selected another UDR type that contains an any, or a map field, you can select to put any unmapped fields into the field you select in this list. All fields of any or map type in the selected UDR type will be available. If set to (None), any unmapped fields will be lost.
Schema PathEnter the path to the JSON schema you want to use in this field.
CSV Specific Settings
UDR TypeClick Browse to select the UDR type you want the Decoder to send out. You can either select one of the predefined UDRs or the DynamicCsv UDR if the CSV format is not known.
FormatSelect the CSV format you want to use; Unix, Mac, Windows, or Excel, or select to define your own customized format. If you select Custom, the following four settings will be enabled.
DelimiterEnter the delimiter character(s) for the fields in the CSV.
Use QuoteSelect this option if quotes are used in the CSV.
QuoteIf Use Quote is selected, enter the type of quotes used in the CSV.
Line BreakEnter how line breaks are expressed in the CSV.

Decompression Tab

In the Decompression tab you specify if you want to decompress the files or not.

 Disk agent configuration dialog  - Decompression tab

SettingDescription
Compression Type

Select the required decompression algorithm:

  • No Compression: The agent will not decompress the files. This is the default setting.
  • GZIP: The agent will decompress GZIP formatted files. No configuration required.
  • LZO: The files are compressed using LZO compression. If you select this option, the Command Line field is displayed and populated by default with the command lzop -d -c -q. This command allows you to use the lzop unix command to decompress standard input with the LZO algorithm and write the output on standard output. You can enter an alternative lzop command if required. For further information on the lzop command, see https://www.lzop.org/lzop_man.php.

Execution Tab

In the Execution tab, you configure how often the workflow will be executed, how decoding errors will be handled and what action is to be taken when Cancel Batch messages are called. For furher information on Cancel Batch, see Batch-Based Real-Time Agents - Transaction Behavior.

Disk agent configuration dialog - Execution tab

Note!

Cancel Batch messages are sent:

  • If a decoding error occurs, and you have selected the Cancel Batch option in the agent configuration
  • If you have selected for files to be decompressed in the Decompression tab of agent configuration and it fails, e g because a file is corrupt or is not compressed.
SettingDescription
Run OnceSelect this option if you want the workflow to run once.
Repeat Every X Seconds

Select this option if you want the workflow to be run repeatedly with an interval of a specific number of seconds. The default value is 5 seconds.

If you have selected this option, when the agent encounters an error, the workflow does not abort. The error is reported in the System Log and the agent retries at the next repeat.

Decoding Error Handling

Select one of the error handling options to control how to react upon decoding errors:

  • Cancel Batch - Processing of the current batch is stopped and skips to the next batch.

  • Route Raw Data - Route the remaining, undecodable, data as raw data. This option is useful if you want to implement special error handling for batches that are partially processed.

Abort Immediately

If enabled, the workflow immediately aborts on the first Cancel Batch message from any agent in the workflow. The erroneous data batch is kept in its original place and must be moved/deleted manually before the workflow can be started again.

Abort After X Consecutive Cancel Batch

If enabled, the value of X indicates the number of allowed Cancel Batch calls, from the collection agent before the workflow will be aborted. The counter is reset between each successfully processed data batch. Thus, if 5 is entered, the workflow will abort on the 6th data batch in a row that is reported erroneous.

Never Abort

The workflow will never abort. However, as with the other error handling options, the System Log is always updated for each Cancel Batch message.

Additional Configurations for Forwarding Agents

When the forwarding agents are included in a real-time workflow, there is also one additional tabs for each of the agent configurations: File Closing Criteria. See the example image below.

Disk agent configuration dialog

SettingDescription

(bytes)

When the file size has reached the number of bytes entered in this field, the file will be closed as soon as the current bytearray has been included, and stored in the storage directory. This means that the file size may actually be larger than the set value since the system will not cut off any bytearrays. If nothing is entered, this file closing criteria will not be used.

Volume (UDRs)

When the file contains the number of UDRs entered in this field, the file will be closed and stored in the storage directory. If nothing is entered, this file closing criteria will not be used.

Timer (sec)

When the file has been open for the number of seconds entered in this field, the file will be closed and stored in the storage directory. If nothing is entered, this file closing criteria will not be used.