Decoder Agent Configuration

Decoder Agent Configuration

To open the Decoder agent configuration dialog from a workflow configuration, do either of the following:

  • Double-click the agent icon

  • Select the agent icon and click the Edit button

Decoder_config.png
Configuration options with a custom decoder

Select the Browse button to choose a decoder type. Either search for a custom-built decoder in the folder it was saved in or select from the following built-in decoders.

  • CSV Format

  • JSON Format

  • MZ Tagged Format

  • Avro Format

Different settings are available depending on the selected decoder:

decodertypes.png
Built-in Decoder Type settings

Decoder Configuration Settings

Setting

Description

Setting

Description

General Settings - All Decoders

Decoder

Click Browse to select from a list of available decoders created in the Ultra Format Editor and the built-in decoders.

On Error

Options to control how to react upon decoding errors.

  • Cancel Batch - The entire batch is cancelled. This is the standard behavior.

  • Route Raw Data - Route the remaining, undecodable data as raw data. This option is useful if you want to implement special error handling for batches that are partially processed.

Custom Decoder settings (Ultra Format)

Full Decode

Select this option to fully decode each UDR before it is sent out from the Decoder agent. All fields are decoded and validated immediately, which ensures that all decoding errors are detected at the decoding stage, but this may hurt performance since not all fields are necessarily used in the workflow. If you must detect all decoding errors in the Decoder, select this option.

If this checkbox is cleared (default), the amount of work needed for decoding is minimized using lazy decoding of field content. In lazy decoding, the Decoder primarily uses the structural information in the Ultra definition (for example, terminated_by and static_size) to split the input into fields and validate the overall record structure. The actual field contents are only decoded when they are first accessed later in the workflow, which improves performance but means that some corrupt data may not be detected during the initial decoding stage and can instead cause a workflow to abort when those specific fields are used.

Hint!
For development and testing, enabling Full Decode helps you verify that both the decoder definition and the incoming data are correct, since every field is checked up front. In production, we recommend leaving Full Decode disabled in most cases and relying on lazy decoding for better performance. With lazy decoding, only the record structure is validated when data is read in, and each field’s contents are decoded on demand when they are actually referenced in the workflow. This reduces unnecessary work for fields that are never used, at the cost of potentially discovering some data errors later in the processing chain rather than at the Decoder stage.

CSV Format Specific Settings

UDR Type

Click Browse to select the UDR type you want the Decoder to send out. You can either select one of the predefined UDRs or the DynamicCsv UDR if the CSV format is not known.

Format

Select the CSV format you want to use: Unix, Mac, Windows, or Excel, or select to define your own customized format. If you select Custom, the following four settings will be enabled.

Delimiter

Enter the delimiter character(s) for the fields in the CSV.

Use Quote

Select this option if quotes are used in the CSV.

Quote

If Use Quote is selected, enter the type of quotes used in the CSV.

Line Break

Enter how line breaks are expressed in the CSV.

JSON Format Specific Settings

UDR Type

Click Browse to select the UDR type you want the Decoder to send out. You can either select one of the predefined UDRs or the DynamicJson UDR, which allows you to add a field of type any for including payload.

Unmapped Fields

If you have selected DynamicJson as UDR Type, you can select the option data in this field to include the payload. If you have selected another UDR type that contains an any or a map field, you can select to put any unmapped fields into the field you select in this list. All fields of ‘any’ or ‘map’ type in the selected UDR type will be available. If set to (None), any unmapped fields will be lost.

Schema Path

Enter the path to the JSON schema to use for validating the input.

MZ Tagged Format Specific Settings

Tagged UDR Type

Click the Add button to select from a list of available internal UDR formats stored in the Ultra and Code servers, to reprocess UDRs of an internal format and send them out.

If the compressed format is used, the decoder automatically detects this.

Avro Format Specific Settings

Schema Registry URL

URL to the schema registry used to obtain the Avro schema for decoding.

Example - Schema Registry URL format

Use the following pattern:
<http> or <https>//<host address>:<port>/<schema_register_endpoint_path>

For example:

"http://localhost:8081/schemas/ids"

Note!
It is possible to use a proxy to contact a schema registry. For information on how to configure the proxy on the execution context level, see https://infozone.atlassian.net/wiki/spaces/MD95/pages/575152568.

Schema Field

Name of the field in the JSON‑formatted schema registry response that contains the Avro schema definition.

Note!
The use and settings of private threads for an agent, enabling multi-threading within a workflow, are configured in the Thread Buffer tab. For further information, see https://infozone.atlassian.net/wiki/spaces/MD95/pages/575013622/Workflow+Template#Thread-Buffer-Tab