Versions Compared

Old Version 3

changes.mady.by.user Henrik Pålsson

Saved on

compared with

New Version Current

changes.mady.by.user Jenni Wallin

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

In Batch workflows, the Encoder agent's configuration contains four tabs:

  • Encoder

  • Header

  • Trailer

  • Thread Buffer

Encoder

The Encoder tab contains the following settings:

Setting

Description

Suppress Encoding

If you select this option, the agent will not encode the incoming data. It will expect a raw bytearray as the input type and will pass it through untouched. This mode is used only when a header and/or a trailer is added to a data batch.

Encoder

Select which encoder to use in the list of available encoders either created in the Ultra Format Editor, or one of the default built-in encoders:

  • CSV Format

  • JSON Format

  • MZ Tagged Format

  • MZ Tagged Format (compressed)

  • Avro Format

CSV Format Specific Settings

UDR Type

Select the UDR type you want the Decoder to send out. You can either select one of the predefined UDRs or the DynamicCsvUDR if the CSV format is not known.

Format

Select the CSV format you want to use: 

  • Unix

  • Mac

  • Windows

  • Excel

  • Custom

If you select Custom, the following four settings will be enabled:

  • Delimiter

  • Use Quote

  • Quote

  • Line Break

Delimiter

Specify the delimiter character(s) for the fields in the CSV.

Use Quote

Select this option if you want to use quotes in the CSV. The Quote field is enabled.

Quote

If Use Quote is selected, enter the type of quote you want to use in the format.

Line Break

Specify how line breaks are stated in the CSV.

JSON Format Specific Settings

Indented Output

Select this option if you want the output to be indented.

Inline Map Field

Select this option if you want to process the dynamic field values from the Unmapped Fields setting in the Decoder. If you select Inline Map Field, you must specify:

  • UDR Type

  • Field

UDR Type

Specify the UDR Type that is sent out by the Decoder. Click Browse to view the available UDR Types.

Field

Select the UDR Field that you want to include.

JSON

AVRO Format Specific Settings

Indented OutputSelect this option if you want the output to be indented.

Schema Register URL

Url to a schema registry used for obtaining Avro schema used for decoding.

Format: <http>or<https>//<host>:<port>/<schema_registry_endpoint_path> Example: http://localhost:8081/schemas/ids

Info

It is possible to use a proxy to contact a schema registry. See HTTP Proxy Support for information
on how to configure the proxy on the execution context level.

Schema Field

Field in the JSON formatted schema registry response containing the schema

Header/Trailer

The batch Encoder agent uses two specialized services that allows allow you to add header and trailer information into each data batch.

They both offer the possibility of using MIM values, constants, and user-defined values in the header or trailer. When you select MIM resources, note that MIM values used in the data batch header are gathered when a new batch begins, while MIM values used in the data batch trailer are gathered when a batch ends. Thus, the numbers of outbound bytes, or UDRs, for any agent will always be zero if they are referred to in data batch headers.

The tabs for both Header and Trailer are identical and contains contain the following settings:

Setting

Description

Suppress On No Data

Select this option to avoid adding header or trailer to batches that do not contain any data (UDRs or bytearrays).

Value

Click on the Add button to add MIMs and their respective configurations

into

to the Value section. They will be added in the order they are specified.

Add Header Content Settings

MIM Defined

Select this option to add a MIM value as part of the header.  Size and Padding must be entered as well.

Note
title

Note!

For data batch headers, the MIM values are gathered at beginBatch .

Size

Specify a

Specify a fixed length of the item. It can only be omitted if the User Defined option is selected, in which case it will be calculated automatically.

Padding

Select which character to pad any remaining space with. You can either specify a user-defined character or select from the following characters:

  • Carriage return

  • Line feed

  • Space

  • Tabulator

User Defined

Select this option if you want to enter a user defined constant. If the value defined in the

 Size 

 Size field is empty or less than the number of characters in the constant, the value of Size is set to the number of characters in the constant. If  Size  is greater than the length of the constant,  Padding must be entered as well.

Pad Only

Select this option if you want to add a string according to the value entered in the Size field, filled with characters defined in the

Padding 

Padding field.

Alignment

Left or right alignment within the allocated field size.

Date Format

Enabled when a MIM of type date is selected.

A Date

A Date Format

Chooser dialog

Chooser dialog is opened, where a date format may be entered.

Thread Buffer

By default, a batch workflow uses one active thread at a time. By configuring a buffer storage for an agent, it will be possible for yet another thread to be created, this is also called multithreading. One thread will be populating the buffer, and another pulling it for data. Adding yet another buffer for another agent will add yet another thread, and so on.

This is especially useful in complex workflows with many agents. All batch agents that receive UDRs can use this functionality.

title
Note

Note!

A workflow that is configured with multithreading can only handle data of the UDR type. If bytearrays are routed into an agent using this service, an exception will be thrown.

The Thread Buffer tab contains the following settings:

Setting

Description

Use Buffer

Select this option to enable multithreading.

Print Statistics

Select this option to generate statistics that you can use to decide where to use the thread buffer in the workflow. After each batch execution, the full and empty percentage of the threads using the buffer is logged in the event area

in

at the bottom of the Workflow Monitor.

For information on how to interpret the results, see the section below, Analyzing Thread Buffer Statistics.

Analyzing Thread Buffer Statistics

By using the Print Statistics option in the Thread Buffer tab, buffer statistics are logged for the whole batch execution and show the full and empty percentage for the threads using the thread buffer.

Info
title

Example - A thread buffer printout

Code Block
languagetextthemeEclipse
11:03:04: Buffer usage statistics [5] for 2089 turnover(s) of UDRs: 
Incoming queue: Available 54%. Full 46%. 
Outgoing queue: Available 59%. Empty 41%.

  • The number within brackets, which is [5]in the example, is the batch counter id.

  • Turnover is the total number of UDRs that have passed through the buffer.

  • Available indicates how often (of the total turnover time) the buffer has been available for the incoming queue to forward a UDR and for the outgoing queue to fetch a UDR.

  • Incoming queue:

    Full is logged for the incoming thread and indicates how often (of the total turnover time) the buffer has been full and an incoming UDR had to wait for available buffer space.

    In the example, Full indicates that for 46% of the incoming UDRs there was a delay because of a full buffer.

  • Outgoing queue:

    Empty is logged for the outgoing thread and indicates how often (of the total turnover time) an outgoing queue had to wait for data because of an empty buffer.

    In the example, Empty indicates that for 41% of the attempts to fetch a UDR, the buffer was empty.

The percentage values for Empty and Full must be as low as possible, and as equal as possible. The latter may be hard to achieve, since the agents may differ too much in processing complexity. If possible, add and configure another agent to take over some of the processing steps from the most complex agent.

Input/Output Data

Input Data

Selected predefined UDRs or bytarrays (if Suppress Encoding is selected).

Output Data

bytearrays

Transaction Behavior

The transaction behavior describes what commands the Encoder agent emits and retrieves that will alter the state of the file that is being processed.

...

The Encoder agent does not emit any commands.

Retrieves

The Encoder agent retrieves the following commands from other agents:

Command

Description

Begin Batch

Possible headers defined in the Header tab are created and dispatched on all outgoing routes before the first UDR is encoded.

End Batch

Possible trailers defined in the Trailer tab are created and dispatched on all outgoing routes after the last UDR has been encoded.