In Batch workflows, the Encoder agent's configuration contains four tabs:
- Encoder
- Header
- Trailer
- Thread Buffer
Encoder
The Encoder tab contains the following settings:
| Setting | Description |
|---|---|
Suppress Encoding | If you select this option, the agent will not encode the incoming data. It will expect a raw bytearray as the input type and will pass it through untouched. This mode is used only when a header and/or a trailer is added to a data batch. |
Encoder | Select which encoder to use in the list of available encoders either created in the Ultra Format Editor, or one of the default built-in encoders:
|
| CSV Format Specific Settings | |
| UDR Type | Select the UDR type you want the Decoder to send out. You can either select one of the predefined UDRs or the DynamicCsvUDR if the CSV format is not known. |
| Format | Select the CSV format you want to use:
If you select Custom, the following four settings will be enabled:
|
| Delimiter | Specify the delimiter character(s) for the fields in the CSV. |
| Use Quote | Select this option if you want to use quotes in the CSV. The Quote field is enabled. |
| Quote | If Use Quote is selected, enter the type of quote you want to use in the format. |
| Line Break | Specify how line breaks are stated in the CSV. |
| JSON Format Specific Settings | |
| Indented Output | Select this option if you want the output to be indented. |
| Inline Map Field | Select this option if you want to process the dynamic field values from the Unmapped Fields setting in the Decoder. If you select Inline Map Field, you must specify:
|
| UDR Type | Specify the UDR Type that is sent out by the Decoder. Click Browse to view the available UDR Types. |
| Field | Select the UDR Field that you want to include. |
Header/Trailer
The batch Encoder agent uses two specialized services that allows you to add header and trailer information into each data batch.
...
| Setting | Description | |||||
|---|---|---|---|---|---|---|
Suppress On No Data | Select this option to avoid adding header or trailer to batches that do not contain any data (UDRs or bytearrays). | |||||
Value | Click on the Add button to add MIMs and their respective configurations into the Value section. They will be added in the order they are specified. | |||||
| Add Header Content Settings | ||||||
MIM Defined | Select this option to add a MIM value as part of the header. Size and Padding must be entered as well.
| |||||
Size | Specify a fixed length of the item. It can only be omitted if the User Defined option is selected, in which case it will be calculated automatically. | |||||
Padding | Select which character to pad any remaining space with. You can either specify a user defined character or select from the following characters:
| |||||
User Defined | Select this option if you want to enter a user defined constant. If the value defined in the Size field is empty or less than the number of characters in the constant, the value of Size is set to the number of characters in the constant. If Size is greater than the length of the constant, Padding must be entered as well. | |||||
Pad Only | Select this option if you want to add a string according to the value entered in the Size field, filled with characters defined in the Padding field. | |||||
Alignment | Left or right alignment within the allocated field size. | |||||
Date Format | Enabled when a MIM of type | |||||
Thread Buffer
By default, a batch workflow uses one active thread at a time. By configuring a buffer storage for an agent, it will be possible for yet another thread to be created, this is also called multithreading. One thread will be populating the buffer, and another pulling it for data. Adding yet another buffer for another agent will add yet another thread, and so on.
...
The percentage values for Empty and Full must be as low as possible, and as equal as possible. The latter may be hard to achieve, since the agents may differ too much in processing complexity. If possible, add and configure another agent to take over some of the processing steps from the most complex agent.
Input/Output Data
Input Data
Selected predefined UDRs or bytarrays (if Suppress Encoding is selected).
Output Data
bytearrays
Transaction Behavior
The transaction behavior describes what commands the Encoder agent emits and retrieves that will alter the state of the file that is being processed.
Emits
The Encoder agent does not emit any commands.
Retrieves
The Encoder agent retrieves the following commands from other agents:
Command | Description |
Begin Batch | Possible headers defined in the Header tab are created and dispatched on all outgoing routes before the first UDR is encoded. |
End Batch | Possible trailers defined in the Trailer tab are created and dispatched on all outgoing routes after the last UDR has been encoded. |
In Batch workflows, the Encoder agent's configuration contains four tabs:
- Encoder
- Header
- Trailer
- Thread Buffer
Encoder
The Encoder tab contains the following settings:
...
Suppress Encoding
...
If you select this option, the agent will not encode the incoming data. It will expect a raw bytearray as the input type and will pass it through untouched. This mode is used only when a header and/or a trailer is added to a data batch.
...
Encoder
...
Select which encoder to use in the list of available encoders either created in the Ultra Format Editor, or one of the default built-in encoders:
- CSV Format
- JSON Format
- MZ Tagged Format
- MZ Tagged Format (compressed)
...
Select the CSV format you want to use:
- Unix
- Mac
- Windows
- Excel
- Custom
If you select Custom, the following four settings will be enabled:
- Delimiter
- Use Quote
- Quote
- Line Break
...
Select this option if you want to process the dynamic field values from the Unmapped Fields setting in the Decoder. If you select Inline Map Field, you must specify:
- UDR Type
- Field
...
Header/Trailer
The batch Encoder agent uses two specialized services that allows you to add header and trailer information into each data batch.
They both offer the possibility of using MIM values, constants and user defined values in the header or trailer. When you select MIM resources, note that MIM values used in the data batch header are gathered when a new batch begins, while MIM values used in the data batch trailer are gathered when a batch ends. Thus, the numbers of outbound bytes, or UDRs, for any agent will always be zero if they are referred to in data batch headers.
The tabs for both Header and Trailer are identical and contains the following settings:
...
Suppress On No Data
...
Select this option to avoid adding header or trailer to batches that do not contain any data (UDRs or bytearrays).
...
Value
...
Click on the Add button to add MIMs and their respective configurations into the Value section. They will be added in the order they are specified.
...
MIM Defined
Select this option to add a MIM value as part of the header. Size and Padding must be entered as well.
| Note | ||
|---|---|---|
| ||
For data batch headers, the MIM values are gathered at |
...
Size
...
Specify a fixed length of the item. It can only be omitted if the User Defined option is selected, in which case it will be calculated automatically.
...
Padding
...
Select which character to pad any remaining space with. You can either specify a user defined character or select from the following characters:
- Carriage return
- Line feed
- Space
- Tabulator
...
User Defined
...
Select this option if you want to enter a user defined constant. If the value defined in the Size field is empty or less than the number of characters in the constant, the value of Size is set to the number of characters in the constant. If Size is greater than the length of the constant, Padding must be entered as well.
...
Pad Only
...
Select this option if you want to add a string according to the value entered in the Size field, filled with characters defined in the Padding field.
...
Alignment
...
Left or right alignment within the allocated field size.
...
Date Format
...
Enabled when a MIM of type date is selected. A Date Format Chooser dialog is opened, where a date format may be entered.
Thread Buffer
By default, a batch workflow uses one active thread at a time. By configuring a buffer storage for an agent, it will be possible for yet another thread to be created, this is also called multithreading. One thread will be populating the buffer, and another pulling it for data. Adding yet another buffer for another agent will add yet another thread, and so on.
This is especially useful in complex workflows with many agents. All batch agents that receive UDRs can use this functionality.
| Note | ||
|---|---|---|
| ||
A workflow that is configured with multithreading can only handle data of the UDR type. If bytearrays are routed into an agent using this service, an exception will be thrown. |
The Thread Buffer tab contains the following settings:
...
Use Buffer
...
Select this option to enable multithreading.
...
Print Statistics
...
Select this option to generate statistics that you can use to decide where to use the thread buffer in the workflow. After each batch execution, the full and empty percentage of the threads using the buffer is logged in the event area in the bottom of the Workflow Monitor.
For information on how to interpret the results, see the section below, Analyzing Thread Buffer Statistics.
Analyzing Thread Buffer Statistics
By using the Print Statistics option in the Thread Buffer tab, buffer statistics are logged for the whole batch execution and show the full and empty percentage for the threads using the thread buffer.
...
| title | Example - A thread buffer printout |
|---|
| Code Block | ||||
|---|---|---|---|---|
| ||||
11:03:04: Buffer usage statistics [5] for 2089 turnover(s) of UDRs:
Incoming queue: Available 54%. Full 46%.
Outgoing queue: Available 59%. Empty 41%. |
The number within brackets, which is
[5]in the example, is the batch counter id.Turnoveris the total number of UDRs that have passed through the buffer.Availableindicates how often (of the total turnover time) the buffer has been available for the incoming queue to forward a UDR and for the outgoing queue to fetch a UDR.Incoming queue:Fullis logged for the incoming thread and indicates how often (of the total turnover time) the buffer has been full and an incoming UDR had to wait for available buffer space.In the example,
Fullindicates that for 46% of the incoming UDRs there was a delay because of a full buffer.Outgoing queue:Emptyis logged for the outgoing thread and indicates how often (of the total turnover time) an outgoing queue had to wait for data because of an empty buffer.In the example,
Emptyindicates that for 41% of the attempts to fetch a UDR, the buffer was empty.
The percentage values for Empty and Full must be as low as possible, and as equal as possible. The latter may be hard to achieve, since the agents may differ too much in processing complexity. If possible, add and configure another agent to take over some of the processing steps from the most complex agent.
Input/Output Data
Input Data
Selected predefined UDRs or bytarrays (if Suppress Encoding is selected).
Output Data
bytearrays
Transaction Behavior
The transaction behavior describes what commands the Encoder agent emits and retrieves that will alter the state of the file that is being processed.
Emits
The Encoder agent does not emit any commands.
Retrieves
The Encoder agent retrieves the following commands from other agents:
...
Command
...
Description
...
Begin Batch
...
Possible headers defined in the Header tab are created and dispatched on all outgoing routes before the first UDR is encoded.
...
End Batch
...
Possible trailers defined in the Trailer tab are created and dispatched on all outgoing routes after the last UDR has been encoded.