9.18.3.1.1 Disk Collection Agent Configuration - Real-Time

Owned by Jenni Wallin
Last updated: May 11, 2023 by Former user (Deleted)
5 min read

You open the Disk collection agent configuration dialog from a workflow configuration: you can right-click the agent icon and select Configuration..., or double-click the agent icon.


Disk Tab

The Disk tab contains settings related to the placement and handling of the source files to be collected by the agent.

Disk collection agent - Disk tab

File Information

SettingDescription
Directory

Enter the path of the source directory on the local file system of the EC, where the source files reside. The path can be absolute or relative to the $MZ_HOME environment variable.

During processing of a file, it will be temporarily stored in a subdirectory under DR_TMP_DIR in the specified directory.

Filename

Enter an expression that matches the source files on the local file system. Regular expressions according to Java syntax applies. For further information, see: http://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html

Example.

 To match all filenames beginning with TTFILE, type:

ready_TTFILE.*

Note!

Collecting a file while it is open for writing in another application may cause loss of data. For this reason, it is recommended that you rename files after moving them to the source directory. The renamed files should include a suffix or prefix that is also included in the Filename expression.

Compression

Select Gzip to decompress files before collection and insertion into the workflow. If the collected files are not compressed, select No Compression.

Polling Interval (ms)

Enter the interval, in milliseconds, at which the source directory is to be scanned for new files.

File Reader SizeThe agent moves files to the temporary directory for processing in bulk. Enter the maximum number files that can be included in each bulk move operation.
Read Size (b)

Enter the size buffer size to be used for for reading files.

When a decoder is selected the agent produces FileSend UDRs that contains one decoded UDR. On the other hand, when a decoder is not selected, the FileSend UDRs contain bytearrays and these will be split according to the value of Read Size (b).

Note!

For performance reasons, it is recommended to set this value to a multiple of 1024.

No of Thread(s)

Enter the number of worker threads that should be used for file collection. This field determines the number of files that agent can process concurrently. You should typically set the number of threads to match the number of workflow threads. If you want to ensure that the files are processed in timestamp order you should set No of Thread(s) to 1 and also select the check box Sort Files.

Sort Files

Select this check box to sort the files that are moved to the temporary directory for processing. The files are sorted according to timestamp in ascending order. Due to multithreading, the files may not be processed in this order.


Timeout Handling

SettingDescription
Timeout (ms)

When the agent routes the first partial data set to the workflow, a timeout counter starts. The timeout counter is reset when the workflows acknowledge reception of the complete file. If the timeout counter is exceeded the collected file is either moved to a user-specified directory or After Collection strategy is applied.

Enter the timeout value in milliseconds.

Move Files on Timeout

Select this check box to move timed out files to an automatically created subdirectory that you specify in the Directory setting.

Note!

When this check box is not selected and a timeout occurs, the After Collection strategy will be applied.

Directory

When Move Files on Timeout is selected, enter the target directory for files that are subject to timeout handling.


After Collection

SettingDescription
Move/Rename

If you select this radio button, the source files will be moved to the directory that you specify in the Destination field, after the collection.

If you use the Prefix or Suffix fields, the file will be renamed as well.

Enter the path of the directory on the local file system of the EC into which the source files will be moved after collection. The path can be absolute or relative to the $MZ_HOME environment variable.

Note!

It is possible to move collected files from one file system to another, however it will have a negative impact on performance.

Prefix/Suffix

Enter the prefix and/or suffix that will be appended to the beginning respectively the end of the name of the source files, after the collection. These fields are only enabled if you have selected Move or Rename.

Note!

If you have selected Rename, the source files will be renamed in the current directory. Make sure not to assign a Prefix or Suffix, giving files new names that matches the regular expression in Filename, or else the files will be collected over and over again.

Search/Replace

Note!

 To apply Search and Replace, select Move/ Rename.


Search: Enter the part of the filename that you want to replace.

Replace: Enter the replacement text.

Search and Replace operate on your entries in a way that is similar to the Unix sed utility.

The identified filenames are modified and forwarded to subsequent agents in the workflow.

This functionality enables you to perform advanced filename modifications, as well:

  • Use regular expression in the Search entry to specify the part of the filename that you want to extract.

Note!

A regular expression that fails to match the original file name will abort the workflow.

  • Enter Replace with characters and meta characters that define the pattern and content of the replacement text.
RemoveIf this radio button is selected, the source files will be removed from the source directory after collection.

Decoder Tab

The Decoder tab contains settings related to decoding of the collected data.

Disk collection agent - Decoder tab

SettingDescription
Decoder

List holding available decoders introduced via the Ultra Format Editor.

The decoders are named according to the following syntax:
<decoder> (<module>)

The option MZ Format Tagged UDRs indicate that the expected UDRs are stored in one of the built-in formats.
Select this option to make the Tagged UDR Type list accessible for configuration. If this option is selected, the Tagged UDR Type list will be enabled.

Tagged UDR type

List of available internal UDR formats stored in the Ultra and Code servers. The formats are named according to the following syntax:

<internal> (<module>)

If the decoder is to reprocess UDRs of an internal format, the Decoder MZ format tagged UDRs has to be selected to enable this list. Once enabled, the internal format may be selected.

Full Decode

If this check box is selected, the UDR will be fully decoded before it is inserted into the workflow. This may have a negative impact on performance since all fields may not be accessed in the workflow, making decoding of all fields unnecessary.

If disabled (default), the amount of work needed for decoding is minimized, using a "lazy" method decoding sub fields. This means the actual decoding work may not be done until later in the workflow, when the field values are accessed for the first time.

Corrupt data (that is, data for which decoding fails) may not be detected during the decoding stage and could cause the UDR to be discarded at a later processing stage.

Next:

9.18.3.1.2 UDR Types