Merge Files Collection Configuration (2.3)

Owned by Jenni Wallin
Last updated: Apr 28, 2023 by Former user (Deleted)
3 min read

You open the Merge Files agent configuration dialog from a workflow configuration: Right-click the agent icon and select Configuration..., or double-click the agent icon. Parts of the configuration may be done in the Sort Order tab. For further information, see Sort Order tab in Workflow Template.

Merge Files Tab

The  Merge Files  tab contains configurations related to the location and handling of the source files collected by the agent.

FieldDescription

Base Directory

Pathname of the source base directory on the local file system of the execution context, where the source files reside.

Filename

Name of the source files collected from the sub directory. Regular expressions according to Java syntax applies. For further information see https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/regex/Pattern.html.

Example - Filename

To match all log filenames beginning with  INF , type:  ^INF.*

Sub Directories

Name of the sub directory from where files will be collected (the Base Directory will always be a match).

Note!

When input is provided from a Usage Engine forwarding agent, you must change the sub directories pattern to exclude the DR_TMP_DIR file by entering for example the following:

(?!DR_TMP_DIR$)(.*)

Compression

Compression type of the source files. Determines if the agent will decompress the files before passing them on into the workflow.

  • No Compression - agent does not decompress the files. Default setting.

  • Gzip - agent decompresses the files using gzip.

File Limit

The maximum number of files processed in each batch.

Byte Limit

The maximum number of bytes processed in each batch.

Note!

Note that limits are set per directory, that is, the batch will be closed when the last file of a sub directory has been processed even if the File Limit or Byte Limit closing condition has not been reached.

Move To Temporary Directory

If enabled, the source files are moved to the automatically created subdirectory DR_TMP_DIR under the directory from which they originated, prior to collection. This option supports safe collection of source files.

Inactive Source Warning (hours)

If the specified value is greater than zero, and if no file has been collected during the specified number of hours, the following message is logged:

The source has been idle for more than <n> hours, the last 
inserted file is <file>.

Move to

If enabled, the files will after collection be moved to the sub directory specified in the Directory filed. If Move Before Collecting is selected the file will be moved from the directory DR_TMP_DIR. to a sub directory relative to the files original location.

The fields Prefix, Suffix and Keep (days) will be enabled when the Move to are set. Information about them will follow.

Rename

If enabled, the source files will after the collection be renamed and kept in the source directory from which they were collected. If using Move Before Collecting the files will after the renaming be moved from the DR_TMP_DIRdirectory back to the original location.

Note!

If  Rename  is enabled, the source files will be renamed in the current directory (source or  DR_TMP_DIR ). Make sure that the new name does  not  match the regular expression or the file will be collected over and over again.

Remove

If enabled, the source files will, after successfully being processed, be removed from the source directory (or from the DR_TMP_DIR directory if Move Before Collecting is used).

Ignore

If enabled, the source files will remain in the source directory.

Destination

Pathname relative to the current position of a file where the source files will be moved.

This field is only enabled if Move to is selected.

Prefix/Suffix

Prefix and/or suffix that will be appended to the beginning respectively the end of the name of the source files.

These fields are only enabled if Move to to or Rename is selected.

Keep (days)

Number of days to keep source files after the collection. In order to delete the source files, the workflow has to be executed (scheduled or manually) again, after the configured number of days.

Note, a date tag is added to the filename, determining when the file may be removed. This field is only enabled if Move to or Rename is selected.

After each successful execution of the workflow the agent will search recursively under Base Directory for files to remove.

Force Single UDR

If this is disabled the output files will automatically be divided in multiple UDRs per file. The output files will be divided in suitable block sizes.