Merge Files Collection Configuration

Owned by Kevin Wu
Last updated: Jan 03, 2024
3 min read

To open the Merge Files collection agent configuration, click  Build → New Configuration . Select  Workflow  from the Configurations dialog. When prompted to Select workflow type , select  Batch. Click Add agent and select Merge Files in the Collection tab in the Agent Selection dialog. Double-click the agent icon or right-click the icon and select  Edit agent , to display the Agent Configuration dialog. 

Parts of the configuration may be done in the Sort Order tab, see  3.1.6 Workflow Template for more information.

Merge Files Tab

The  Merge Files  tab contains configurations related to the location and handling of the source files collected by the agent.

Merge Files collection agent configuration - Merge Files tab.

Field

Description

Field

Description

File Information Settings

Base Directory

Pathname of the source base directory on the local file system of the execution context, where the source files reside.

Filename

Name of the source files collected from the sub directory. Regular expressions according to Java syntax applies. For further information, see

http://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html

Example - Filename

To match all log filenames beginning with  INF , type:  ^INF.*

Sub Directories

Name of the sub directory from where files will be collected (the Base Directory will always be a match).

Note!

When input is provided from a forwarding agent, you must change the sub directories pattern to exclude the DR_TMP_DIR file by entering the following:

(?!DR_TMP_DIR$)(.*)

Compression

Compression type of the source files. Determines if the agent will decompress the files before passing them on into the workflow.

  • No Compression - agent does not decompress the files. Default setting.

  • Gzip - agent decompresses the files using gzip.

Merge Closing Condition Settings

File Limit

The maximum number of files processed in each batch.

Byte Limit

The maximum number of bytes processed in each batch.

Note!

Note that limits are set per directory, that is, the batch will be closed when the last file of a sub directory has been processed even if the File Limit or Byte Limit closing condition has not been reached.



Before Collection Settings

Move To Temporary Directory

If enabled, the source files are moved to the automatically created subdirectory DR_TMP_DIR under the directory from which they originated, prior to collection. This option supports safe collection of source files.

Inactive Source Warning (hours)

If the specified value is greater than zero, and if no file has been collected during the specified number of hours, the following message is logged:

The source has been idle for more than <n> hours, the last inserted file is <file>.

After Collection Settings

Move to

If enabled, the files will after collection be moved to the sub directory specified in the Directory filed. If Move to Temporary Directory is selected the file will be moved from the directory DR_TMP_DIR. to a sub directory relative to the files original location.

The fields Prefix, Suffix and Keep (days) will be enabled when the Move to are set. Information about them will follow.

Rename

If enabled, the source files will after the collection be renamed and kept in the source directory from which they were collected. If using Move to Temporary Directory the files will after the renaming be moved from the DR_TMP_DIRdirectory back to the original location.



Remove

If enabled, the source files will, after successfully being processed, be removed from the source directory (or from the DR_TMP_DIR directory if Move to Temporary Directory is used).

Ignore

If enabled, the source files will remain in the source directory.

Destination

Pathname relative to the current position of a file where the source files will be moved.

This field is only enabled if Move to is selected.

Prefix/Suffix

Prefix and/or suffix that will be appended to the beginning respectively the end of the name of the source files.

These fields are only enabled if Move to to or Rename is selected.

Keep (days)

Number of days to keep source files after the collection. In order to delete the source files, the workflow has to be executed (scheduled or manually) again, after the configured number of days.

Note, a date tag is added to the filename, determining when the file may be removed. This field is only enabled if Move to or Rename is selected.

After each successful execution of the workflow the agent will search recursively under Base Directory for files to remove.

Backward Compatibility Settings

Force Single UDR

If this is disabled the output files will automatically be divided in multiple UDRs per file. The output files will be divided in suitable block sizes.