11.2 Control File Collection Strategy

This section includes a description of the complementary Control File Collection Strategy that is applied in   for the Disk, FTP, SFTP, and SCP Collection agents. 

Overview

The collection strategy makes it possible to collect files for which a corresponding control file exist. If the control file does not exist, the file is ignored.

The Control File Collection Strategy controls which further configuration options that are available in the Source tab. If no strategy is selected, the default strategy is used.

Collection Strategy - Control File tab

Note!

The Collection Strategy drop down list will only be visible if there are other collection strategies available in the system, apart from the default collection strategy available.

SettingDescription

Collection Strategy

Select the Control File option in this list.

Directory

Enter the absolute path name of the source directory on the remote host, where the source files reside. The path name may also be entered relative to the home directory of the User Name account.

Filename

Enter the name of the source files on the remote host.

Regular expressions according to Java syntax can be used.

For further information, see http://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html.


Example

To match all file names beginning with TTFILE, enter: TTFILE.*

Compression

Select compression type for the source files. This selection determines if the agent will decompress the files before passing them on in the workflow.

  • No Compression - the agent will  not  decompress the files.

  • Gzip - the agent will decompress the files using gzip.

Position

The control filename consists of an extension added either before or after the shared filename part. Select one of the choices: Prefix or Suffix.

Prefix means that the text entered inte the Control File Extension field will be searched for before the shared filename part, and Suffix means that the text entered in the Control File Extension field, will be searched for after the shared filename part.

Control File Extension

The Control File Extension is used to define when the data file should be collected. A data file with filename FILE will only be collected if the corresponding control file exists. A possible control filename can be FILE.ok.

The text entered in this field is the expected extension of the shared filename. The Control File Extension will be attached to the beginning or the end of the shared filename, depending on the selection made in the Position list, above.

Data File Extension

The Data File Extension will only be applicable if  Position  is set to Suffix.

There can be cases where a more strict definition of which files should be collected is needed. This is defined in the Data File Extension field.

Consider a data file called FILE.dat. If .dat is entered in the Data File Extension field the corresponding Control file will be called FILE.ok if .ok is entered in the Control File Extension field.

Note!

Consider a directory containing 5 files:

  • FILE1.dat

  • FILE2.dat

  • FILE1.ok

  • ok.FILE1

  • FILE1

  1. The  Position  field is set to Prefix and the Control File Extension field is set to .ok.

    The control file is ok.FILE1 and FILE1 will be the file collected.
     

  2. The  Position  field is set to Suffix and the Control File Extension field is set to .ok.

    The control file is FILE1.ok and FILE1 will be be the file collected.
     

  3. The  Position  field is set to Suffix and the Control File Extension field is set to .ok and the Data File Extension field is set to .dat.

    The control file is FILE1.ok and FILE1.dat will be the file collected.

After collection, the control file is handled in the same way as the collected file is configured to be handled, that is the system should delete/rename/move/ignore it.

Move to Temporary Directory

If this option is selected, the source files will be moved to the automatically created subdirectory DR_TMP_DIR in the source directory, before collection. This option supports safe collection when source files repeatedly use the same name.

Inactive Source Warning (h)

If this option is selected, a warning message (event) will appear in the System Log and Event Area when the configured number of hours have passed without any file being available for collection:

The source has been idle for more than <n> hours,
       the last inserted file is <file>.

Move to

If this option is selected, the source files will be moved from the source directory (or from the directory DR_TMP_DIR if using Move to Temporary Directory), to the directory specified in the  Destination  field, after collection.

Note!

The  Destination must be located in the same file system as the collected files at the remote host. Additionally, absolute path names must be defined (relative path names cannot be used).

Rename

If this option is selected, the source files will be renamed after the collection, and remain (or moved back from the directory DR_TMP_DIR if using Move Before Collecting) in the source directory from which they were collected.

Remove

If this option is selected, the source files will be removed from the source directory (or from the directory DR_TMP_DIR, if using Move Before Collecting), after the collection.

Ignore

If this option is selected, the source files will remain in the source directory after the collection. This field is  not available if Move Before Collecting is enabled.

Destination

If the  Move to option has been selected, enter the full path name of the directory on the remote host into which the source files will be moved after the collection in this field. If any of the other After Collection options have been selected, this option will not be available.

Prefix and Suffix

If any of the  Move to  or  Rename  options have been selected, enter the prefix and/or suffix that will be appended to the beginning and/or end of the name of the source files, respectively, after the collection, in these fields. If any of the other After Collection options have been selected, this option will not be available.

Note!

If  Rename  is enabled, the source files will be renamed in the current (source or  DR_TMP_DIR ) directory. Ensure that you do not assign a  Prefix  or Suffix , giving files new names that still match the  Filename regular expression. That will cause the files to be collected over and over again.

Keep (days)

If any of the  Move to  or  Rename  options have been selected, enter the number of days to keep moved or renamed source files on the remote host after the collection in this field. In order to delete the source files, the workflow has to be executed (scheduled or manually) again, after the configured number of days. If any of the other After Collection options have been selected, this option will not be available.

Note!

A date tag is added to the filename, determining when the file may be removed.