SCP Collection Agent Configuration(4.3)

The agent configuration consists of the following tabs:

  • Connection

  • Source

  • Advanced

  • Security


Note!

You can configure part of the parameters in the Filename Sequence or Sort Order service tabs. For further information, see Workflow Template (3.0).

Connection Tab

The Connection tab contains configuration settings related to the remote host and authentication.


SettingDescription

Host

Enter the primary host name or IP-address of the remote host to be connected. If a connection cannot be established to this host, the Additional Hosts, that you can specify in the Advanced tab, are tried.

File System Type

Enter the type of file system on the remote host. This information is used for naming the remote files.

  • Unix - remote host using a Unix file system. This is the default setting.

  • Windows NT - remote host using a Windows NT file system.

Authenticate With

Choose between authentication using a password or a private key.

Username

Enter the username for an account on the remote host, enabling the SCP session to login.

Password

Enter the password for the specified Username. This option only applies if you have selected password authentication.

Private Key

Click Select... to display a window where you enter the private key. If the private key is protected by a passphrase, you must also enter the passphrase in Key Password. This option only applies if you have selected private key authentication. 

Enable Collection Retries

Select this check box to enable repeated attempts to connect and start a file transfer.

When you select this option, the agent tries to connect to the host as many times as you enter in the Max Retries field described below. If the connection fails, a new attempt is made after the number of seconds that you enter in the Retry Interval (s) field described below.

Retry Interval (s)

Enter the time interval in seconds, between retries.

If a connection problem occurs, the actual time interval before the first attempt to reconnect is the time that you set in the Timeout field in the Advanced tab plus the time set in the Retry Interval (s) field. For the remaining attempts, the actual time interval is the number seconds entered in this field.

Max Retries

Enter the maximum number of retries to connect.

If more than one connection attempt has been made, the number of used retries is reset as soon as a file transfer is completed successfully.

Note!

This number does not include the original connection attempt.

Source Tab

The Source tab contains settings related to the remote host, source directories and source files. The configuration available is modified depending on which Collection Strategy you select. The following text describes the configuration options available when no custom strategy has been chosen.


SettingDescription

Collection Strategy

If there is more than one collection strategy available in the system a Collection Strategy drop down list is also visible. From the drop down list you can choose between the Default Collection Strategy and Duplicate Filter.

Your selection determines the settings to be configured on the tab.

Directory

Enter the absolute pathname of the source directory on the remote host, where the source files reside. You can also enter the pathname relative to the home directory of the Username account.

Filename

Enter the name of the source files on the remote host. Regular expressions according to Java syntax applies. For further information, see https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/regex/Pattern.html.


Example

To match all filenames beginning with  TTFILES , enter:  TTFILES.* .

Compression

Select the compression type of the source files. This setting determines if the agent decompresses the files before passing them on in the workflow or not.

  • No Compression - The agent does not decompress the files. This is the default setting.

  • Gzip - The agent decompresses the files using gzip.

When you select Default Collection Strategy:

SettingDescription

Move to Temporary Directory

If you select this option, the source files are moved to the automatically created subdirectory DR_TMP_DIR in the source directory, prior to collection. This option supports safe collection of a source file reusing the same name.

Append Suffix to Filename

Enter the suffix that you want to add to the file name prior to collecting it.

Important!

Before you execute your workflow, make sure that none of the file names in the collection directory include this suffix.

Inactive Source Warning (h)

If the value that you enter is greater than zero, and if no file has been collected during the specified number of hours, the following message is logged:

The source has been idle for more than <n> hours, the last inserted file is <file>.
Move to

If you select this option, the source files are moved from the source directory (or from the directory DR_TMP_DIR, if you are using Move to Temporary Directory) to the directory specified in the Destination field, after collection.

If the Prefix or Suffix fields are set, the file is also renamed.

Note!

If a file with the same filename already exists in the target directory, the existing file is overwritten and the workflow does not abort.

Destination

Enter the absolute pathname of the directory on the remote host which the source files are to be moved to after collection.

This field is only available if Move to is selected.

Note!

The Directory must be located in the same file system as the collected files at the remote host. In addition, absolute pathnames must be defined. Relative pathnames cannot be used.

RenameIf you select this option, the source files are renamed after collection, remaining in the source directory from which they were collected, or moved back from the directory DR_TMP_DIR, if you are using Move To Temporary Directory.

Prefix and Suffix

Enter the prefix and/or suffix to be appended to the names of the source files after collection.

These fields are only available if Move to or Rename is selected.

Note!

If Rename is enabled, the source files are renamed in the current directory (source or  DR_TMP_DIR ). Ensure that you do not assign a prefix or suffix, giving files new names that still match the filename regular expression, otherwise the files will be collected over and over again.

Search and Replace

To apply Search and Replace, select either Move to or Rename.

  • Search: Enter the part of the filename that you want to replace.
  • Replace: Enter the replacement text.

Search and Replace operate on your entries in a way that is similar to the Unix sed utility. The identified filenames are modified and forwarded to the following agent in the workflow.

This functionality also allows you to make advanced filename modifications:

  • Use regular expression in the Search entry to specify the part of the filename that you want to extract.

    Note!

    A regular expression that fails to match the original file name aborts the workflow.

  • Enter Replace with characters and meta characters that define the pattern and content of the replacement text.

    Search and Replace Examples

    To rename the file file1.new to file1.old, use:

    • Search.new
    • Replace.old

    To rename the file JAN2011_file to file_DONE, use:

    • Search([A-Z]*[0-9]*)_([a-z]*)
    • Replace$2_DONE

    Note!

    The search value divides the file name into two parts by using brackets. The replace value applies the second part by using the place holder $2.

Keep (days)

Enter the number of days to keep source files on the remote host after collection. To delete the source files, the workflow has to be executed again (scheduled or manually), after the number of days that you enter.

Note!

A date tag is added to the filename, determining when the file can be removed. This field is only available if Move to or Rename is selected.

Remove

If you select this option, the source files are removed from the source directory, or from the directory DR_TMP_DIR, if you are using Move To Temporary Directory, after collection.

Ignore

If enabled, the source files will remain in the source directory after the collection. This option is not available if Move Before Collecting is enabled.

When you select Duplicate Filter:

SettingDescription
FilenameSelect this option to have only the filename checked for duplicates. If the filename is in the list of files which have already been collected once, the file is ignored by the agent.
Filename and TimestampSelect this option to have both the filename and the time stamp of the last modification checked for duplicates. If the file has already been collected once, it is collected again only if the duplicate check reveals that the file has been updated since the previous collection.
File List Size

Enter a value to determine the maximum size of the list of files already collected. This list of files is compared to the input files to detect duplicates and prevent them from being collected by the agent.

When this collection strategy is used with multiple server connection strategy, each host has its own duplicate list. If a server is removed from the multiple server configuration, the collection strategy automatically drops the list of duplicates for that host in the next successful collection.

Note!

If the number of files to be collected is greater than the file list size, files older than the oldest file in the list are not collected.

Advanced Tab

The  Advanced  tab contains configurations related to more specific use of the SCP service.


SettingDescription
Max Async RequestAdvance setting to tune the maximum asynchronous request a SFTP server can be handle, leave it empty, will use the default value(100) from library.

Port

Enter the port number that the SCP service is to use on the remote host.

Timeout (s)

Enter the maximum time, in seconds, to wait for a response from the server. 0 (zero) means to wait forever.

Accept New Host Keys

If you select this check box, the agent overwrites the existing host key when the host is represented with a new key. The check box is empty by default so that the default behavior is to abort when the key does not match.

Note!

Selecting this option causes a security risk since the agent accepts new keys even if they may belong to another machine.

Enable Key Re-Exchange

Use this check box to enable and disable automatic re-exchange of session keys during ongoing connections. This can be useful if you have long lived sessions as you may experience connection problems for some servers if one side initiates a key re-exchange during the session.

Additional Hosts

You can add additional host names or IP-addresses that may be used to establish a connection. These hosts are tried, in sequence from top to bottom, if the agent fails to connect to the remote host set on the Connection tab.

Use the buttons AddEditRemoveMove up and Move down to configure the host list.

Security Tab

The Security tab contains configurations related to the advanced security options for SCP. You can modify the configuration available by selecting the Advanced Security Option check box. If the Advanced Security Option is not enabled, the cipher mode defaults to aes128-ctr and the HMac type defaults to hmac-sha2-256. If the Advanced Security Option is selected but the fields are left empty, the Cipher Mode defaults to aes128-ctr and the HMac Type defaults to hmac-sha2-256.

Note!

 has upgraded its Maverick library to version 1.7.30, this has resulted in a change in the default handling for SFTP advanced security. Users should take note of the behaviour change for the Advanced Security Option for the SFTP agents. The Advanced Security Option will be disabled by default. Users will have to enable it on their own accord from the Security Tab in the SFTP agents configuration.

With Advanced Security Option disabled, Maverick will manage the connection between the SFTP agent and the server. Maverick will attempt to connect with the STRONG security level. Failing to do so, it will auto downgrade the security level to WEAK and attempt to connect, this behaviour will allow our agents to work well with backwards compatibility for servers with older instances of the Maverick library.  Furthermore, having STRONG security level will result in a performance degradation for .

However, when a user manually enables the Advanced Security Option from the security tab, Maverick will instead assign the WEAK security level, which will not be as strict or resource intensive as the STRONG security level.

For more information about security levels, you can refer to this page: https://www.jadaptive.com/managed-security-in-our-java-ssh-apis/

SettingDescription
Cipher Mode

Select the required algorithm for the cipher mode. The list includes the block cipher modes supported by the SCP agent. Your selection allows the agent to determine which algorithm for the block cipher is to be used when communicating with the servers.

  • 3des-cbc
  • 3des-ctr
  • blowfish-cbc
  • aes128-cbc
  • aes192-cbc
  • aes256-cbc
  • aes128-ctr
  • aes192-ctr
  • aes256-ctr
  • arcfour
  • arcfour128
  • arcfour256

HMac Type

Select the methods of encryption for key exchange. Your selection allows the agent to determine the method of encryption to be used when the keys are exchanged between the servers and the SCP agent.

  • hmac-sha1
  • hmac-sha1-96
  • hmac-sha1-etm@openssh.com
  • hmac-md5
  • hmac-md5-96
  • hmac-md5-etm@openssh.com
  • hmac-sha2-256
  • hmac-sha2-256-96
  • hmac-sha2-256-etm@openssh.com
  • hmac-sha2-512
  • hmac-sha2-512-96
  • hmac-sha2-512-etm@openssh.com
  • hmac-ripemd160
  • hmac-ripemd160-etm@openssh.com