FTPS Collection Agent Configuration

To open the FTPS collection agent configuration, click  Build → New Configuration. Select Workflow from the Configurations dialog. When prompted to Select workflow type, select Batch. Click Add agent and select Ftps from the Agent Selection dialog. Double-click the agent icon or right-click the icon and select Edit agent, to display the Agent Configuration dialog.

Part of the configuration may be done in the  Filename Sequence or  Sort Order tabs described in Workflow Template.

Connection Tab

The  Connection  tab contains configuration data that is relevant to a remote server.

The FTPS collection agent configuration - Connection tab

SettingDescription

Server Information Provider

The standard behavior is Single Server. If your system is installed with the Multi Server File functionality, you can configure the FTPS agent to collect from more than one server. For further information, contact your System Administrator. 

Host

Primary host name or IP address of the remote host to be connected. If a connection cannot be established to this host, the Additional Hosts specified in the Advanced tab, are tried.

Username

Username for an account on the remote host, enabling the FTPS session to login.

Password

Password related to the Username.

Transfer Type

Data transfer type to be used during file retrieval.

  • Binary - The agent uses binary transfer type. Default setting.

  • ASCII - The agent uses ASCII transfer type.

Encryption Type

You can select an encryption type:

  • Implicit - Full encryption of FTP connection from the start of the session.
  • Explicit - Explicitly define any required security/encryption mechanisms. 

File System Type

Type of file system on the remote host.

  • Unix - Remote host using Unix file system. Default setting.

  • Windows NT - Remote host using Windows NT file system.

  • VAX/VMS - Remote host using VAX/VMS file system.

Client AuthenticationIf you select this checkbox, you must enter the keystore file location (JKS format) and password to connect to the FTPS Server. The connection fails if the file is invalid or not found, or if the key in the file is not the same as the file stored in the FTPS server.
Keystore LocationEnter the keystore file location in JKS format. You must populate this field if you have selected the Client Authentication checkbox.
Keystore PasswordEnter the keystore password. You must populate this field if you have selected the Client Authentication checkbox.
Reuse SSL/TTL SessionSelect this checkbox to reuse the SSL/TTL session.
Collection Retries Settings

Enable

Select this checkbox to enable repetitive attempts to connect and start a file transfer.

When this option is selected, the agent attempts to connect to the host as many times as is stated in the Max Retries field. If the connection fails, a new attempt is made after the number of seconds entered in the Retry Interval(s) field.

Retry Interval(s)

Enter the time interval in seconds, between retries.

If a connection problem occurs, the time interval before the first attempt to reconnect is the time set in the Timeout field in the Advanced tab plus the time set in the  Retry Interval(s) field. For the remaining attempts, the time interval is the number seconds entered in this field.

Max Retries

Enter the maximum number of retries to connect.

If more than one connection attempt is made, this number is reset as soon as a file transfer is completed successfully.

Note!

This number does not include the original connection attempt.

Restart Retries Settings

Enable 

Select this checkbox to enable the agent to send a RESTART command if the connection has been broken during a file transfer. The RESTART command contains information about where in the file you want to resume the file transfer.

Before selecting this option, ensure that the FTPS server supports the RESTART command.

When this option is selected, the agent attempts to re-establish the connection, and resumes the file transfer from the point in the file stated in the RESTART command, as many times as is entered in the Max Restarts field. When a connection has been re-established, a RESTART command is sent after the number of seconds entered in the Retry Restarts Interval(s) field.

Note!

The Restart Retries settings will not work if you have selected to decompress the files in the Source tab, see he section Source Tab below.

Note!

RESTART is not always supported for transfer type ASCII.

For further information about the RESTART command, see http://www.w3.org/Protocols/rfc959/.

Retry Restarts Interval(s)

Enter the time interval, in seconds, you want to wait before initiating a restart in this field. This time interval are applied for all restart retries.

If a connection problem occurs, the time interval before the first attempt to send a RESTART command is the time set in the Timeout field in the Advanced tab plus the time set in the  Retry Restarts Interval(s) field. For the remaining attempts, the time interval is the number seconds entered in this field.

Max Restarts

Enter the maximum number of restarts per file you want to allow.

In case more than one attempt to send the RESTART command is made, the number of used retries is reset as soon as a file transfer is completed successfully.

Source Tab

The  Source tab contains configurations related to the remote host, source directories, and source files. The following text describes the configuration options available when no custom strategy has been chosen.

The FTPS collection agent configuration - Source tab

SettingDescription

Collection Strategy

If there is more than one collection strategy available in the system, a Collection Strategy drop-down list is also visible. For more information, see Appendix 4 - Collection Strategies.

Directory

Absolute path of the source directory on the remote host, where the source files reside. If the FTPS server is of UNIX type, the path name can also be given relative to the home directory of the User Name account.

Include SubfoldersSelect this checkbox to include the subfolders of the path specified in the Directory setting.

Filename

Name of the source files on the remote host. Use regular expressions according to Java syntax. See also http://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html.


Example

To match all filenames beginning with TTFILE, type:  TTFILE.*

Note!

When collecting files from VAX file systems, the names of the source files include both path and filename, which has to be considered when entering the regular expression.

Compression

Compression type of the source files:

  • No Compression - The agent does not  decompress the files before passing them on in the workflow.

  • Gzip - The agent decompresses the files using gzip before passing them on.

Move to Temporary Directory

Select this checkbox to move the source files to the automatically created subdirectory DR_TMP_DIR in the source directory, before collection. This option supports safe collection when source files repeatedly use the same name.

Append Suffix to Filename

Enter a suffix that you want to add to the filename prior to collecting it.

Important!

Before you execute your workflow, make sure that none of the filenames in the collection directory include this suffix.

Inactive Source Warning (h)

Enter number of hours after which to display a warning message (event) in the System Log and Event Area if no file has been available for collection for the configured number of hours:

The source has been idle for more than <n> hours, the last inserted file is <file>.

Move to

If enabled, the source files will be moved from the source directory (or from the directory DR_TMP_DIR if using  Move to Temporary Directory ), to the directory specified in the Destination field, after collection.

Note!

The Directory has to be located in the same file system as the collected files at the remote host. Also, absolute pathnames must be defined (relative pathnames cannot be used).

If a file with the same filename, but with a different content, already exists in the target directory, the workflow will abort.

If a file with the same filename, AND the same content, already exists in the target directory, this file will be overwritten and the workflow will not abort.

DestinationEnter the full path of the remote host directory into which the source files should be moved after the collection. This field is only available if Move to is enabled.

Rename

Select this checkbox to rename the source file after the collection, and let them remain (or be moved back from the directory DR_TMP_DIR if using  Move to Temporary Directory ) in the source directory from which they were collected.

Note!

When the File System Type for VAX/VMS is selected, the following must be considered. If a file is renamed after collection on a VAX/VMS system, the filename might become too long. In that case the following rules apply:

A VAX/VMS filename consists of <filename>.<extension>;<version>, where the maximum number of characters for each variable is:

  • <filename>: 39 characters

  • <extension>: 39 characters

  • <version>: 5 characters

If the new filename turns out to be longer than 39 characters, the agent moves part of the filename to the extension. If the total sum of the filename and extension exceeds 78 characters, the last characters are truncated.

Example:

 A_VERY_LONG_FILENAME_WITH_MORE_THAN_39_ CHARACTERS.DAT;5

 Is converted to:

 A_VERY_LONG_FILENAME_WITH_MORE_THAN_39_. CHARACTERSDAT;5

Note!

If a new file is created on the FTPS server, with the same filename as the original file but with different content, the workflow aborts.

If a new file with the same filename AND content is created, the file is overwritten.


Enter the prefix and/or suffix to append to the beginnings and/or ends of the source filenames, respectively, after the collection. These fields are only available if Move to  or  Rename is enabled.

Warning!

If  Rename is enabled, the source files are renamed in the current (source or DR_TMP_DIR) directory. Be sure not to assign a  Prefix or  Suffix that gives the files new names that match the Filename regular expression, as this will cause the files to be collected over and over again.

Search and Replace

Select either  Move to  or  Rename to enable  Search and  Replace .

  • Search : Enter the section of the filename to replace.

  • Replace : Enter the replacement text.

Search and  Replace operate on entries in a way similar to the Unix sed utility. The identified filenames are modified and forwarded to the following agent in the workflow.

This functionality enables you to perform advanced filename modifications:

  • Use a regular expression in the Search field to specify the section of the filename that you want to extract.

    Note!

    A regular expression that fails to match the original filename aborts the workflow.

  • Enter characters and meta characters that define the pattern and content of the replacement text into the Replace field.

Search and Replace Examples

To rename the file file1.new to file1.old, use:

  • Search .new
  • Replace .old

To rename the file JAN2011_file to file_DONE, use:

  • Search ([A-Z]*[0-9]*)_([a-z]*)
  • Replace $2_DONE

Note that the search value divides the filename into two parts by using parentheses. The replace value applies to the second part by using the placeholder $2.

Remove

Select this checkbox to remove the source files from the source directory (or from the directory DR_TMP_DIR, if using  Move to Temporary Directory ), after the collection.

Keep (days)

This field is only available if Move to  or  Rename is enabled.

Enter the number of days to keep moved or renamed source files on the remote host after the collection. To delete the source files, the workflow has to be executed (scheduled or manually) again, after the configured number of days.

Note!

A date tag is added to the filename, which determines when the file can be removed. 

Ignore

Select this checkbox to let the source files remain in the source directory after the collection. This field is not available if  Move to Temporary Directory  is enabled.

Route FileReferenceUDR

Select this checkbox to forward the data to an SQL Loader agent. See SQL Loader Agent for further information.

Advanced Tab

The  Advanced  tab contains configurations related to the use of the FTPS service.

For example, in case the used FTPS server does not return the file listed in a well-defined format the  Disable File Detail Parsing  option can be useful. For information refer to that section.

The FTPS collection agent configuration - Advanced tab

SettingDescription

Command Port

Enter the port number that the FTPS service should use to connect to the remote host.

Timeout (s)

Enter the maximum time, in seconds, to wait for a response from the server. Zero (0) means to wait forever.

Passive Mode (PASV)

This checkbox must be selected if FTPS passive mode is used for data connection.

In passive mode, the channel for data transfer between client and server is initiated by the client instead of by the server. This is useful when a firewall is situated between the client and the server.

Disable File Detail Parsing

Select this checkbox to disable file detail parsing for information received from the FTPS server. This enhances the compatibility with unusual FTPS servers but disables some functionality.

If file detail parsing is disabled, file modification timestamps are not available to the collector. The collector cannot distinguish between directories and simple files. For this reason, sub-directories in the input directory must not match the Filename regular expression. The agent assumes that a file named DR_TMP_DIR is a directory because a directory named DR_TMP_DIR is used when Move to Temporary Directory in the Source tab is selected. Therefore, you cannot name a regular file DR_TMP_DIR in the collection directory.

Note!

When collecting files from a VAX file system, this checkbox must be selected.

Additional Hosts

Select Add to enter additional hosts or IP addresses from where to collect the source files. If the agent fails to connect to the remote Host specified in the Connection tab, it tries to connect to the hosts listed here, in sequence from top to bottom.

Use the  Add Edit Remove , U p, and  Down buttons to configure the host list.