SFTP

Owned by Former user (Deleted)
Last updated: Dec 05, 2024 by Casey Han
7 min read

The SFTP Functions enable communication with an SFTP server. This provides a secure way to transfer files from and to a remote location.

Disclaimer!

File storage maintenance of the SFTP server, including storage, cleanup, and permissions, is your responsibility. As the function may involve transferral or removal of files in the SFTP server, it is recommended that you perform the necessary maintenance of your file storage before you run any streams with this function.

The following SFTP Functions are available in Usage Engine: 

To configure an SFTP collector or forwarder function, you must keep the hostname or IP address and the user credentials at hand.

In Connection, enter the following common options available to both collector and forwarder functions:

Field 

Description

Field 

Description

Host name or IP address

The host name or IP address of the SFTP server you want to connect to.

Port

Port number of the SFTP server.

In Credentials, enter the following common options available to both collector and forwarder functions:

Field 

Description

Field 

Description

Authentication method

Select a method to authenticate the SFTP Client. The functions support both password and private key authentication.

Username

The username of the SFTP account.

Password

Note!

The Password option is applicable only for the Password authentication method.

The password of the SFTP account.

Private key

Note!

The Private key option is applicable only for the Private key authentication method.

Private key to be used for SFTP Client authentication.

Passphrase (for encrypted private key)

Passphrase used to encrypt/decrypt the private key. If the private key is protected by a passphrase, the passphrase must be provided as well.
This is an optional field.

All errors are logged in the system logs. See Logs for more information. 

Public and Private Keys

If a stream uses the SFTP function to use private key authentication, the creator of the stream must also create both a public key and a private key.

The public key must be shared, for example by email, with the SFTP server owner. The SFTP server owner then imports the public key into the SFTP server.

When you use the SFTP client Function, copy and paste the private key into the Private Key field in the SFTP client. Once this is done the stream can be run and connect the SFTP client to the SFTP server.

SFTP Collector

The SFTP collector function enables you to collect data from an SFTP server. 

To configure the SFTP collector function, take the following steps:

  1. In Connection, specify the connection details to connect to the server.

  2. In Credentials, specify the connection details and user credentials to connect to the server. To avoid manual entry of credentials, you can enable the Secrets Wallet toggle to select your SFTP credentials stored in the Secrets wallet. 

  3. In File location, specify the Path to the folder that you want to collect data from. 

  4. In Collection strategy, you can decide how you wish to collect your files.

    1. Select the Collect files according to one of these options:

      • In alphabetical order (A-Z)

      • In reverse alphabetical order (Z-A)

      • By last modification (most recent first)

      • By last modification (oldest first)

    2. Optionally, you can check the Exclude files check box to exclude files based on the Last modification date and set the Condition, Value, and Duration for exclusion. For example, you can exclude files that have been modified before (Anterior to) or after (Posterior to) the Last modification date.

  5. Enable the After Collection toggle if you want to perform an action on the source file after the operation of the collector has completed. You can check one of the following actions below:

Action on source file

Action description

Action on source file

Action description

Remove

Removes the source file after the operation of the collector is completed.

Move to

Moves the source file to a destination folder. When this option is selected, you have to specify the folder path in the Folder field. The destination folder must be in the same SFTP.

You can also check the Overwrite if file already exists check box, if you want to overwrite the file with the same name found in the folder.

  1. In File information, you can select the file selection method, the file format and additional options for those formats. Under File selection options, specify whether you want to collect data from All files in path, Specific file, or Based on regular expression.

File selection options

File format

File selection options

File format

All files in path

Select the File format. The following options are available:

CSV: Specify a Delimiter for the CSV file format. The default value is ' , '.

Excel: Specify the Sheet selection options. The following options are available:

  • All sheets in file: Select all sheets in the file.

  • Specific sheet(s): Enter the sheet name(s) in Select sheet(s).

JSON: Select this option to collect a file in the JSON format.

XML : Select this option to collect a file in the XML format.

Specific file(s)

Specify the Filename and select the File format. One or multiple files can be used.

The following options are available:

CSV: Specify a Delimiter for the CSV file format. The default value is ' , '.

Excel: Specify the Sheet selection options. The following options are available:

  • All sheets in file: Select all sheets in the file.

  • Specific sheet(s): Enter the sheet name(s) in Select sheet(s).

JSON: Select this option to collect a file in the JSON format.

XML : Select this option to collect a file in the XML format.

Based on regular expression

If the file name is in the regular expression format (regex format), you must select the Based on regular expression, and specify the File name pattern.

  1. Select the Include table header check box if there are headers in the CSV or Excel file(s) that you are collecting data from.

SFTP Forwarder

The SFTP forwarder function enables you to forward data to an SFTP server.

To configure the SFTP forwarder function, take the following steps:

  1. In Connection Details, specify the connection details and user credentials to connect to the server. To avoid manual entry of credentials, you can enable the Secrets Wallet toggle to select your SFTP credentials stored in the Secrets Wallet. 

  2. In File Information, specify the path of the file in Path. The folder path can begin or end with ' / '. If you do not specify '/' at the beginning of the folder path, the output file will be created in the home directory.

  3. In Output file Information, specify how you want to handle the output file(s) in Filename options. You can select from the following options:

File Output options

Description

File Output options

Description

Collector filename

Select Collector filename if you want to keep the same filename as your input file(s).

If the input file does not have a filename, the system generates a name based on the collector function, for example, Counter.

The function appends the file format extension to the filename based on the file format selected in Step 5, if the selected extension is not in the filename.

Custom filename

Select Custom filename to define a new filename for all the output files.

The function appends the file format extension to the filename based on the file format selected in Step 5, so you do not need to add any extensions to the filename.

  1. In File name options, select Append timestamp to append the timestamp with the name of the output file.

  2. In File format, select the format of the output file(s) from the following options:

File format options

Description

File format options

Description

CSV

Select to send the output file in CSV format. Select Include table header to include the table header in the output file(s). Specify a Delimiter for the CSV file format. The default value is ' , '.

Excel

Select to send the output file in Excel format. Specify the name of the sheet in Sheet name.

JSON

Select to send the output file in JSON format. Select the preferred output format, Action on records: one file with All in one array, one file with All in one array with key, or One file per record.

Buffer

Select to send the output file in Buffer format. If you are reading or processing files containing binary data (Buffer format), for example for performance or other reasons, you can write these files through the STFP Forwarder.

  1. To output JSON files in a more compact form, deselect the Output file in pretty print  checkbox. By default, pretty print is on.

  2. To compress the output file, select the Compress file checkbox and specify the Compression format. The supported formats are:

    • Zip

    • Gzip

SFTP Metadata

You can view and access the following metadata properties of SFTP. To view the metadata, use the meta object as mentioned in the Script function.

Property name 

Description

Property name 

Description

fileName

 

Name of the file.

filePath

The path from where the file needs to be collected. File can be in Excel or CSV format.

Path format: folder/file

fileSize

Size of the file (in bytes)

sheetName

Name of the sheet in the Excel file

collectionTime

Timestamp when the file is read

Â