SFTP

Owned by Former user (Deleted)
Last updated: Aug 29, 2024 by Casey Han
6 min read

The SFTP Functions enable communication with an SFTP server. This provides a secure way to transfer files from and to a remote location. 

The following SFTP Functions are available in Usage Engine: 

  • SFTP Collector

  • SFTP Forwarder

To configure an SFTP collector or forwarder function, you must keep the hostname or IP address and the user credentials at hand.

To avoid manual entry of credentials, you can enable the Secrets Wallet toggle to select your SFTP credentials stored in the Secrets Wallet.

In Connection Details, enter the following common options available to both collector and forwarder functions:

Field 

Description

Field 

Description

Host name or IP address

The host name or IP address of the SFTP server you want to connect to.

Port

Port number of the SFTP server.

Authentication by

Select a method to authenticate the SFTP Client. Supports both password and private key authentication.

Password

The password of the SFTP account

Private key

Private key to be used for SFTP Client authentication.

Passphrase

Passphrase used to encrypt/decrypt the private key. If the private key is protected by a passphrase, the passphrase must be provided as well.
This is an optional field. Applies only if the Private Key authentication is enabled.

All errors are logged in the system logs. See Logs for more information. 

Public and Private Keys

If a stream uses the SFTP function to use private key authentication, the creator of the stream must also create both a public key and a private key.

Note!

The private key must be handled with care and it must be regarded as highly sensitive information. Keep it safe at all times.  

The public key must be shared, for example by email, with the SFTP server owner. The SFTP server owner then imports the public key into the SFTP server.

When you use the SFTP client Function, copy and paste the private key into the Private Key field in the SFTP client. Once this is done the stream can be run and connect the SFTP client to the SFTP server.

SFTP Collector

The SFTP collector function enables you to collect data from an SFTP server. 

To configure the SFTP collector function, take the following steps:

  1. In Connection Details, specify the connection details and user credentials to connect to the server. To avoid manual entry of credentials, you can enable the Secrets Wallet toggle to select your SFTP credentials stored in the Secrets Wallet. 

  2. In File Information, specify the Path to the folder that you want to collect data from. 

  3. In File selection options, specify whether you want to collect data from All files in folder or Specific file. You can specify if want to collect data from CSV, Excel or XML file formats.

     

     

  4. Select the Include table header checkbox if there are headers in the CSV or Excel file(s) that you are collecting data from.

  5. In Collection strategy, you can decide how you wish to collect your files. There are multiple options such as:

    1. In alphabetical order (A-Z)

    2. In reverse alphabetical order (Z-A)

    3. By creation date (most recent first)

    4. By creation date (oldest first)

    5. By last modified (most recent first)

    6. By last modified (oldest first)

  6. In After Collection, you can choose the action to be performed on the source file after collecting data. You can choose to either Ignore or Delete the file.

SFTP Forwarder

The SFTP forwarder function enables you to forward data to an SFTP server.

To configure the SFTP forwarder function, take the following steps:

  1. In Connection Details, specify the connection details and user credentials to connect to the server. To avoid manual entry of credentials, you can enable the Secrets Wallet toggle to select your SFTP credentials stored in the Secrets Wallet. 

  2. In File Information, specify the path of the file in Path. The folder path can begin or end with ' / '. If you do not specify '/' at the beginning of the folder path, the output file will be created in the home directory.

  3. In Output file Information, specify how you want to handle the output file(s) in Filename options. You can select from the following options:

File Output options

Description

File Output options

Description

Collector filename

Select Collector filename if you want to keep the same filename as your input file(s).

If the input file does not have a filename, the system generates a name based on the collector function, for example, Counter.

The function appends the file format extension to the filename based on the file format selected in Step 5, if the selected extension is not in the filename.

Custom filename

Select Custom filename to define a new filename for all the output files.

The function appends the file format extension to the filename based on the file format selected in Step 5, so you do not need to add any extensions to the filename.

Note!

For existing streams, if you would like to use the new filename saving method as described above, you can check the New filename saving method check box.

  1. In File name options, select Append timestamp to append the timestamp with the name of the output file.

  2. In File format, select the format of the output file(s) from the following options:

File format options

Description

File format options

Description

CSV

Select to send the output file in CSV format. Select Include table header to include the table header in the output file(s). Specify a Delimiter for the CSV file format. The default value is ' , '.

Excel

Select to send the output file in Excel format. Specify the name of the sheet in Sheet name.

JSON

Select to send the output file in JSON format. Select the preferred output format, Action on records: one file with All in one array, one file with All in one array with key, or One file per record.

Buffer

Select to send the output file in Buffer format. If you are reading or processing files containing binary data (Buffer format), for example for performance or other reasons, you can write these files through the STFP Forwarder.

  1. To output JSON files in a more compact form, deselect the Output file in pretty print  checkbox. By default, pretty print is on.

  2. To compress the output file, select the Compress file checkbox and specify the Compression format. The supported formats are:

    • Zip

    • Gzip

SFTP Metadata

You can view and access the following metadata properties of SFTP. To view the metadata, use the meta object as mentioned in the Script function.

Property name 

Description

Property name 

Description

fileName

 

Name of the file.

filePath

The path from where the file needs to be collected. File can be in Excel or CSV format.

Path format: folder/file

fileSize

Size of the file (in bytes)

sheetName

Name of the sheet in the Excel file

collectionTime

Timestamp when the file is read