SFTP

The SFTP functions enable communication with an SFTP server. This provides a secure way to transfer files from and to a remote location.

Note!

To view a list of encryption and key exchange algorithms supported by the SFTP functions, read the GitHub - mscdex/ssh2: SSH2 client and server modules written in pure JavaScript for node.js documentation. This source provides the most accurate and up-to-date details on supported algorithms.

Caution!

File storage maintenance of the SFTP server, including storage, cleanup, and permissions, is your responsibility. As the function may involve transferring or removing files from the SFTP server, it is recommended that you perform the necessary maintenance on your file storage before running any streams with this function.

1 Public and private keys
2 SFTP collector
3 SFTP forwarder
4 SFTP metadata

To configure an SFTP collector or forwarder function, you must have the hostname or IP address and user credentials readily available.

In Connection, enter the following common options available to both the collector and forwarder functions:

Field	Description

Field	Description
Host name or IP address	The host name or IP address of the SFTP server you want to connect to.
Port	Port number of the SFTP server.

In Credentials, enter the following common options available to both collector and forwarder functions:

Field	Description

Field	Description
Authentication method	Select a method to authenticate the SFTP client. The functions support both password and private key authentication.
Username	The username of the SFTP account.
Password	Note! The Password option is applicable only for the password authentication method. The password of the SFTP account.
Private key	Note! The Private key option is applicable only for the Private key authentication method. Private key to be used for SFTP Client authentication.
Passphrase (for encrypted private key)	Note! The Passphrase (for encrypted private key) option is applicable only for the Private key authentication method. Passphrase used to encrypt/decrypt the private key. If the private key is protected by a passphrase, the passphrase must be provided as well. This is an optional field.

Note!

To avoid manual entry of credentials, you can enable the Secrets wallet toggle to select your SFTP credentials stored in the Secrets wallet.

All errors are logged in the system logs. See Logs for more information.

Public and private keys

If a stream uses the SFTP function to use private key authentication, the creator of the stream must also create both a public key and a private key.

Note!

The private key must be handled with care and it must be regarded as highly sensitive information. Keep it safe at all times.

The public key must be shared, for example by email, with the SFTP server owner. The SFTP server owner then imports the public key into the SFTP server.

When you use the SFTP client function, copy and paste the private key into the Private key field in the SFTP client. Once this is done the stream can be run and connect the SFTP client to the SFTP server.

SFTP collector

The SFTP collector function enables you to collect data from an SFTP server.

To configure the SFTP collector function, take the following steps:

In Connection, specify the connection details to connect to the server.
In Credentials, specify the connection details and user credentials to connect to the server. To avoid manual entry of credentials, you can enable the Secrets wallet toggle to select your SFTP credentials stored in the Secrets wallet.
In File location, specify the Path to the folder that you want to collect data from.
In Collection strategy, you can decide how you wish to collect your files.
1. Select the Collect files according to one of these options:
  - In alphabetical order (A-Z)
  - In reverse alphabetical order (Z-A)
  - By last modification (most recent first)
  - By last modification (oldest first)
2. Optionally, you can check the Exclude files check box to exclude files based on the Last modification date and set the Condition, Value, and Duration for exclusion. For example, you can exclude files that have been modified before (Anterior to) or after (Posterior to) the Last modification date.
Enable the After Collection toggle if you want to perform an action on the source file after the operation of the collector has completed. You can check one of the following actions below:

Action on source file	Action description

Action on source file

Action description

Remove

Removes the source file after the operation of the collector is completed.

Move to

Moves the source file to a destination folder. When this option is selected, you have to specify the folder path in the Folder field. The destination folder must be in the same SFTP.

You can also check the Overwrite if file already exists check box, if you want to overwrite the file with the same name found in the folder.

In File information, you can select the file selection method, the file format and additional options for those formats. Under File selection options, specify whether you want to collect data from All files in path, Specific file, or Based on regular expression.

File selection options	File format

File selection options

File format

All files in path

Select the File format. The following options are available:

CSV: Specify a Delimiter for the CSV file format. The default value is ' , '.

Note!

The Delimiter option is applicable only for the CSV file format.

Excel - Specify the Sheet selection options. The following options are available:

All sheets in file: Select all sheets in the file.
Specific sheet(s): Enter the sheet name(s) in Select sheet(s).

Note!

You can only preview Excel files of 10MB or less.

JSON - Select this option to collect a file in the JSON format.

XML - Select this option to collect a file in the XML format.

Note!

You can only preview XML files of 20MB or less.

Specific file(s)

Specify the Filename and select the File format. One or multiple files can be used.

The following options are available:

CSV: Specify a Delimiter for the CSV file format. The default value is ' , '.

Note!

The Delimiter option is applicable only for the CSV file format.

Excel - Specify the Sheet selection options. The following options are available:

All sheets in file: Select all sheets in the file.
Specific sheet(s): Enter the sheet name(s) in Select sheet(s).

Note!

You can only preview Excel files of 10MB or less.

JSON: Select this option to collect a file in the JSON format.

XML : Select this option to collect a file in the XML format.

Note!

You can only preview XML files of 20MB or less.

Based on regular expression

If the file name is in the regular expression format (regex format), you must select the Based on regular expression, and specify the File name pattern.

Note!

During collection, the following occurs:

Compressed files are automatically de-compressed.
The type of archive file format is automatically identified based on the contents of the file instead of the file extension. The supported archive file formats are ZIP, gzip and zlib.

For all supported archive file formats,

The archive must contain only a single file that is compressed.
The archive must not contain any directories.

Select the Include table header check box if there are headers in the CSV or Excel file(s) that you are collecting data from.

SFTP forwarder

The SFTP forwarder function enables you to forward data to an SFTP server.

To configure the SFTP forwarder function, take the following steps:

In Connection, specify the connection details and user credentials to connect to the server. To avoid manual entry of credentials, you can enable the Secrets Wallet toggle to select your SFTP credentials stored in the Secrets Wallet.
In File location, specify the path of the file in Path. The folder path can begin or end with ' / '. If you do not specify '/' at the beginning of the folder path, the output file will be created in the home directory.
In Output file information, specify how you want to handle the output file in Filename options. You can select from the following options:

File Output options	Description

File Output options

Description

Collector filename

Select Collector filename if you want to keep the same filename as your input file.

If the collector does not have a filename, the function generates a name based on the function, for example, Counter.

If you use the Script function and send the data out in the Flush that is called at the end of each transaction, the original filename is lost, and a new filename is generated based on the name of the Script function.

If you use the Aggregate function, it will merge all the payloads and generate a new filename based on the name of the Aggregate function.

The function appends the file format extension to the filename based on the file format selected in Step 5, if the selected extension is not in the filename.

Custom filename

Select Custom filename to define a new filename for all the output files.

The function appends the file format extension to the filename based on the file format selected in Step 5, so you do not need to add any extensions to the filename.

Note!

By default, the Append timestamp checkbox is checked. Select the Append timestamp checkbox to append the timestamp to the name of the output file. For example, if you use a CSV file, the filename will then be <myfile>_<timestamp>.csv.

Note!

For existing streams, if you would like to use the new filename saving method as described above, you can check the New filename saving method check box.

Note!

If a file with the same name already exists in the SFTP path, the operation results in an error. In this case, you encounter an error with code 4. Common scenarios that can cause an error with code 4 include:

Renaming a file to match the name of an existing file.
Creating a directory that is already present.
Moving a remote file to another file system.
Uploading a file when the file system is full.

These are only some possible reasons for the error.

Note!

There are exceptions to the filename patterns when it comes to the collector function.

Count and other collector functions that do not read files generate a new filename based on the name of the collector function.
Script - If the data is flushed out by the Script function, a new filename will be created based on the name of the Script function.
Aggregate - A filename will be applied based on the name of the Aggregate function.

In File format, select the format of the output file from the following options:

File format options	Description

File format options	Description
CSV	Select to send the output file in CSV format. Select Include table header to include the table header in the output file. Specify a Delimiter for the CSV file format. The default value is ' , '. Note! SFTP forwarder follows a stricter definition of CSV when writing files. The forwarder assumes an equal number of columns for all rows to ensure consistency and compatibility with downstream processes. This means that while the SFTP collector can read CSV files with varying numbers of fields, the forwarder requires a uniform structure when writing CSV files. The snippet below is an example of a CSV file output that is produced by the forwarder with the same number of columns for each row. Columns with no value are represented by no inputs between the separators. `1,2,3,4 1,,, 1,,2,4` Note! The order of columns in a CSV file may not always match what you set in a function. For example, if your headers include both numbers and letters, Usage Engine will place numeric headers first, followed by alphabetical ones.
Excel	Select to send the output file in Excel format. Specify the name of the sheet in Sheet name.
JSON	Select to send the output file in JSON format. Select the preferred output format you want to use in the Action on record: drop-down list; one file with All in one array one file with All in one array with key or One file per record Note! If One file per record is selected, you must also select Append timestamp. NDJSON (Newline delimited JSON) Note! If NDJSON is selected, Output file in pretty print is not applicable and therefore this option will be hidden.
Buffer	Select to send the output file in Buffer format. If you are reading or processing files containing binary data (Buffer format), for example for performance or other reasons, you can write these files through the STFP forwarder.

To output JSON files in a more compact form, deselect the Output file in pretty print checkbox. By default, pretty print is on.

Note!

Selecting pretty print increases the size of the output file.

To compress the output file, select the Compress file checkbox and specify the Compression format. The supported formats are:
- Zip
- Gzip

SFTP metadata

You can view and access the following metadata properties of SFTP. To view the metadata, use the meta object as mentioned in the Script function.

Property name	Description

Property name	Description
fileName	Name of the file.
filePath	The path from where the file needs to be collected. File can be in Excel or CSV format. Path format: folder/file
fileSize	Size of the file (in bytes).
sheetName	Name of the sheet in the Excel file.
collectionTime	Timestamp when the file is read.