The SFTP Functions enable communication with an SFTP server. This provides a secure way to transfer files from and to a remote location. 

The following SFTP Functions are available in Usage Engine: 

• SFTP Collector
• SFTP Forwarder

To configure an SFTP collector or forwarder Function, you must keep the hostname or IP address and the user credentials at hand.

In Connection Details, enter the following common options available to both collector and forwarder functions:

All errors are logged in the system logs. See Logs for more information. 

Public and Private Keys

If a stream uses the SFTP Function to use private key authentication, the creator of the stream must also create both a public key and a private key.

The public key must be shared, for example by email, with the SFTP server owner. The SFTP server owner then imports the public key into the SFTP server.

When you use the SFTP client Function, copy and paste the private key into the Private Key field in the SFTP client. Once this is done the stream can be run and connect the SFTP client to the SFTP server.

SFTP Collector

The SFTP collector Function enables you to collect data from an SFTP server. 

To configure the SFTP collector function, take the following steps:

1. In Connection Details, specify the connection details and user credentials to connect to the server. To avoid manual entry of credentials, you can enable the Secrets Wallet toggle to select your SFTP credentials stored in the Secrets Wallet. 
2. In File Information, specify the Path to the folder that you want to collect data from. 

3. In File selection options, specify whether you want to collect data from All files in folder or Specific file. You can specify if want to collect data from CSV, Excel or XML file formats.
   
   

   File selection options	File format
   All files in path	Select the File format. The following options are available: CSV: Specify a Delimiter for the CSV file format. The default value is ' , '. Excel: Specify the Sheet selection options. The following options are available: All sheets in file: Select all sheets in the file. Specific sheet(s): Enter the sheet name(s) in Select sheet(s). JSON: Select this option to collect a file in the JSON format. XML : Select this option to collect a file in the XML format.
   Specific file(s)	Specify the File Name and select the File format. One or multiple files can be used. If the file name is in the regular expression format (regex format), you must select the Use Regex check box and specify the File name pattern. The following options are available: CSV: Specify a Delimiter for the CSV file format. The default value is ' , '. Excel: Specify the Sheet selection options. The following options are available: All sheets in file: Select all sheets in the file. Specific sheet(s): Enter the sheet name(s) in Select sheet(s). JSON: Select this option to collect a file in the JSON format. XML : Select this option to collect a file in the XML format.

   Note!

   During collection,

   • Compressed files are automatically de-compressed.
   • The type of archive file format is automatically identified based on the contents of the file instead of the file extension. The supported archive file formats are ZIP, gzip and zlib.

   For all supported archive file formats,

   • The archive must contain only a single file that is compressed.
   • The archive must not contain any directories.

   
   

4. Select the Include table header checkbox if there are headers in the CSV or Excel file(s) that you are collecting data from.
   

5. In Collection strategy, you can decide how you wish to collect your files. There are multiple options such as:
   1. In alphabetical order (A-Z)
   2. In reverse alphabetical order (Z-A)
   3. By creation date (most recent first)
   4. By creation date (oldest first)
   5. By last modified (most recent first)

   6. By last modified (oldest first)

      By selecting the Exclude files checkbox, you can exclude files based on the Last modification date and set the Condition for exclusion. For example, you can choose to exclude files that have been modified before or after the Last modification date.
6. In After Collection, you can choose the action to be performed on the source file after collecting data. You can choose to either Ignore or Delete the file.

SFTP Forwarder

The SFTP forwarder Function enables you to forward data to an SFTP server.

To configure the SFTP forwarder Function, take the following steps:

1. In Connection Details, specify the connection details and user credentials to connect to the server. To avoid manual entry of credentials, you can enable the Secrets Wallet toggle to select your SFTP credentials stored in the Secrets Wallet. 
2. In File Information, specify the path of the file in Path. The folder path can begin or end with ' / '. If you do not specify '/' at the beginning of the folder path, the output file will be created in the home directory.

3. In Output file Information, specify how you want to handle the output file(s) in Filename options. You can select from the following options:
   
   

   File Output options	Description
   Collector filename	Select Collector filename if you want to keep the same filename as your input file(s). If the collector does not have a filename, for example, Counter, the system generates a name based on the Function.
   Custom filename	Select Custom filename to define a new filename for all the output files.

   
   

4. In File name options, select Append timestamp to append the timestamp with the name of the output file.
   
   

   Note!

   If Append timestamp is not selected, the existing file at the destination can be overwritten by the output file.

   There are exceptions to the filename patterns when it comes to the Collector function. 

   • Count and other collector Functions that do not read files generate a new filename based on the name of the Collector Function. 
   • Script — If the data is flushed out by the Script Function, a new filename will be created based on the name of the Script Function.
   • Aggregate — A filename will be applied based on the name of the Aggregate Function.

   
   

5. In File format, select the format of the output file(s) from the following options:
   

   File format options	Description
   CSV	Select to send the output file in CSV format. Select Include table header to include the table header in the output file(s). Specify a Delimiter for the CSV file format. The default value is ' , '.
   Excel	Select to send the output file in Excel format. Specify the name of the sheet in Sheet name.
   JSON	Select to send the output file in JSON format. Select the preferred output format, Action on records: one file with All in one array, one file with All in one array with key, or One file per record. Note! If One file per record is selected, you must also select Append timestamp, otherwise, the files will be overwritten.
   Buffer	Select to send the output file in Buffer format. If you are reading or processing files containing binary data (Buffer format), for example for performance or other reasons, you can write these files through the STFP Forwarder.

6. To output JSON files in a more compact form, deselect the Output file in pretty print  checkbox. By default, pretty print is on.

   Note!

   Selecting pretty print increases the size of the output file.

7. To compress the output file, select the Compress file checkbox and specify the Compression format. The supported formats are:
   • Zip
   • Gzip

SFTP Metadata

You can view and access the following metadata properties of SFTP. To view the metadata, use the meta object as mentioned in the Script Function.