Versions Compared

Old Version 10

changes.mady.by.user Former user

Saved on

compared with

New Version Current

changes.mady.by.user Casey Han

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

The following SFTP Functions are available in Usage Engine: 

  • SFTP Collector

  • SFTP Forwarder

To configure an SFTP collector or forwarder Functionfunction, you must keep the hostname or IP address and the user credentials at hand.

...

In Connection Details, enter the following common options available to both collector and forwarder functions:

Field 

Description

Host name or IP address

The host name or IP address of the SFTP server you want to connect to.

Port

Port number of the SFTP server.

Authentication by

Select a method to authenticate the SFTP Client. Supports both password and private key authentication.

Password

The password of the SFTP account

Private key

Private key to be used for SFTP Client authentication.

Passphrase

Passphrase used to encrypt/decrypt the private key. If the private key is protected by a passphrase, the passphrase must be provided as well.
This is an optional field. Applies only if the Private Key authentication is enabled.

All errors are logged in the system logs. See Logs for more information. 

...

If a stream uses the SFTP Function function to use private key authentication, the creator of the stream must also create both a public key and a private key.

title
Note

Note!

The private key must be handled with care and it must be regarded as highly sensitive information. Keep it safe at all times.  

...

When you use the SFTP client Function, copy and paste the private key into the Private Key field in the SFTP client. Once this is done the stream can be run and connect the SFTP client to the SFTP server.

...

SFTP Collector

The SFTP collector Function function enables you to collect data from an SFTP server. 

To configure the SFTP collector function, take the following steps:

  1. In Connection Details, specify the connection details and user credentials to connect to the server. To avoid manual entry of credentials, you can enable the Secrets Wallet toggle to select your SFTP credentials stored in the Secrets Wallet. 

  2. In File Information, specify the Path to the folder that you want to collect data from. 

  3. In File selection options, specify whether you want to collect data from All files in folder or Specific file. You can specify if want to collect data from CSV, Excel or XML file formats.

    File selection optionsFile format
    All files in path

    Select the File format. The following options are available:

    CSV: Specify a Delimiter for the CSV file format. The default value is ' , '.

    Excel: Specify the Sheet selection options. The following options are available:

    • All sheets in file: Select all sheets in the file.
    • Specific sheet(s): Enter the sheet name(s) in Select sheet(s).

    JSON: Select this option to collect a file in the JSON format.

    XML : Select this option to collect a file in the XML format.

    Specific file(s)

    Specify the File Name and select the File format. One or multiple files can be used.

    If the file name is in the regular expression format (regex format), you must select the Use Regex check box and specify the File name pattern.

    The following options are available:

    CSV: Specify a Delimiter for the CSV file format. The default value is ' , '.

    Excel: Specify the Sheet selection options. The following options are available:

    • All sheets in file: Select all sheets in the file.
    • Specific sheet(s): Enter the sheet name(s) in Select sheet(s).

    JSON: Select this option to collect a file in the JSON format.

    XML : Select this option to collect a file in the XML format.

    Note
    titleNote!

    During collection,

    • Compressed files are automatically de-compressed.
    • The type of archive file format is automatically identified based on the contents of the file instead of the file extension. The supported archive file formats are ZIP, gzip and zlib.

    For all supported archive file formats,

    • The archive must contain only a single file that is compressed.
    • The archive must not contain any directories.

  4. Select the Include table header checkbox if there are headers in the CSV or Excel file(s) that you are collecting data from.

  5. In Collection strategy, you can decide how you wish to collect your files. There are multiple options such as:

    1. In alphabetical order (A-Z)

    2. In reverse alphabetical order (Z-A)

    3. By creation date (most recent first)

    4. By creation date (oldest first)

    5. By last modified (most recent first)

    6. By last modified (oldest first)

      Info
      By selecting the Exclude files checkbox, you can exclude files based on the Last modification date and set the Condition for exclusion. For example, you can choose to exclude files that have been modified before or after the Last modification date.

  6. In After Collection, you can choose the action to be performed on the source file after collecting data. You can choose to either Ignore or Delete the file.

...

SFTP Forwarder

The SFTP forwarder Function function enables you to forward data to an SFTP server.

To configure the SFTP forwarder Functionfunction, take the following steps:

  1. In Connection Details, specify the connection details and user credentials to connect to the server. To avoid manual entry of credentials, you can enable the Secrets Wallet toggle to select your SFTP credentials stored in the Secrets Wallet. 

  2. In File Information, specify the path of the file in Path. The folder path can begin or end with ' / '. If you do not specify '/' at the beginning of the folder path, the output file will be created in the home directory.

  3. In Output file Information, specify how you want to handle the output file(s) in Filename options. You can select from the following options:

File Output options

Description

Collector filename

Select Collector filename if you want to keep the same filename as your input file(s).

If the

...

input file does not have a filename, the system generates a name based on the collector function, for example, Counter

...

.

The function appends the file format extension to the filename based on the file format selected in Step 5, if the selected extension is not in the filename.

Custom filename

Select Custom filename to define a new filename for all the output files.

The function appends the file format extension to the filename based on the file format selected in Step 5, so you do not need to add any extensions to the filename.

Note

Note!

For existing streams, if you would like to use the new filename saving method as described above, you can check the New filename saving method check box.

  1. In File name options, select Append timestamp to append the timestamp with the name of the output file.

    Note
    titleNote!

    If Append timestamp is not selected, the existing file at the destination can be overwritten by the output file.

    There are exceptions to the filename patterns when it comes to the Collector function. 

    • Count and other collector Functions that do not read files generate a new filename based on the name of the Collector Function. 
    • Script — If the data is flushed out by the Script Function, a new filename will be created based on the name of the Script Function.
    • Aggregate — A filename will be applied based on the name of the Aggregate Function.

  2. In File format, select the format of the output file(s) from the following options:

File format options

Description

CSV

Select to send the output file in CSV format. Select Include table header to include the table header in the output file(s). Specify a Delimiter for the CSV file format. The default value is ' , '.

Excel

Select to send the output file in Excel format. Specify the name of the sheet in Sheet name.

JSON

Select to send the output file in JSON format. Select the preferred output format, Action on records: one file with All in one array, one file with All in one array with key, or One file per record.

Note

...

Note!

If One file per record is selected, you must also select Append timestamp, otherwise, the files will be overwritten.

Buffer

Select to send the output file in Buffer format. If you are reading or processing files containing binary data (Buffer format), for example for performance or other reasons, you can write these files through the STFP Forwarder.

  1. To output JSON files in a more compact form, deselect the Output file in pretty print  checkbox. By default, pretty print is on.

    Note
    titleNote!

    Selecting pretty print increases the size of the output file.

  2. To compress the output file, select the Compress file checkbox and specify the Compression format. The supported formats are:

    • Zip

    • Gzip

SFTP Metadata

You can view and access the following metadata properties of SFTP. To view the metadata, use the meta object as mentioned in the Script Function function.

Property name 

Description

fileName

Name of the file.

filePath

The path from where the file needs to be collected. File can be in Excel or CSV format.

Path format: folder/file

fileSize

Size of the file (in bytes)

sheetName

Name of the sheet in the Excel file

collectionTime

Timestamp when the file is read