Versions Compared

Old Version 1

changes.mady.by.user Casey Han

Saved on

compared with

New Version Current

changes.mady.by.user Casey Han

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

See Stream using Amazon S3 Functions - Acme EV for an example how the Amazone S3 collector and forwarder can be used in a stream.

Anchor
common
common
Common Configurations

For both functions, you need to configure AWS Credentials and File location.

AWS Credentials

The AWS Credentials can either be specified directly in the configuration dialog for the functions, or in an AWS Secret in the Secrets Wallet, and must contain:

Setting

Description

Access Key

The identifier used to sign the requests sent to Amazon S3, is referenced by S3 as the Access Key ID. 

Secret Key

The Secret Key is used in conjunction with the Access Key to cryptographically sign Amazon AWS requests. When you create the Access Key in Amazon S3 you can view and download the Secret Key. S3 references this as the Secret Access Key.

File location

The File location settings include:

Setting

Description

Bucket

The Bucket is the public cloud storage resource available in Amazon S3.

Folder

Enther the path to the folder you want to collect data from and push data to when using the Amazon S3 functions in this field.

Note

Note!

If the path to the folder is not specified, the root folder of the Amazon S3 bucket is selected by default.

Anchor
AWSS3Collector
AWSS3Collector
Amazon S3 Collector Configuration

With the Amazon S3 Collector you can collect data from your AWS S3 bucket for processing in your stream.

...

Note

Note!

During collection, the following applies:

  • Compressed files are automatically decompressed.

  • The type of archive file format is automatically identified based on the contents of the file instead of the file extension. The supported archive file formats are ZIP, gzip and zlib.

For all supported archive file types, the following applies:

  • The archive must contain only a single file that is compressed.

  • The archive must not contain any directories.

There are exceptions to the filename patterns when it comes to collector functions. 

Anchor
AWSS3Forwarder
AWSS3Forwarder
Amazon S3 Forwarder

With the Amazon S3 Forwarder function you can send data to an Amazon S3 bucket from your stream.

...

  1. In Output file Information section, specify how you want to handle the output file(s) in Filename options:

Filename option

Action

Collector filename

Select this option if you want to keep the same filename as on the input file(s).

Custom filename

Select this option to define a new filename for the output files, see Configuring Dynamic Naming in Fields for information on how to define file names with additional flexibilit

...

File format

Depending on what option you select in the this drop-down, you will see different complimentary settings beneath the drop-down list.

CSV

Select this option to send the files in CSV format. Select the Include table header check box to include the table header in the output file(s).

Select the delimiter for the CSV file(s) in the Delimiter field. Comma is selected by default but you can also select Tab, Semicolon, Space, or Other. If you select Other you can specify a custom delimiter in the Custom delimiter field that will then be displayed.

You can select to compress the files by selecting the Compress file option. A drop-down list will then be displayed in which you can select either Zip or GZip.

Select the Bucket owner full control option to enable full control over any new objects that are written by other AWS accounts.

Excel

Select this option to send the output file in Excel format. You can then select to specify the name of the sheet in the Sheet name field that is displayed. The default name is “Sheet 1”.

You can select to compress the files by selecting the Compress file option. A drop-down list will then be displayed in which you can select either Zip or GZip.

Select the Bucket owner full control option to enable full control over any new objects that are written by other AWS accounts.

Buffer

Select this option to send the output file in Buffer format, which is useful if you are reading or processing files containing binary data for performance reasons, for example.

You can select to compress the files by selecting the Compress file option. A drop-down list will then be displayed in which you can select either Zip or GZip.

Select the Bucket owner full control option to enable full control over any new objects that are written by other AWS accounts.

JSON

Select this option to send the output file in the JSON format.

Note

Note!

It is recommended to select the JSON format if the record has arrays.

You can then select the output format you want to use in the Action on record: drop-down list;

  • one file with All in one array

  • one file with All in one array with key

  • or One file per record.

Note

Note!

If One file per record is selected, you must also select Append timestamp, otherwise, the files will be overwritten.

You can select to compress the files by selecting the Compress file option. A drop-down list will then be displayed in which you can select either Zip or GZip.

Select the Bucket owner full control option to enable full control over any new objects that are written by other AWS accounts.

You can also select the Output file in pretty print option to format the JSON files for easier reading.

Note

Note!

Selecting the Output file in pretty print option will increase the size of the output file.

...

You have now configured the Amazon S3 Forwarder function.

Anchor
S3metadata
S3metadata
Amazon S3 Metadata

You can view and access the following metadata properties of Amazon S3. To view the metadata, see information on the meta object in the Script function documentation.

Property name

Description

fileName

Name of the file.

Syntax
Code Block
log.info(meta.fileName);

filePath

Path from where the file is collected. The file can be either in the Excel or CSV format.

Path format: bucket/folder/file

Syntax
Code Block
log.info(meta.filePath);

fileSize

Size (in bytes) of the file.

Syntax
Code Block
log.info(meta.fileSize);

sheetName

Name of the sheet of the Excel file.

Syntax
Code Block
log.info(meta.sheetName);

collectionTime

Timestamp when the file is collected.

Syntax
Code Block
log.info(meta.collectionTime);

...