The Amazon S3 Function allows data to be sent or received from/to Amazon S3 buckets. To use this function, it is necessary to have prepared an Amazon S3 bucket in advance.
...
With the Amazon S3 Collector and Amazon S3 Forwarder functions you can send and receive data to and from Amazon S3 buckets.
Before using the functions you must ensure that the Amazon S3 bucket(s) are setup using an IAM policy with the following minimum content:
| Code Block |
|---|
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowGetPutDelete",
"Effect": "Allow",
"Action": [
"s3:GetObject*",
"s3:PutObject*",
"s3:DeleteObject*",
"s3:AbortMultipartUpload",
"s3:ListMultipartUploadParts"
],
"Resource": "arn:aws:s3:::<bucket-name>/*"
},
{
"Sid": "AllowListingBucket",
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:ListBucketMultipartUploads"
],
"Resource": "arn:aws:s3:::<bucket-name>"
}
]
} |
"Resource": "arn:aws:s3:::<bucket-name>/*" means that actions in a statement are applicable to all objects in the S3 bucket arn:aws:s3:::<bucket-name>.
If you only want to share a certain key (folder) in the S3 bucket, you can specify the folder after the bucket name like this: arn:aws:s3:::<bucket-name>/<directory-to-share>/* in the Resource section, but this is only applicable for the AllowGetPutDelete statement.
| Note |
|---|
Warning! It is not recommended to use the AWS managed policy AmazonS3FullAccess since this will allow all actions on all buckets and objects! |
See https://docs.aws.amazon.com/s3/ for more information on how to setup IAM policies.
To connect to your bucket, you need to have your Access Key and Secret Access Key at hand. For information on how to find this information, see https://docs.aws.amazon.com/general/latest/gr/managing-aws-access-keys.html for information on how to find this information. You also need to enter the folder from or to which you are collecting data or sending data.
For an example of how to create a stream that uses these two functions, take a look at See Stream using Amazon S3 Functions - Acme EV for an example how the Amazone S3 collector and forwarder can be used in a stream.
To configure the Amazon S3 Functions, the following is required from your AWS account:
...
Common Configurations
For both functions, you need to configure AWS Credentials and File location.
AWS Credentials
The AWS Credentials can either be specified directly in the configuration dialog for the functions, or in an AWS Secret in the Secrets Wallet, and must contain:
Setting | Description |
|---|---|
Access Key | The identifier |
used to sign the requests sent to Amazon S3, is referenced by S3 as the Access Key ID. | |
Secret Key | The |
Secret Key is used in conjunction with the Access Key to cryptographically sign Amazon AWS requests. When you create the |
Access Key in Amazon S3 |
you can view and download the Secret Key. S3 references this as the Secret Access Key. |
File location
The File location settings include:
Setting | Description |
|---|---|
Bucket | The Bucket is the public cloud storage resource available in Amazon S3. |
Folder |
Enther the path to the folder you want to collect data from and push data to when using the Amazon S3 |
There are two types of Amazon S3 Functions available in Usage Engine:
...
functions in this field.
|
Amazon S3 Collector Configuration
With the Amazon S3 Collector you can collect data from your AWS S3 bucket for processing in your stream.
To configure the Amazon S3 collector, take the following steps:
...
:
Place the function in your stream and double-click to open the Configuration dialog.
Configure the Common Configurations as described above.
All files in the subfolder(s)
...
| title | Note! |
|---|
...
to the
...
main folder stated in the Folder field will be collected. You can also choose to not include files from any subfolders by
...
selecting the Do not include files from subfolders
...
checkbox.
...
In the After Collection
...
section you can select to Remove files from Amazon server after collection.
In File information
...
section you can specify the selection criteria for your files
...
including how to Select files and the File format:
Setting | Description |
|---|---|
File selection | |
...
All files in folder | |
...
Select this option |
...
to collect all files in the specified folder. |
...
Based on filename |
...
Select this option to collect files based on specific file |
...
names. You can specify one or several file names. |
Based on regular expression |
...
Select this option to collect files based on file name patterns that you state using regular expressions. |
Based on list of files |
...
Select this option to collect files using a meta file containing the path to another file |
...
. If there is more than |
...
one meta |
...
file |
...
present, they will be processed |
...
simultaneously. |
...
...
You need to specify the following for the meta file: |
...
...
 There are also three mandatory prerequisites for this option: |
...
|
...
|
...
|
...
|
...
|
...
|
File format |
...
Depending on what option you select in the this drop-down, you will see different complimentary settings beneath the drop-down list. | |
CSV | Select this option to collect files in CSV format. |
...
Select the Include table header check box to include the table header in the collected file(s). |
...
Select the delimiter for the CSV file |
...
(s) in the Delimiter field. Comma is selected by default but you can also select Tab, Semicolon, Space, or Other. If |
...
you select Other you can |
...
specify a custom delimiter in |
...
the Custom delimiter field that will then be displayed. |
Excel |
...
Select this option to collect files in Excel format. |
...
Select the Include table header check box to include the table header in the collected file(s). Select the All sheets in file |
...
option to collect everything in the file, or the Specific Sheet(s) |
...
option to specify which sheets you want to collect in the Select sheet(s) field that will be displayed. |
JSON |
...
Select this option to collect files in |
...
XML format. |
XML |
...
Select this option to collect files in XML format |
You have now configured the Amazon S3 Collection function.
| Note |
|---|
...
Note! During collection, the following applies:
For all supported archive file types, the following applies:
There are exceptions to the filename patterns when it comes to collector functions. |
...
Amazon S3 Forwarder
The Amazon With the Amazon S3 Forwarder function allows you to can send data to your Amazon an Amazon S3 bucket from your stream.
To configure the Amazon S3 forwarder, take the following actions:
...
:
Place the function in your stream and double-click to open the Configuration dialog.
Configure the Common Configurations as described above.
The folder path stated in the Folder field cannot begin or end with a '/'.
| Note |
|---|
...
Note! If the path to the folder is not specified, the root folder of the S3 bucket is selected by default. |
...
In Output
...
file Information section, specify how you want to handle the output file(s) in Filename
...
options:
Filename option | Action |
|---|---|
Collector filename | Select |
...
this option if you want to keep the same filename as |
...
the input file(s). |
...
The function appends the file format extension to the filename based on the |
...
file format selected in Step 5, if the selected extension is not in the filename. | |
Custom filename | Select |
...
this option to define a new filename for |
...
the output files. |
...
...
for information on how to define filenames with additional flexibility. The function appends the file format extension to the filename based on the file format selected in Step 5, so you do not need to add any extensions to the filename. |
| Note |
|---|
Note! For existing streams, if you would like to use the new filename saving method as described above, you can check the New filename saving method check box. |
Select the Append timestamp checkbox to append the timestamp to the name of the output file. For
...
a CSV file, the
...
file name will
...
then be <
myfile>_<timestamp>.csv.
| Note |
|---|
...
Note! If you have selected Append timestamp |
...
, the existing file at the destination can be overwritten by the output file. |
In File format, select the format of the output file(s) from the following options:
File format |
|---|
...
Depending on what option you select in the this drop-down, you will see different complimentary settings beneath the drop-down list. | |
|---|---|
CSV | Select this option to send the |
...
files in CSV format. |
...
Select the Include table header |
...
check box to include the table |
...
header in the output file(s). |
...
Select the delimiter for the CSV file |
...
Compression can be toggled by enabling the Compress file option. A dropdown menu is used to select the format – Zip and GZIP are supported.
An additional option toggles the Bucket owner full control.
...
(s) in the Delimiter field. Comma is selected by default but you can also select Tab, Semicolon, Space, or Other. If you select Other you can specify a custom delimiter in the Custom delimiter field that will then be displayed. You can select to compress the files by selecting the Compress file option. A drop-down list will then be displayed in which you can select either Zip or GZip. Select the Bucket owner full control option to enable full control over any new objects that are written by other AWS accounts. | |
Excel | Select this option to send the output file in Excel format. You can |
...
then select to specify the name of the sheet in the Sheet name field that is displayed. The default |
...
name is |
...
“Sheet 1”. |
...
You can select to compress the files by selecting the Compress file option. A |
...
drop-down list will then be displayed in which you can select either Zip or GZip. Select the Bucket owner full control option to enable full control over any new objects that are written by other AWS accounts. | |
Buffer | Select this option to send the output file in |
...
Buffer format |
...
, which is useful if you are reading or processing files containing binary data |
...
for |
...
performance |
...
reasons, |
...
for example. |
...
You can select to compress the files by selecting the Compress file option. A |
...
drop-down list will then be displayed in which you can select either Zip or GZip. Select the Bucket owner full control |
...
option to enable full control over any new objects that are written by other AWS accounts. | |||
JSON | Select this option to send the output file in the JSON format.
|
...
You can then select the output format you want to use in the Action on record: drop-down list;
|
...
|
...
|
...
|
...
You can select to compress the files by selecting the Compress file option. A |
...
An additional option toggles the Bucket owner full control.
...
drop-down list will then be displayed in which you can select either Zip or GZip. Select the Bucket owner full control option to enable full control over any new objects that are written by other AWS accounts. You can also select the Output file in pretty print option |
...
to format the JSON files for easier reading.
|
...
|
...
|
...
- Zip
- GZip
...
When you select the Bucket owner full control
...
option, the option Access Control List (ACL)
...
is enabled
...
: ACL = bucket-owner-full-control.
...
You have now configured the Amazon S3 Forwarder function.
Amazon S3 Metadata
You can view and access the following metadata properties of Amazon S3. To view the metadata, see information on the documentation of the meta object mentioned in the Script function function documentation.
Property name | Description | |
|---|---|---|
fileName | Name of the file. Syntax
|
| ||
filePath | Path from where the file is collected. The file can be either in the Excel or CSV format. Path format: Syntax
|
| title | Syntax |
|---|
| ||
fileSize | Size (in bytes) of the file. Syntax
|
| ||
sheetName | Name of the sheet of the Excel file. Syntax
|
| title | Syntax |
|---|
| ||
collectionTime | Timestamp when the file is collected. Syntax
|
|