Page Comparison

Download and extract the sample data file (.zip)

Create an S3 bucket in Amazon Web Services (AWS).

Make a note of the bucket name, access key id, and secret access key.

For further information about creating and accessing S3 buckets, see the Amazon Web Services (AWS) documentation.

Create folders named in and out in the S3 bucket. Upload the extracted files (.csv) to the in folder. 

Create or open an existing solution.

When you open your solution, click Create Stream.

Give your stream a name and click Create.

You are redirected to the stream editor where you can start building your stream.

Using the library of functions on the left, click and drag the Amazon S3 function from the Collectors list onto the stream editor.

Double-click the function to open and configure it.

1. Enter your Access Key and Secret Key.
   
   

   
   
   

2. In Bucket, enter the name of the S3 bucket that you want to collect data from. In Folder, enter the folder path for the file that you want to collect. If the path to the folder is not specified, the root folder of the S3 bucket is selected by default.
   
   
   
   
3. Select Based on filename and enter the full name of the file (with extension) to be collected or select the Based on regular expression if the file name is in the regular expression format (regex format) and specify the File name pattern. Select the desired file format. Since the CSV file contains headers, select Include table header. The default delimiter is (,).
   
   
   
   
   

4. In After Collection, select Do nothing. Select Remove to remove the source file after collection is complete.
   
   

To see a preview of the collected data to be collected, click Preview. 

From the functions library on the left, in the Processors list, drag and drop the Route function onto the canvas. Once placed link it to the Amazon S3 collector.

Double-click the function to open and configure it. 

To setup the appropriate Route function conditions a preliminary setup must be made as follows:

1. Click on the edit icon to rename "Condition #1" to "Complete"
2. From the dropdown menu of the Compare with selection choose "type" to enable processing for the given data
3. Set the Expression option to "Equal to" and Value to "Complete"
4. Set Handling of unmatched records to "Create new output"

Add a Log function to output the default data (which is not used further in this tutorial). Click on the title of the Log function and rename it to "Start and Partial Data".

You can press on Preview to see the data flow. 

Add a Data Aggregator function and connect it to the Router output previously named "Complete".

Configure the following options to set the required output settings.

Flush settings:

Click Preview to display the Data Aggregator output.

Add an Amazon S3 Forwarder function and link it to the Data Aggregator function.

Double-click the function to open and configure it.

Enter your Access key and Secret key.

Enter the name of the bucket and the folder where you want to store the file(s). 

Select Collector filename if you want to keep the same filename as your input file(s), or select Custom filename to  to provide a new filename for the output file(s).

By default, the Append timestamp checkbox is selected. If you clear the checkbox then the existing file at the destination can be overwritten by output file.

For more information, see .

To run your stream, click Start.

Alternatively, you can run your stream from the Streams list. Select your stream and click Start.

The Status indicates when the stream has run to completion.

To see the logs, after your stream has run to completion, click View Log.

Download the result from the out folder in the Amazon S3 bucket.

After importing the result to Excel, the example file may look as follows:

You have now successfully collected data from an S3 bucket, which you've aggregated and stored in your S3 bucket!