Versions Compared

Old Version 3

changes.mady.by.user Jenni Wallin

Saved on

compared with

New Version Current

changes.mady.by.user Jenni Wallin

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

To open the Amazon S3 collection agent configuration dialog from a workflow configuration, you can do the following:

  • right-click the agent icon and select Configuration...

  • select the agent icon and click the Edit button

Note

...

Note!

If you are using the Amazon S3 collection agent in a Batch workflow, part of the configuration may be done in the Filename Sequence and Sort Order tabs in Workflow Template(4.

...

3).

For Real-Time workflow, additional configurations in Decoder, Decompression and Execution tabs are available in Batch-Based Real-Time Agents - Agent Configuration(4.3).

The Amazon S3 tab contains settings related to the placement and handling of the source files to be collected by the agent.Image Removed

...

Amazon S3 collection agent configuration - Amazon S3 tab

Setting

Description

Profile

Select the File System profile you want the agent to use, see File System Profile - Amazon S3 (4.3) for further information about this profile.

File Information

Directory

The absolute pathname of the source directory on the location stated in the File System profile, where the source files reside.

Include Subfolders

Select this check box if you have subfolders in the source directory from which you want files to be collected.

If you select Enable Sort Order in the Sort Order tab, the sort order selected will also apply to subfolders.

Filename

Name of the source files in the location stated in the File System profile. Regular expressions according to Java syntax applies. For further information, see http://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html.

Info
title

Example

To match all filenames beginning with TTFILE , type: TTFILE.*

Compression

Select the compression type of the source files:

  • No Compression - agent does not decompress the files. This is the default setting.

  • Gzip - agent decompresses the files using gzip.

This determines if the agent will decompress the files before passing them on in the workflow.

Before Collection

Move to Temporary Directory

If enabled, the source files will be moved to the automatically created subdirectory DR_TMP_DIR in the source directory, prior to collection. This option supports safe collection of a source file reusing the same name.

Append Suffix to Filename

Enter the suffix that you want added to the file name prior to collecting it.

Warning
title

Important!

 Before you execute your workflow, make sure that none of the file names in the collection directory include this suffix.

Inactive Source Warning (h)

If the specified value is greater than zero, and if no file has been collected during the specified number of hours, the following message is logged:

The

source

has

been

idle

for

more

than

<n>

hours,

the

last

inserted

file

is

<file>.

After Collection

Move to

If enabled, the source files will be moved from the source directory (or from the directory DR_TMP_DIR, if using Move Before Collecting) to the directory specified in the >Destination field, after the collection.

If the Prefix or Suffix fields are set, the file will be renamed as well.

Note
title

Note!

It is only possible to move files within the same bucket.

Destination

The absolute pathname of the directory on the location specified in the referenced File System profile into which the source files will be moved after collection.

This field is only enabled if Move to is selected.

Rename

If enabled, the source files will be renamed after the collection, remaining in the source directory from which they were collected (or moved back from the directory DR_TMP_DIR, if using the Move To Temporary Directory setting).

Prefix/Suffix

Prefix and/or suffix that will be appended to the beginning respectively the end of the name of the source files, after the collection.

These fields are only enabled if Move to or Rename is selected.

Note
title

Note!

If Rename is enabled, the source files will be renamed in the current directory (source or DR_TMP_DIR). Be sure not to assign a prefix or suffix, giving files new names, still matching the filename regular expression, or else the files will be collected over and over again.

Search and Replace

To apply Search and Replace, select either the Move to or Rename setting.

Search: Enter the part of the filename that you want to replace.

Replace: Enter the replacement text.

Search and Replace operate on your entries in a way that is similar to the Unix sed utility. The identified filenames are modified and forwarded to the following agent in the workflow.

This functionality enables you to perform advanced filename modifications, as well:

Use regular expression in the Search entry to specify the part of the filename that you want to extract.

Note
title

Note!

A regular expression that fails to match the original file name will abort the workflow.

Enter Replace with characters and meta characters that define the pattern and content of the replacement text.

Info
title

Search and Replace Examples

To rename the file file1.new to file1.old, use:

Search: .new

Replace: .old

To rename the file JAN2011_file to file_DONE, use:

Search: ([A-Z]*[0-9]*)_([a-z]*)

Replace: $2_DONE

Note
titleNote!

The search value divides the file name into two parts with brackets. The replace value applies the second part with the place holder $2. 

Remove

If enabled, the source files will be removed from the source directory (or from the directory DR_TMP_DIR, if using the Move To Temporary Directory setting), after the collection.

Keep (days)

Number of days to keep source files after the collection. In order to delete the source files, the workflow has to be executed (scheduled or manually) again, after the configured number of days.

Note
title

Note!

A date tag is added to the filename, determining when the file may be removed. This field is only enabled if Move to or Rename is selected.

Ignore

If enabled, the source files will remain in the source directory after collection.

UDR Type

Route FileReferenceUDR

Select this check box if you want to forward the data to an SQL Loader agent. See SQL Loader(4.3) for further information.