9.2.2.1 ADLS2 File Collection Agent Configuration
- Jenni Wallin
- Former user (Deleted)
- Former user (Deleted)
- Former user (Deleted)
You open the ADLS2 file collection agent configuration dialog from a workflow configuration: you can right-click the agent icon and select Configuration..., or double-click the agent icon
Part of the configuration may be done in the Filename Sequence or Sort Order tab described in 3.1.6 Workflow Template.
The ADLS File tab contains settings related to the placement and handling of the source files to be collected by the agent.
ADLS2 File collection agent configuration - ADLS2 File tab
| Setting | Description |
|---|---|
Profile | Select the Azure profile you want the agent to use, see 8.4 Azure Profile for further information about this profile. |
| Container Name | Enter the name of the container where the files will be collected from. The container name can be found in the Containers section of the storage account that has been configured in the Azure Profile. Example An example container name.
|
Collection Strategy | If there are more than one collection strategy available in the system a Collection Strategy drop down list will also be visible. For more information about the nature of the collection strategy, refer to 11. Appendix 4 - Collection Strategies. |
| Directory | The absolute pathname of the source directory on the location where the source files reside. |
| Include Subfolders | Select this check box if you have subfolders in the source directory from which you want files to be collected. If you select Enable Sort Order in the Sort Order tab, the sort order selected will also apply to subfolders. |
Filename | Name of the source files in the location stated in the File System profile. Regular expressions according to Java syntax applies. For further information, see http://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html. Example To match all filenames beginning with |
Compression | Compression type of the source files. Determines if the agent will decompress the files before passing them on in the workflow. - No Compression - agent does not decompress the files. This is the default setting. - Gzip - agent decompresses the files using gzip. |
Move to Temporary Directory | If enabled, the source files will be moved to the automatically created subdirectory |
Append Suffix to Filename | Enter the suffix that you want to be added to the file name prior to collecting it. Important! Before you execute your workflow, make sure that none of the file names in the collection directory include this suffix. |
Inactive Source Warning (hours) | If the specified value is greater than zero, and if no file has been collected during the specified number of hours, the following message is logged: The source has been idle for more than <n> hours, the last inserted file is <file>. |
Move to | If enabled, the source files will be moved from the source directory (or from the directory If the Prefix or Suffix fields are set, the file will be renamed as well. Note! It is only possible to move files within the same bucket. |
Destination | The absolute pathname of the directory on the location specified in the referenced File System profile into which the source files will be moved after collection. This field is only enabled if Move to is selected. |
Rename | If enabled, the source files will be renamed after the collection, remaining in the source directory from which they were collected (or moved back from the directory |
| Prefix/Suffix | Prefix and/or suffix that will be appended to the beginning respectively the end of the name of the source files, after the collection. These fields are only enabled if Move to or Rename is selected. Note! If Rename is enabled, the source files will be renamed in the current directory (source or |
| Search and Replace | To apply Search and Replace, select either the Move to or Rename setting. Search: Enter the part of the filename that you want to replace. Replace: Enter the replacement text. Search and Replace operate on your entries in a way that is similar to the Unix sed utility. The identified filenames are modified and forwarded to the following agent in the workflow. This functionality enables you to perform advanced filename modifications, as well: Use regular expression in the Search entry to specify the part of the filename that you want to extract. Note! A regular expression that fails to match the original file name will abort the workflow. Enter Replace with characters and meta characters that define the pattern and content of the replacement text. Search and Replace Examples To rename the file Search: Replace: To rename the file Search: Replace: Note! The search value divides the file name into two parts with brackets. The replace value applies the second part with the place holder $2. |
Remove | If enabled, the source files will be removed from the source directory (or from the directory |
Ignore | If enabled, the source files will remain in the source directory after collection. |
Keep (days) | Number of days to keep source files after the collection. In order to delete the source files, the workflow has to be executed (scheduled or manually) again, after the configured number of days. Note! A date tag is added to the filename, determining when the file may be removed. This field is only enabled if Move to or Rename is selected. |
| Route FileReferenceUDR | Select this check box if you want to forward the data to an SQL Loader agent. See 9.78 SQL Loader Agent for further information. |
