...
| Item | Description | |||||
|---|---|---|---|---|---|---|
Enable Duplicate Check | When selected, the agent will store every collected URL in a (configurable) number of days. The storage will be checked to make sure that no URL is collected again as long as it remains in the storage. | |||||
Database Profile | Each collected URL will be stored in the database defined in the profile selected. The schema must contain a table called "duplicate_check", for more information about this table refer to 9.40.5 HTTP Batch Appendix - Database Requirements for Duplicate Check. | |||||
Max Cache Age (Days) | The number of days to keep collected URLs in the database. When the workflow starts, it will delete entries that are older than this number of days.
|
| Scroll ignore | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||