Workflow Types(3.2)
- Yeok Hong Ng
Workflows are a collection of various types of agents to process data. The agents in a workflow are either collection agents, processing agents or forwarding agents. The list of agents that you can see, depends on your license for Usage Engine.
There are four types of workflow configurations:
- Batch
- Real-Time
- Task
- System Task
Batch Workflow
A batch workflow processes input from a specific source, often a file. The workflow creates batches from the data and processes them one by one.
In a batch workflow, data is collected by a single collecting agent in a transaction-safe manner. A single batch is collected (only once) and is fully processed by the workflow before the next batch is collected.
A batch workflow:
- Processes batches, one at a time
- Is started either manually or by a scheduled trigger
Stops either when it finishes processing the input, or when aborted
Note!
If the workflow aborts, the current batch can be reprocessed.
Real-Time Workflow
Real-time workflows are used in systems where instant processing requests need to be addressed as they occur.
In real-time workflows most of the collecting agents communicate in a two-way manner; they receive requests and provide replies.
A real-time workflow:
Can have more than one collecting agent
Note!
Multithreading operations differ from batch workflows in the following ways:
The order of UDR processing cannot be guaranteed.
Only global MIMs are used.Can process several UDRs simultaneously. See Multithreading (3.2)
- Once started, the real-time workflows remain active. A real-time workflow is started either manually or by a scheduled trigger and stops either manually or due to an error.
- Processes reside in memory. Transaction safety must be handled prior to collection and after distribution.
- Errors are registered in the System Log and the workflow continues to run. Real-time workflow error handling rarely leads to workflow aborts.
Note!
Real-time workflows may use the Inter Workflow or Kafka agents to forward data to a batch workflow.
Task Workflows
Task workflows are single-agent workflows that execute user-defined SQL, shell scripts, or unit tests.
System Task Workflows
A number of System Task workflows are included in Usage Engine. System Task workflows and workflow groups that include System Task workflows enable you to perform background routines such as log and run-time data clean up from the platform.
System Task workflows include:
This section includes information about:
Opening a System Task Workflow
Modifying a System Task Workflow Configuration
Opening a System Task Workflow
- The System Task Workflow is open from the workflow configurations list (Build menu).
- To open a System Task, double-click a System Task workflow or workflow group in the Configurations pane.
Modifying a System Task Workflow Configuration
All System Task Workflow configurations are modified at the template level. The workflow properties are set to Final and cannot be modified.
- You can modify a System Task configuration, including its scheduling criteria, but you cannot create or remove a System Task configuration.
- The Archive Cleaner Workflow lets you modify only its scheduling criteria. For further information see Scheduling in Managing a Workflow Group.
To Modify a System Task Workflow:
- Open the System Task Workflow that you want to modify.
System Task Workflows list - Double-click the agent icon. The agent's configuration dialog opens.
- Perform the changes and click OK.
- Save the workflow.
Archive Cleaner
The Archive Cleaner System Task in a workflow
The Archive Cleaner System Task enables you to remove old archived files from the file system. Archive Cleaner operates according to data that it receives from the Archive profiles.
Note!
You can only modify the scheduling criteria of the Archive Cleaner. See Archive Profile (3.2). Since scheduling can only be applied to workflow groups, you modify the Archive Cleaner scheduling from the workflow group configuration.
Configuration Cleaner
The Configuration Cleaner enables you to specify the maximum age of an old configuration before it is removed.
Note!
You cannot remove the most recent configuration with the Configuration Cleaner, only historical ones.
To configure the Configuration Cleaner:
- Select a configuration type from the table and click the entry in the Keep column; a drop-down list appears.
- Select one of the following options: Always, Days, or Versions.
The Configuration Cleaner System Task
| Item | Description |
|---|---|
Type | The icon represents the configuration type. |
Name | The name of the configuration type. |
Keep | Select the time unit to specify how long the old configurations should remain in the system. The time units are as follows:
|
Value | Specify the number of days or versions that represent the period during which configurations are kept. |
Data Veracity Maintenance
The Data Veracity Maintenance System Task in a workflow
The Data Veracity Maintenance system task removes outdated Data Veracity data, provided that the state is REPROCESSED or DELETE_APPROVED. For more information see Data Veracity Maintenance System Task(3.2).
The Data Veracity Maintenance Task
ECS Maintenance
The ECS Maintenance enables you to remove old ECS data from the file system, provided that the state is REPROCESSED. For more information see ECS Maintenance System Task(3.2).
The ECS Maintenance Task
Statistics Cleaner
The Statistics Cleaner enables you to remove old statistics data that has been collected by the Statistics server and stored in the database.
The Statistics Cleaner Task
| Item | Description |
|---|---|
Minute Level Records | Specifies the number of days during which a minute-level record should be kept in the database. |
Hour Level Records | Specifies the number of days during which an hour-level record should be kept in the database. |
Day Level Records | Specifies the number of days during which a day-level record should be kept in the database. |
System Backup
The System Backup enables you to create a backup of all the configurations in the system. A backup file is saved on the host machine where the Platform application is installed.
$MZ_HOME/backup/yyyy_MM, where yyyy_MM is translated to the current year and month, and $MZ_HOME/backup is the default path. The system saves a backup file and names it according to the following format: backup_<date>.zip. In addition, System Backup lets you specify the maximum age of backup files before they are removed. You can change this location using the following command:<property name="mz.backup.path" value="${mz.home}/backup"/>
Refer to Persistent Storage (3.2) to know about how you can configure the persistent storage.
The System Backup configuration dialog contains two tabs:
System Backup: This is to enable the backup option.
Cleanup: This is to configure the time during which a backup should be kept on disk before it is deleted.
The System Backup System Task - System Backup tab
| Item | Description |
|---|---|
Enable System Backup | Select this checkbox to enable the system backup. This setting is selected by default. |
Use Encryption | Select this checkbox to enable encryption of the backup. |
Password | Enter a password. |
The System Backup System Task - Cleanup tab
| Item | Description |
|---|---|
Imported Files | Every time the System Importer imports a configuration, the system saves it as a backup on the Platform. Enter the period, in days, during which the imported files should remain on disk. |
System Backup Files | Defines the maximum age of system backup files before they are removed from the host disk. |
System Log Cleaner
The System Log Cleaner deletes the System Log periodically. You set the frequency values for deleting different message types in the System Log Cleaner configuration.
The System Log Cleaner Task
| Item | Description |
|---|---|
Error/Disaster | Enter the number of days to keep Error and Disaster messages before they are removed from the database. Max value: 99 days Default value: 30 days |
Warning | Enter the number of days to keep Warning messages in the database. Max value: 99 days Default value: 20 days |
Information | Enter the number of days to keep Information messages in the database. Max value: 99 days Default value: 10 days |
User Cleanup
The User Cleanup Task can be used to automatically deleting inactive users. If SSO is turned on, a user logged in through SSO is added in the system upon login, but the user is not removed when it is deleted from the OIDC server. To resolve this, the User Cleanup Task can be used to delete any user that has not logged in for a configurable number of days.
The User Cleanup Task
| Item | Description |
|---|---|
| Apply only to SSO users | This setting is enabled by default and limits the function only to SSO users. |
| Retention Period | The default value is 0 (zero). This setting specifies the number of days to wait before a user is deleted after their last login session. A value of 0 (zero) disables the cleanup function. |
| Default Successor | If a certain user is deleted from the platform, a successor needs to be designated. This user will take ownership of all previously owned configurations. |