Workflows are a collection of various types of agents to process data. The agents in a workflow are either collection agents, processing agents or forwarding agents. The list of agents that you can see, depends on your license for Usage Engine.

There are four kinds of workflows:

• Batch
• Realtime
• Task
• System Task

Batch Workflow

A batch workflow processes input that originates in a specific source, often a file. The workflow creates batches from the data and processes them one by one.  

In a batch workflow data is collected by a single collecting agent in a transaction-safe manner: a single batch is collected (only once) and is fully processed by the workflow before the next batch is collected.  

A batch workflow:  

• Processes batches, one at a time
• Is started either manually or by a scheduled trigger  

• Stops either when it finishes processing the input, or when aborted

  Note!

  If the workflow aborts, the current batch can be reprocessed.

Real-Time Workflow

  Real-time workflows are applicable in systems where instant processing requests need to be addressed as they occur.

  In real-time workflows most of the collecting agents communicate in a two-way manner; they receive requests and provide replies.

  A real-time workflow:

•   Can have more than one collecting agent

•   Can process several UDRs simultaneously. See Multithreading(3.0).

  Note!

  Due to multithreading in real-time workflows, unlike batch workflows:
   

  1. The order of UDR processing cannot be guaranteed.
      

  2. Only global MIMs are used.

•   Once started, is always active. A real-time workflow is started either manually or by a scheduled trigger, and stops either manually or due to an error.
•   Processes in memory. Transaction safety must be handled prior to collection and after distribution.
•   Real-time workflow error handling rarely leads to aborting the workflow. Errors are registered in System Log and the workflow continues to run.
  
  

Task Workflows

Task workflows are single agent workflows that execute user defined SQL, shell scripts or unit tests.

System Task Workflows

A number of System Task workflows are included in Usage Engine. System Task workflows and workflow groups that include System Task workflows enable you to perform background routines such as log- and run-time data clean up from the platform.

System Task workflows include:

• Archive Cleaner

• Configuration Cleaner

• Data Veracity Maintenance

• ECS Maintenance

• Statistics Cleaner

• System Backup

• System Log Cleaner

• User Cleanup

This section includes information about:

• Opening a System Task Workflow

• Modifying a System Task Workflow Configuration

Opening a System Task Workflow

You open a System Task Workflow from the workflow configuration.

To open a System Task, double-click a System Task workflow or workflow group in the Configuration Browser pane.

Modifying a System Task Workflow Configuration

You modify all the System Task Workflow configurations at template level. The workflow properties are all set to Final and cannot be modified.

•   You can modify a System Task configuration, including its scheduling criteria, but you cannot create or remove a System Task configuration.
•   The Archive Cleaner Workflow lets you modify only its scheduling criteria. For further information see Scheduling in Managing a Workflow Group(3.0) .  

  To Modify a System Task Workflow:  

1.   Open the System Task Workflow that you want to modify.
   
   System tasks list
   
   
2.   Double-click the agent icon.
   
   The agent's configuration dialog opens.
   
   
3.   Perform the changes and click OK .
   
   
4.   Save the workflow.

Archive Cleaner

The Archive Cleaner System Task enables you to remove old archived files from the file system. Archive Cleaner operates according to data that it receives from the Archive profiles.

The Archive Cleaner System Task placed in a workflow

Configuration Cleaner

Enables you to specify the maximum age of an old configuration before it is removed. 

To configure the Configuration Cleaner:

1.   Select a configuration type from the table and then click the entry in the Keep column; a drop-down list appears.
   
   
2.   Select one of the following: Always, Days, or Versions.

The Configuration Cleaner System Task

Data Veracity Maintenance

The Data Veracity Maintenance Task

The Data Veracity Maintenance system task removes outdated Data Veracity data, provided that the state is REPROCESSED or DELETE_APPROVED. For more information about the Data Veracity Maintenance System Task workflow see Data Veracity Maintenance System Task(3.0). 

ECS Maintenance

The ECS Maintenance System Task

Enables you to remove old ECS data from the file system, provided that the state is REPROCESSED. For information about the ECS_Maintenance System Task workflow see ECS Maintenance System Task(3.0).

Statistics Cleaner

Enables you to remove old statistics data that has been collected by the Statistics server and stored in the database.

The Statistics Cleaner System Task

System Backup

Enables you to create a backup of all the configurations. A backup file is saved on the host machine where the platform application is installed. 
The System Backup files are stored under $MZ_HOME/backup/yyyy_MM, where yyyy_MM is translated to the current year and month and $MZ_HOME/backup is the default path. The system saves a backup file and names it according to the following format: backup_<date>.zip. System Backup also enables you to specify the maximum age of backup files before they are removed from the host disk. You can change this location using the following command:

<property name="mz.backup.path" value="${mz.home}/backup"/>

Refer to Persistent Storage (3.0) to know about how you can configure persistent storage.

The System Backup configuration dialog box is made up of two tabs:

• System Backup: Lets you enable the backup option

• Cleanup: Lets you configure the time during which a backup should be kept on disk before it is deleted
   

The System Backup System Task workflow - the System Backup tab

The System Backup Cleaner System Task - the Cleanup tab

System Log Cleaner

The System Log Cleaner deletes the System Log periodically. You set the frequency values for deleting different message types, on the System Log Cleaner dialog box.

The System Log Cleaner System Task

User Cleanup Task

The User Cleanup Task can be used for automatically deleting users who are no longer active. If SSO is turned on, any user logged in through SSO will be added upon login, but the user will not be removed when it is removed from the OIDC server. Because of this, the User Cleanup Task can be used for cleaning out any user that has not logged in for a configurable number of days.

The User Cleanup Task