Kafka

The Apache Kafka framework uses a high-throughput, distributed, publish-subscribe messaging model. The framework is scalable and it can be elastically and transparently expanded without downtime. Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of a single machine and to allow clusters of coordinated consumers.

The real-time Kafka agents enable MediationZone Image Added to act as a Kafka cluster. Kafka producer agents publish messages (containing UDRs) and these are subscribed to by any number of Kafka collector agents in the workflows that are to be connected. These agents can also be used to connect to external Kafka interfaces.

...

• There is a need to connect real-time workflows that are running on the same, or in different, MediationZone Image Added installations.
• Minimization of data loss is prioritized.
• One-way communication is sufficient (one-to-one or one-to-many).

...

A Workflow Bridge agent acts as a bridge for communication between real-time workflows, or between batch and real-time workflows, within the same MediationZone Image Added system. There are several benefits of using Workflow Bridge:

• Using real-time-processing capabilities while keeping transaction safety when using the bridge between batch and real-time workflows

• Scaling the processing of high volume streaming data by distributing the processing load between execution contexts when bridging between an upstream data collection workflow and one or more downstream real-time workflow(s). When sending data to multiple workflows, the UDRs can be sent in a load-balancing scenario, where specific data can be distributed to a specific workflow. Data can also be broadcast to all downstream workflows.

Image RemovedImage Added

Batch to real-time Workflow Bridge

...

In order to optimize performance, it is possible to collect and send data in a bulk from the forwarding agent. When the Workflow Bridge real-time collection agent has received the data bulk, it is unpacked and forwarded as separate UDRs by the agent. The bulk is created by the Workflow Bridge forwarding agent after a configured number of UDRs has been reached, or after a configured time interval. This is specified in the Workflow Bridge profile.