1. Start all three of the predefined SCs mapped to Zookeeper, zk1 zk2 zk3 , and all three of the predefined SCs mapped to Kafka, sc1 sc2 sc3 . Then start the services defined in $MZ_HOME/common/config/cell/default/master/services/custom.conf (using  service start --scope  custom), in this case, Zookeeper and Kafka, as Kafka requires Zookeeper to keep track of its cluster. To do this, use the following commands:

   $ mzsh system start
   $ mzsh service start --scope custom

2. You must create a Kafka topic and one or more partitions to write to. You use the kafka command  to do this as described in the MZSH Command Line Tool User's Guide.  Refer to the example below to create a topic named  mytopic  with three partitions and a replication factor of two.
   
   

   Example - Creating a topic in Kafka

   $ mzsh kafka --service-key kafka1 --create --topic mytopic --partitions 3 --replication-factor 2       

   Note!

   It is not recommended that you create a greater number of topics than is necessary. Bear in mind the following:

   • If you have 100 workflow configurations with a Kafka forwarding agent connected to a Kafka collection agent in each case, you do not require 100 topics. The characteristics of your data must determine what topics to use, e g if 10 of your connected workflow configurations handle very critical data, those 10 should each have their own topic with suitable retention, replication profile settings etc, to ensure high security.
   • There must be a Kafka profile for each topic. This means that a large number of topics leads to a large number of profiles. A large number of profiles must in turn be modified when you tune Kafka properties for connected workflow configurations.

   • By default, you cannot delete a topic. To change this default behavior, set the property delete.topic.enable to true in broker-default.properties located in the directory $MZ_HOME/common/config/templates/kafka/<version>/custom. You can then use the mzsh kafka command to delete a topic as described in 2.2.11 kafka. For further information, see the section below, Modifying Service Provider Properties.

3. You can now create your Kafka profile in the  Desktop. Refer to 9.48.3 Kafka Profile. Enter the Kafka Topic which you have created. Select the Use Embedded Kafka check box, and enter  'kafka1' as the Kafka Service Key. When you proceed to creating Kafka forwarding and collection agents, you can then refer to the profile you have created. 
   
    
   Embedded Kafka profile configuration dialog

Configuring Embedded Kafka

In addition to creating topics and partitions, there are a number of other kafka command options available. It is important to note that all of the commands take the Kafka service key as an input parameter. To find out which service key to use when you start your Kafka services, refer to $MZ_HOME/common/config/cell/default/master/services/custom.conf. For further information, see MZSH Command Line Tool User's Guide.

It is recommended that you replace the sc-list property with the deployment-info property in custom.conf that is located in the $MZ_HOME/common/config/cell/default/master/services directory. This allows you to provide a brokerid specific to an SC, which can be persisted even when SC topology changes. If both the sc-list property and the deployment-info property are included in custom.conf, the deployment-info property takes precedence. Specify a brokerid for an SC as follows:

deployment-info: [{ sc:"sc1",brokerid:1},{ sc:"sc2",brokerid:2},{ sc:"sc3",brokerid:3}]

By default, the topic logs are stored in $MZ_HOME/storage/kafka. The customizable property selecting this storage path is in $MZ_HOME/common/config/templates/kafka/<version>/custom/template.conf. In addition, other Kafka broker properties, such as log retention rules, are stored in $MZ_HOME/common/config/templates/kafka/<version>/custom/broker-defaults.properties.

Modifying Service Provider Properties

If you require to modify service provider properties, e g the broker-defaults.properties which is part of the Kafka Service Provider, you copy the basic folder located in the standard template folder to the custom template folder, where you can modify the properties file as required. You must then indicate in the custom.conf file located in the $MZ_HOME/common/config/cell/default/master/services directory, that you are using a template in the custom folder. For example, if you require to modify broker properties so that you can delete topics, follow the steps below:

1. Go to the $MZ_HOME/common/config/templates/kafka/<version>/standard/basic directory and copy the basic directory to the $MZ_HOME/common/config/templates/kafka/<version>/custom/basic directory. 
    
2. Go to the broker-defaults.properties file in the basic directory that you have just created, and set the property delete.topic.enable to true. Save your changes.
3. Use the mzsh topo activate command to activate your manual changes.

4. Use the mzsh topo set command to indicate template must refer to 1/custom/basic instead of 1/standard/basic:

   $ mzsh topo set topo://services:custom/val:kafka.kafka1.template 1/custom/basic

5. Restart the platform and SCs:

   $ mzsh restart platform 
   $ mzsh system restart

6. Start the custom kafka service:

   mzsh service start -s custom