/
Prerequisites for using Kafka functions

Prerequisites for using Kafka functions

    Before you use the Kafka Consumer or Kafka Producer functions in UsageCloud, you must ensure that your Kafka cluster is reachable and that the client identity used has the correct Access Control Lists (ACLs).

    By configuring these Kafka ACLs in advance, you ensure that Kafka functions can connect to brokers and perform operations securely and reliably.

    Resource type

    Resource Name

    Operations

    Description

    Resource type

    Resource Name

    Operations

    Description

    Kafka Consumer permissions and operations

    TOPIC

    All or specific topics.

    READ

    The Kafka Consumer function reads data from Kafka topics. To do this, it requires access to resources of type TOPIC in the Kafka cluster.

    UsageCloud applies topic-level ACL to control client access to Kafka topics. You can configure this ACL to:

    • Read messages from the configured topics.

    GROUP

    All groups.

    READ

    The Kafka Consumer function uses consumer groups to track which messages it has read. Each consumer group maintains its own offset history:

    • If consumer group group1 reads messages for the first time, it consumes from the beginning, for example, message1 through message5.

    • If you then switch to consumer group group2, it starts again from the beginning because it has a separate offset history.

    • If you later return to group1, it continues from where it stopped, after message5.

    In UsageCloud, the consumer group ID is either:

    • A consumer group ID that you configure for normal processing, or

    • A randomly generated group ID, for example, a UUID that is used in Preview and other testing scenarios.

    Note!

    Previews and tests use temporary or unique group IDs that you do not know in advance. Configure ACLs on the GROUP resource to allow all consumer groups for the required operations to ensure the main consumer group and any preview or test groups use the same permissions without additional ACL changes.

    Kafka Producer permissions and operations

    TOPIC

    All or specific topics.

    CREATE

    The Kafka Producer function sends data to Kafka topics. To do this, it requires access to resources of type TOPIC in the Kafka cluster.

    UsageCloud applies topic-level ACLs to control client access to Kafka topics. You can configure these ACLs to:

    • Allow a client to access every topic by using a wildcard pattern.

    • Limit a client to a defined set of topics, for example, topic1 and topic2, and deny access to topic3.

    To support Kafka Producer functions in UsageCloud, you must grant the client the following permissions on the TOPIC resource type:

    • Write messages to the configured topics.

    • Describe topics and their configurations so that the client fetches topic metadata and the offsets that it commits.

    TOPIC

    All or specific topics.

    DESCRIBE_CONFIGS

    TOPIC

    All or specific topics.

    WRITE

    TRANSACTIONAL_ID

    All TRANSACTIONAL_ID

    DESCRIBE

    The Kafka Producer function uses transactions to publish messages reliably. To do this, it requires access to TRANSACTIONAL_ID resources in the Kafka cluster.

    UsageCloud generates TRANSACTIONAL_IDautomatically. You do not control the exact TRANSACTIONAL_ID values, ACLs do not target a single TRANSACTIONAL_ID. Instead, you must grant permissions for all TRANSACTIONAL_ID that the client uses on the TRANSACTIONAL_ID resource type. This access allows the client to:

    • Initialize and use transactional producers.

    • Commit or abort transactions when sending data to Kafka.

    TRANSACTIONAL_ID

    All TRANSACTIONAL_ID

    WRITE