kcat-kafka

Configuring and connecting Kcat with OpenShift Streams for Apache Kafka

As a developer of applications and services, you can use Kcat to test and debug your Kafka instances in OpenShift Streams for Apache Kafka. Kcat is a command-line utility for messaging in Apache Kafka 0.8 and later. With Kcat, you can produce and consume messages directly from the command line. You can also list topic and partition information for your Kafka instances.

Prerequisites

• You have a running Kafka instance in Streams for Apache Kafka (see Getting started with OpenShift Streams for Apache Kafka).

• You have a command-line terminal application.

• JDK 11 or later is installed. (The latest LTS version of OpenJDK is recommended.)

• You’ve installed the latest supported version of Kcat for your operating system. To verify your Kcat version, enter the following command:

  $ kcat -V

  You see output similar to the following example:

  kcat - Apache Kafka producer and consumer tool
  https://github.com/edenhill/kcat
  Copyright (c) 2014-2021, Magnus Edenhill
  Version 1.7.0 (librdkafka 1.3.0 builtin.features=gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer)

Configuring Kcat to connect to a Kafka instance

To enable Kcat to access a Kafka instance, you must configure a connection. The configuration must include the bootstrap server endpoint for the Kafka instance and the credentials for your OpenShift Streams for Apache Kafka service account.

You can either pass these values to the kcat command or use a configuration file. In this task, you set environment variable values and then pass them to the kcat command.

For more information about Kcat configuration options, see Configuration in the Kcat documentation.

Note	Kcat does not yet fully support SASL/OAUTHBEARER authentication, which requires you to specify an access token in addition to service account credentials. Therefore, you use SASL/PLAIN authentication to connect to your Kafka instance with just service account credentials.

Prerequisites

• You have the bootstrap server endpoint for your Kafka instance. To get the server endpoint, select your Kafka instance in the OpenShift Streams for Apache Kafka web console, select the options icon (three vertical dots), and click Connection.

• You have the generated credentials for your service account. To reset the credentials, use the Service Accounts page.

• You’ve set permissions for your service account to access resources in the Kafka instance. To verify the current permissions, select your Kafka instance in the OpenShift Streams for Apache Kafka web console and click the Access tab. To learn more about setting permissions, see Managing account access in OpenShift Streams for Apache Kafka.

Procedure

• On the command line, set the Kafka instance bootstrap server and client credentials as environment variables to be used by Kcat. Replace the values with your own server and credential information.

  Setting environment variables for bootstrap server and client credentials

  $ export KAFKA_HOST=<bootstrap_server>
  $ export RHOAS_SERVICE_ACCOUNT_CLIENT_ID=<client_id>
  $ export RHOAS_SERVICE_ACCOUNT_CLIENT_SECRET=<client_secret>

Producing messages in Kcat

You can use Kcat to produce messages to Kafka topics in several ways, such as reading them from standard input (stdin) on the command line, or from a file. In this task, you produce messages from input on the command line. For more examples of Kcat producer messaging, see Examples in the Kcat documentation.

Prerequisites

• Kcat is installed.

• You have a running Kafka instance in OpenShift Streams for Apache Kafka (see Getting started with OpenShift Streams for Apache Kafka).

• You have a topic in your Kafka instance that you can use to produce and consume messages.

• You’ve set the Kafka bootstrap server endpoint and your service account credentials as environment variables.

Procedure

1. On the command line, enter the following command to start Kcat in producer mode. This mode enables you to produce messages to your Kafka topic. Replace <kafka-topic> with your own topic name.

   Starting Kcat in producer mode

   $ kcat -t <kafka-topic> -b "$KAFKA_HOST" \
    -X security.protocol=SASL_SSL -X sasl.mechanisms=PLAIN \
    -X sasl.username="$RHOAS_SERVICE_ACCOUNT_CLIENT_ID" \
    -X sasl.password="$RHOAS_SERVICE_ACCOUNT_CLIENT_SECRET" -P

   Note	Kcat does not yet fully support SASL/OAUTHBEARER authentication, which requires you to specify an access token in addition to service account credentials. Therefore, you use SASL/PLAIN authentication to connect to your Kafka instance with just service account credentials.

   Note	A Kcat producer might behave differently based on the operating system. For example, on Mac operating systems, messages are sent by redirecting them to the Kcat binary in the format echo "message" | kcat …​.

2. With Kcat running in producer mode, enter messages that you want to produce to the Kafka topic, as shown in the following example:

   First message
   Second message
   Third message

3. To finish producing the messages you entered, press CTRL+D.

4. Keep the producer running so that you can use it again later, when you create a consumer.

   Note	On Mac operating systems, pressing CTRL+D produces the messages you entered and then stops the producer. To use the producer again, you must restart it, as shown earlier in this task.

Verification

• Verify that your producer is still running without any errors in the terminal.

Consuming messages in Kcat

You can also use Kcat to consume messages from Kafka topics. In this task, you use Kcat to consume the messages that you previously produced to your topic.

Prerequisites

• Kcat is installed.

• You have a running Kafka instance in OpenShift Streams for Apache Kafka (see Getting started with OpenShift Streams for Apache Kafka).

• You used Kcat to produce example messages to a topic in your Kafka instance.

Procedure

1. Open a second terminal window or tab, separate from your producer.

2. On the command line, enter the following command to start Kcat in consumer mode. This mode enables you to consume messages from your Kafka topic. Replace <kafka-topic> with the name of the topic that you previously produced messages to.

   Starting Kcat in consumer mode

   $ kcat -t <kafka-topic> -b "$KAFKA_HOST" \
    -X security.protocol=SASL_SSL -X sasl.mechanisms=PLAIN \
    -X sasl.username="$RHOAS_SERVICE_ACCOUNT_CLIENT_ID" \
    -X sasl.password="$RHOAS_SERVICE_ACCOUNT_CLIENT_SECRET" -C

   You see output that looks like the following example. The message values are the ones you previously sent using the producer.

   First message
   Second message
   Third message
   % Reached end of topic <kafka-topic> [0] at offset 3

3. If your producer is still running in a separate terminal, continue entering messages in the producer terminal and observe the messages being consumed in the consumer terminal.

Note	You can also use the Streams for Apache Kafka web console to browse messages in the Kafka topic. For more information, see Browsing messages in the OpenShift Streams for Apache Kafka web console.

Verification

1. Verify that your consumer is running without any errors in the terminal.

2. Verify that the consumer displays the messages from your Kafka topic.