rhoas-cli-getting-started-connectors

Managing Connectors by using the OpenShift Application Services CLI

As a developer of OpenShift Connectors, you can use the rhoas command-line interface (CLI) to create and configure connections between OpenShift Streams for Apache Kafka and third-party systems.

You can use a source connector to send data from an external system to Streams for Apache Kafka. You can use a sink connector to send data from Streams for Apache Kafka to an external system.

For the example in this guide, you create a source connector that sends data from a simple data generator to a Kafka topic. You also create a sink connector that sends data from the Kafka topic to an HTTP site.

Prerequisites

• You’ve installed the latest version of the rhoas CLI as described in Installing and configuring the OpenShift Application Services CLI.

• You’ve completed the following tasks:

  Note: You can find detailed instructions for these tasks in Creating Kafka resources by using the OpenShift Application Services CLI.

  • Create a Kafka instance.

    $ rhoas kafka create --name=my-kafka-instance

  • Verify that the Kafka instance is in the Ready state.

    $ rhoas context status kafka

  • Create a service account and copy the service account ID and secret. You must use a service account to connect and authenticate your Connectors instances with your Kafka instance.

    $ rhoas service-account create --file-format=json --short-description=test-service-account

  • For your Kafka instance, set the permissions for the service account to enable Connectors instances (that are configured with the service account credentials) to produce and consume messages in any topic in the Kafka instance.

    $ rhoas kafka acl grant-access --producer --consumer --service-account=<service-acct-id> --topic all --group all

  • Create a Kafka topic named test-topic. The Kafka topic stores messages sent by producers (data sources) and makes them available to consumers (data sinks).

    $ rhoas kafka topic create --name=test-topic

Creating a namespace to host your Connectors instances

A Connectors namespace hosts your Connectors instances.

The namespace that you use depends on your OpenShift Dedicated environment.

If you’re using the hosted preview environment

You must create a preview namespace before you can create Connectors instances. A preview namespace is active for 48 hours.

If you’re using a Trial cluster in your own OpenShift Dedicated environment

The namespace is created when you add the Connectors service to your Trial cluster, as described in Adding the Red Hat OpenShift Connectors add-on to your OpenShift cluster. Your OpenShift Dedicated Trial cluster namespace is active for 60 days.

Prerequisites

• You’re logged in to rhoas.

  $ rhoas login

Procedure

1. If you’re using a Trial cluster in your own OpenShift Dedicated environment, skip to Step 2.

   If you’re using the Connectors evaluation site, create an evaluation namespace.

   $ rhoas connector namespace create --name=eval-namespace

2. Verify that your namespace is listed.

   $ rhoas connector namespace list

Building connector configuration files

Before you can create a Connectors instance, you must build a configuration file that is based on a supported connector type that is listed in the Connectors catalog.

For this example, you want to create two types of connectors: a data generator (a source connector) and an HTTP sink connector.

You must build a configuration file for each connector type that you want to create. When you build a configuration file, the default file name is connector.json. Optionally, you can specify a different configuration file name.

Prerequisites

• Your current local directory is the place where you want to save your Connectors configuration files. For example, if you want to save your configuration files in a directory named my-connectors, ensure that the my-connectors directory is your current directory.

  $ cd my-connectors

• You’re logged in to rhoas.

• For the sink connector example, open the free Webhook.site page in a browser window. The Webhook.site page provides a unique URL that you can use for the example HTTP data sink.

Procedure

1. Decide which type of connector you want to create.

   1. View a list of the supported connector types that are available in the Connectors catalog. The default number of connector types listed is set to 10. To see all Connectors types, specify a limit value of 100.

      $ rhoas connector type list --limit=100

   2. For this example, find the data generator source connector by specifying "Generator" in the search flag.

      $ rhoas connector type list --search=%Generator%

      The result is as follows:

      {
        "name": "Data Generator source",
        "id": "data_generator_0.1",
        "description": "A data generator (for development and testing purposes)."
      }

   3. For this example, find the HTTP sink connector by specifying "HTTP" in the search flag.

      $ rhoas connector type list --search=%HTTP%

      The first result is the HTTP sink.

      {
        "name": "HTTP sink",
        "id": "http_sink_0.1",
        "description": "Send data to an HTTP endpoint."
      }

2. Build a configuration file for the data_generator_0.1 connector type. Specify test-generator as the Connectors instance name and test-generator.json as the configuration file name.

   $ rhoas connector build --name=test-generator --type=data_generator_0.1 --output-file=test-generator.json

   Note: By default, the configuration file is in JSON format. Optionally, you can specify YAML format by adding -o yaml to the connector build command.

3. Answer the prompts for configuration values.

   1. For Format, press ENTER to accept the default (application/octet-stream).

   2. For Error handling method, select stop. The Connectors instance stops running if it encounters an error.

   3. For Topic Names, type test-topic.

   4. For Content Type, accept the default.

   5. For Message, type Hello World!.

   6. For Period, accept the default (1000).

4. Build a configuration file for the http_sink_0.1 connector type and specify test-http as the configuration file name.

   $ rhoas connector build --name=test-http --type=http_sink_0.1 --output-file=test-http.json

5. Answer the prompts for configuration values.

   1. For Format, press ENTER to accept the default (application/octet-stream).

   2. For Error handling method, select stop.

   3. For Method, accept the default (POST).

   4. For URL, paste your unique URL that you copied from the Webhook.site page.

   5. For Topic Names, type test-topic.

6. Verify that the configuration files were built.

   $ ls

   The result shows the test-generator.json and test-http.json files.

7. Optionally, you can edit a configuration file in an editor of your choice.

   Note: To prevent saving sensitive data to disk, the values for the service account and the namespace are not included in the configuration file. You’re prompted to specify those values when you create an Connectors instance.

Creating Connectors instances

After you build a configuration file based on a connector type, you can use the configuration file to create a Connectors instance.

For this example, you create two Connectors instances: a data generator source Connectors instance and an HTTP sink Connectors instance.

Prerequisites

• You have built configuration files based on each type of connector that you want to create.

• The configuration files are saved in your current directory.

• You have a Connectors namespace.

• You have an OpenShift Streams for Apache Kafka instance running and have a topic called test-topic.

• You have a service account created that has read and write access to the Kafka topic, and you know the credentials (ID and secret).

Procedure

1. Create a source Connectors instance by specifying the source connector’s configuration file. For example, the data generator configuration file is test-generator.json.

   $ rhoas connector create --file=test-generator.json

2. Answer the prompts for details about the Connectors instance.

   1. For Set the Connectors namespace, select your namespace from the list. For example, select eval-namespace.

   2. For Service Account Client ID, type or paste your ID.

   3. For Service Account Client Secret, type or paste your secret.

      A message states "Successfully created the Connectors instance".

3. Wait until the status of the Connectors instance is Ready.

   To check the status:

   $ rhoas connector list

4. Verify that your source Connectors instance is producing messages.

   $ rhoas kafka topic consume --name=test-topic --partition=0 --wait

5. Create a sink Connectors instance by specifying the sink connector’s configuration file. For example, the HTTP sink configuration file is test-http.json.

   $ rhoas connector create --file=test-http.json

6. Answer the prompts for details about the Connectors instance.

   1. For Set the Connectors namespace, select your namespace from the list. For example, select eval-namespace.

   2. For Service Account Client ID, type or paste your ID.

   3. For Service Account Client Secret, type or paste your secret.

      A message states "Successfully created the Connectors instance".

7. Wait until the status of the Connectors instance is Ready.

   To check the status:

   $ rhoas connector list

8. Verify that your sink Connectors instance is receiving messages by viewing your Webhook.site page in a web browser.

Managing Connectors instances

The following rhoas connector help commands describe additional rhoas connector commands that you can use to manage your Connectors instances:

• rhoas connector namespace -h for managing Connectors namespaces

• rhoas connector type -h for viewing the available types of connectors

• rhoas connector list -h` for listing Connectors instances

• rhoas connector build -h for building configuration files

• rhoas connector create -h for creating Connectors instances

Additional resources

• CLI command reference (rhoas)