Kafka Operator

Information about the Kafka Operator

Overview

Apache Kafka is an open-source distributed event streaming platform used for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. The Kafka Operator is a Kubernetes operator to automate provisioning, management, autoscaling, and operations of Apache Kafka clusters deployed to Kubernetes. It works by watching custom resources, such as KafkaClusters, KafkaUsers, and KafkaTopics, to provision underlying Kubernetes resources (i.e StatefulSets) required for a production-ready Kafka Cluster.

Install

Follow these steps to install the Kafka operator in a workspace. This procedure results in a Kafka operator running in a workspace namespace, ready to manage and create Kafka clusters in any project namespaces. See the Kafka custom resource documentation for more information on creating Kafka clusters.

NOTE: Only install the Kafka operator once per workspace.

  1. Follow the generic installation instructions for workspace catalog applications on the Application Deployment page.

  2. Within the AppDeployment, update the appRef to specify the correct kafka-operator App. You can find the appRef.name by listing the available Apps in the workspace namespace:

    kubectl get apps -n ${WORKSPACE_NAMESPACE}

Refer to the Kafka operator Helm Chart documentation for details on custom configuration for the operator.

Uninstall via the CLI

Uninstalling the Kafka operator does not affect existing KafkaCluster deployments. After uninstalling the operator, you must manually remove any remaining Custom Resource Definitions (CRDs) from the operator.

  1. Delete all of the deployed Kafka custom resources (see Deleting Kafka Custom Resources).

  2. Uninstall a Kafka operator AppDeployment:

    kubectl -n <workspace namespace> delete AppDeployment <name of AppDeployment>

  3. Remove Kafka CRDs:

    NOTE: The CRDs are not finalized for deletion until you delete the associated custom resources.

    kubectl delete crds kafkaclusters.kafka.banzaicloud.io kafkausers.kafka.banzaicloud.io kafkatopics.kafka.banzaicloud.io

Resources