Install on-premises

Install Konvoy in an on-premises environment

The topics in this section guide you through the basic steps to prepare your environment and install Konvoy in an on-premises environment.

Prerequisites

Before installing, verify that your environment meets the following basic requirements:

  • Docker version 18.09.2 or later. You must have Docker installed on the host where the Konvoy command line interface (CLI) will run. For example, if you are installing Konvoy on your laptop computer, be sure the laptop has a supported version of Docker.

  • kubectl v1.20.6 or later. You must have kubectl installed on the host, where the Konvoy command line interface (CLI) runs, to enable interaction with the running cluster.

  • Konvoy cannot be run from a host that belongs to a Kubernetes cluster, you must have a separate host.

Control plane nodes

  • You should have at least three control plane nodes.

  • Each control plane node should have at least:

    • 4 cores
    • 16 GiB memory
    • Approximately 80 GiB of free space for the volume used for /var/lib/kubelet and /var/lib/containerd.
    • Disk usage must be below 85% on the root volume.

Worker nodes

  • You should have at least four worker nodes. The specific number of worker nodes required for your environment can vary depending on the cluster workload and size of the nodes.

  • Each worker node should have at least:

    • 8 cores
    • 32 GiB memory
    • Approximately 80 GiB of free space for the volume used for /var/lib/kubelet and /var/lib/containerd.
    • Disk usage must be below 85% on the root volume.

  • If you plan to use local volume provisioning to provide persistent volumes for the workloads, you must mount at least four volumes to the /mnt/disks/ mount point on each node. Each volume must have at least 55 GiB of capacity if the default addon configurations are used.

Operating system and services for all nodes

For all hosts that are part of the cluster, except the deploy host, verify the following:

  • A supported operating system is installed.
  • Firewalld is disabled.
  • Containerd is uninstalled.
  • Docker-ce is uninstalled.
  • Swap is disabled.
  • The hostnames for all the machines in the Kubernetes cluster are unique within a single cluster.
  • If you are using the XFS file system for the volume that mounts the /var/lib/containerd/ directory, it must be formatted with the ftype=1 option.

NOTE: This is the default in most OS distributions. You can use the xfs_info utility to verify that ftype=1 has been used.

Networking

Make sure the following domains are accessible from the control plane nodes and worker nodes.

  • docker.elastic.org
  • download.docker.com
  • gcr.io
  • github.com
  • grafana.com
  • k8s.gcr.io
  • kubernetes.github.io
  • mesosphere.github.io
  • mirror.centos.org
  • ntp.org
  • nvidia.github.io
  • packages.cloud.google.com
  • prometheus-community.github.io
  • quay.io
  • raw.githubusercontent.com
  • registry.hub.docker.com
  • stakater.github.io
  • storage.googleapis.com

For the deploy host, make sure domains registry.hub.docker.com, mesosphere.github.io, and github.com are accessible.

NOTE: In an airgap installation, these domains do not need to be accessible.

For the deploy host, make sure domain registry.hub.docker.com, mesosphere.github.io, and github.com are accessible.

Edit the inventory file

To start the Konvoy installation, you first need an Ansible inventory file in your current working directory to describe the hosts where you want to install Konvoy. Konvoy will automatically generate the skeleton of the inventory file for you during initialization:

  1. Create an empty working directory on the computer you are using as the deploy host.

    For example, you might run the following:

    mkdir konvoy-deploy
cd konvoy-deploy

  2. Run the following commands to initialize Konvoy in the current working directory:

    konvoy init --provisioner=none [--cluster-name <your-specified-name>]

    NOTE: The cluster name may only contain the following characters: a-z, 0-9, . - and _.

    Running the konvoy init command generates an inventory file skeleton inventory.yaml and a default cluster.yaml configuration file in the current working directory.

  3. Open inventory file inventory.yaml in a text editor to specify the hosts.

The inventory file inventory.yaml follows the standard Ansible inventory file yaml format for hosts and groups.

For Konvoy, specify two groups

  • control-plane
  • node

The control-plane group defines the host IP addresses for your control plane nodes. The node group defines the host IP addresses for your worker nodes.

Specifying IP addresses and host names

The IP addresses you specify in the inventory file can be the private IP addresses you use in your internal network. The primary requirement is that all of the hosts in the cluster can communicate with each other using the IP addresses you specify. Note that placing all of the hosts in the same subnet (for example, 10.0.50.0/24) can simplify the cluster configuration significantly.

For each host, you can also optionally specify the ansible_host attribute if you want Ansible to use different host names to reach the hosts.

Ensuring connectivity

Make sure that the computer you are using as the deploy host can open secure shell (SSH) connections to communicate with each host specified in the inventory file. To ensure a successful installation, the ansible_user account must be able to open a secure shell on each host without typing password.

You can use ssh-agent to pass identity keys and passphrases for authentication.

Sample inventory file

The following inventory.yaml file will be generated after running konvoy init:

control-plane:
  hosts:
    <ip-address>:
      ansible_host: <ssh-address>
      ansible_port: <optional>
      node_pool: control-plane
node:
  hosts:
    <ip-address>:
      ansible_host: <ssh-address>
      ansible_port: <optional>
      node_pool: worker
bastion: {}
all:
  vars:
    ansible_port: 22
    ansible_ssh_private_key_file: ""
    ansible_user: ""
    control_plane_endpoint: ""
    order: sorted
    version: v1beta1

  • <ip-address> is the host’s private IP address that will be used by Kubernetes for the Node’s identity.

  • ansible_host is the IP used by Konvoy to SSH into the host, if removed the <ip-address> will be used instead.

  • ansible_port is an optional port used by Konvoy to SSH into the host, the default value is 22.

  • node_pool is used to group nodes into pools, Konvoy will apply a Kubernetes label based on this value and use it internally when selecting Nodes based on their pool.

  • ansible_ssh_private_key_file is the name of the private SSH key used by Konvoy to SSH into the hosts.

  • ansible_user is the user used by Konvoy to SSH into the hosts.

  • control_plane_endpoint is an address for a loadbalancer used by all of the Nodes to reach the Kubernetes API.

  • The values of order: sorted and version: v1beta1 should not be modfied.

The following example illustrates a simple inventory.yaml file with three control plane nodes and three worker nodes.

control-plane:
  hosts:
    10.0.50.232:
    10.0.50.233:
    10.0.50.234:

node:
  hosts:
    10.0.50.108:
      node_pool: worker
    10.0.50.109:
      node_pool: worker
    10.0.50.110:
      node_pool: worker
    10.0.50.111:
      node_pool: worker

all:
  vars:
    ansible_port: 22
    ansible_ssh_private_key_file: "mysshkey.pem"
    ansible_user: "centos"
    control_plane_endpoint: "10.0.50.100:6443"
    order: sorted
    version: v1beta1

In this example:

  • The keys ansible_host and ansible_port were removed, the IPs will be used instead to SSH into the hosts on the default port 22.
  • The ansible_ssh_private_key_file is set to mysshkey.pem and is expected to be in working directory.
  • The ansible_user is the centos user account that has administrative privileges.

Specifying local addons repositories

When using Konvoy with its default addons options, the tool tries to fetch the list of available addons from a public GitHub kubernetes-base-addons repo, kubeaddons-kommander repo and kubeaddons-dispatch repo when initializing and validating the cluster.yaml file. If in your environment access to that repo is blocked, you may also use a local clone of the above repo.

Assuming that the repo was cloned in the local directory to ./kubernetes-base-addons and kubeaddons-kommander, use the --addons-repositories flag with the konvoy init, konvoy up, konvoy provision commands.

This will result in your cluster.yaml containing the details below:

kind: ClusterConfiguration
apiVersion: konvoy.mesosphere.io/v1beta2
spec:
  addons:
  - configRepository: /opt/konvoy/artifacts/kubernetes-base-addons
    configVersion: stable-1.20-4.3.0
    addonsList:
    ...
  - configRepository: /opt/konvoy/artifacts/kubeaddons-dispatch
    configVersion: stable-1.20-1.4.6
    addonsList:
    - name: dispatch
      enabled: false
  - configRepository: /opt/konvoy/artifacts/kubeaddons-kommander
    configVersion: stable-1.20-1.4.3
    addonsList:
    - name: kommander
      enabled: true

You can also specify remote git repos hosted in your organization using the same --addons-repositories flag.

Configure the Kubernetes cluster

After you edit the inventory file, edit the generated cluster.yaml file. The cluster.yaml file provides the configuration details for creating your Konvoy cluster.

Configure the RPM and DEB package repository

By default Konvoy adds new RPM and DEB repositories to the control-plane and worker hosts that are required to install a container runtime and a Kubernetes cluster. If the required repositories are already configured in your environment, you can disable this behavior by setting the value of enableAdditionalRepositories to false.

kind: ClusterConfiguration
apiVersion: konvoy.mesosphere.io/v1beta2
spec:
 osPackages:
   enableAdditionalRepositories: false

Below is the list of all package repositories that are added by Konvoy.

The RPM repositories:

[konvoy-packages]
name = Konvoy Containerd package repository
baseurl = https://packages.d2iq.com/konvoy/stable/linux/centos/7/x86_64
gpgcheck = 1
gpgkey = https://packages.d2iq.com/konvoy/stable/linux/centos/7/x86_64/gpg.asc

[kubernetes]
name=Konvoy Kubernetes package repository
baseurl=https://packages.d2iq.com/konvoy/rpm/stable/centos/7/x86_64
gpgcheck=1
gpgkey=https://packages.d2iq.com/konvoy/rpm-gpg-pub-key

For any Nvidia GPU enabled machines there are additional repositories:

[libnvidia-container]
baseurl = https://nvidia.github.io/libnvidia-container/centos7/x86_64
enabled = 1
gpgcheck = 0
gpgkey = https://nvidia.github.io/libnvidia-container/gpgkey
name = libnvidia-container Repository
repo_gpgcheck = 1
sslcacert = /etc/pki/tls/certs/ca-bundle.crt
sslverify = 1

[nvidia-container-runtime]
baseurl = https://nvidia.github.io/nvidia-container-runtime/centos7/x86_64
enabled = 1
gpgcheck = 0
gpgkey = https://nvidia.github.io/nvidia-container-runtime/gpgkey
name = nvidia-container-runtime Repository
repo_gpgcheck = 1
sslcacert = /etc/pki/tls/certs/ca-bundle.crt
sslverify = 1

The RPM packages installed by Konvoy:

  • yum-plugin-versionlock
  • chrony
  • openssl
  • libseccomp
  • container-selinux, on RHEL you can install the Centos RPM directly
  • containerd.io
  • nfs-utils
  • kubectl
  • kubernetes-cni
  • kubelet
  • cri-tools
  • kubeadm
  • nvme-cli (only on AWS)
  • nvidia-container-runtime (for GPU enabled machines)

There may be additional dependencies that need to be installed that can be found in the standard CentOS/RHEL repositories

The DEB repositories:

deb https://packages.cloud.google.com/apt/ kubernetes-xenial main

https://packages.cloud.google.com/apt/doc/apt-key.gpg

deb https://packages.d2iq.com/konvoy/stable/linux/ubuntu/xenial xenial main

https://packages.d2iq.com/konvoy/stable/linux/ubuntu/xenial/gpg.asc

deb https://packages.d2iq.com/konvoy/stable/linux/ubuntu/bionic bionic main

https://packages.d2iq.com/konvoy/stable/linux/ubuntu/bionic/gpg.asc

deb https://packages.d2iq.com/konvoy/stable/linux/debian/stretch stretch main

https://packages.d2iq.com/konvoy/stable/linux/debian/stretch/gpg.asc

deb https://packages.d2iq.com/konvoy/stable/linux/debian/buster buster main

https://packages.d2iq.com/konvoy/stable/linux/debian/buster/gpg.asc

For any Nvidia GPU enabled machines there are additional repositories:

deb https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64 /

https://nvidia.github.io/libnvidia-container/gpgkey

deb https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64 /

https://nvidia.github.io/nvidia-container-runtime/gpgkey

The DEB packages installed by Konvoy:

  • apt-transport-https
  • chrony
  • openssl
  • libseccomp2
  • containerd.io
  • nfs-common
  • kubectl
  • kubernetes-cni
  • kubelet
  • cri-tools
  • kubeadm
  • nvme-cli (only on AWS)
  • xfsprogs (only on AWS)
  • nvidia-container-runtime (for GPU enabled machines)

There may be additional dependencies that need to be installed that can be found in the standard Debian repositories.

Configure the control plane

Konvoy supports Kubernetes control plane high availability (HA) out-of-the-box for on-premises deployments if you do not have a third-party load balancer.

The default control plane load balancer for Konvoy is based on Keepalived, which will be deployed on all control-plane nodes as static Kubernetes pods.

To use keepalived control plane load balancing:

  • Identify and reserve an unused virtual IP (VIP) address from your networking infrastructure. During the installation, the Konvoy installer will check if the designated IP is free to use as a Keepalived’s VIP by pinging it.

  • Configure your networking infrastructure so that the reserved virtual IP address is reachable:

    • from all hosts specified in the inventory file.
    • from the computer you are using as the deploy host.

    If all cluster hosts and the reserved virtual IP address are in the same subnet, you typically do not need to perform any additional configuration to your networking infrastructure. If you are using more than one subnet for the cluster, however, you should work with your networking team to ensure connectivity between all hosts and the reserved virtual IP address.

The following example illustrates the configuration if the reserved virtual IP address is 10.0.50.20:

kind: ClusterConfiguration
apiVersion: konvoy.mesosphere.io/v1beta2
spec:
  kubernetes:
    controlPlane:
      controlPlaneEndpointOverride: "10.0.50.20:6443"
      keepalived:
        interface: ens20f0 # optional
        vrid: 51           # optional

You could set spec.kubernetes.controlPlane.keepalived.interface to specify the network interface you want to use for the Keepalived VIP. This field is optional. If not set, Konvoy will try to automatically detect the network interface to use based on the route to the VIP. If Konvoy fails to detect the correct network interface, the Konvoy installation may fail when deploying kubeadm.

You could also set spec.kubernetes.controlPlane.keepalived.vrid to specify the Virtual Router ID used by Keepalived. This field is optional. If not set, Konvoy will randomly pick a Virtual Router ID for you.

If you are not setting any of the optional values, set spec.kubernetes.controlPlane.keepalived: {} to enable the default values.

Configure pod and service networking

The following example illustrates how you can configure the pod subnet and service subnet in the cluster.yaml configuration file:

kind: ClusterConfiguration
apiVersion: konvoy.mesosphere.io/v1beta2
spec:
  kubernetes:
    networking:
      podSubnet: 192.168.0.0/24
      serviceSubnet: 10.0.51.0/24

When configuring these settings, you should make sure that the values you set for podSubnet and serviceSubnet do not overlap with your node subnet and your keepalived virtual IP address.

Configure MetalLB load balancing

Konvoy supports Service type LoadBalancer out-of-the-box for on-premises deployments if you do not have a third-party load balancer.

The default load balancer service for addons is based on MetalLB.

To use MetalLB for addon load balancing:

  • Identify and reserve a range of virtual IP addresses (VIPs) from your networking infrastructure.

  • Configure your networking infrastructure so that the reserved virtual IP addresses are reachable:

    • from all hosts specified in the inventory file.
    • from the computer you are using as the deploy host.

Set the range of virtual IP addresses in your cluster.yaml metallb addon configuration values at configInline.address-pools.addresses.

If all cluster hosts and the reserved virtual IP addresses are in the same subnet, you typically do not need to perform any additional configuration to your networking infrastructure. If you are using more than one subnet for the cluster, however, you should work with your networking team to ensure connectivity between all hosts and the reserved range of virtual IP addresses.

MetalLB can be configured in two modes - layer2 and bgp.

The following example illustrates the layer2 configuration in the cluster.yaml configuration file:

kind: ClusterConfiguration
apiVersion: konvoy.mesosphere.io/v1beta2
spec:
  addons:
    addonsList:
    - name: metallb
      enabled: true
      values: |-
        configInline:
          address-pools:
          - name: default
            protocol: layer2
            addresses:
            - 10.0.50.25-10.0.50.50

The following example illustrates the BGP configuration in the cluster.yaml configuration file:

kind: ClusterConfiguration
apiVersion: konvoy.mesosphere.io/v1beta2
spec:
  addons:
    addonsList:
    - name: metallb
      enabled: true
      values: |-
        configInline:
          peers:
          - my-asn: 64500
            peer-asn: 64500
            peer-address: 172.17.0.4
          address-pools:
          - name: my-ip-space
            protocol: bgp
            addresses:
            - 172.40.100.0/24

The number of virtual IP addresses in the reserved range determines the maximum number of services with a type of LoadBalancer that you can create in the cluster.

-NOTE*: Once this configuration is used to create a LoadBalancer Service, MetalLB will reject reconfiguration that would cause the Service’s address to be invalidated. To force reconfiguration you must delete all metallb-controller pods.

Add storage to worker nodes

Konvoy supports local persistent volume provisioning out-of-the-box if you do not have a third-party storage vendor.

This default storage provisioning option allows operators to mount local volumes at a specific location on each host. For a Konvoy cluster, the local volume mount point is /mnt/disks.

Mounted volumes in the /mnt/disks location are detected automatically. Once detected, corresponding persistent volume objects are created in the API server for your stateful workloads. Konvoy uses the static local volume provisioner to perform this task.

To mount local volumes:

  1. Format and mount the volume by running commands similar to the following:

    sudo mkfs.ext4 /dev/path/to/disk
DISK_UUID=$(blkid -s UUID -o value /dev/path/to/disk)
sudo mkdir /mnt/disks/$DISK_UUID
sudo mount -t ext4 /dev/path/to/disk /mnt/disks/$DISK_UUID

  2. Persist the mount entry by adding it to /etc/fstab as follows:

    echo UUID=`sudo blkid -s UUID -o value /dev/path/to/disk` /mnt/disks/$DISK_UUID ext4 defaults 0 2 | sudo tee -a /etc/fstab

  3. Verify that local volumes stay mounted after the host is rebooted.

For more information on how to mount local volumes, see the Operations guide for Kubernetes.

Note that if your stateful workload is using a local persistent volume, it cannot be moved to a different node. If the node fails, the stateful workload might lose its data. If you cannot tolerate this limitation, you should consider other storage options.

Configure NTP on nodes

By default, Konvoy will install and configure chrony as an NTP client used to synchronize time on the control-plane and worker machines. You can disable this by configuring your cluster as follows:

kind: ClusterConfiguration
apiVersion: konvoy.mesosphere.io/v1beta2
spec:
  ntp:
    autoConfigure: false

Pre-flight checks

After you have completed the basic configuration in the cluster.yaml file, you should run the Konvoy pre-flight checks before you run the installation command. The pre-flight checks help to ensure that your on-premises environment has everything ready for installing Konvoy.

To perform the pre-flight checks:

  1. Run the following command:

    konvoy check preflight

  2. Fix any issue reported by the pre-flight check.

  3. Re-run the konvoy check preflight command.

  4. Repeat the previous steps until pre-flight checks pass with a return status of OK.

Install Konvoy

After verifying your infrastructure, you can create a Konvoy Kubernetes cluster by running the following command:

konvoy up

This command installs Kubernetes, and installs default addons to support your Kubernetes cluster.

Specifically, the konvoy up command does the following:

This set of configuration options is the recommended environment for small clusters.

As the konvoy up command runs, it displays information about the operations performed. For example, you can view the command output to see when Ansible connects to the hosts and installs Kubernetes. Once the Kubernetes cluster is up, the konvoy up command installs the addons specified for the cluster.

Deploying Additional Kubernetes Resources

It is possible to provide additional Kubernetes resources that will be deployed after the base cluster is provisioned but before any of the addons are deployed.

To add custom resource files:

  1. Create a new directory named extras/kubernetes/ to contain your custom resource files.

    mkdir -p extras/kubernetes

  2. Create the desired Kubernetes resource files in the extras/kubernetes/ directory.

  3. Run the konvoy up, konvoy deploy or konvoy deploy kubernetes command.

    After [Deploying Kubernetes] and [Adding Node Labels and Taints] phases, a phase will run that will deploy all the resource files provided in `extras/kubernetes/:

    STAGE [Deploying Additional Kubernetes Resources]

secrets/my-secret created
...

Viewing cluster operations

You can access user interfaces to monitor your cluster through the operations portal.

After you run the konvoy up command, if the installation is successful, the command output displays the information you need to access the operations portal.

You should see information similar to this:

Run `konvoy apply kubeconfig` to update kubectl credentials.

Navigate to the URL below to access various services running in the cluster.
  https://10.0.50.25/ops/landing
And login using the credentials below.
  Username: AUTO_GENERATED_USERNAME
  Password: SOME_AUTO_GENERATED_PASSWORD_12345

If the cluster was recently created, the dashboard and services may take a few minutes to be accessible.

Checking the files installed

When the konvoy up completes its setup operations, the following files are generated: