For production clusters, regular maintenance should include routine backup operations on a regular basis to ensure data integrity and reduce the risk of data loss due to unexpected events. Back up operations should include the cluster state, application state, and the running configuration of both stateless and stateful applications in the cluster.

Velero

Konvoy provides the Velero addon by default, to support backup and restore operations for your Kubernetes cluster and persistent volumes.

For on-premises deployments, Konvoy deploys Velero integrated with Minio, operating inside the same cluster.

For production use-cases, it is advisable to provide an external storage class for Minio to use. With the external storage class, you can configure the Velero addon in cluster.yaml as follows:

- name: velero
  enabled: true
  values: |-
     minio:
      persistence:
         storageClass: <external storage class name>

You may also store your backups in Amazon S3 by configuring the Velero addon in cluster.yaml as follows:

- name: velero
  enabled: true
  values: |-
    minioBackend: false
    configuration:
      backupStorageLocation:
        # `name:` must be empty
        bucket: <BUCKET_NAME>
        config:
          region: <AWS_REGION> # such as us-west-2
          s3ForcePathStyle: "false"
          insecureSkipTLSVerify: "false"
          s3Url: ""
    credentials:
      # With the proper IAM permissions with access to the S3 bucket,
      # you can attach the EC2 instances the IAM Role, OR fill in `existingSecret` OR `secretContents` below.
      #
      # Name of a pre-existing secret (if any) in the Velero namespace
      # that should be used to get IAM account credentials.
      existingSecret:
      # The key must be named `cloud`, and the value corresponds to the entire content of your IAM credentials file.
      # Here is a list of documentation for plugins maintained by the Velero team:
      # [AWS] https://github.com/vmware-tanzu/velero-plugin-for-aws/blob/main/README.md
      secretContents: {}
        # cloud: |
        #   [default]
        #   aws_access_key_id=<REDACTED>
        #   aws_secret_access_key=<REDACTED>

Install the Velero command-line interface

Although installing the Velero command-line interface is optional and independent of deploying a Konvoy cluster, having access to the command-line interface provides several benefits. For example, you can use the Velero command-line interface to back up or restore a cluster on demand, or to modify certain settings without changing the Velero platform service configuration.

By default, Konvoy sets up Velero to use Minio over TLS using a self-signed certificate. Due to this, when using certain commands, you may be asked to use the --insecure-skip-tls-verify flag. Again, the default setup is not suitable for production use-cases.

Instructions to install the Velero command-line interface can be found here.

Enable or disable the backup addon

NOTE: Before disabling the Velero addon, be sure you have a recent backup that you can use to restore the cluster in the event that there is a problem converting to the new backup service.

You can enable the Velero addon using the following settings in the ClusterConfiguration section of the cluster.yaml file:

addons:
- name: velero
  enabled: true
...

If you want to replace the Velero addon with a different backup addon service, you can disable the velero addon by modifying the ClusterConfiguration section of the cluster.yaml file as follows:

addons:
- name: velero
  enabled: false
...

Before disabling the Velero platform service addon, however, be sure you have a recent backup that you can use to restore the cluster in the event that there is a problem converting to the new backup service.

After making changes to your cluster.yaml, you must run konvoy up to apply them to the running cluster.

Regular backup operations

For production clusters, you should be familiar with the following basic administrative functions Velero provides:

Set a backup schedule
Run on-demand backups
Restore from a backup archive

Set a backup schedule

By default, Konvoy configures a regular, automatic backup of the cluster’s state in Velero. The default settings do the following:

create backups on a daily basis
save the data from all namespaces

These default settings take effect after the cluster is created. If you install Konvoy with the default platform services deployed, the initial backup starts after the cluster is successfully provisioned and ready for use.

Alternate backup schedules

The Velero CLI provides an easy way to create alternate backup schedules. For example:

velero create schedule thrice-daily --schedule="@every 8h"

To change the default backup service settings:

Check the backup schedules currently configured for the cluster by running the following command:
```
velero get schedules
```
Delete the velero-kubeaddons-default schedule by running the following command:
```
velero delete schedule velero-kubeaddons-default
```
Replace the default schedule with your custom settings by running the following command:
```
velero create schedule velero-kubeaddons-default --schedule="@every 24h"
```

You can also create backup schedules for specific namespaces. Creating a backup for a specific namespace can be useful for clusters running multiple apps operated by multiple teams. For example:

velero create schedule system-critical --include-namespaces=kube-system,kube-public,kubeaddons --schedule="@every 24h"

The Velero command line interface provides many more options worth exploring. You can also find tutorials for disaster recovery and cluster migration on the Velero community site.

Back up on demand

In some cases, you might find it necessary create a backup outside of the regularly-scheduled interval. For example, if you are preparing to upgrade any components or modify your cluster configuration, you should perform a backup immediately before taking that action.

Create a backup by running the following command:

velero backup create BACKUP_NAME

Restore a cluster

Before attempting to restore the cluster state using the Velero command-line interface, you should verify the following requirements:

The backend storage, Minio, is still operational.
The Velero platform service in the cluster is still operational.
The Velero platform service must be set to a restore-only-mode to avoid having backups run while restoring.

To list the available backup archives for your cluster, run the following command:

velero backup get

To set Velero to a restore-only-mode, modify the Velero addon in the ClusterConfiguration of the cluster.yaml file:

addons:
...
- name: velero
  enabled: true
  values: |-
    configuration:
      restoreOnlyMode: true
...

Then you may apply the configuration change by running:

konvoy deploy addons -y

Finally, check your deployment to verify that the configuration change was applied correctly:

helm get values velero-kubeaddons

To restore cluster data on demand from a selected backup snapshot available in the cluster, run a command similar to the following:

velero restore create --from-backup BACKUP_NAME

Backup service diagnostics

You can check whether the Velero service is currently running on your cluster through the operations portal, or by running the following kubectl command:

kubectl get all -n velero

If the Velero platform service addon is currently running, you can generate diagnostic information about Velero backup and restore operations. For example, you can run the following commands to retrieve, back up, and restore information that you can use to assess the overall health of Velero in your cluster:

velero get schedules
velero get backups
velero get restores
velero get backup-locations
velero get snapshot-locations

Back up and restore

Back up and restore the Konvoy cluster