# Backup Process for Kubernetes

Commvault provides several backup types for Kubernetes.

## Backup Types

### Full Cluster Backups

A full cluster backup is a special application group that selects all namespaces (including System namespaces) and cluster-scoped API resources for protection. Protection then occurs according to these steps:

### Namespace Backups

A namespace backup is the default backup method for applications. A namespace backup protects all API resources within the parent namespace of a protected application. Namespaces may be selected by label or by discrete selection by browsing the cluster live.

### Application Backups

Application backups are invoked when an individual application (Pod, DaemonSet, Deployment, StatefulSet, or Helm Chart) is selected for backup.

Application backups take an advanced application-centric approach to protection and intelligently infer related API resources/objects (such as Secrets, ConfigMaps, Namespaces).

## Backup Modes

Full cluster, namespace, and application backups can use either snapshot-based backups or streaming backups. If a Kubernetes cluster has a VolumeSnapshotClass object configured, then backups are snapshot-based. If the cluster does not have VolumeSnapshotClass object configured, then backups are streaming.

### CSI Volume Snapshot-Based Backups

#### Container and Application Workloads

A Container Storage Interface (CSI) volume snapshot-based backup uses the VolumeSnapshotClass object configured in the Kubernetes cluster to take snapshots of the persistent volumes and backup data. The CSI snapshot creation process runs in parallel for each application. Persistent volume backups do not occur as a consistency group across all PVs for an app. Instead, snapshots are created sequentially by the access node, and then protected in parallel.

A snapshot-based backup operation includes the following steps:

1. Discover applications based on the application group content.

2. For each PersistentVolumeClaim (PVC) that is associated with applications, in the backup namespace:

   1. Determine the VolumeSnapshotClass (for the PVC).

      By default, the Commvault software uses the VolumeSnapshotClass associated with the StorageClass of PVC. To use a specific VolumeSnapshotClass, add the `cv-snapshotter: <volumesnapshotclass-name>` label to the StorageClass of PVC.

      To add label to the StorageClass, run the following command:

          kubectl patch  storageclass  $[storageclass-name] --patch '{"metadata":{"labels":{"cv-snapshotter":"$[volumesnapshotclass-name]"}}}'

      where:
      - `<storageclass-name>`: The name of the StorageClass.
      - `<volumesnapshotclass-name>`: The name of the VolumeSnapshotClass that you want to use for creating VolumeSnapshot.

   2. Create a snapshot (VolumeSnapshot) of the PVC.

   3. Create a temporary PVC from the snapshot created.

      By default, the Commvault software uses the same StorageClass as that of the source PVC to create a temporary PVC. To use a specific StorageClass, add the `cv-backup-storageclass:<storage_class_name>` label to the VolumeSnapshotClass.

      To add label to the VolumeSnapshotClass, run the following command:

          kubectl patch volumesnapshotclass $[volumesnapshotclass-name] --patch '{"metadata":{"labels":{"cv-backup-storageclass":"$[storage_class_name]"}}}'

      where:
      - `<volumesnapshotclass-name>`: The name of the VolumeSnapshotClass.
      - `<storageclass-name>`: The name of the StorageClass that you want to use for creating PVC.

   4. Create a temporary pod with temporary PVC mounted, to read data through Kubernetes API Server.

3. After the backup completes, delete the temporary pod, temporary VolumeSnapshot, and the temporary PVC.

#### Virtual Machine Workloads

Commvault protects virtual machines (VMs) running on OpenShift Virtualization (KubeVirt) by using CSI snapshots and an in cluster Commvault access node. This approach captures VM data at the storage layer without requiring in-guest packages, making it simple and scalable for cloud-native environments. Backups are crash-consistent and occur outside the guest operating system. This avoids performance overhead inside the VM and aligns with Kubernetes-native storage workflows.

The Commvault software takes a snapshot of the underlying persistent volumes and reads the data through an access node that runs inside the cluster.

The VM backup operation includes the following steps:

1. Discover VMs by querying the Kubernetes API server. During discovery, Commvault does the following:

   - Identifies KubeVirt resources such as VirtualMachine and VirtualMachineInstance

   - Collects namespace and labels, VM configuration metadata and associated disks.

   - Each VM disk is backed by a DataVolume, which maps to a PersistentVolumeClaim (PVC). These PVCs are the units of protection during backup.

2. For each DataVolume or PersistentVolumeClaim (PVC) associated with VM:

   1. Create a CSI VolumeSnapshotClass for PVC. This operation is storage provider native and does not require VM downtime. The software briefly freezes and unfreezes the VM to maintain consistency.

   2. Create a temporary PVC using the snapshot as the data source. This PVC represents a point-in-time copy of the VM disk.

   3. Attach the temporary PVC to the access node running in the cluster. This enables direct access to the snapshot data without affecting the original VM.

   4. Reads data from the mounted volume and stream it to the MediaAgent.

3. After the backup completes, unmount the volume, and delete the temporary PVC and snapshots.

### Streaming Backups

A streaming backup is a legacy backup process that is used when the Kubernetes cluster does not have VolumeSnapshotClass object configured.

A streaming backup operation includes the following steps:

1. Discover applications based on the application group content.

2. For each PersistentVolumeClaim that is associated with the applications, in the backup namespace, create a temporary pod and mount the volume, to read data through Kubernetes API Server.

3. After the backup completes, delete the temporary pod.

## How Commvault Handles Failures During Backups and Restores

### What happens when a failure occurs during a backup or a restore?

By default, Commvault handles failures as follows:

- If the backup or restore for an individual application fails, then the Kubernetes access node restarts the application-specific job from the beginning, starting with rescheduling a new Commvault temporary worker pod. Backup data from the previous failed backup is discarded and must be re-transferred.

- If a child Kubernetes access node fails, then the operations that it was handling are rescheduled on another access node.

- If the coordinator Kubernetes access node fails, then a partial or complete failure status is registered for the job. The job is not restarted.

- If the Commvault temporary worker pod fails, then the job is not restarted and is marked as Completed with Errors.

For more information about job restarts, see Job Status and Control for Virtual Machines. This page contains information for the Virtual Server Agent, which is the agent that the Commvault software uses to manage Kubernetes jobs.

### How many times, and how frequently, does Commvault try to restart a failed job?

By default, restarts are enabled for all job types. For backup jobs, the default number of restarts is 10. For restore jobs, the default number of restarts is 144. For both backup and restore jobs, the default restart interval is 20 minutes.

For more information about job restarts, see Job Status and Control for Virtual Machines. This page contains information for the Virtual Server Agent, which is the agent that the Commvault software uses to manage Kubernetes jobs.

### If resources are created for failed attempts to run the job, are those resources cleaned up?

If a Kubernetes backup or restore operation is interrupted, the Commvault access node might not have an opportunity to remove temporary VolumeSnapshots, volumes, and Commvault temporary worker pods.

For a workaround and other information, see "Cleanup of Temporary Resources" in Restrictions and Known Issues for Kubernetes.

### What happens when the storage snapshotter pod fails during a backup job?

If the Container Storage Interface (CSI) components (specifically, the external snapshotter pod) fail, the following occur:

1. Commvault attempts to issue snap commands fail.

2. The failures are reported by the kube-apiserver as failures of Commvault attempts to snap.

3. Commvault fails the backup for that container.

4. Commvault tries to restart the application snapshot job 5 times, with the following results:

   - If snapshots are already created, the backup proceeds. However, the cleanup of Commvault-created PVCs/snapshots might fail. For more information, see "Cleanup of Temporary Resources" in Restrictions and Known Issues for Kubernetes.

   - If snapshots are not created yet, the backup job fails, and Commvault does not attempt to restart the job.

×

Loading...