To implement application-consistent backups for Kubernetes, you can execute tasks before and after taking volume snapshots.
Tasks are scripts/commands that run before or after certain job phases. During backups, the script files are copied and executed inside the application pods. The tasks can be written in any scripting language, such as bash, Perl, or Python. To use Perl or Python scripts, you must pre-install Perl or Python in the pod.
CvTasks Method vs Pre/Post-Script Method
Both the new CvTasks method described on this page and the old method of executing .prescript and .postscript files from access nodes (for Metallic, the access nodes must be on-premises) are supported. If both methods are configured, the new method is used.
Requirements
For a given Kubernetes cluster, the following CRD must be applied. The CRD can be obtained from Commvault / Kubernetes on GitHub.
- kubectl create -k AppConsistency/deploy/
Custom Resources
The following custom resources are used:
-
CvTask:
-
Defines a task. A task represents a unit of execution.
-
Can be a single command, a script text, or a local script (residing in the Pod).
-
-
CvTaskSet:
-
Links a Kubernetes application to the tasks that need to be executed.
-
Prescribes an order in which these tasks will be executed.
-
CvTask
For task execution, CvTask has the following ways to execute, which can be specified in commandType:
-
Command: Direct command is executed on the Pod.
-
LocalScript: Local script that is present on the Pod is executed. (You must provide absolute paths.)
-
ScriptText: Custom script (in the command field of CvTask) is copied and executed inside the Pod.
CvTaskSet
CvTaskSet must include one of the following fields to function as a selector for CvTasks to be executed on the application pods:
-
appName
-
appNamespace
-
labelSelectors
CvTaskSet can have multiple tasks to be executed. Each task is mapped to a CvTask. So, if you want to execute multiple tasks for an application, you can create multiple CvTasks and add them to the tasks in CvTaskSet.
The following is a sample YAML for CvTaskSet. In this example, for all applications with the name wordpress
in the namespace wordpress-mysql
, the tasks cvtask001
and cvtask002
are executed in the same order.
apiVersion: k8s.cv.io/v1
kind: CvTaskSet
metadata:
name: cvtaskset001
namespace: cv-config
spec:
appName: wordpress
appNamespace: wordpress-mysql
tasks:
- cvTaskName: cvtask001
cvTaskNamespace: cv-config
id: testid0001
isDisabled: false
executionOrder: 1
- cvTaskName: cvtask002
cvTaskNamespace: cv-config
id: testid0002
isDisabled: false
executionOrder: 2
Process Steps
-
For each application that is configured as content in the application group, during the backup, Commvault searches CvTaskSets in the
cv-config
namespace for matching applications. -
If Commvault finds one or more matching CvTaskSets, Commvault executes the tasks that are specified in CvTaskSet on applications, in the specified order.
Database Detection Script
You can use the CVK8SDiscoverDB.sh script to detect MySQl, PostgreSQL, KeyDB and MariaDB database pods that run inside the Kubernetes cluster. The script generates CVTask and CVTaskSet custom resource YAML files to quiesce these databases. The YAML files are placed in the 'manifests' folder inside the script directory. You can also use the script to apply the YAML files to the cluster.
When you run the script, you are prompted to enter the credentials for the respective databases to quiesce or unquiesce them. You can also specify these credentials as kubernetes secrets.
To get the script, go to DBDetection in Commvault / Kubernetes on GitHub.
Syntax for the Database Detection Script
CVK8SDiscoverDB.sh --namespace <name / all > --db <mysql / postgresql / keydb / mariadb / all>
Examples for the Database Detection Script
The following command detects MySQL database pods that run under the 'test' namespace:
CVK8SDiscoverDB.sh --namespace test --db mysql
The following command detects PostgreSQL pods that run under the 'test1' and 'test2' namespaces:
CVK8SDiscoverDB.sh --namespace test1, test2--db postgresql
The following command detects MySQL and PostgreSQL pods that run under all namespaces:
CVK8SDiscoverDB.sh --namespace all --db all
Custom Resource Examples
For more examples, see AppConsistency in Commvault / Kubernetes on GitHub.
CvTask Example 1: Command
In this example:
-
The command
mysql
with arguments-hHOSTNAME
,-uUSER
,-pPASSWORD
,DATABASE
, and-e"FLUSH TABLES WITH READ LOCK;"
is called before the volume snapshot is taken. -
The command
mysql
with arguments-hHOSTNAME
,-uUSER
,-pPASSWORD
,DATABASE
, and-e"UNLOCK TABLES;"
is called after the volume snapshot is taken.apiVersion: k8s.cv.io/v1 kind: CvTask metadata: name: cvtask001 namespace: cv-config spec: postBackupSnapshot: args: - -hHOSTNAME - -uUSER - -pPASSWORD - DATABASE - -e"UNLOCK TABLES;" command: mysql commandType: Command preBackupSnapshot: args: - -hHOSTNAME - -uUSER - -pPASSWORD - DATABASE - -e"FLUSH TABLES WITH READ LOCK;" command: mysql commandType: Command
CvTask Example 2: ScriptText
In this example:
-
The whole script after
preBackupSnapshot
is copied on to the application pod and executed before the volume snapshot is taken. -
The whole script after
postBackupSnapshot
is copied on to the application pod and executed after the volume snapshot is taken.apiVersion: k8s.cv.io/v1 kind: CvTask metadata: name: cvtask009-python-scripttext namespace: cv-config spec: preBackupSnapshot: args: [] command: | #!/usr/local/bin/python import MySQLdb import os import time import datetime dt=datetime.datetime.now().strftime("%I:%M%p on %B %d, %Y") file1 = open("/scripts/pre-freeze.log","a+" ) try: conn = MySQLdb.connect ('localhost' , 'root' , 'password' ) cur = conn.cursor() cur.execute ("select version()") data = cur.fetchone() file1.write (dt) file1.write ("-------------------------------------------\n") file1.write ("-------------------------------------------\n") file1.write ("\t MySQL version is %s: "%data) file1.write ("-------------------------------------------\n") file1.write ("-------------------------------------------\n") except: file1.write (dt) file1.write("\t unable to connect to MySQL server\n") file2 = open ('/tmp/freeze_snap.lock', 'w') file2.close() try: cur.execute (" flush tables with read lock ") file1.write (dt) file1.write ("\t using quiesce.py script - quiesce of database successful \n") except: file1.write(dt) file1.write( "\n unexpected error from MySQL, unable to do flush tables with read lock, Please check MySQL error logs for more info\n") while True: check = os.path.exists ("/tmp/freeze_snap.lock") if check == True: continue else: break commandType: ScriptText postBackupSnapshot: args: [] command: | #!/usr/local/bin/python import MySQLdb import os import time import datetime dt=datetime.datetime.now().strftime("%I:%M%p on %B %d, %Y") file1 = open("/scripts/post-thaw.log","a+" ) try: os.remove('/tmp/freeze_snap.lock') time.sleep(2) except Exception, e: print e try: conn = MySQLdb.connect ('localhost' , 'root' , 'password' ) cur = conn.cursor() cur.execute ("select version()") data = cur.fetchone() file1.write (dt) file1.write ("-------------------------------------------\n") file1.write ("-------------------------------------------\n") file1.write ("\t MySQL version is %s: "%data) file1.write ("-------------------------------------------\n") file1.write ("-------------------------------------------\n") except: file1.write (dt) file1.write("\t unable to connect to MySQL server\n") try: file1.write (dt) file1.write ("\t executing query to unquiesce the database \n") cur.execute ("unlock tables") file1.write (dt) file1.write ("\t Database is in unquiesce mode now \n") except: file1.write(dt) file1.write( "\n unexpected error from MySQL, unable to unlock tables. Please check MySql error logs for more info \n") cur.close() conn.close() commandType: ScriptText
CvTask Example 3: LocalScript
In this example:
-
The local script
/root/scripts/mysqldb-actions
with argumentquiesce
is called before the volume snapshot is taken. -
The local script
/root/scripts/mysqldb-actions
with argumentunquiesce
is called after the volume snapshot is taken.apiVersion: k8s.cv.io/v1 kind: CvTask metadata: name: cvtask003-local namespace: cv-config spec: postBackupSnapshot: args: - unquiesce command: /root/scripts/mysqldb-actions commandType: LocalScript preBackupSnapshot: args: - quiesce command: /root/scripts/mysqldb-actions commandType: LocalScript
CvTaskSet Example 1: appName
In this example, for applications in the namespace wordpress
that have the name wordpress-mysql
, task cvtask001
is executed, followed by cvtask002
. Task cvtask003
is skipped because it is disabled.
apiVersion: k8s.cv.io/v1
kind: CvTaskSet
metadata:
name: cvtaskset001
namespace: cv-config
spec:
appName: wordpress-mysql
appNamespace: wordpress
tasks:
- cvTaskName: cvtask001
cvTaskNamespace: cv-config
executionOrder: 1
id: custom-testcase-123
isDisabled: false
- cvTaskName: cvtask002
cvTaskNamespace: cv-config
executionOrder: 3
id: custom-testcase-123
isDisabled: false
- cvTaskName: cvtask003
cvTaskNamespace: cv-config
executionOrder: 2
id: custom-testcase-123
isDisabled: true
CvTaskSet Example 2: labelSelectors
In this example, for applications in the namespace wordpress
that have labels tier=frontend
and app=database
, tasks cvtask001
and cvtask002
are executed in the same order. Task cvtask003
is skipped because it is disabled.
apiVersion: k8s.cv.io/v1
kind: CvTaskSet
metadata:
name: cvtaskset001
namespace: cv-config
spec:
labelSelectors:
- -tier=frontend
- -app=database
appNamespace: wordpress
tasks:
- cvTaskName: cvtask001
cvTaskNamespace: cv-config
executionOrder: 1
id: custom-testcase-123
isDisabled: false
- cvTaskName: cvtask002
cvTaskNamespace: cv-config
executionOrder:
id: custom-testcase-123
isDisabled: false
- cvTaskName: cvtask003
cvTaskNamespace: cv-config
executionOrder: 2
id: custom-testcase-123
isDisabled: true