You can stop and restart a single node in a HyperScale cluster when required. To maintain data integrity and prevent data loss, always follow the shutdown and startup procedures outlined on this page when stopping or starting a node. Use these steps to safely shut down and restart a single node for maintenance purposes.
Caution
- If more than one node must be shut down, shut down the entire cluster using the procedure described in Stopping and Starting the Cluster. Improperly shutting down multiple nodes individually can result in data integrity issues, data loss, and restore failures.
- Before stopping a node, verify the cluster status by running the following command to confirm that all services are fully online on the remaining nodes:
After starting a node, it may take up to one hour for all services to come online. Do not stop the next node until you confirm the services are fully operational. Stopping another node prematurely may result in two nodes being offline simultaneously, which can compromise data integrity./usr/local/hedvig/scripts/showmembers.exp
Stopping a Node
-
Stop or suspend all running jobs that are associated with the Backup Gateway installed in the node that you plan to shut-down. For more information, see Controlling Jobs.
-
Set the Backup Gateway on maintenance mode. For more information, see Setting Backup Gateway in Maintenance Mode.
-
Stop the Commvault Cloud services, and verify that the Commvault File System (CVFS) vdisk is unmounted on the node:
-
Using an SSH client program, like PuTTy on Windows, login to any one of the nodes using the
root
credentials. -
Stop the services using the following command:
# commvault stop
-
To verify that the vdisk is unmounted successfully, run the following command:
The output should not appear like below:# df -h
127.0.0.1:/exports/CVLTBackupSMC43002153 269T -1.5T 271T 0% /ws/hedvig/CVLTBackupSMC43002153 127.0.0.1:/exports/CVLTBackupSMC43002153 269T -1.5T 271T 0% /ws/hedvig/CVLTBackupSMC43002153-r
-
-
Stop the nodes using the following commands:
Note
On nodes with HyperScale 2.x installations, before you stop the nodes, switch to the admin user using the command,
# su - admin
.# export HV_PUBKEY=1 # /opt/hedvig/bin/hv_deploy --show_all_clusters
This will display the cluster name, as shown in the following sample output:
Cluster name: HV13102021054000 - owner: unowned - version: v-4.5.0.0.2.3597.00320e0ba9:06aca75bc4db06170a13ef14cbfe8446
Use the cluster name to stop the node using the following command:
# /opt/hedvig/bin/hv_deploy --stop_node fqdn hostname --cluster_name Clustername
For example:
# /opt/hedvig/bin/hv_deploy --stop_node hs2300m5.company.com --cluster_name HV13102021054000 ... provisioning as a hsx cluster. details: ApplianceType HS2300 ... logged into commvault cluster Ensuring password-less SSH connections have been added for all cluster machines. RUNNING: stop_node ************************************************************* RUNNING PLAY 1 [stop hedvig hblock] ******************************************** RUNNING PLAY 2 [stop hedvig pages] ********************************************* RUNNING PLAY 3 [stop hedvig hpod] ********************************************** COMPLETED PLAY 3 *************************************************************** ACTION RECAP: ****************************************************************** hs2300m5.company.com : ok=9 changed=0 unreachable=0 failed=0
Starting a Node After a Reboot
-
Reboot the node using the following command:
# reboot
After a reboot all the services will be automatically restarted and the CVFS vdisk will be mounted.
-
Disable the maintenance mode for the Backup Gateway to bring it online. For more information, see Setting Backup Gateway in Maintenance Mode.
Starting a Node Manually
-
Using an SSH client program, like PuTTy on Windows, login to a node using the
root
credentials. -
Start the nodes using the following commands:
Note
On nodes with HyperScale 2.x installations, before you stop the nodes, switch to the admin user using the command,
# su - admin
.# export HV_PUBKEY=1 # /opt/hedvig/bin/hv_deploy --show_all_clusters
This will display the cluster name, as shown in the following sample output:
Cluster name: HV13102021054000 - owner: unowned - version: v-4.5.0.0.2.3597.00320e0ba9:06aca75bc4db06170a13ef14cbfe8446
Use the cluster name to start the node using the following command:
# /opt/hedvig/bin/hv_deploy --start_node fqdn hostname --cluster_name Clustername
For example:
# /opt/hedvig/bin/hv_deploy --start_node hs2300m5.company.com --cluster_name HV13102021054000 ... provisioning as a hsx cluster. details: ApplianceType HS2300 ... logged into commvault cluster Ensuring password-less SSH connections have been added for all cluster machines. RUNNING: start_node ************************************************************ RUNNING PLAY 1 [collect deploy server facts] *********************************** RUNNING PLAY 2 [start node] **************************************************** COMPLETED PLAY 2 *************************************************************** ACTION RECAP: ****************************************************************** hs2300m5-1a.company.com : ok=16 changed=10 unreachable=0 failed=0 localhost : ok=2 changed=0 unreachable=0 failed=0
-
Start the Commvault Cloud services. (This will also mount the CVFS vdisk.)
# commvault start
-
Disable the maintenance mode for the Backup Gateway to bring it online. For more information, see Setting Backup Gateway in Maintenance Mode.