Replacing Metadata Drives on HyperScale X Nodes

When the metadata drives (NVME or SSD drives) on a HyperScale X node fail, you can replace them with new empty drives.

Each node contains 2 metadata drives. One of the drives store the Commvault metadata that is associated with the index cache and the deduplication database (DDB) and the other drive stores the metadata for Commvault File System (CVFS).

You can configure the new drive to store the metadata for Distributed Storage and the Commvault metadata.

Before You Begin

Verify and ensure that the capacity of the new disk is the same as the disk you want to replace.

Procedure

Stop or suspend all running jobs that are associated with the MediaAgent installed in the node that you plan to shut-down. For more information, see Controlling Jobs.
Set the MediaAgent on maintenance mode. For more information, see Setting the MediaAgent on Maintenance Mode.
Stop the Commvault services and unmount the Commvault File System (CVFS) vdisk on the node as follows:
1. Using an SSH client program, like PuTTy on Windows, login to any one of the nodes using the root credentials.
2. Stop the services using the following command:
```
# commvault stop
```
3. To verify that the vdisk is unmounted successfully, run the following command:
```
# df -h
```
  The output should not appear like below:
```
127.0.0.1:/exports/CVLTBackupSMC43002153 269T -1.5T 271T 0% /ws/hedvig/CVLTBackupSMC43002153 
127.0.0.1:/exports/CVLTBackupSMC43002153 269T -1.5T 271T 0% /ws/hedvig/CVLTBackupSMC43002153-r
```

Stop the node using the following commands:

# export HV_PUBKEY=1
# /opt/hedvig/bin/hv_deploy --show_all_clusters

This will display the cluster name, as shown in the following sample output:

Cluster name: HV13102021054000 - owner: unowned - version: v-4.5.0.0.2.3597.00320e0ba9:06aca75bc4db06170a13ef14cbfe8446

Use the cluster name to stop the node using the following command:

# /opt/hedvig/bin/hv_deploy --stop_node <fqdn hostname> --cluster_name <Clustername>

For example:

# /opt/hedvig/bin/hv_deploy --stop_node hs2300m5.company.com --cluster_name HV13102021054000
... provisioning as a hsx cluster. details: ApplianceType HS2300
... logged into commvault cluster
Ensuring password-less SSH connections have been added for all cluster machines.
RUNNING: stop_node *************************************************************
RUNNING PLAY 1 [stop hedvig hblock] ********************************************
RUNNING PLAY 2 [stop hedvig pages] *********************************************
RUNNING PLAY 3 [stop hedvig hpod] **********************************************
COMPLETED PLAY 3 ***************************************************************
ACTION RECAP: ******************************************************************
hs2300m5.company.com : ok=9    changed=0    unreachable=0    failed=0

Remove the defective disk containing the metadata.
Attach the new replacement disk to the MediaAgent that has the failed disk.
Reboot the node using the following command:
```
# reboot
```
Note

Do not mount the disk on the MediaAgent after the reboot.
Access the node using the iRMC management console in your environment and navigate to the following folder:
```
/opt/commvault/Base/
```

Run the following command to identify the unused drives that were replaced:

./CVSDS -S

The sample output list the unused drives.

Unused Disks: [metadata]
============================
   Disk Name | Disk Size(GB)
============================
/dev/nvme1n1 |   5961.631348
============================

To configure the new metadata drive for CVFS, run the following command and press Enter:

./CVSDS -M -I /dev/nvme1n1 -V CDS

where nvme1n1 is the name of metadata drive for CVFS.

The following output is displayed. At each prompt, enter Y to confirm and proceed with the process.

Please confirm that the node has been rebooted after replacing the damaged metadata drive.
Press 'y' to confirm and continue with the process [y/n]: y
Unused Disks: [metadata]
============================
   Disk Name | Disk Size(GB)
============================
/dev/nvme1n1 |   5961.631348
============================
Following disks will be formatted and used for metadata:
/dev/nvme1n1
Press 'y' to confirm and continue with the process [y/n]: y
Step: Pre-processing
Ransomware protection will be paused on nodes - <smhs43dbxxxx.company.com> - for the duration of the operation.
Press 'y' to confirm and continue with the process [y/n]: y
Pausing Ransomware protection
Successfully paused ransomware protection
Step: Enumerating Block Devices
Step: Validating configuration
Step: Configuring the storage
Successfully created storage layout
Press 'y' to proceed with reconstruction of CDS Metadata [y/n]: y
Step: Reconstruct CDS Metadata
CDS reconstruction is initiated from node - <smhs43dbxxxx.company.com>.
It can take upto 20 min. Please do not exit the process.
Restarting commvault services.
 Step: Post-processing
Resuming Ransomware protection
Successfully resumed ransomware protection

To configure the new metadata drive for Commvault index cache and DDB metadata, run the following command and press Enter:

[root@smhs43db0102 Base]# ./CVSDS -M -I /dev/nvme0n1 -V CV

where nvme0n1 is the name of metadata drive for Commvault metadata.

The following output is displayed. At each prompt, enter Y to confirm and proceed with the process.

Please confirm that the node has been rebooted after replacing the damaged metadata drive.
Press 'y' to confirm and continue with the process [y/n]: y
Unused Disks: [metadata]
============================
   Disk Name | Disk Size(GB)
============================
/dev/nvme0n1 |   5961.631348
============================
Following disks will be formatted and used for metadata:
/dev/nvme0n1
Press 'y' to confirm and continue with the process [y/n]: y
Step: Pre-processing
Ransomware protection will be paused on nodes - <smhs43dbxxxx.company.com> - for the duration of the operation.
Press 'y' to confirm and continue with the process [y/n]: y
Pausing Ransomware protection
Successfully paused ransomware protection
Step: Enumerating Block Devices
Step: Validating configuration
Step: Configuring the storage
Successfully created storage layout
Restarting commvault services.
Step: Post-processing
Resuming Ransomware protection
Successfully resumed ransomware protection

The command will mount the drive, and the Index Cache and/or the deduplicated database (DDB) will be automatically reconstructed in the new disk, if it is a control node. The automatic reconstruction of DDB may take some time. To speed up the DDB reconstruction, you can manually recover the DDB. For more information on manually recovering the DDB, see Manually Recovering the Deduplication Database.

If the CVFS and Commvault metadata was on the same drive that you need to replace, run the following command and press Enter:
```
./CVSDS -M -I /dev/nvme1n1 -V CV_CDS
```
Reboot the node using the following command:
```
# reboot
```
After a reboot all the services will be automatically restarted and the CVFS vdisk will be mounted.
Remove the MediaAgent from maintenance mode. For more information, see Setting the MediaAgent on Maintenance Mode.