HyperScale Storage Pool - Recovering a Node

When there are hardware failures in a node, such as disk issues or corrupted operating system, only the drive containing the operating system (which can be reinstalled and rebuilt) cannot be recovered. Drives containing the backup and other data, such as the Deduplication Database (DDB) and Index Cache can be preserved.

Nodes setup with MediaAgents pointing to an external CommServe can be recovered. (Nodes setup with both the CommServe and MediaAgents not supported.)

Before You Begin

Perform the following steps if the node was setup using Service Pack 9:

  • Login to the node and navigate to the following folder:


  • Run the following command:

    ./CVSDS -r

    This command will enumerate all data disk drives and stamp the missing extended attributes.


  1. Download the ISO as follows:
    1. Open a Web browser, and then go to https://store.commvault.com.
    2. Determine the appropriate ISO that must be downloaded, based on the following:
      • RHEL version used in the other nodes.
      • Appliance model
    3. Download the appropriate ISO.
  2. Use a USB flash drive to copy the ISO and make it bootable as follows:
    1. On a Linux system, copy the ISO locally, and connect a USB drive to the system.
    2. Create a bootable USB using the following command:

      # dd if=<path of ISO> of=/dev/sdx bs=4m


      X (in sdx) is the drive letter assigned by Linux to the USB drive, and not a partition on the USB.

    3. Once the process completes, attach the USB drive to a node.
  3. Boot the node with the USB drive.

    The Hyperscale Configuration Detected screen is displayed.

  4. Click Next.

    The Preserve Data screen is displayed.

  5. Choose the Preserve Drives option and then click Next.

    The system will start re-imaging the node

    This process will take some time to complete.

  6. Reboot the node.
  7. After the reboot, verify the following network settings for the node and click OK.
    • Data Protection Network FQDN
    • Data Protection Network IP Address, Data Protection Network Netmask and Data Protection Network Default Gateway
    • Domain Name Server 1, Domain Name Server 2 and Domain Name Server 3
    • StoragePool Network IP Address and StoragePool Network Netmask
    • Root password

      Note: Verify that the IP addresses are resolvable to and from the other nodes in the storage pool.

  8. When prompted, choose the Not a last node in the block option and then select OK.
  9. Navigate to the following folder:


  10. Register the HyperScale Client using the following command:


  11. When prompted, provide the following information:
    • CommServe name: Type the fully qualified host name for the CommServe computer.
    • CommServe user name and password: Type the credentials to access this CommServe
  12. Open the CommCell Console and perform a readiness check and verify that the node is accessible. See Viewing the Check Readiness Report for more information on performing the readiness check.

    Perform the following steps if the readiness fails:

    1. Delete contents of /opt/commvault/Base/certificates folder in the node.
    2. Revoke the active certificates for the client (node) as described in Revoking a Client Certificate.
    3. Create a temporary certificate and renew the certificate as described in Renew a Revoked Certificate.

      Make sure to copy and paste the renewed certificate in the /opt/commvault/Base/certificates/export.txt file.

    4. Restart the services on the node. See Controlling Services on UNIX for more information on restarting services.
    5. Add the necessary host entries for the node in the other 2 MediaAgents to the /etc/hosts file.
  13. Open the CommCell Console and add the MMEnableStoragepoolNodeRefresh additional setting with the value 1. The additional setting must be created on the CommServe. See Adding or Modifying Additional Settings for more information.

    This additional setting will enable the Refresh option for the storage pool in the Command Center.

  14. Open the Command Center and navigate to Storage > Storage Pools.
  15. From the Storage Pools page, click the name of the <Storage Pool> in which the reconstructed node is located.
  16. In the Storage area, click the Actions button against the reconstructed node, and then click Refresh. Click Confirm.

    This will refresh the node and the node will be displayed as Ready in the Status column.

  17. Verify that the node is refreshed. Check the following details:
    • Storage target state, total capacity, and mount path status
    • Deduplication database summary and details.
    • Node status and the respective drives state.

    If the Index Cache is not up to date, it will be reconstructed automatically as part of any operations which queries the index, such as backup, browse and find.

    Similarly, if the DDB is detected as offline, unreadable, or unavailable, it will automatically recovered from the latest DDB backup.

  18. Recommended: Open the CommCell Console and disable the MMEnableStoragepoolNodeRefresh additional setting, by setting the value to 0.

This completes the steps for recovering a node.

Last modified: 12/9/2020 12:43:07 PM