Recovering From Stuck Failovers

Updated

Symptoms

When one or more of the following situations occur:

  • The failover progress does not report any new activity in the Process Manager associated with the SQL Server client, in the Failover Assistant tab.

  • Both the production and standby CommServe hosts are down and the Node Information in the Failover Assistant tab appears red or blank.

  • Production CommServe host is functional but the Process Manager associated with the SQL Server client does not display any information in the Failover Assistant tab.

  • The failover is interrupted, and the subsequent rollback operation also fails. Services are not started in the production CommServe host.

  • Force a failover to a standby CommServe host, when an unplanned failover fails.

    Caution: Use extreme caution while forcing a failover to a standby CommServe host. Data loss can be experienced with this process as the SQL Replication job will not run automatically before performing the failover. Use this option as a last resort and only when all the other options do not work.

Resolution - Windows

  1. Navigate to the following folder associated with the SQL client:

    \Program Files\Commvault\ContentStore2\Base

    Note: ContentStore2 corresponds to the installation folder associated with the SQL client or Instance002. If you have installed the SQL client in a different instance, navigate to the corresponding folder.

  2. Run the following command:

    CvFailover -OpType ResetFailoverConfig -FailoverType Production -TargetNode ProductionSQLClient
  3. If failover operation does not succeed or fail and still in progress, run the following command:

    CvFailover -OpType ResetFailoverOperation -FailoverType Production -TargetNode ProductionSQLClient
  4. Re-run the failover operation.

  5. If the failover operation continues to be stuck, run the following command:

    CvFailover -OpType Failover -FailoverType Production -TargetNode ProductionSQLClient -forceUnplannedFailover

    Caution:If there is a failure during a forced unplanned failover, contact Commvault Customer Support. Do not perform any manual or additional steps to recover from the failure.

Resolution - Linux

  1. Navigate to the following folder associated with the SQL client:

    opt/commvault2/Base

    Note: commvault2 corresponds to the installation folder associated with the SQL client or Instance002. If you have installed the SQL client in a different instance, navigate to the corresponding folder.

  2. Run the following command:

    ./CvFailover -OpType ResetFailoverConfig -FailoverType Production -TargetNode ProductionSQLClient
  3. If failover operation does not succeed or fail and still in progress, run the following command:

    ./CvFailover -OpType ResetFailoverOperation -FailoverType Production -TargetNode ProductionSQLClient
  4. Re-run the failover operation.

  5. If the failover operation continues to be stuck, run the following command:

    ./CvFailover -OpType Failover -FailoverType Production -TargetNode ProductionSQLClient -forceUnplannedFailover

    Caution:If there is a failure during a forced unplanned failover, contact Commvault Customer Support. Do not perform any manual or additional steps to recover from the failure.

Additional Information

CvFailover Command Options

  • <OpType> can be one of the following operations:

    Failover - To perform a failover operation.

    GetFailoverConfig - To display the failover configuration status.

    ResetFailoverOperation - To reset a partial failover operation.

    ResetFailoverConfig - To reset an existing failover configuration. (To be used in case of xml corruption.)

  • <Failover Type> can be one of the following failover types:

    Production - To perform a production failover.

    ProductionMaintenance - To perform a maintenance failover.

    MaintenanceFailback option - To reset maintenance failover.

    Test - To perform a test failover.

    TestFailback - To reset a test failover.

  • <Target Node Name> is the name of the target SQL client in the CommServe host to which the operation must be failed over.

  • -forceUnplannedFailover can be used when the setup is stuck and only when the target node is the node with latest database, to avoid major data loss.