Recovering From Stuck Failovers

Symptoms

When one or more of the following situations occur:

The failover progress does not report any new activity in the Process Manager associated with the SQL Server client, in the Failover Assistant tab.
Both the production and standby CommServe hosts are down and the Node Information in the Failover Assistant tab appears red or blank.
Production CommServe host is functional but the Process Manager associated with the SQL Server client does not display any information in the Failover Assistant tab.
The failover is interrupted, and the subsequent rollback operation also fails. Services are not started in the production CommServe host.
Force a failover to a standby CommServe host, when an unplanned failover fails.
Caution: Use extreme caution while forcing a failover to a standby CommServe host. Data loss can be experienced with this process as the SQL Replication job will not run automatically before performing the failover. Use this option as a last resort and only when all the other options do not work.

Navigate to the following folder associated with the SQL client:
\Program Files\Commvault\ContentStore2\Base
Note: ContentStore2 corresponds to the installation folder associated with the SQL client or Instance002. If you have installed the SQL client in a different instance, navigate to the corresponding folder.

Run the following command:

CvFailover -OpType ResetFailoverConfig -FailoverType Production -TargetNode ProductionSQLClient

If failover operation does not succeed or fail and still in progress, run the following command:

CvFailover -OpType ResetFailoverOperation -FailoverType Production -TargetNode ProductionSQLClient

Re-run the failover operation.
If the failover operation continues to be stuck, run the following command:
```
CvFailover -OpType ResetFailoverOperation -FailoverType Production -FailoverSubType forceUnplanned -TargetNode ProductionSQLClient
```
Caution:If there is a failure during a forced unplanned failover, contact Commvault Customer Support. Do not perform any manual or additional steps to recover from the failure.

Navigate to the following folder associated with the SQL client:
opt/commvault2/Base
Note: commvault2 corresponds to the installation folder associated with the SQL client or Instance002. If you have installed the SQL client in a different instance, navigate to the corresponding folder.

Run the following command:

./CvFailover -OpType ResetFailoverConfig -FailoverType Production -TargetNode ProductionSQLClient

If failover operation does not succeed or fail and still in progress, run the following command:

./CvFailover -OpType ResetFailoverOperation -FailoverType Production -TargetNode ProductionSQLClient

Re-run the failover operation.
If the failover operation continues to be stuck, run the following command:
```
./CvFailover -OpType ResetFailoverOperation -FailoverType Production -TargetNode ProductionSQLClient -forceUnplannedFailover
```
Caution:If there is a failure during a forced unplanned failover, contact Commvault Customer Support. Do not perform any manual or additional steps to recover from the failure.

<OpType> can be one of the following operations:
Failover - To perform a failover operation.
GetFailoverConfig - To display the failover configuration status.
ResetFailoverOperation - To reset a partial failover operation.
ResetFailoverConfig - To reset an existing failover configuration. (To be used in case of xml corruption.)
<Failover Type> can be one of the following failover types:
Production - To perform a production failover.
ProductionMaintenance - To perform a maintenance failover.
MaintenanceFailback option - To reset maintenance failover.
Test - To perform a test failover.
TestFailback - To reset a test failover.
<Target Node Name> is the name of the target SQL client in the CommServe host to which the operation must be failed over.
-forceUnplannedFailover can be used when the setup is stuck and only when the target node is the node with latest database, to avoid major data loss.