Symptoms
When one or more of the following situations occur:
-
The failover progress does not report any new activity in the Process Manager associated with the SQL Server client, in the Failover Assistant tab.
-
Both the production and standby CommServe hosts are down and the Node Information in the Failover Assistant tab appears red or blank.
-
Production CommServe host is functional but the Process Manager associated with the SQL Server client does not display any information in the Failover Assistant tab.
-
The failover is interrupted, and the subsequent rollback operation also fails. Services are not started in the production CommServe host.
-
Force a failover to a standby CommServe host, when an unplanned failover fails.
Caution
Use extreme caution while forcing a failover to a standby CommServe host. Data loss can be experienced with this process as the SQL Replication job will not run automatically before performing the failover. Use this option as a last resort and only when all the other options do not work.
Resolution - Windows
-
Navigate to the following folder associated with the SQL client:
\Program Files\Commvault\ContentStore2\Base
Note
ContentStore2 corresponds to the installation folder associated with the SQL client or Instance002. If you have installed the SQL client in a different instance, navigate to the corresponding folder.
-
Run the following command:
CvFailover -OpType ResetFailoverConfig -FailoverType Production -TargetNode ProductionSQLClient
-
If failover operation does not succeed or fail and still in progress, run the following command:
CvFailover -OpType ResetFailoverOperation -FailoverType Production -TargetNode ProductionSQLClient
-
Re-run the failover operation.
-
If the failover operation continues to be stuck, run the following command:
CvFailover -OpType ResetFailoverOperation -FailoverType Production -FailoverSubType forceUnplanned -TargetNode ProductionSQLClient
Caution
If there is a failure during a forced unplanned failover, contact Commvault Customer Support. Do not perform any manual or additional steps to recover from the failure.
Resolution - Linux
-
Navigate to the following folder associated with the SQL client:
opt/commvault2/Base
Note
commvault2 corresponds to the installation folder associated with the SQL client or Instance002. If you have installed the SQL client in a different instance, navigate to the corresponding folder.
-
Run the following command:
./CvFailover -OpType ResetFailoverConfig -FailoverType Production -TargetNode ProductionSQLClient
-
If failover operation does not succeed or fail and still in progress, run the following command:
./CvFailover -OpType ResetFailoverOperation -FailoverType Production -TargetNode ProductionSQLClient
-
Re-run the failover operation.
-
If the failover operation continues to be stuck, run the following command:
./CvFailover -OpType ResetFailoverOperation -FailoverType Production -TargetNode ProductionSQLClient -forceUnplannedFailover
Caution
If there is a failure during a forced unplanned failover, contact Commvault Customer Support. Do not perform any manual or additional steps to recover from the failure.
Additional Information
CvFailover Command Options
-
<OpType>
can be one of the following operations:Failover - To perform a failover operation.
GetFailoverConfig - To display the failover configuration status.
ResetFailoverOperation - To reset a partial failover operation.
ResetFailoverConfig - To reset an existing failover configuration. (To be used in case of xml corruption.)
-
<Failover Type>
can be one of the following failover types:Production - To perform a production failover.
ProductionMaintenance - To perform a maintenance failover.
MaintenanceFailback option - To reset maintenance failover.
Test - To perform a test failover.
TestFailback - To reset a test failover.
-
<Target Node Name>
is the name of the target SQL client in the CommServe host to which the operation must be failed over. -
-forceUnplannedFailover can be used when the setup is stuck and only when the target node is the node with latest database, to avoid major data loss.