Replacing a Cassandra Dead Node

You can replace a Cassandra dead node with another node that will have all the data of the dead node. Use this procedure when a node in the Cassandra cluster cannot be recovered, for example there is a hardware failure.

If you enabled incremental backups on the Cassandra cluster, then after the restore or dead node replacement, Cassandra might create several files under the directory named "backups" as part of the compaction process. If you disable incremental backups on the replacement node before the restore, then you can reduce the following:

• The space used for the files that are created under the "backups" directory

• The application size of the next incremental backup

Before You Begin

1. Create a placeholder node in the same Cassandra cluster as the dead node. This node will replace the dead node. For more information see, Configuring Cassandra Nodes.

2. On the replacement node, make sure that the following Cassandra files are the same as the files on the dead node.

   • The configuration file (cassandra.yaml)

   • The datacenter and rack information file (cassandra-rackdc.properties)

   • The network topology file (cassandra-topology.properties)

3. Do not start Cassandra on the replacement node.

4. To disable incremental backups, on the command line, type the following command:

   nodetool disablebackup
       
   

Procedure

1. From the CommCell Browser, expand Client Computers > client > Big Data Apps > instance.

2. Right-click the default_subclient, and then click Browse and Restore.

3. On the Browse and Restore Options dialog box, select Latest Backup, and then click View Content.

4. On the Browse page, determine what data to restore.

   The Browse page displays all the cluster nodes in the left pane.

   1. Expand the cluster.

   2. Select the dead node.

   3. Click Recover All Selected.

      The Restore Options dialog box appears.

5. On the General tab, select the restore options:

   1. Select the Out of place restore check box.

   2. From the Destination Instance list, select the source instance.

   3. In the Target column for a node, select the placeholder node.

   4. Select the Replace Dead Node check box.

   5. Select Recover.

   6. In the Number of Streams box, type the number of streams the software uses for the restore operation.

6. Click OK.

Result

The software replaces the dead node with the target node. All data of the dead node is now available on the target node.

Cassandra synchronizes the system keyspaces and the replicated data, but it might take some time.

What to Do Next

1. Edit the Cassandra configuration file (cassandra.yaml) on the replacement node and remove the initial_token parameter.

2. On the replacement node, remove the following line from the Cassandra-env.sh file:

   JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=address_of_dead_node
       
   

3. Run a discover operation or run a backup operation on any subclient that does an auto discover. The discover operation automatically removes the dead node and replaces it with a new node.

4. If you disabled incremental backups, use the following command to enable them:

   nodetool enablebackup
       
   

5. Run a full backup for the replacement node so that you can perform incremental backups. For more information, see Backing Up Cassandra Databases.