You can replace a Cassandra dead node with another node that will have all the data of the dead node. Use this procedure when a node in the Cassandra cluster cannot be recovered, for example there is a hardware failure.
If you enabled incremental backups on the Cassandra cluster, then after the restore or dead node replacement, Cassandra might create several files under the directory named "backups" as part of the compaction process. If you disable incremental backups on the replacement node before the restore, then you can reduce the following:
-
The space used for the files that are created under the "backups" directory
-
The application size of the next incremental backup
Before You Begin
-
On the replacement node, make sure that the following Cassandra files are the same as the files on the dead node.
-
The configuration file (cassandra.yaml)
-
The datacenter and rack information file (cassandra-rackdc.properties)
-
The network topology file (cassandra-topology.properties)
-
-
Do not start Cassandra on the replacement node.
-
To disable incremental backups, on the command line, type the following command:
nodetool disablebackup
Procedure
-
From the navigation pane, go to Protect > Big data.
The Big data page appears.
-
In the Name column, click the cluster that you want to replace the node for.
The cluster page appears.
-
On the Configuration tab, select the dead node that you want to replace, and then click Replace dead node.
The Replace dead node dialog box appears.
-
Select a target node that you want to replace the dead node with.
-
In the Java path box, enter the full path to the Cassandra gateway node java installation path, for example, /opt/java/jreXX.xx/bin.
-
In the Cassandra configuration file path box, enter the full path to the location of the Cassandra configuration file, either cassandra.yaml file or scylla.yaml file.
You do not need to enter the file, just the path to the file, for example, /etc/cassandra/conf or /etc/scylla.
-
In the Cassandra data files directory box, enter the full path to the location where the data is stored, for example, /var/lib/cassandra/data or /var/lib/scylla/data.
-
Click Save.
Result
The software replaces the dead node with the target node. All data of the dead node is now available on the target node.
Cassandra synchronizes the system keyspaces and the replicated data, but it might take some time.
What to Do Next
-
Edit the Cassandra configuration file (cassandra.yaml) on the replacement node and remove the initial_token parameter.
-
On the replacement node, remove the following line from the Cassandra-env.sh file:
JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=address_of_dead_node -
Run a discover operation or run a backup operation on any subclient that does an auto discover. The discover operation automatically removes the dead node and replaces it with a new node.
-
If you disabled incremental backups, use the following command to enable them:
nodetool enablebackup