Monitoring Disk and Network Health for HyperScale X

Disk and network metrics and other useful information for Commvault HyperScale X nodes can be obtained using the cv_disk_net_monitor.py script.

The following commands are used to generate the outputs:

• For network monitoring

  Outputs from ifconfig, nfsstat and netstat commands.

• For disk monitoring

  Output from df command for all the data and metadata disks on the node.

Procedure

You can view the saved logs from each node by navigating to the following log folder:

#cd /var/log/commvault/Log_Files/CVFSPerfMon/
    

OPTIONALLY, you can manually run the script as follows:

• Navigate to the following folder:

  # cd /opt/commvault/MediaAgent
      
  

• Run the following command:

  # ./python3 cv_disk_net_monitor.py
      
  

Output

The logs are saved either as comma separated values (CSV) or normal readable log files. (CSV files provide a curated view of the relevant information, while the log files provide the raw outputs from the commands.)

The following log files are available in each node:

• Network Monitoring

  1. NetworkStatusFiltered.log containing the output of the following 3 commands

     • Output from ifconfig -a | grep 'flags\|RX errors\|TX errors command, to capture the flags, MTU, TX/RX errors for all interfaces on the node. Useful to identify network retransmissions, etc.

     • Output from netstat -s | grep -i retrans command, to capture the retransmission statistics on the node. Useful for identifying network congestion/issues.

     • Output from nfsstat -l | grep 'read\|write\|access\|getattr\|lookup\|readdir to capture nfs calls for read, write, access, getattr, setattr, lookup, lookup_root and readdir. Useful to estimate and map file operations over a period of time.

  2. ifconfigCSV.csv

     CSV file containing date, time, interface, MTU, TX/RX stats (total, errors, drops and overruns) and flags for bond1, bond2 and their slaves.

  3. nfsstatCSV.csv

     CSV file containing date, time and the nfs calls for read, write, access, getattr, setattr, lookup, lookup_root and readdir.

  4. netstatCSV.csv

     CSV file containing date, time and retransmission stats (retransmitted segments, TCPLostRetransmit, retransmits in slow start, SACK retransmits failed, TCPRetransFail and TCPSynRetrans.

• Disk Monitoring

  1. DiskStatus.log

     Contains df output for all the data and metadata disks in KB. Useful to monitor disk usage.

  2. DiskStatusCsv.csv

     Same content as DiskStatus.log but in csv format.