VMW0067: Change tracking for Virtual Machine [VMservername] has reported an error (VMware)

Symptom

When Change Block Tracking (CBT) is unavailable, backups revert to Cyclic Redundancy Check (CRC) to determine changed blocks. Since the Virtual Server Agent needs to read the entire virtual machine disk, CRC incremental backups may take almost as long as full backups, even though the amount of data transferred and stored by an incremental backup is limited to the changed blocks within the virtual disk.

Cause

If a virtual machine is cloned or migrated using cold migration, vmotion, or svmotion methods, CRC may become the default scan mechanism to protect the machine until CBT is reinstated. In each backup cycle, an attempt is made to enable and use CBT.

To check for this condition, review the vsbkp.log file or generate a Job Summary report.

• For an installation of the Virtual Server Agent using the default installation path, the vsbkp.log file is located in the following folder:

  C:\Program Files\CommVault\Simpana\Log Files

  Look for log entries that indicate CRC is being used, as shown in the following example:

  vsbkp::getCRCFilePaths() - Using CRC from job id [...]
  vsbkp::initCRCTables() - Check sum table has been initialize successfully
  [C:\Program  Files\CommVault\Simpana\iDataAgent\JobResults\CV_JobResults\iDataAgent\VirtualServerAgent\2\164\vsidaCRC_[...]_(virtualserverguid)_(virtualservervmdkname.vmdk_VCB4)]
  vsbkp::initCRCTables() - Check sum table has been initialize successfully
  [C:\Program Files\CommVault\Simpana\iDataAgent\JobResults\CV_JobResults\iDataAgent\VirtualServerAgent\2\164\vsidaCRC_[...]_(virtualserverguid)_(virtualservervmdkname.vmdk_VCB4)]
  vsbkp::crcExtentProcessor() - Done...

• To generate a Job Summary report:

  1. On the CommCell Console, click Reports | Summary.

  2. In the Reports Selection window, highlight Job Summary.

  3. On the General tab, confirm that Backup is selected and Group By is set to Client.

  4. On the Computers tab, choose one of the following:

     • Click the Modify button under Computers to select individual clients or a group of virtual clients.

     • Click the Modify button under Agent Types to select the Virtual Server agent type. You do not have to change the Computers field section when using agent type.

  5. In the Selection tab's Advanced section, ensure that Include Protected VMs is selected.

     The Options and Time Range tabs do not have to be modified from the defaults, but can be depending on your reporting needs.

  6. Click Run.

If you do not see any CRC entries, check the vsbkp.log file for the following error:

QueryChangedDiskAreas --- Exception during QueryChangedDiskAreas Error caused by file /vmfs/volumes/VolumeGUID/GUID Instance (VMservername)/GUID Instance .vmdk

If necessary, contact VMware support for issues with the QueryChangedDiskAreas API.

Resolution

To take full advantage of VADP-based incremental backups, correct CBT issues on the problematic virtual machine as quickly as possible.

Review the following VMware links to verify and reset change tracking:

• Enabling Changed Block Tracking (CBT) on virtual machines (1031873)

• Changed Block Tracking (CBT) on virtual machines (1020128)

• Reverting to a preexisting snapshot can cause incremental backups based on Changed Block Tracking to become inconsistent (1021607)

• Virtual machine freezes temporarily during snapshot removal on an NFS datastore in a ESX/ESXi 4.1 host (1031106)

• VMware Data Recovery fails to backup virtual machine with dedicated swap file disk (2013450)

To make CBT available again, the virtual machine must be power cycled by logging in to the virtual machine, accessing the Guest OS, and shutting the machine down. Upon powering on the VMware client, CBT should be reactivated.

VMware states that a stun/unstun cycle is required for CBT to be reactivated on cloned or migrated virtual machines; but a power cycle is the only definitive method to ensure that CBT is reactivated.