VMW0068: Reset Changed Block Tracking (CBT) for VMware backups (VMware)

Symptom

If CBT is disabled or not functioning properly, the size of incremental backups can be larger than expected. In some cases issues with CBT can cause a full backup to take longer and, when restoring a lazy zero thick provisioned disk, might result in the disk becoming eager zero thick provisioned.

You can verify CBT issues in the CommCell Console by checking the Status Details dialog for a virtual machine backup job. The CBT Status field shows Disabled.

To display the Status Details dialog, right-click a backup job and select View Job Details, then click the Virtual Machine Status tab and double-click a virtual machine entry.

The vsbkp.log file includes information such as the following:

4168 6 08/15 13:37:42 98795 ### QueryChangedDiskAreas --- Exception during QueryChangedDiskAreas Error caused by file /vmfs/volumes/50f432d3-ba380107-2b26-90b11c070562/VM/VM-000003.vmdk

Cause

Changed Block Tracking (CBT) is a VMware feature that identifies blocks of data that have changed or are in use. It enables incremental backups to identify changes from the last previous backup, writing only changed or in-use blocks. If CBT is disabled or not functioning properly, an incremental backup might back up a complete virtual machine disk instead of only changed blocks, or revert to a full backup.

Resolution

You can disable CBT to restart change tracking for subsequent backups. This resolution is applicable for ESX hosts 4.0 or later, and for virtual machines at hardware version 7 and later.

Before resetting CBT, ensure that there are no snapshots on the virtual machine.

To reset CBT for a specific virtual machine, you must disable CBT, power cycle the virtual machine, and run a new backup. When the new backup is run, CBT is automatically re-enabled.

You can also reset CBT on multiple virtual machines using a PowerCLI script that resets CBT without requiring that virtual machines be powered down.

See the following articles:

• Resetting Changed Block Tracking for VMware vSphere virtual machines

• Changed Block Tracking (CBT) on virtual machines

As a result of resetting CBT, the time required for backups increases because the first backup after resetting CBT backs up all blocks for any virtual machines on which CBT was reset. This change can also increase disk storage requirements for backups. The change does not affect scheduled backup cycles or data aging.

Increase in Backup Time After Fix for VMware CBT Error for expanded VMDKs

VMware ESXi 4.x or ESXi 5.x can return incorrect information about virtual machine disk sectors after a virtual machine disk (VMDK) is expanded beyond 128 GB or any doubling of 128 GB, and Changed Block Tracking (CBT) can produce unrecoverable backups. This is a flaw in VMware APIs used by backup software, and can lead to data loss if not addressed immediately. Any virtual machine is at risk if VMDKs have been expanded beyond 128 GB, 256 GB, 512 GB, 1024 GB, or any doubling of 128 GB. For example, the issue can occur after expanding a VMDK from 100 GB to 150 GB, or after expanding a VMDK from 150 GB to 300 GB. Any backup performed since the disk expansion can contain invalid data, including full backups. For more information, including patches for ESXi 5.x, see QueryChangedDiskAreas command returns incorrect sectors after extending virtual machine vmdk file with Change Block Tracking (CBT) enabled (2090639).

Commvault Version 11, or Version 10 Service Pack 9 and later, include a fix for this issue to prevent data loss. The Commvault software automatically detects expanded disks and resets CBT:

1. The first backup job resets CBT for all virtual machines with a disk size greater than 128 GB.

   The CBT Status for the affected virtual machines is displayed as Reset.

2. The software stores the size of each VMDK file in the .vmx file for each virtual machine.

3. For any backup jobs going forward, the backup job compares the current size of each VMDK file with the size recorded in the .vmx file.

4. If a disk has been expanded, the software resets CBT and backs up all blocks for the VMDKs attached to the virtual machine.

For older service packs for Commvault Version 10 and for Commvault Versions 9 or 8, you can use the CheckChangeTracking tool to reset CBT on specific affected virtual machines. For more information, see KB article VMW0009. The tool automatically re-enables CBT for the next backup job, which will back up all blocks for VMDKs even if an incremental backup is requested. CBT is also enabled for future incremental backups.

If you apply VMware patch ESXi600-201511001 to the VMware ESXi 6.0, 6.0 Update 1, or 6.0 Update1a hosts, Commvault detects the new ESXi build version and reverts back to using CBT. The next incremental backup will accurately protect any blocks that have changed since the last successful backup.