HotAdd TransportThe term HotAdd refers to the way the backups are completed. In HotAdd mode, the data volumes containing the virtual machines to be backed up are automatically mounted to the proxy, so they can be accessed by the proxy as a local disk. The ESX host the proxy is running on must have access to all datastores for the virtual machine. If the virtual machine and the proxy are not on the same host, all datastores must be shared between the hosts. If SAN mode is not available, HotAdd mode can achieve close to SAN mode performance.
To enable incremental backups of virtual disks, Changed Block Tracking (CBT) must be used for the first full backup. (CBT is enabled for backups by default.)
In vSphere 5.0, the SCSI HotAdd feature is enabled only for vSphere editions Enterprise and higher, which have Hot Add licensing enabled. No separate Hot Add license is available for purchase as an add-on. In vSphere 4.1, Hot Add was also enabled in the Advanced edition. Customers with vSphere Essentials or Standard editions are not able to perform proxy-based backup, which relies on SCSI HotAdd. Those customers must use alternate transport modes.
Backup of NFS Datastores
You can perform backups even when VMware datastores are configured on NFS volumes. The Virtual Server Agent and MediaAgent are configured on a Windows virtual machine. During backup, the VSA mounts the VMDKs of the source VMs to itself, using a process called HotAdd. The source VMDKs then appear as a local disk to the VSA. The VSA module reads the VMDK locally, performs deduplication, and sends the data to the MediaAgent module on the same VM to be written to the disk library.
While not as efficient as SAN mode, this configuration ensures that no data is transferred over the IP network during backup. VM restores are also performed locally. In this mode, a single VSA and deduplication node running on a virtual machine with the recommended resources can provide end-to-end protection for up to 30 TB of VM data.
HotAdd Installations - Virtualized Agent and Virtualized MediaAgent
In this configuration, the Virtual Server Agent and MediaAgent are both installed in HotAdd mode. The backup destination is typically network based (CIFS/NFS) or another VMFS datastore and tape out options are very limited. Shared storage is required for HotAdd backups of virtual machines living on other ESX hosts.
When To Use
- All virtual environment where physical servers are not preferred.
- Datastores are configured on NFS.
- Each MediaAgent can write directly to a mount point presented to the virtual machine.
- Able to segregate backup and production traffic on the network.
When Not To Use
- Direct backup to tape is required.
- Backup traffic cannot be segregated from production traffic.
HotAdd Installations - Virtualized Agent and Physical MediaAgent
In this configuration, the Virtual Server Agent is installed in HotAdd mode while the MediaAgent is installed on a physical computer. Data is transferred over the LAN to the physical MediaAgent. This model also allows for the use of a centralized Windows MediaAgent and Linux MediaAgents. Shared storage is required for HotAdd mode backups of virtual machines living on other ESX hosts.
When To Use
- Direct backup to tape or SILO to cloud is required.
- Segregation of backup and production data is possible.
- Physical MediaAgent is required for other protection tasks and secondary operations.
When Not To Use
- Segregation of backup and production data is not possible.
- SAN-based datastores on Fibre Channel or iSCSI are in use. Use an all physical configuration instead.
SCSI Controllers and Device Nodes
HotAdd relies on the SCSI protocol. Use the LSI SCSI controller. HotAdd does not support IDE disks or the paravirtual SCSI (PVSCSI) controller.
A single SCSI controller can have up to 15 disks attached, with at least one SCSI device node reservation for the virtual machine OS. If you run concurrent backup jobs that include more than 15 disks, you might need to add SCSI controllers to the VSA proxy that is responsible for hot adding disks. Having insufficient SCSI device nodes can cause delays for backup or backup copy operations and create a backlog of snapshots.
If the backup operation for a disk cannot reserve a SCSI device node, one of the following events occurs:
- If the transport mode is specified as HotAdd, the backup job fails.
- If the transport mode is specified as Auto, the backup job switches to NBD mode for the remaining disks.
To ensure that enough SCSI device nodes are available, the best option is to load balance backups across multiple VSA proxies.
Adding SCSI Controllers by Adding a Hard Disk
You can also increase the capacity for a proxy by adding a hard disk on the VSA proxy and specifying a SCSI reservation that is one more than the previous reservation. For example, after 0:0 you can add 1:0 for a second disk, and then 2:0 for a third disk. After adding SCSI controllers using this method, you can remove the disks without removing the SCSI controllers.
SCSI Port Number Sequence
VMware uses SCSI ports in sequence, but sorts digit by digit rather than by the total port number, Port numbers might be assigned incorrectly in the VMX file for a VMware VM. For example:
scsi0.pciSlotNumber = "160"
scsi1.pciSlotNumber = "224"
scsi2.pciSlotNumber = "256"
scsi3.pciSlotNumber = "1184"
In this example, VMware would use port 1184 before 224, causing backups to fail.
Changing the ports as in the following example would resolve this issue:
scsi0.pciSlotNumber = "160"
scsi1.pciSlotNumber = "1184"
scsi2.pciSlotNumber = "224"
scsi3.pciSlotNumber = "256"
You can use the following process to test HotAdd attachments for a VM:
- Create a snapshot on the VM that you planning to back up.
- Attach the base disk from that VM to the VSA proxy as an independent or non-persistent disk.
- If the attach is successful, the VSA proxy will be able to perform a HotAdd operation.
- Detach the disk from the VSA proxy.
- Remove the snapshot for the VM.
Best Practices for HotAdd
- Helper virtual machines are not required for HotAdd proxies using VADP.
- When deploying the Virtual Server Agent for HotAdd backups, choose the datastore with the largest VMFS block size to ensure backups can mount and back up virtual machines residing on all datastores. For VMFS-3, a proxy that needs to back up and restore very large virtual disks should be deployed on volumes that support a large block size, with the block size of the proxy datastore matching the block size of the datastore for the disk being backed up. VMFS-5 uses a consistent file block size and can handle volumes up to about 60TB.
- When HotAdd disks are backed up, a snapshot and redo log are created. While the HotAdd disk is still attached, do not remove the snapshot or the virtual machine being backed up. If the VM is removed while the disk is still attached, clean up of redo logs fails, and you must remove virtual disks from the backup appliance manually. If the snapshot is removed, the redo log could be left in an unconsolidated state.
- If you use the vSphere Client to remove all disks on a controller, the entry for the controller is also removed from vSphere.
- Virtual Windows disks created by HotAdd backup or restore operations might have different disk signatures than the original virtual disks.