VM Dispatch and Proxy Selection for VMware Backups

Virtual Server Agent (VSA) hypervisors use a common framework to distribute backup jobs, as described in Backup Workload Distribution.

This topic describes considerations that are specific to VMware.

To adjust the default behavior, see Performance Tuning for Backups.

Dispatch Logic

The basic process for matching virtual machines to proxies includes the following stages:

1. The coordinator prioritizes the virtual machines for each backup job.

2. The coordinator identifies specific proxies that can be used to back up each virtual machine. For each proxy, the list of VMs that can be assigned to the proxy includes all virtual machines that are on subclients where the proxy is available.

   You can create different proxies for different backup methods (for example, different proxies for HotAdd, SAN, or NBD) and associate those proxies with the appropriate subclients. When a job starts, the coordinator identifies all virtual machines that can be backed up by each proxy to create the proxy queue.

3. Each VSA proxy for the instance provides the coordinator information about the total memory and number of CPU cores for the proxy. The coordinator uses this information to calculate the amount of work that each proxy can handle, and the number of backup streams that can be assigned for the proxy.

4. The coordinator assigns each virtual machine to an appropriate VSA proxy and continues to assign virtual machines to proxies until all VMs are processed.

5. Depending on the flow of work, datastore and volume activity, how quickly backups are completed, and the availability of proxies, VM-to-proxy assignments can change over the course of a job:

   • When a proxy finishes backing up a VM and has no more VMs waiting to be backed up in the local queue (including statically assigned VMs), it sends the coordinator a request for another VM to back up. The coordinator selects the next VM from the dynamic assignment list for the proxy.

   • The process controller for each proxy keeps track of the number of VMs, number of disks, and size of VMs that are being processed, as well as the number of streams in use. The proxy communicates any change in workload to other proxies immediately, and sends changes for the number of disks and VMs being processed to the coordinator when requesting another VM to back up. When the number of streams in use changes, the proxy sends that information to the coordinator immediately.

   • The coordinator updates the proxy queue with current information about VM priorities.

   • The coordinator updates the queue as long as there are additional VMs to back up.

If only one proxy is available to back up a VM, the VM is statically assigned to that proxy.

Proxy Selection

The virtualization client enables automatic load balancing of backup jobs across multiple VSA proxies.

The CommServe system attempts to contact the coordinator node (the first in the list of proxies). If the coordinator node is not available, the system tries the next proxy in the list until finding an available proxy. Once a proxy is successfully contacted, it acts as the coordinator for the remainder of the job. The coordinator proxy contacts the remaining proxies in the list. Any proxies that are not available are not used for the current job attempt.

Once all proxies to be used for the job are known, the coordinator distributes the virtual machines among the available proxies. The decision of which proxy to use for a given virtual machine is based on the best path for backing up that virtual machine:

1. Virtual machines are backed up by proxies running as physical machines that have SAN connectivity to the datastore.

2. Virtual machines are backed up by proxies that are running as virtual machines on the same host.

3. Virtual machines that are part of a vSAN stretched cluster are backed up by proxies that are in the same fault domain.

4. Virtual machines are backed up by proxies that are running as virtual machines on hosts that have access to the datastore.

5. Virtual machines are backed up by proxies that are on the same subnet as the virtual machine's host.

6. The remaining virtual machines are distributed evenly across the proxies.

For example, if a vCenter has ESX servers ESX1, ESX2 and ESX3:

• ESX1 hosts 2 virtual machines (VM1, VM2) and hosts Proxy1.

• ESX2 hosts 1 virtual machine (VM3), and hosts Proxy2.

• ESX3 hosts 1 virtual machine (VM4).

Then:

• VM1 is backed up by Proxy1 because they are on the same host.

• VM2 is backed up by Proxy1 because they are on the same host.

• VM3 is backed up by Proxy2 because they are on the same host.

• VM4 is backed up by Proxy2 because it only has one VM assigned.

If one of the proxies becomes unavailable during the backup, the virtual machines allocated to that proxy are distributed among the remaining proxies. If the proxy becomes available again while the same backup job is running, it is not used for backups unless you suspend the backup job and resume it again.

Datastore Throttling

When assigning virtual machines to proxies, the coordinator assigns virtual machines from different datastores before assigning an additional VM that resides on the same datastore. This approach reduces the chance that there are multiple streams against the same datastore. The coordinator only considers the primary datastore for each virtual machine.

Datastore throttling takes precedence over the virtual machine priority sequence. For example, if proxy 1 is backing up a virtual machine that resides on datastore 1, proxy 2 will be assigned a VM that resides on a different datastore, even if another VM on datastore 1 has a higher priority.

Transport Mode

By default, backups use the transport mode specified for the virtualization client or the subclient, or select the best transport automatically if none is specified.

The supported transport mode for a proxy is used to determine which proxies can back up each VM. If proxies that use different transport modes can back up a VM, the number of SAN proxies is considered first, followed by the number of HotAdd proxies if there are no proxies that support SAN transport, and then NBD or NBDSSL proxies if no proxies support SAN or HotAdd transport.